qi& dZddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZmZmZmZmZmZmZddlmZmZmZmZdd lmZmZmZmZmZm Z m!Z!m"Z"e jFe$Z%e"d Z&d e'd e(e'e'ffdZ)dde e*e+fde,d e'fdZ-ddeede,d e'fdZ. dde/e eefde,de,d e/e'fdZ0 dde1de/e'de'de*de*de,d dfdZ2e&jgdgd  dd!ee'e jhd" fd#ee,e jjd$d% fd&ee,e jjd'd( fded)ed df d*Z6d e'd e,fd+Z7e&jgd,gd- ddddejpddfd eee'e jhd. fdee,e jjd/d0d1 fd2ee,e jjd3d4 fd5ee,e jjd6d7d8 fd9eded)ed dfd:Z9d;ee'de,d2e,d5e,d9ede,d)ee'd dfd<Z:d e'de,d2e,d5e,d9ede,d)ee'd dfd=Z;e&jgd>d?d@g  dd!ee'e jhdA fded)ed dfdBZe&jgdYgdZ  dd[ee'e jhd\ fd]ee'e jhd^ fd)ed dfd_Z?e&jgd`gda  ddbeee'e jhdc fddeee'e jhde fdCee,e jjdf fdgee,e jjdhdi fdjee,e jjdkdl fdmeee'e jjdn fdoeee'e jjdp fdQee,e jjdRdq fdTeee/e'e jjdr fdVeee/e'e jjds fdteee'e jjdu fdvee,e jjdwdx fdyee,e jjdzd{ fd|ee,e jjd}d~d fdee,e jjddd fd)ed df"dZ@e&jgdgd  ddee'e jhd fdeee'e jhd fded)ed df dZAy)z7Contains commands to interact with buckets via the CLI.N)datetime) AnnotatedOptionalUnion)logging) BUCKET_PREFIX BucketFile BucketFolder FilterMatcher_is_bucket_path_parse_bucket_path_split_bucket_id_and_prefix) StatusLineare_progress_bars_disableddisable_progress_barsenable_progress_bars) FormatOpt OutputFormatQuietOptTokenOptapi_object_to_dict get_hf_apiprint_list_output typer_factoryz"Commands to interact with buckets.)helpargumentreturnc|jtr t|S t|S#t$rt d|dtdwxYw)zParse a bucket argument accepting both 'namespace/name(/prefix)' and 'hf://buckets/namespace/name(/prefix)'. Returns: tuple: (bucket_id, prefix) where bucket_id is "namespace/bucket_name" and prefix may be empty string. zInvalid bucket argument: -. Must be in format namespace/bucket_name or znamespace/bucket_name) startswithrr r ValueError)rs M/opt/pipecat/venv/lib/python3.12/site-packages/huggingface_hub/cli/buckets.py_parse_bucket_argumentr$8sb =)!(++ *844  'z2 /!6 8   s - A Fsizehuman_readablecz|s t|SdD]$}|dkr|dk(r |d|cS|dd|cS|dz}&|ddS)zFormat a size in bytes.)BKBMBGBTBir( z.1fz PB)str)r%r&units r# _format_sizer0Isj 4y- $;s{q''3Zq' '    3Zs mtimecP|y|r|jdS|jdS)z0Format mtime datetime to a readable date string.z %b %d %H:%Mz%Y-%m-%d %H:%M:%S)strftime)r2r&s r# _format_mtimer6Ws, }~~m,, >>- ..r1itemsquietc i}|D]e}|jjd}|}|ddD]}||vrdii||<||d}|d}t|tr ||vsWdii||<_d|i||<gd} d} d} |s{|D]i}t|tst |j |} t| t| } t|j|} t| t| } k| dkDr| dz| z} g}t||d| | | |S) aBuild a tree representation of files and directories. Produces ASCII tree with size and date columns before the tree connector. When quiet=True, only the tree structure is shown (no size/date). Args: items: List of BucketFile/BucketFolder items human_readable: Whether to show human-readable sizes and short dates quiet: If True, show only the tree structure without sizes/dates Returns: List of formatted tree lines /N __children____item__rr4 prefix_widthmax_size_widthr&) pathsplit isinstancer r r0r%maxlenr6r2 _render_tree)r7r&r8treeitempartscurrentpart final_partr@rAmax_date_widthsize_strdate_strliness r# _build_treerR`sS&D 5 $#2J 4D7"!/ 4 dmN3G 4 2Y dL )('5r&: ##-t"4GJ  5 LNN  DD$ +' >B!$^S]!C(^D!$^S]!C  D A )A->LE   !%%  Lr1noderQindentr@rAc .t|j}t|D]\}\}} |t|dz k(} | rdnd} d| v} | j di} |dkDrz| rd|z}nQ| j d}|9t |j |}t|j|}|d |d d |}nd|z}|j|d || || rd nd n|j|| || rd nd | s|| rd ndz}t| |||||y)z:Recursively render a tree structure with size+date prefix.ru └── u ├── r<rr-r=N>r4 r:z u│ r?) sortedr7 enumeraterFgetr0r%r6r2appendrG)rSrQrTr@rAr&r7inamevalueis_last connectoris_dirchildrenprefixrIrOrP child_indents r#rGrGsL 4::< E%e,=D%s5zA~%$+L 5(99^R0 ! |+yy,#+DII~FH,TZZHH (>*: (:;2hZHF </F LLF82fXi[fsRT>UV W LLF8I;tfFS4KL M !wVHEL )--  1r1create)zhf buckets create my-bucketz hf buckets create user/my-bucketz-hf buckets create hf://buckets/user/my-bucketz*hf buckets create user/my-bucket --privatez+hf buckets create user/my-bucket --exist-ok)r]examples bucket_idzTBucket ID: bucket_name, namespace/bucket_name, or hf://buckets/namespace/bucket_nameprivatez --privatezCreate a private bucket.exist_okz --exist-okz3Do not raise an error if the bucket already exists.tokenct|}|jtr3 t|\}}|r t j d|dtd|}|j||r|nd|} |rt| jytd| jd| jd y#t$r#}t j t|d}~wwxYw) zCreate a new bucket.rjNz-Cannot specify a prefix for bucket creation: . Use namespace/bucket_name or namespace/bucket_name.)rhrizBucket created: z (handle: )) rr!rr$r"typer BadParameterr. create_bucketprinthandleurl) rgrhrir8rjapi parsed_idrce bucket_urls r#reresF 5 !CM* - 6y A Iv $$? {K11>?UW  """#J   j    0 :;L;L:MQOP# -$$SV, , -sB,, C5CCcd|jtr|ttd}d|vS|}d|vS)zFCheck if argument is a bucket ID (namespace/name) vs just a namespace.Nr:)r!rrF)rrBs r# _is_bucket_idr{s?=)M*,- $; $;r1z list | ls) zhf buckets listzhf buckets list huggingfacezhf buckets list user/my-bucketz!hf buckets list user/my-bucket -Rz!hf buckets list user/my-bucket -hz%hf buckets list user/my-bucket --treez(hf buckets list user/my-bucket --tree -hz+hf buckets list hf://buckets/user/my-bucketz%hf buckets list user/my-bucket/sub -RzyNamespace (user or org) to list buckets, or bucket ID (namespace/bucket_name(/prefix) or hf://buckets/...) to list files.z--human-readablez-hz$Show sizes in human readable format.as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).formatc t|duxr t|}|rt|||||||yt|||||||y)zList buckets or files in a bucket. When called with no argument or a namespace, lists buckets. When called with a bucket ID (namespace/bucket_name), lists files in the bucket. N)rr&r|r}r~r8rj) namespacer&r|r}r~r8rj)r{ _list_files _list_buckets)rr&r|r}r~r8rj is_file_modes r#list_cmdrsXr4'CM(,CL)  ) r1rc*|rtjd|rtjd|8|jtr#|t td}|j d}t |}|j|Dcgc] }t|} }| s<|s:|tjk7r'||n|jd} td| d ygd } d td ttffd } ddd} t!| ||| | | ycc}w)zList buckets in a namespace.z'Cannot use --tree when listing buckets.z,Cannot use --recursive when listing buckets.Nr:rl)rr]z"No buckets found under namespace ''.)idrhr% total_files created_atrIrc ddlm}||jd||jdt|jdd||jd||jd gS) Nr) _format_cellrrhr%rr&rr) _cli_utilsrrZr0)rIrr&s r#row_fnz_list_buckets..row_fnsj, $ ( ), - &!,^ L -0 1 ,/ 0   r1right)r%r)r~r8headersr alignments)rprqr!rrFrstripr list_bucketsrrjsonwhoamirsdictlistr.r)rr&r|r}r~r8rjrvbucketresultsresolved_namespacerrrs ` r#rras  !JKK  !OPP!5!5m!Dc-023 $$S) 5 !C8;8H8HS\8H8]^f!&)^G^ <#4#44.7.CV\I]  67I6J"M N DG  T  d3i  "':JgfE7SYfpq-_s Dc X|r(|tjk(rtjdt |} t |\}} t|j|| xsd|} | s tdytd| D} |tjk(r:| D cgc] } t| }} ttj|dn|r!t| || }|D] }t|n|rF| D]@} t| t rt| j"d ,t| j"Bn| D]} t| t r7t%| j&|}td d d |dd | j"d Jt)| j*|}t%| j,|}t|d d |dd | j"|s| rt/j1dyyy#t $r#} tjt| d} ~ wwxYwcc} w)zList files in a bucket.z%Cannot use --tree with --format json.rlNrcr}z(empty)c3<K|]}t|tywN)rDr ).0rIs r# z_list_files..sKT*T<8Ksr>rT)r&r8r:r4z>12rWz>19z!Use -R to list files recursively.)rrrprqrr$r"r.rlist_bucket_treersanyrdumpsrRrDr rBr6 uploaded_atr0r%r2rdone)rr&r|r}r~r8rjrvrgrcrxr7has_directoriesrIr tree_linesline mtime_strrOs r#rrs 6\...  !HII 5 !C)28< 6   >T   E  iKUKKO """8=>%d+>> djj+,  ~US  D $K   !D$ - 1o&dii  ! GD$ -)$*:*:NK C9S/DII;a@A' >B)$**nE #b3r$))EF G  =>)9W )  Q(()&?sG8H'8 H$HH$infozhf buckets info user/my-bucketz+hf buckets info hf://buckets/user/my-bucketzFBucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namec<t|} t|\}}|j |}|rt|jyttjt|dy#t$r#}tjt |d}~wwxYw)zGet info about a bucket.rlNr>r) rr$r"rprqr. bucket_inforsrrrr)rgr8rjrvrw_rxrs r#rrs$ 5 !C)-i8 1__Y 'F  fii djj+F3A>? )  Q(()sA// B8BBdelete)z hf buckets delete user/my-bucketz-hf buckets delete hf://buckets/user/my-bucketz&hf buckets delete user/my-bucket --yesz-hf buckets delete user/my-bucket --missing-okyesz--yesz-yzSkip confirmation prompt. missing_okz --missing-okz3Do not raise an error if the bucket does not exist.c*|jtr4 t|\}}|r t j d|dtd|}n$d|vr t j d|dtd|s:t jd|d }|std t jt| } | j|| |r t|ytd |y#t$r#}t j t |d}~wwxYw)zDelete a bucket. This deletes the entire bucket and all its contents. Use `hf buckets rm` to remove individual files. Nz-Cannot specify a prefix for bucket deletion: rmrnr:zInvalid bucket ID: r z(Are you sure you want to delete bucket ''?Aborted.rl)rzBucket deleted: ) r!rr$r"rprqr.confirmrsAbortr delete_bucket) rgrrr8rjrwrcrxrrvs r#rrs*LM* - 6y A Iv $$? {K11>?UW  I   !)-;;H/I_ a  --"J9+UW XY * ++-  5 !CiJ7  i   ,-3 -$$SV, , -sC&& D/D  Dz remove | rm)z)hf buckets remove user/my-bucket/file.txtz2hf buckets rm hf://buckets/user/my-bucket/file.txtz.hf buckets rm user/my-bucket/logs/ --recursivez:hf buckets rm user/my-bucket --recursive --include "*.tmp"z8hf buckets rm user/my-bucket/data/ --recursive --dry-runzBucket path: namespace/bucket_name/path or hf://buckets/namespace/bucket_name/path. With --recursive, namespace/bucket_name is also accepted to target all files.z0Remove files recursively under the given prefix.dry_runz --dry-runz8Preview what would be deleted without actually deleting.includezQInclude only files matching pattern (can specify multiple). Requires --recursive.excludezLExclude files matching pattern (can specify multiple). Requires --recursive.c t|\}} | dk(r|stjd|d|d|s|r|stjdt |} |r"t | } | jd g} | j|| jd xsdd D]B}t|ts| j|| jd t| dD| jd t| d|s|r9t||}| Dcgc] }|j!|j"s|"}}n| }|Dcgc]}|j"}}t%d|D}t'|d }|s|s t)dyt|d|}|sV|sT|s|D]}t)d|tj*d|d|d}|st)dtj,|r.|D]}t)dt.|d |t)d|dy| j1|||r|D] }t)|y|D]}t)dt.|d |t)d|d|dy| jd }|stjd|r#t)dt.|d |t)d y|s=tj*d!|d"|d}|st)dtj,| j1||g|r t)|yt)dt.|d |y#t$r#} tjt | d} ~ wwxYwcc}wcc}w)#zbRemove files from a bucket. To delete an entire bucket, use `hf buckets delete` instead. Nr4z?No file path specified. To remove files, provide a path (e.g. 'zd/FILE') or use --recursive to remove all files. To delete the entire bucket, use `hf buckets delete z`.z,--include and --exclude require --recursive.rl)enabledzListing files from remoter:TrzListing files from remote (z files))include_patternsexclude_patternsc34K|]}|jywr)r%)rfs r#rzremove..s7A7srzNo files to remove.z file(s) totaling rWzRemove z from 'rrzdelete: z (dry run) z would be removed.)rzRemoved rzFile path cannot be empty.z"(dry run) 1 file would be removed.zRemove 'z' from ')r$r"rprqr.rrupdaterrrDr r[rFrr matchesrBsumr0rsrrrbatch_bucket_files)rr}rrrrr8rjrgrcrxrvstatus all_filesrImatcherr matched_files file_paths total_sizerO count_labelrBr file_paths r#remover7s|)28< 6|I   k"DDM;b R  7I  !OPP 5 !CI. 12&( (( ==%-)  UD $ +  & ;C N;K7ST U  1#i.1AIJ g#WwWG(1M1W__QVV5LQMMM%M&34aff4 4777  4@+, Z));H:F 7&'DBtf+&'mmgk]')B$OPGj!kkm# " D {!D6BC D J{m+=> ?  y< " d  # D {!D6BC D H[M 2> ?MM#& $$%AB B  H]OI;a {C D 6 7 mmhyk)B$OPGj!kkm# y)=  )  H]OI;a {C Dw )  Q(()@N5s)M M4=M4 M9 M1M,,M1move)z/hf buckets move user/old-bucket user/new-bucketz/hf buckets move user/my-bucket my-org/my-bucketzIhf buckets move hf://buckets/user/old-bucket hf://buckets/user/new-bucketfrom_idzMSource bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_nameto_idzRDestination bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namec$t|\}}|r tjd|dtdt|\}}|r tjd|dtdt |}|j ||t d|d|y) z2Move (rename) a bucket to a new name or namespace.z)Cannot specify a prefix for bucket move: rmrnrl)rrzBucket moved:  -> N)r$rprqrr move_bucketrs)rrrjparsed_from_id from_prefix parsed_to_id to_prefixrvs r#rrs2#9"ANK  7yA--:O;Q S  5U;L)  7w?--:O;Q S  5 !COON,O? N>*$|n =>r1sync)z2hf buckets sync ./data hf://buckets/user/my-bucketz2hf buckets sync hf://buckets/user/my-bucket ./dataz;hf buckets sync ./data hf://buckets/user/my-bucket --deletez^hf buckets sync hf://buckets/user/my-bucket ./data --include "*.safetensors" --exclude "*.tmp"zIhf buckets sync ./data hf://buckets/user/my-bucket --plan sync-plan.jsonlz'hf buckets sync --apply sync-plan.jsonlzhf buckets cp my-config.json hf://buckets/user/my-bucket/logs/zKhf buckets cp my-config.json hf://buckets/user/my-bucket/remote-config.jsonz7hf buckets cp - hf://buckets/user/my-bucket/config.jsonsrcz9Source: local file, hf://buckets/... path, or - for stdindstz?Destination: local path, hf://buckets/... path, or - for stdoutc  t|}t|}|duxr t|}|dk(}|dk(}|r|rtjd|s0|s.|s,|tjdtjd|r|stjd|r?|r=|Jt |\} } | dk(s| j d rtjd |r|stjd |s6|s4t jj|rtjd |rt |\} } | dk(s| j d rtjd | jd dd} |rt} | s t tj5}t jj|| }|j| | |fgt!|d5}t"j$j&j)|j+dddddd| st-y|| }nrt jj|s0|j t j.s|j d r!t jj|| }n|}t jj1|}|rt j2|d|r t |j| | |fg|r t- |st5d|d|yyy|rxt |\} }t"j6j&j+}|r t |j9| ||fg|r t- |st5d|yyt jj;|stjd|t |\} } | dk(r t jj=|}n6| j d r#| t jj=|z}n| }|r t |j9| ||fg|r t- |st5d|dt>| d |yy#1swY{xYw#1swYxYw#| s t-wwxYw#|r t-wwxYw#|r t-wwxYw#|r t-wwxYw)z'Copy a single file to or from a bucket.rlN-z$Remote-to-remote copy not supported.z2Missing destination. Provide a bucket path as DST.z;One of SRC or DST must be a bucket path (hf://buckets/...).z+Stdin upload requires a bucket destination.r4r:zAStdin upload requires a full destination path including filename.z"Cannot pipe to stdout for uploads.zNSource must be a file, not a directory. Use `hf buckets sync` for directories.zJSource path must include a file name, not just a bucket or directory path.rr;rbT)riz Downloaded: r)addzUploaded: stdin -> zSource file not found: z Uploaded: ) rr rprqr endswithosrBisdirrsplitrrtempfileTemporaryDirectoryjoindownload_bucket_filesopensysstdoutbufferwritereadrsepdirnamemakedirsrsstdinrisfilebasenamer)rrr8rjrv src_is_bucket dst_is_bucket src_is_stdin dst_is_stdoutrrcrgfilenamepbar_was_disabledtmp_dirtmp_pathr local_path parent_dir remote_pathdatas r#rrsu, 5 !C#C(MtO<(H--i68:L9MNh-: ))//9:: )(*{% s#s||BFF';s||C?PWW\\#x8  4J J6%' +)))vz6J5KL(* SEj\:;1)6 !3C!8 ;yy$$&  ! # '  " "9D+3F2G " H$& 'u- . ww~~c"$$'>se%DE E.s3 6 R<''**3/K __S ! 277#3#3C#88K K  ! # '  " "9C3E2F " G$& Jse4 yk;-P QI:::: )(*)*(* $&2$&saR4'AR')8R!R')R4:S7SS*R$ R''R1,R44SSS'*S9)F)FF)rrF)FFFN)FN)FFFNNFNr)NNFFFNNFNNNFFFFN)NFN)B__doc__rrrrrtypingrrrrphuggingface_hubrhuggingface_hub._bucketsrr r r r r rhuggingface_hub.utilsrrrrrrrrrrrrr get_logger__name__logger buckets_clir.tupler$intfloatboolr0r6rrRrrGcommandArgumentOptionrer{tablerrrrrrrrrr1r#rs > -- #vu      H %!EF  S U38_ " uS%Z( $ 3 /(+/T/c/!< j,./ 0<< < #Y <F ) ) 9) ) )  )  ) )X   .  -/Q g  /Q  +  /Q  F  /Q* +/Q, -/Q. //Q /QdCD    .    $**G@  W  @    7  @ $  F  %@ 2   C  3@ B C@ D E@ F G@ H I@  @ F-r}-r-r-r -r  -r  -r C=-r -r`>?>?>?>? >?  >?  >? C=>? >?B  (5@ Y  @ @ @ @@2  .  /9. Y  9.    ,   9.  F  9., -9.. /9.0 19.9.x   6    ]QE a  QE   C  QE$    ,   %QE4  K  5QEBc d  CQENc _  OQEZ [QE\ ]QE^ _QE QEh  *"?  `  "?  e   "? "? "?"?T    &          Ix  ^   x  c   x   B   x&  N  'x4  O  5xB   P   CxN   6   OxZ  H  [xhc I  ixtc I  ux@  ;  AxL  T  MxZ  V  [xh   8  ixx    "   yxH IxJ Kx x@    "  R 3,ghh iR  ~u~~+lmm R  R  R R Rr1