o 0jj @s~UddlZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZddlmZddlmZdd lmZmZdd lmZmZmZdd lmZmZddlZddlZdd lmZm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddlZ>m?Z?m@Z@ddlAmBZBe GdddZCe GdddeCZDe GdddeCZEeFejGZHeeId<GdddeHZJGd d!d!ejGeJd"ZKGd#d$d$ejLjMZNGd%d&d&ejLjMZOd'ePd(ePfd)d*ZQd+ePd(ePfd,d-ZRd.ePd/eSdBd(efd0d1ZTd2eKd.ePd3ePd4eUd5ePf d6d7ZVd8d9ZWeKZXdS):N)deque)IterableIterator) ExitStack)deepcopy) dataclassfield)datetime)chain)Path PurePosixPath)AnyNoReturnUnion)quoteunquote)_DEFAULT_CALLBACK NoOpCallback TqdmCallback) apply_config) isfilelike) constants)CommitOperationCopyCommitOperationDelete)BucketNotFoundErrorEntryNotFoundErrorHfHubHTTPErrorRepositoryNotFoundErrorRevisionNotFoundError) hf_hub_urlhttp_get)SPECIAL_REFS_REVISION_REGEX BucketFile BucketFolderHfApiLastCommitInfoRepoFile RepoFolder)HFValidationErrorhf_raise_for_status http_backoffhttp_stream_backoff)md5c@s0eZdZUdZeed<eed<defddZdS)HfFileSystemResolvedPathz_Top level Data structure containing information about a resolved Hugging Face file system path.rootpathreturncCs|jd|jdS)N/)r/r0rstripselfr6_/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py unresolve+sz"HfFileSystemResolvedPath.unresolveN)__name__ __module__ __qualname____doc__str__annotations__r8r6r6r6r7r.$s r.c@sveZdZUdZeed<eed<eed<eed<eddZeed<eddZeed <ed dd Z ed Bed <d dZ d S)"HfFileSystemResolvedRepositoryPathzLData structure containing information about a resolved path in a repository. repo_typerepo_idrevision path_in_repoFinitr/r0N)defaultrepr _raw_revisioncCsftj|jd|j}|jr|d|j|_n|jtjkr*|dt |j|_n||_|j |_ dS)N@) rREPO_TYPES_URL_PREFIXESgetr@rArHr/rBDEFAULT_REVISION safe_revisionrCr0)r5 repo_pathr6r6r7 __post_init__=s  z0HfFileSystemResolvedRepositoryPath.__post_init__) r9r:r;r<r=r>rr/r0rHrPr6r6r6r7r?/s  r?c@s4eZdZUdZeed<eddZeed<ddZdS) HfFileSystemResolvedBucketPathzHData structure containing information about a resolved path in a bucket. bucket_idFrDr/cCsd|j|_dS)Nzbuckets/)rRr/r4r6r6r7rPOsz,HfFileSystemResolvedBucketPath.__post_init__N) r9r:r;r<r=r>rr/rPr6r6r6r7rQHs  rQ _cached_basecs(eZdZdZfddZddZZS)_Cacheda Metaclass for caching HfFileSystem instances according to the args. This creates an additional reference to the filesystem, which prevents the filesystem from being garbage collected when all *user* references go away. A call to the :meth:`AbstractFileSystem.clear_instance_cache` must *also* be made for a filesystem instance to be garbage collected. This is a slightly modified version of `fsspec.spec._Cached` to improve it. In particular in `_tokenize` the pid isn't taken into account for the `fs_token` used to identify cached instances. The `fs_token` logic is also robust to defaults values and the order of the args. Finally new instances reuse the states from sister instances in the main thread. cstj|i|i|_dSN)super__init___cache)clsargskwargs __class__r6r7rWgs z_Cached.__init__c Ost||}|dd}|j|tg|Ri|}|j|tjg|Ri|}|s<|jr<||jvr<||_ |j|St j |g|Ri|}|sh|jrh||jvrh|j| }| D] \}} t||| q]||_||_||_|jr~|s~||_ ||j|<|S)NZskip_instance_cacheF)rpop _tokenize threading get_ident main_threadidentZcachablerXZ_latesttype__call___get_instance_stateitemssetattrZ _fs_token_ storage_argsstorage_options) rYrZr[skipZfs_tokenZfs_token_main_threadobjinstance_stateattr state_valuer6r6r7reos&      z_Cached.__call__)r9r:r;r<rWre __classcell__r6r6r\r7rTWs rTcs^eZdZdZdZdZddddddedBdeeBdBdedBd edBffd d Z e d ed efddZ dedededBd e ee dBffddZded e ee dBffddZ dTdededBd eeBfddZdTdedBd dfddZ   dUdedededBdedBd edf d d!ZdTdededBd dfd"d#Z $  dVded%ed&edBdedBd df d'd(Z dWded*ed+ededBd eeeeefBf d,d-Z $ $   dXded%ed+ededBd edBd&edBf d.d/Zded0ed%ed eeeBfd1d2Z ded e!e eeeeefffd3d4 Z"dTded&edBd eeffd5d6 Z#  $ $ $ dYded&edBd7ed*ed+ededBd eeeeeeeffBfd8d9Z$dTd:ed;ededBd dfdd?Z'dZded+ededBd eeeffd@dAZ(dBdCZ)dDdEZ*dFdGZ+ded efdHdIZ,e-dfd[fdJdK Z.e/dLdMZ0dNdOZ1dPdQZ2dRdSZ3Z4S)\ HfFileSystemaz Access a remote Hugging Face Hub repository as if were a local file system. > [!WARNING] > [`HfFileSystem`] provides fsspec compatibility, which is useful for libraries that require it (e.g., reading > Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility > layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible. The file system supports paths for the `hf://` protocol, which follows those URL schemes: * Models, Datasets and Spaces repositories: ``` hf://[@]/ hf://datasets/[@]/ hf://spaces/[@]/ ``` * Buckets (generic storage): ``` hf://buckets// ``` Note: when using the [`HfFileSystem`] directly, passing the `hf://` protocol prefix is optional in paths. Args: endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . token (`bool` or `str`, *optional*): A valid user access token (string). Defaults to the locally saved token, which is the recommended method for authentication (see https://huggingface.co/docs/huggingface_hub/quick-start#authentication). To disable authentication, pass `False`. block_size (`int`, *optional*): Block size for reading and writing files. expand_info (`bool`, *optional*): Whether to expand the information of the files. **storage_options (`dict`, *optional*): Additional options for the filesystem. See [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.__init__). Usage: ```python >>> from huggingface_hub import hffs >>> # List files >>> hffs.glob("my-username/my-model/*.bin") ['my-username/my-model/pytorch_model.bin'] >>> hffs.ls("datasets/my-username/my-dataset", detail=False) ['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json'] >>> # Read/write files >>> with hffs.open("my-username/my-model/pytorch_model.bin") as f: ... data = f.read() >>> with hffs.open("my-username/my-model/pytorch_model.bin", "wb") as f: ... f.write(data) ``` Specify a token for authentication: ```python >>> from huggingface_hub import HfFileSystem >>> hffs = HfFileSystem(token=token) ``` rIZhfN)endpointtoken block_size expand_inforrrsrtrucsTtj|i||p tj|_||_t||d|_||_||_ i|_ i|_ i|_ dS)N)rrrs) rVrWrENDPOINTrrrsr%_apirtru_repo_and_revision_exists_cache_bucket_exists_cachedircache)r5rrrsrtrurZrjr\r6r7rWs  zHfFileSystem.__init__threading_identr1cs\dptjd<dd<fddtD|||f}tt|}|S)zDeterministic token for cachingrrrsci|]}||qSr6r6).0keyr[r6r7 z*HfFileSystem._tokenize..)rLrrvsortedr-r=encode hexdigest)rYr{rZr[Z tokenize_argshr6rr7r_s  zHfFileSystem._tokenizer@rArBc Cs|||f|jvrqz |jj|||tjdWnKttfy=}zd|f|j|||f<d|f|j||df<WYd}~n8d}~wty`}zd|f|j|||f<d|j||df<WYd}~nd}~wwd|j|||f<d|j||df<|j|||fS)N)rBr@timeoutFTN)rxrwZ repo_inforZHF_HUB_ETAG_TIMEOUTrr)r)r5r@rArBer6r6r7_repo_and_revision_exists"   z%HfFileSystem._repo_and_revision_existrRc Csb||jvr,z|j|Wnty&}z d|f|j|<WYd}~n d}~wwd|j|<|j|S)NFr)ryrwZ bucket_infor)r5rRrr6r6r7_bucket_exists s   zHfFileSystem._bucket_existsr0cCsdtdBdtdBdtdBfdd}||}|std|dd d krQd|dd d }d|dd d}||\}}|sKt||t||d S|dd dtj vrud|vrgtd|dd \}}tj |}ntj }| dd kr=dd|dddvr|dd \}} d| vrt| } | dur|d| fvrtd| d} | } n | dd \} } nd} |t| |}||||\} }| st||nd} d|ddd} d|ddd}|dd }d|dd d}| }|} ||||\} }| s._align_revision_in_path_with_revisionz [!WARNING] > Note: When possible, use `HfApi.delete_file()` for better performance. Args: path (`str`): Path to delete. recursive (`bool`, *optional*): If True, delete directory and all its contents. Defaults to False. maxdepth (`int`, *optional*): Maximum number of subdirectories to visit when deleting recursively. revision (`str`, *optional*): The git revision to delete from. r)rrrBc"g|] }|s|jqSr6isdirrr0r}r0r4r6r7 "z#HfFileSystem.rm..rcrr6rrr4r6r7rrcSsg|]}t|dqS))rC)r)r}rCr6r6r7rrzDelete  z recursively rINz up to depth rr)rAr@rs operationsrBrrr)rZ expand_pathrrQrwrrR create_commitrAr@rsrBrLrr8) r5r0rrrBr[rpathsrZ paths_in_reporrr6r4r7rms(    zHfFileSystem.rmTdetailrefreshc s|j|d}|z|jf||d|}WnBty[|js(tdz|j|f||d|}Wn tyDg}Ynwfdd|D}t|dkrYtdYnw|r`|Sdd|DS)a List the contents of a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.ls). > [!WARNING] > Note: When possible, use `HfApi.list_repo_tree()` for better performance. Args: path (`str`): Path to the directory. detail (`bool`, *optional*): If True, returns a list of dictionaries containing file information. If False, returns a list of file paths. Defaults to True. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `list[Union[str, dict[str, Any]]]`: List of file paths (if detail=False) or list of file information dictionaries (if detail=True). r)rrBNcg|] }|dkr|qSnamer6r}orr6r7rz#HfFileSystem.ls..rcSsg|]}|dqSrr6rr6r6r7rs)rr8_ls_treerr0rrlen)r5r0rrrBr[routr6rr7lss$  "    zHfFileSystem.lsc sT|dur|n jdurjnd}j||d}|}|j}|r"|nd}g} |jvr |s j|} | | g} |rdtfdd| D} | r| \} |dusZ|kr| djvri| | dnj| d} | | | fdd| D| sLg}|rt |t rfd d| D}|r| s|r|rt j | |}|d s||ks|t| |vr|d n||durԈt|dd }||8}fd d| D} tjD]}|d rj|dqjd| j|d |||d | St |trj|j|j |d}njj|j|j |||j|jd}|D]z}|d |j }t |t rN||j!d|j"|j#|j$|j%|j&d}n-t |t'rb||j!d|j$|j(|j)d}nt |t*rs|dd|j+|j%d}n|dd|j)d}|d}j,|g||t|dd |dus|kr| |q-| S)NFrrrcs g|] }|ddkr|fqSrd directoryr6r} path_infodepthr6r7r7s z)HfFileSystem._ls_tree..rcs$g|]}|ddkrd|fqS)rdrrr6rrr6r7rBs   cs&g|]}|ddur|dqS) last_commitNr)rrr4r6r7rLs&r2cs"g|] }|dds|qS)rr2) startswithr) common_pathr6r7r]rT)rrrBrur)prefixr)rexpandrBr@filersizerdblob_idlfsxet_hashrsecurity)rrrdrmtime uploaded_atrrrrrdtree_idr)rrrdr)-rurr8r/rzextendrpopleftappendrr?osr0 commonprefixendswithr r3rrrlistrr^rrQ_list_bucket_tree_with_foldersrRrwZlist_repo_treerArBr@r'rrrrrrr#rrr(r setdefault)r5r0rrrBrurr root_pathrZcached_path_infosZdirs_not_in_dircacheZ dirs_to_visitdir_infoZdirs_not_expanded common_prefixZcommon_path_depthZ cached_pathtreer cache_pathZcache_path_info parent_pathr6)rrr5r7rs        E      zHfFileSystem._ls_treerc Cs|jj|||d}i}|rd|dnd}g}|D]O}|||s$qtt|jjd| dD]5} tdt | |j d} | j|vrO|| | || j<q3| j durh|| j} | j dusd| j | j krh| j | _ q3q|swt d|d |d |S) z|j||dd}tj|g|Ri|EdHdS)a Return all files below the given path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk). Args: path (`str`): Root path to list files from. Returns: `Iterator[tuple[str, list[str], list[str]]]`: An iterator of (path, list of directory names, list of file names) tuples. rBrN)rrLr8rVwalk)r5r0rZr[r\r6r7rs $zHfFileSystem.walkc s0|j||dd}tj|fd|i|S)ah Find files by glob-matching. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob). Args: path (`str`): Path pattern to match. Returns: `list[str]`: List of paths matching the pattern. rBrr)rrLr8rVglob)r5r0rr[r\r6r7rs zHfFileSystem.globwithdirsc s|dur |dkr td|j||d}|}z|j|fd||d|Wn-tySz|j|fd|i|dd krB|iiniWn tyPiYnwYn(w|s^d d Dn|j|fi|} | dd krr| gnd dDt} |s| Sfdd| DS)a List all files below path. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find). Args: path (`str`): Root path to list files from. maxdepth (`int`, *optional*): Maximum depth to descend into subdirectories. withdirs (`bool`, *optional*): Include directory paths in the output. Defaults to False. detail (`bool`, *optional*): If True, returns a dict mapping paths to file information. Defaults to False. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to list from. Returns: `Union[list[str], dict[str, dict[str, Any]]]`: List of paths or dict of file information. Nrzmaxdepth must be at least 1rT)rrrrBrdrcSsg|] }|ddkr|qSrr6rr6r6r7r"rz%HfFileSystem.find..rcSsi|]}|d|qSrr6rr6r6r7r'rz%HfFileSystem.find..cr|r6r6)r}rrr6r7r,r)rrr8rrinfoFileNotFoundErrorr) r5r0rrrrrBr[rrnamesr6rr7finds4     zHfFileSystem.findpath1path2c KsH|j||d}|j||d}t|tst|trtd|j|jko'|j|jk}|rSd|d|}|jj|j|j|j| d|| ddt |j |j |jdgd n?|j |d |jd } | } Wd n1skwYd|d|}|jj| |j |j|j|j|j| d|| dd |j|d |j|d d S)a Copy a file within or between repositories. > [!WARNING] > Note: When possible, use `HfApi.upload_file()` for better performance. Args: path1 (`str`): Source path to copy from. path2 (`str`): Destination path to copy to. revision (`str`, *optional*): The git revision to copy from. rz)Copy from/to buckets is not available yetzCopy z to rrrI)Zsrc_path_in_reporCZ src_revision)rAr@rBrrrrNZpath_or_fileobjrCrArsr@rBrrr)rrrQrr@rArwrrBrLrrCopenread upload_filersrr8) r5rrrBr[Zresolved_path1Zresolved_path2Z same_reporfcontentr6r6r7cp_file.sN      zHfFileSystem.cp_filecKs|j|fii|ddi}d|vr&|ddur!td|d|djSd|vr2|dr2|dSd|vr>|dr>|dStd |d |d ) ak Get the last modified time of a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified). Args: path (`str`): Path to the file. Returns: `datetime`: Last modified time of the file. ruTrNz9'modified' is not implemented for repository paths like 'rrrz'Cannot determined 'modified' for path 'z ' (info: ))rrdate)r5r0r[rr6r6r7modifiedjs   zHfFileSystem.modifiedc sN|j|d}||d|jdur|jnd}|jsRdddt|tr+dd<t|trQ|rQ|jj|j |j |j d d }idt |j |j|jd d nt|tr{|}|j||d fdd|j|D} | svtd| dnd|}|s||jvr||||jvrfdd|j|D} | std| d|sdus|rrddur|jj|j |j||j |j d} | std| d} t|j |j |j d|jd} t| tr| d| j| jd| j| j| j| j| jdn| d| jdd| j| jd|sfdddDdus%JS)aS Get information about a file or directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info). > [!WARNING] > Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()` for better performance > (or `HfApi.get_bucket_paths_info()` or `HfApi.bucket_info()` for buckets) Args: path (`str`): Path to get info for. refresh (`bool`, *optional*): If True, bypass the cache and fetch the latest data. Defaults to False. revision (`str`, *optional*): The git revision to get info from. Returns: `dict[str, Any]`: Dictionary containing file information (type, size, commit info, etc.). rruNFrr)rrrdr)r@rB)oidtitler)rr)rcrrr6rrr6r7rrz%HfFileSystem.info..crrr6rrr6r7rr)rrBr@rI)rCrHr2rrrcr|r6r6)r}krr6r7rrz%HfFileSystem.info..) rr8rLrur0rr?rwZlist_repo_commitsrAr@rBr& commit_idrZ created_atrQrrrzrZget_paths_inforCrHr'rrrrrrr) r5r0rrBr[rrurrZout1Z paths_inforrr6)rr0r7rs           &    zHfFileSystem.infocKsDz|ddr |||j|fi|WdSty!YdSw)a Check if a file exists. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists). > [!WARNING] > Note: When possible, use `HfApi.file_exists()` for better performance. Args: path (`str`): Path to check. Returns: `bool`: True if file exists, False otherwise. rFT)rLrrOSError)r5r0r[r6r6r7existss   zHfFileSystem.existscC*z ||ddkWStyYdSw)ai Check if a path is a directory. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir). Args: path (`str`): Path to check. Returns: `bool`: True if path is a directory, False otherwise. rdrFrr r5r0r6r6r7r    zHfFileSystem.isdircCr )a` Check if a path is a file. For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile). Args: path (`str`): Path to check. Returns: `bool`: True if path is a file, False otherwise. rdrFr rr6r6r7isfilerzHfFileSystem.isfilecCsl||}t|tr|jd|jdt|j}nt|j|j |j |j |jd}| |r4| ddd}|S)z Get the HTTP URL of the given path. Args: path (`str`): Path to get URL for. Returns: `str`: HTTP URL to access the file or directory on the Hub. z /buckets/z /resolve/)r@rBrrz/tree/r)rrrQrrrRrr0r rArCr@rBrreplace)r5r0rurlr6r6r7r.s   zHfFileSystem.urlc sT|d}t|dh}t|ttfrt|dkr)tj||f||d|St |r0|}n| |r>t j |dddSt|t tfrPt j t j|ddd}|dur]t|d}d}|} |j||d } |j||d d } || z+t|| ||| d|jt|tr|jndd || W|r|dSdS|r|ww) ad Copy single remote file to local. > [!WARNING] > Note: When possible, use `HfApi.hf_hub_download()` or `HfApi.download_bucket_files` for better performance. Args: rpath (`str`): Remote path to download from. lpath (`str`): Local path to download to. callback (`Callback`, *optional*): Optional callback to track download progress. Defaults to no callback. outfile (`IO`, *optional*): Optional file-like object to write to. If provided, `lpath` is ignored. rBr)callbackoutfileT)exist_okNFwbrr)r temp_fileZdisplayed_filename expected_sizeZ resume_sizeheadersZ _tqdm_bar)rLsetkeysrrrrrVget_filerrrmakedirsr=r r0dirnamertellrrZset_sizer!rr8rw_build_hf_headersZtqdmseekclose) r5ZrpathZlpathrrr[rBZunhandled_kwargsZ close_fileZ initial_posZresolve_remote_pathrr\r6r7rHsF        zHfFileSystem.get_filecCtd)zA context within which files are committed together upon exit Requires the file class to implement `.commit()` and `.discard()` for the normal and exception cases. (Transactional commits are not supported.rr4r6r6r7 transactions zHfFileSystem.transactioncCr#)z@Begin write transaction for deferring files, non-context versionr$r%r4r6r6r7start_transactionszHfFileSystem.start_transactioncCstt||j|j|ffSrU) make_instancerdrirjrfr4r6r6r7 __reduce__s zHfFileSystem.__reduce__cCst|jt|jt|jdS)N)rzrxry)rrzrxryr4r6r6r7rfsz HfFileSystem._get_instance_staterU)rNN)FNN)TFN)FFNNN)NFFFN)FNr1N)5r9r:r;r<Z root_markerprotocolr=boolintrW classmethodr_tuple Exceptionrrr?rQrrrrrrrdictr rrrr#r$rrrrrrr rrr rrrrrpropertyr&r'r)rfrpr6r6r\r7rqsB     h$  3 .   ,($ =<(n>   rq) metaclasscseZdZddedededBffdd ZfddZd ed ed efd d Z dddZ dde d dfddZ dfdd Z d efddZZS)rNfsr0rBc stz |j||d|_Wnty'}zd|ddvr"t|d|d}~wwtj||jfi||dS)NrwrrIB. Make sure the repository and revision exist before writing data.)rrrrLrVrWr8)r5r4r0rBr[rr\r6r7rWszHfFileSystemFile.__init__cst|dsdStSNr)hasattrrV__del__r4r\r6r7r9s  zHfFileSystemFile.__del__startendr1cCsLdd|d|di|jj}|}td||tjd}t||jS)Nrangebytes=-rGETrr) r4rwr rr+rHF_HUB_DOWNLOAD_TIMEOUTr*r)r5r:r;rrrr6r6r7 _fetch_ranges zHfFileSystemFile._fetch_rangecCstjddd|_dS)Nzhffs-F)rr)tempfileNamedTemporaryFilerr4r6r6r7_initiate_uploadsz!HfFileSystemFile._initiate_uploadFfinalc Cs|jd|j}|j||rg|jt|jtr1|j j j |jj |jj |jjfgdn#|j j j|jj |jj|jj|j j|jj|jj|jd|jddt|jj |j j|jddSdS)Nr)addrrrr)bufferr!rrwriter"rrrQr4rwrrRrr0rrCrArsr@rBr[rLrremoverr8)r5rGblockr6r6r7 _upload_chunks0         zHfFileSystemFile._upload_chunkrcs|jdkr;|dus |dkr;|jdkr;|jj|jddd}|}|jt|7_|WdS1s6wYt|S)zRead remote file. If `length` is not provided or is -1, the entire file is downloaded and read. On POSIX systems the file is loaded in memory directly. Otherwise, the file is downloaded to a temporary file and read from there. rNrr)rt)rlocr4rr0rrrV)r5lengthrrr\r6r7rs$ zHfFileSystemFile.readcC|j|jSrUr4rr0r4r6r6r7rzHfFileSystemFile.urlrUr*)Fr)r9r:r;rqr=rWr9r-bytesrCrFr,rMrrrpr6r6r\r7rs    rc seZdZ    d!dededededBd ed ef fd d Zd"d edefddZd#defddZd#de e dede fddZ defddZ fddZ ddZdd ZZS)$rrNrnoner4r0rrBrt cache_typec s|dkr td||dkrtd|d|vr"td|dz |j||d|_Wn"tyN}zd|d d vrDt|d |WYd}~nd}~ww|jdd |_tj||jf|||d |d|_ |t |_ d|_ t |_dS)Nrz:HfFileSystemStreamFile only supports block_size=0 but got rUz?HfFileSystemStreamFile only supports cache_type='none' but got r5z;HfFileSystemStreamFile only supports reading but got mode='rrrrIr6)rrrrtrV)rrrrrLr8detailsrVrWresponser _exit_stack_stream_iterator bytearray_stream_buffer) r5r4r0rrBrtrVr[rr\r6r7rWs>    zHfFileSystemStreamFile.__init__rNwhencecCs2|dkr |dkr dS||jkr|dkrdStd)NrrzCannot seek streaming HF file)rNr)r5rNr^r6r6r7r!s zHfFileSystemStreamFile.seekrrOcCs|jdur |d} z |jdus|jdurWdS||j|}|jt|7_|WStyH|jdur=|j|r@|d}Ynwq )zRead the remote file. If the file is already open, we reuse the connection. Otherwise, open a new connection and read from it. If reading the stream fails, we retry with a new connection. NFT)rY_open_connectionr[_read_from_streamrNrr0r")r5rOZ retried_oncerr6r6r7rs&    zHfFileSystemStreamFile.readiteratorr1cCs|dkrdS|dkr"t|j}|j|D]}||qt|S|t|jkr:t|jd|}|jd|=|St|j}|j|D]+}|t|}|t|krZ||qF||d||j||dt|St|S)zRead up to `length` bytes from stream buffer and stream. If length < 0, read until EOF. If EOF is reached before length, fewer bytes may be returned. rr_N)r\r]rrrTr)r5rbrObufchunkresultZneedr6r6r7ra:s,         z(HfFileSystemStreamFile._read_from_streamcCrPrUrQr4r6r6r7r\rRzHfFileSystemStreamFile.urlcs"t|dsdS|jtSr7)r8rZr"rVr9r4r\r6r7r9_s   zHfFileSystemStreamFile.__del__cCst|j|j|j|j|jjffSrU)reopenr4r0r blocksizecacherr4r6r6r7r)fsz!HfFileSystemStreamFile.__reduce__c Cs|jd|_|}|jj}|jdkr d|jd|d<|j t d||t j d|_ zt|j WntyR}z|j jdkrMd|_ WYd}~dSd}~ww|j |_dS) z%Open a connection to the remote file.Nrr=r>Ranger?r@i)r]rr[rr4rwr rNrZ enter_contextr,rrArYr*r status_codeZ iter_bytes)r5rrrr6r6r7r`is0     z'HfFileSystemStreamFile._open_connection)rNrrU)rrS)r9r:r;rqr=r-rWr!rrrTrarr9r)r`rpr6r6r\r7rs2#" rrBr1cCst|r|St|SrU)r"r safe_quoterr6r6r7rNsrNscCs t|ddS)NrI)safe)r)rmr6r6r7rls rlr0rcCsN|}t|tr |d}nt|tr|d}n t|tr"|d}t||)Nz (repository not found)z (revision not found)z (invalid repository id))rrrr)r)r0rmsgr6r6r7rs       rr4rrtrVcCs|j||||dS)NrW)r)r4r0rrtrVr6r6r7rfsrfcCs0||i|}|D] \}}t|||q |SrU)rgrh)rYrZr[rmr4rnror6r6r7r(sr()YrrDr` collectionsrcollections.abcrr contextlibrcopyr dataclassesrrr itertoolsr pathlibr r typingr rr urllib.parserrZfsspecZhttpxZfsspec.callbacksrrrZ fsspec.configrZ fsspec.utilsrrIrZ _commit_apirrerrorsrrrrrZ file_downloadr r!Zhf_apir"r#r$r%r&r'r(utilsr)r*r+r,Zutils.insecure_hashlibr-r.r?rQrdZAbstractFileSystemrSr>rTrqspecZAbstractBufferedFilerrr=rNrlr0rr-rfr(Zhffsr6r6r6r7sf         $   5$K