o 0j+7@sdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZd d lmZmZmZmZmZmZd d lmZd d lmZmZe r^d dlm Z e!e"Z#e$dZ%dZ&dddZ'eGdddZ(e    dCdee(de)dBde)de)de)dBde)dBde*e)e)fdBde+e)dBde,e+e*e+e*e)dBffddZ-Gd d!d!eZ.Gd"d#d#eZ/   dDd$d%d&e*de)dBde*e)e)fdBde)dBddf d'd(Z0d)e*fd*d+Z1d,e*fd-d.Z2d/e*fd0d1Z3d$d%d2e)ddfd3d4Z4d$d%d5e*d6e5d2e)ddf d7d8Z6d5e*d9e(d6e5de+e)fd:d;Z7dd?Z8d$d%d@e+e)d6e5de+e*fdAdBZ9dS)Ez.Git LFS related type definitions and utilitiesN)Iterable) dataclass)ceil)getsize) TYPE_CHECKINGBinaryIO TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_urlhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<edefddZ edefd d Z ed e fd d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathcCsZt|}t|d}|ddd}t|}Wdn1s!wY||||dS)Nrbrrr)ropenpeekr)clsrrfilershar#T/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/huggingface_hub/lfs.py from_pathIs   zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr)rrr)rdigestlen)r r&r"r#r#r$ from_bytesQs zUploadInfo.from_bytesfileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nrrr)readseekioSEEK_SETrtell)r r*rr"rr#r#r$ from_fileobjVs zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr%r)rr0r#r#r#r$r6s  r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs|dur|ntj}d}|tjvrtj|}|d||d} d|dur&|nddgdd |Dd d } |dur?d t|i| d <itt|d|pJi}td| || d} t| | } | dd} t | t skt d| d}t |t rw|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|] }|j|jdqS)oidr)rhexr).0rEr#r#r$ s z'post_lfs_batch_info..r) operationrAobjectsZ hash_algonameref)r;POSTr@jsonrNzMalformed response from serverZtransfercSsg|] }d|vrt|qSerror)_validate_batch_actionsrKobjr#r#r$rLcSsg|] }d|vrt|qSrT)_validate_batch_errorrWr#r#r$rLrY)r ZENDPOINTZREPO_TYPES_URL_PREFIXESr LFS_HEADERSr rrrSget isinstancelist ValueErrorr9)r:r;r<r=r>r?r@rAZ url_prefixZ batch_urlpayloadrespZ batch_inforNZchosen_transferr#r#r$post_lfs_batch_info`sB,        rbc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r1r2r3r7r6r9r#r#r#r$rcs  rcc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rIpartsN)r1r2r3r4r9r6r^rcr#r#r#r$rfs rfrMrlfs_batch_actionc CsFt||d}|durtd|jddS|dd}t||dd}|dur2t||di}|d} t|d |d } | durjzt| } Wntt fy`td | d wt ||| | d nt || d|durt|t|d |} t d| t ||d|jj|jjdd} t| t|jddS)a9 Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrEverifyheader chunk_sizehref)r?zTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rMrkrl upload_url)rMrnrQ)r;r@rHrRz: Upload successful)rVr\loggerdebugZ path_in_repo_validate_lfs_actionr r7r_ TypeError_upload_multi_part_upload_single_partrr upload_inforrJrr) rMrhr;r@r?ri upload_action verify_actionrkrlrnZ verify_urlZ verify_respr#r#r$ lfs_uploadsD        rx lfs_actioncCs:t|dtr|ddust|dtstd|S).validates response from the LFS batch endpointrmrkNz"lfs_action is improperly formatted)r]r\r9dictr_)ryr#r#r$rq s rqlfs_batch_actionscCspt|dtrt|dtstd|did}|did}|dur.t||dur6t||S)rzrIrz)lfs_batch_actions is improperly formattedrirErjN)r]r\r9r7r_rq)r|rvrwr#r#r$rVs rVlfs_batch_errorcCsht|dtrt|dtstd|d}t|tr.t|dtr.t|dts2td|S)rzrIrz'lfs_batch_error is improperly formattedrUmessagecode)r]r\r9r7r_r{)r} error_infor#r#r$rZ"s  rZrncCsH|jdd}td||d}t|WddS1swYdS)aK Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Raises: [`HfHubHTTPError`] If the upload resulted in an error. TZ with_tqdmPUTr&N)as_filerr)rMrnr*responser#r#r$rt0s "rtrkrlcCsHt||j|d}t|||d}td|t||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rkrurl)rMsorted_parts_urlsrlrQ)rSr@N) _get_sorted_parts_urlsru_upload_parts_iterativelyr_get_completion_payloadrrJr[r)rMrkrlrnrresponse_headersZcompletion_resr#r#r$rsDs rsrucCsNddtdd|DdddD}t|}|t|j|kr%td|S)NcSsg|]\}}|qSr#r#)rK_rnr#r#r$rL\sz*_get_sorted_parts_urls..cSs2g|]\}}|rt|dkrt|d|fqS)r )isdigitr(r7)rKZpart_numrnr#r#r$rL_s  cSs|dS)Nrr#)tr#r#r$dsz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr(rrr_)rkrurlZsorted_part_upload_urlsZ num_partsr#r#r$r[s rrrIcCsfg}t|D]'\}}|d}|dus|dkr#td|d|d||d|dq||dS)NrerCzInvalid etag (`z`) returned for part r )rdre)rIrg) enumerater\r_append)rrIrgZ part_numberrkrer#r#r$rms  rrc Csg}|jdd<}t|D].\}}t||||d}td||d}t|||jWdn1s6wYq Wd|S1sGwY|S)NTr)Z seek_fromZ read_limitrr)rrrrrrr@) rMrrlr@r*Zpart_idxZpart_upload_urlZ fileobj_sliceZpart_upload_resr#r#r$r|s&   r)NNNN)NNN):r4r-recollections.abcr dataclassesrmathrZos.pathrtypingrrr urllib.parser Zhuggingface_hubr utilsr r rrrrZ utils._lfsrZ utils.sharrZ _commit_apirZ get_loggerr1rocompileZ OID_REGEXZLFS_MULTIPART_UPLOAD_COMMANDr[rr9r{r^tuplerbrcrfrxrqrVrZrtr7rsrrrr#r#r#r$s            )  V  G