o *j#@sddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZeZd Zd Zd Zd Zd ZdZddZGdddZdS))print_functionN)hash_url_to_filename)HubApi)DataDownloadConfig) MS_CACHE_HOME)"DEFAULT_DATA_ACCELERATION_ENDPOINTMetaDataFields UploadMode) get_loggerZAccessIdZ AccessSecretZ SecurityTokenBucketZ BackupDirZDircs:ddlm}ddlmGfddd|}|S)zCreate a credentials provider for oss2 with lazy import. Returns an instance that subclasses oss2.CredentialsProvider so that the oss2 SDK can call ``get_credentials()`` automatically when the token expires or authentication is needed. r)CredentialsProvider Credentialscs*eZdZfddZfddZdS)z?_create_credential_provider.._CredentialProviderWrappercs&|_|_|_|_t|_dS)N)api dataset_name namespacerevision threadingLock_lock)selfrrrrf/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/msdatasets/utils/oss_utils.py__init__'s zH_create_credential_provider.._CredentialProviderWrapper.__init__cs^|j"|jj|j|jd|jd}|t|t|tdWdS1s(wYdS)ztoss2 SDK will call this method automatically when it finds the token is expired or needs authentication.FrrZ check_cookier)Z access_key_idZaccess_key_secretZsecurity_tokenN) rr!get_dataset_access_config_sessionrrr ACCESS_ID ACCESS_SECRETSECURITY_TOKEN)r oss_configr rrget_credentials.s$zO_create_credential_provider.._CredentialProviderWrapper.get_credentialsN)__name__ __module__ __qualname__rr!rrrrrrrr_CredentialProviderWrapper%sr&)oss2r Zoss2.credentialsr)rrrrr r&rr%r_create_credential_providers  r(c @sXeZdZdZddZeddZdededefd d Z d ed ed e de def ddZ dS) OssUtilitiesz` A utility class for handling Alibaba Cloud OSS operations such as upload and download. cCs||_||_||_t|_|jj|j|jd|jd}tddkr%t|_ n d|dd|_ tj t d|_ t|_d |_d |_d |_d d l}|j|j d|_|j|j d|_t|j|j|j|jd}||}|t|_|j||j |j|ddd|_|t |_!|t"|_#d S)zh Initializes the OssUtilities with the given OSS configuration and dataset information. FrZENABLE_DATASET_ACCELERATIONTruezhttps://Regionz .aliyuncs.comztmp/resumable_storeii rN)rootrzoss-)authendpoint bucket_nameregion)$rrrrrrosgetenvrr/pathjoinrZresumable_store_root_pathmultiprocessing cpu_count num_threads part_sizemultipart_threshold max_retriesr'ZResumableDownloadStoreresumable_store_downloadZResumableStoreresumable_store_uploadr(ZProviderAuthV4BUCKETr0r lstripbucketDIRoss_dirBACK_DIRoss_backup_dir)rrrrr r'Zcredential_providerr.rrrrFsV     zOssUtilities.__init__cCs8|rtdt|t|}td|ddddSdS)Ndz {0}% T)endflush)intfloatprintformat)Zconsumed_bytes total_bytesZraterrr _percentageyszOssUtilities._percentage oss_file_namedownload_configreturnc Cs<|j}tj|j|}tj|j|}|j}d}|r(|j|}|r(|t j }d} ddl } z;|r5|} n |j |r=|n|} t| dd} tj|| } |jsUtj| sh| j|j | | |j|j|j|j|jdW| Sty}z&td|d|d | d 7} | |jkrtd |d |WYd}~nd}~wwq/) a< Downloads a file from OSS to the local cache. Args: oss_file_name (str): The name of the file in OSS to download. download_config (DataDownloadConfig): Configuration for the download process. Returns: str: The local path to the downloaded file. FrNT)etag)r@keyfilenamestoreZmultiget_thresholdr9progress_callbackr8zError downloading : , trying again...zFailed to download  due to exceeded retries.) cache_dirr2r4r5rBrDsplitZ meta_args_mapgetrZ ARGS_BIG_DATAr'r@ object_existsrZforce_downloadexistsZresumable_downloadr<r:r9rNr8 Exceptionloggerwarningr;error)rrOrPr[Z candidate_keyZcandidate_key_backupr\Zbig_dataZ args_dict retry_countr'Z file_oss_keyrTZ local_patherrrdownloadsf          zOssUtilities.downloadoss_object_namelocal_file_pathindicate_individual_progress upload_modec Csd}tj|j|}|r|j}nd}ddl} z2|j|} |tj kr5| r5t d|d|j dW|S|j |j|||j|j|j||jdW|Sty{} z&t d|d | d |d 7}||jkrqt d |d | WYd} ~ nd} ~ wwq)a Uploads a local file to OSS. Args: oss_object_name (str): The name of the object in OSS. local_file_path (str): The local file path to upload. indicate_individual_progress (bool): Whether to show individual progress. upload_mode (UploadMode): The upload mode (e.g., OVERWRITE, APPEND). Returns: str: The OSS object key where the file is uploaded. rNTzSkip z in case of z mode.)r@rSrTrUr:r9rVr8zError uploading rWrXrYzFailed to upload rZ)r2r4r5rBrNr'r@r^r APPENDrainfovalueZresumable_uploadr=r:r9r8r`rbr;rc) rrgrhrirjrdZ object_keyrVr'existrerrruploadsR      zOssUtilities.uploadN) r"r#r$__doc__r staticmethodrNstrrrfboolr rorrrrr)As"3  =r)) __future__rr6r2rZdatasets.utils.file_utilsrZmodelscope.hub.apirZ.modelscope.msdatasets.download.download_configrZmodelscope.utils.config_dsrZmodelscope.utils.constantrrr Zmodelscope.utils.loggerr rarrrr>rCrAr(r)rrrrs&      &