o *jl6 @sddlZddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z m Z ddlmZddlmZeZddZd d Zd efd d Zd efddZd efddZded efddZded efddZddZddeddfddZde eeee fd efddZ ! "d2de eeee fd#e ed$e ed%e e d e!f d&d'Z" ! "d2d(e eefd#ed$e ed%e e d e!f d)d*Z# + ! "d3de eeee fd#e ed,ed$e ed%e e d e!f d-d.Z$d/ed e fd0d1Z%dS)4N)Path)Errorcopy2copystat)BinaryIOOptionalUnion)urlparse) get_loggercCsxt|}|j}|j}|dur|dusdS|jsgn|j}|r*|ddvr*|dn|t|dkr:|ddvr:dSdS)zto decide if a func could receive dict inputs or not Args: func (class): the target function to be inspected Returns: bool: if func only has one arg ``input`` or ``inputs``, return True, else return False NFr)selfcls)inputZinputsT)inspectgetfullargspecvarargsvarkwargspoplen)funcZfull_args_specrrrr\/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/utils/file_utils.pyfunc_receive_dict_inputss rcCstjtddd}|S)z1 default base dir: '~/.cache/modelscope' z.cacheZ modelscopeZhub)ospath expanduserrhomejoinpath)Zdefault_cache_dirrrr get_default_modelscope_cache_dir)srreturncCstjtdtS)zGet modelscope cache dir, default location or setting with MODELSCOPE_CACHE Returns: str: the modelscope cache root. ZMODELSCOPE_CACHE)rrrgetenvrrrrrget_modelscope_cache_dir2s r"cCtjtdS)z[Get model cache root path. Returns: str: the modelscope model cache root. modelsrrjoinr"rrrrget_model_cache_root=sr'cCr#)zGet dataset raw file cache root path. if `MODELSCOPE_CACHE` is set, return `MODELSCOPE_CACHE/datasets`, else return `~/.cache/modelscope/hub/datasets` Returns: str: the modelscope dataset raw file cache root. Zdatasetsr%rrrrget_dataset_cache_rootFsr( dataset_idcC$t}|dur |Stj||dS)zGet the dataset_id's path. dataset_cache_root/dataset_id. Args: dataset_id (str): The dataset id. Returns: str: The dataset_id's cache root path. N/)r(rrr&)r)Z dataset_rootrrrget_dataset_cache_dirQ r,model_idcCr*)zcache dir precedence: function parameter > environment > ~/.cache/modelscope/hub/model_id Args: model_id (str, optional): The model id. Returns: str: the model_id dir if model_id not None, otherwise cache root dir. Nr+)r'rrr&)r. root_pathrrrget_model_cache_dir`r-r0cCs8t|d }|}Wd|S1swY|S)Nr)openread)rftextrrr read_fileos   r6Fc Cst|}|dur|||}nt}tj||dg} |D]} | |vr$qtj|| } tj|| } zUtj| rpt| } |rOt| | t | | | dn7tj | sY|rYWqtj | rjt | | ||||dn|| | ntj | rt | | ||||dn|| | Wqt y}z| |jdWYd}~qd}~wty}z| | | t|fWYd}~qd}~wwzt ||Wn$ty}zt|dddur| ||t|fWYd}~nd}~ww| rt | |S)z7copy from py37 shutil. add the parameter dirs_exist_ok.N)exist_ok)follow_symlinks) dirs_exist_okrwinerror)rlistdirsetmakedirsrr&islinkreadlinksymlinkrexistsisdir copytree_py37rextendrOSErrorappendstrgetattr)srcdstsymlinksignore copy_functionignore_dangling_symlinksr9names ignored_nameserrorsnamesrcnamedstnamelinktoerrwhyrrrrCvsr          rCfile_path_or_objcCsrt|ttfrt|}|jSt|trt|St|tjr5| }| dt j | }| ||St d)NrzCUnsupported type: must be string, Path, bytes, or io.BufferedIOBase) isinstancerGrstatst_sizebytesrioBufferedIOBasetellseekrSEEK_END TypeError)rX file_pathcurrent_positionsizerrr get_file_sizes    rf [Calculating]Tbuffer_size_mb tqdm_desc disable_tqdmc Cs,ddlm}t|}|dkr-d}d}t|ttfr't|tr |nt|}|j}d|d}|dd}t} ||dd d d ||d } t|ttfrd} t |d (} | |} rq| | | t | 7} | t | | |} sXWd n1s{wY| } | |krtd|d| d|| }nst|tr| || } | t |t |}nYt|tjrd} |dtj| |} r| | | t | 7} | t | | |} s| } |dtj| |krtd|d| d|| }n| td| || |dS)aCompute SHA256 hash for a file path, bytes, or file-like object. Args: file_path_or_obj: File path, bytes, or file-like object. buffer_size_mb: Read buffer size in MB. Default 16MB. tqdm_desc: Progress bar description. disable_tqdm: Whether to disable progress bar. Returns: dict with keys: file_path_or_obj, file_hash, file_size. rtqdmi@Fz Large Filez[Validating Hash for ]TB)totalinitial unit_scale dynamic_ncolsunitdescdisablerbN4File size changed during hash computation: declared  bytes, actually hashed % bytes. File may have been modified: z5Input must be str, Path, bytes or a io.BufferedIOBaserX file_hash file_size) tqdm.autormrfrYrGrrRhashlibsha256r2r3updater hexdigestloggerwarningr\r]r^r`rSEEK_SETclose ValueError)rXrirjrkrm declared_sizerRr buffer_sizer}progress bytes_hashedr4Z byte_chunkr~rrr get_file_hashs             rrcc s*ddl}ddl}ddlm}ttj}|dd|jdddgfdd}|j |d d } | t } d} ||||d d d dd }  } | durVn| | | t| 7} | t| qM| | ddur{d| |krtd |d| d| | dS)aCompute SHA256 with async I/O double-buffering for high-latency storage. Uses a producer-consumer pattern: a background thread reads file chunks into a bounded queue while the main thread computes the hash. This overlaps I/O latency with CPU computation. rNrlro)maxsizec szMz&td} |}|sn|q Wdn1s!wYWnty=}z |d<WYd}~n d}~wwWddSWddSdw)NrxTr)r2r3put Exception)r4chunkerZ chunk_queuercZ read_errorrr _producerFs&   z'_get_file_hash_async.._producerT)targetdaemonrp)rqrvrwrtrursZ unit_divisorryrzr{r|)queue threadingrrmrGrrgetsizeQueueThreadstartrrgetrrr&rrrr)rcrirjrkrrrmrrZ reader_threadr}rrrrrr_get_file_hash_async.s`          r async_threshold_mbcCsNt|ttfrtjt|}||ddkrt||||dSt||||dS)aCompute SHA256 hash with automatic optimization for large files. For file paths larger than async_threshold_mb, uses async double-buffered I/O to overlap disk reads with hash computation. For smaller files or non-path inputs (bytes, BinaryIO), falls back to synchronous hashing. Args: file_path_or_obj: File path, bytes, or file-like object. buffer_size_mb: Read buffer size in MB. Default 16MB for NAS optimization. async_threshold_mb: File size threshold for async mode. Default 32MB. tqdm_desc: Progress bar description. disable_tqdm: Whether to disable progress bar. Returns: dict with keys: file_path_or_obj, file_hash, file_size. ro)rirjrk)rYrGrrrrrr)rXrirrjrkr~rrrcompute_file_hash}srurl_or_filenamecCst|jdko tj| S)z5 Check if a given string is a relative path. )r schemerrisabs)rrrris_relative_paths r)rgrhT)rgrrhT)&rrr]rpathlibrshutilrrrtypingrrr urllib.parser Zmodelscope.utils.loggerr rrrrGr"r'r(r,r0r6rCr\intrfbooldictrrrrrrrrs         I _  Q +