o *j d@sUdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZmZmZmZmZmZmZddlmZmZmZmZmZmZmZddlm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/dd l0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBmCZCddlDmEZEddlFmGZGddlHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPddlQmRZRmSZSddlTmUZUddlVmWZWddlXmYZYzddl0mZZ[Wne\yddl'm[Z[YnweYZ]da^e eGe_d<deGfd d!Z`  dDd"d#Za  dDd$ebd%ebd&e e ebd'e ede ebf d(d)Zc dEd%ebd'e ede ebe ebffd*d+Zdd,ebd-ebd'ed.ebdebf d/d0Ze 1dFd2ebd3ebd4ebd.ebd5e ebebebebfd'e ed6e efde e ebebffd7d8Zg   dGd,ebd.ebd'ed9e ebd6e efde)f d:d;Zhde)fdd?Zj   dGde)fd@dAZkde)fdBdCZldS)IacDataset module factory functions and data file resolution for ModelScope. This module provides ModelScope-specific implementations of dataset module loading (both script-based and script-free) and data file pattern resolution. These functions are monkey-patched onto the ``datasets`` library internals by :func:`~hf_datasets_util.load_dataset_with_ctx`. N)partial)Path)DictListOptionalSequenceTupleUnion) BuilderConfigDownloadConfig DownloadModeFeaturesVersionconfig data_files)FILES_TO_IGNORE DataFilesDictEmptyDatasetError_get_data_files_patterns"_is_inside_unrequested_special_dir?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirsanitize_patterns)!_prepare_path_and_storage_options xbasenamexjoin)DataFilesNotFoundError)DatasetInfosDict)BuilderConfigsParameters DatasetModule,create_builder_configs_from_metadata_configsget_dataset_builder_classimport_main_classinfer_module_for_data_files)camelcase_to_snakecase)_MODULE_TO_EXTENSIONS_PACKAGED_DATASETS_MODULES) cached_path is_local_pathrelative_to_absolute_path)MetadataConfigs) tracked_str) filesystem) _un_chain)stringify_path) DatasetCardDatasetCardData)version)HubApi)_HAS_SCRIPT_LOADING_create_importable_file_get_importable_file_path_load_importable_file files_to_hash get_importsinit_dynamic_modulesresolve_trust_remote_code)DEFAULT_DATASET_REVISIONREPO_TYPE_DATASET)is_relative_path)has_attr_in_class) get_logger)_ALL_ALLOWED_EXTENSIONS)ALL_ALLOWED_EXTENSIONS_hub_apireturncCstdur tdddatS)N)timeout max_retries)rAr1rGrGn/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/msdatasets/utils/_module_factories.py _get_hub_apiEs rIcCst|tttfr|stdtt|d}nt|}|pi}|r%||d<t||p*i}i}tt|D].\}}|\}} } |t |dkrOt di| |}q4t di| ||d<| |d<||d<q4|d\} }} t |fi|} | S) Nzempty urlpath sequencerprotocolZtarget_optionsZtarget_protocolforG) isinstancelisttupleset ValueErrorr-r, enumeratereversedlendictr+)Zurlpathstorage_optionsrJZurlpath0chainZinkwargsichurlsZnested_protocolkwpaths_fsrGrGrHget_fs_token_pathsQs*  r_pattern base_pathallowed_extensionsdownload_configc st|r t||}nt|rtj|dtj}nd}t||d\}}t||d}| dd ddp8|j | dd ddt t t |ht|jtrW|jn|jd}|dkrd|dndi}|d krxtjtd krxd |d <z |j|fd di|}Wntytd|dwfdd|D} dur·fdd| D} t| t| krtt | t | } td|d| n| } | sd|d} dur| dt7} t| | S)zResolve data file paths/URLs from a user-supplied pattern. Supports ``*``, ``**``, and fsspec-based remote patterns (e.g. ``hf://``). Hidden files/directories and ``__pycache__`` are excluded by default. rrc)rVz::z://fileZhfz0.20.0FZ expand_infodetailTzUnable to find ''cszg|]9\}}|ddkr;t|vrttj|tjsttj|tjs|r7|n|qS)typerg)rrospathrelpathr startswith).0filepathinfo)files_to_ignore fs_base_path fs_patternprotocol_prefixrGrH s$      z$_resolve_pattern..Ncs8g|]}tfddt|dddDr|qS)c3s|] }d|vVqdS).NrG)rosuffixrbrGrH sz._resolve_pattern...rwrKN)anyrsplit)rorpryrGrHrvs z Some files matched the pattern 'z-' but don't have valid data file extensions: z with any supported extension )r<rr'rkrl splitdriveseprr_r|Z root_markerrPrrrMrJstrrZHF_HUB_VERSIONr0parseglobFileNotFoundErrorritemsrTrNloggerrq) r`rarbrcrVr^rJZ glob_kwargsZtmp_file_pathsZ matched_pathsoutZinvalid_matched_files error_msgrG)rbrrrsrtrurH_resolve_patternnsX     rcCs:tt||d}zt|WStytd|ddw)zGet data file patterns for a dataset directory. Tries ``SPLIT_PATTERN_SHARDED`` first, then falls back to ``ALL_DEFAULT_PATTERNS``. )rarczThe directory at z doesn't contain any data filesN)rrrrr)rarcresolverrGrGrH_get_data_patternss    rrepo_id path_in_reporevisionc Cst}|d\}}|j|td}|r|jdurd|d|_z|j||||d|d}t||d} W| StyM} zd } t | WYd} ~ | Sd} ~ ww) z.Loading z requires executing code from the repository. This is disabled by default for security reasons. If you trust the authors of this dataset, you can enable it with `trust_remote_code=True`.NDownloading extra moduleslibraryz Error in the z script, importing relative z module but z: is the name of the script. Please change relative import zl to another name and add a '# From: URL_OR_PATH' comment pointing to the original relative import file path.r.py)rrrrrzWrong import_typererK dependencies dependencyZthemitZsklearnz scikit-learnZBioZ biopythonzTo be able to use z$, you need to install the following z: z, z. Please install z using 'pip install  z' for instance.)r{rQrIcopyrappendrr&rkrljoin importlib import_module ImportErrorrTvalues)rrrrrrcr local_importsZlibrary_importsZhas_remote_coderr import_name import_path sub_directoryrrlocal_import_pathZneeds_to_be_installedZlibrary_import_nameZlibrary_import_pathZ_dependencies_strZ _them_strrGrGrH_download_additional_moduless     rdynamic_modules_pathc CsX|d\}}|d}t||||d} | s$td|d|d|dt|d||d} t| } t||||| ||d } g} | rG| tj| f|pKt}t | gd d | D}t |d ||d }t j |st||d}|rtd|dt| | | |d |||dntd|dt|d ||d \}}tt}|j|d|d}t|||S)zShared implementation for loading a dataset module from a Hub .py script. Used by both ``get_module_with_script`` (monkey-patch for datasets<4.0) and ``_compat_hub_script_module`` (compat shim for datasets>=4.0). rrrrrcrz Cannot find z in z at revision rw README.md)rrrrrrcrcSg|]}|dqSrKrGrolocrGrGrHrv{z'_load_script_module..datasetsrmodule_namespacesubdirectory_namer)rrz3Use trust_remote_code=True. Will invoke codes from z9. Please make sure that you can trust the external codes. local_pathradditional_filesrrrr download_moderz requires executing the dataset script in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.)r)rar)r|rrr7rrrZREPOCARD_FILENAMEr8r6r4rkrlexistsr9rwarningr3rQr5rinvalidate_cachesrIget_file_base_pathr)rrrcrrrrrZscript_file_nameZlocal_script_pathdataset_readme_pathrrrhash_valimportable_file_pathtrust module_pathrbuilder_kwargsrGrGrH_load_script_moduleGs          rcCs@|j}|jjddp t}t|||j|j|jr|jnd|jdS)zXMonkey-patch target for ``HubDatasetModuleFactoryWithScript.get_module`` (datasets<4.0).rNrrrcrrr) rrcrVgetr:rrrr)selfrrrGrGrHget_module_with_scripts rcCst||pt|p t|||dS)zLLoad a dataset module from a Hub repo .py script (compat for datasets>=4.0).r)rr:r )rlrrcrrrrGrGrH_compat_hub_script_modules rc Cs|}t|j}g}t|}|D]l\}} } } |dkrq|dkrYtjtj|| d} tj| r9|| | fqtj tjtj|| rX|| tjtj|| fq|dkr{t } d| _ t | | d}| durttj|| }|| |fq|pt }t|gdd |D}t|d ||d }tj|st||}|rt||g|d |||d ntd |dt|d ||d \}}tdtt|ji}t|||S)zILoad a dataset module from a local .py script (compat for datasets>=4.0).rrrrrreNcSrrrGrrGrGrHrvrz/_compat_local_script_module..rrrrzS requires executing the dataset script. Set `trust_remote_code=True` to allow this.ra)rstemr7rkrlrdirnameisfilerisdirr rr&r8r6r4rr9r3rQr5rrrresolveparentr)rlrrrrrrrrrrrrel_pathZ dl_configrrrrrrrGrGrH_compat_local_script_modules        rc Csf|jjddp t}d|jd|d|jpdd}|j}|j}t|d||d}|r6t t |j nt }|jd d}t|}t|} |jdurUt|j} n-|r{d tt|vr{|durl||d } n tt|d } t| } nt||jd } tj| |t|jd } t| |j|jd \} }t| dr| jt| d} n| t| } t| \}}|r| dv}t !t"}|||||jd}d|j#vr||d<t"di|\}}nt$|j%dd | i|g}d}t&}|j'|t(d}|j)||d|jt*t |jj| d}|j}|j+durd|_+|dur%t,| dkr%tt| }t-|||| t.|||ddS)zLMonkey-patch target for ``HubDatasetModuleFactoryWithoutScript.get_module``.rNzhf://datasets/@rrdrrrrre)rarbrc)rrlrcfilter) extensions>Z audiofolderZ imagefolder)rmetadata_configsradefault_builder_kwargsrcsupports_metadatar)rr)rarrrzDownloading metadatarK)rbuilder_configsdefault_config_name) dataset_infosZbuilder_configs_parametersrG)/rcrVrr:rdata_dirrstriprrr.loadrdatar/r)Zfrom_dataset_card_datarrrnextiterrrrZ from_patternsr@r"hasattrrr$Zfilter_extensionsr%inspect signaturer parametersr!ZBUILDER_CONFIG_CLASSrIrr;rr#rrTrr)rrrarrcrZdataset_card_dataZ subset_namerrpatternsZsubset_data_filesZdata_files_dict module_namerrr]rZcreate_builder_signatureZin_argsrrrrrrGrGrHget_module_without_scripts               r)NN)N)F)NNN)NNNNN)m__doc__rrrk functoolsrpathlibrtypingrrrrrr rr r r r rrrZdatasets.data_filesrrrrrrrZ,datasets.download.streaming_download_managerrrrZdatasets.exceptionsrZ datasets.inforZ datasets.loadrrrr r!r"Zdatasets.namingr#Zdatasets.packaged_modulesr$r%Zdatasets.utils.file_utilsr&r'r(Zdatasets.utils.metadatar)Zdatasets.utils.trackr*Zfsspecr+Z fsspec.corer,Z fsspec.utilsr-Zhuggingface_hubr.r/ packagingr0Z modelscoper1Z#modelscope.msdatasets.utils._compatr2r3r4r5r6r7r8r9Zmodelscope.utils.constantr:r;Zmodelscope.utils.file_utilsr<Zmodelscope.utils.import_utilsr=Zmodelscope.utils.loggerr>r?r@rrrA__annotations__rIr_rrrrboolrrrrrrrGrGrGrHs   $$           (       E  * _ b  N