o )jɥ'@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZddlmZddlmZmZmZmZmZmZdd lmZdd lmZdd l m!Z!d d l"m#Z#m$Z$d dl%m&Z&d dl'm(Z(d dl)m*Z*m+Z+d dl,m-Z-m.Z.m/Z/d dl0m1Z1d dl2m3Z3m4Z4m5Z5m6Z6m7Z7eZ8dZ9dddddddddddddeddddfde:dee:dee:e dfdeeee:fdee;dee deee:ee:fdeee:ee:fdee:deeee:e:fd eeee:e:fd!ee<d"e:d#ee:d$ee;d%eee&d&ee:d'ee:d(e:f&d)d*Z=eddddddddddd+ddfd,e:dee:dee:e dfdee:deeee:fdee;dee deee:ee:fdeee:ee:fdeeee:e:fd eeee:e:fd$ee;d!ededddddddddd+dddd/d"e:d#ee:dee:dee:e dfdeeee:fdee;dee deee:ee:fdeee:ee:fdee:deeee:e:fd eeee:e:fd!eeeDdeee:deee:deee:d eee:d(eeDf d?d@ZEdd2dddde9fdAdBZFe(ddfdCdDZGdedddddd+dddf d>ee:dEe1dFe:d"e:dGe#dHe:dIe:d#ee:dee:dee deee:ee:fdeee:ee:fdeeee:e:fd eeee:e:fd!er@rA)r4r(r)r*r+r,r-r.r/r1r0r2r7r8) r$rCrDrErrFrGrHrIr%rrJr)rRr(r)r/r*r+r,r-r.r0r1r5r2r7r8rKrLrMrNrNrOdataset_snapshot_downloads>D    $rS)r4r(r)r*r+r,r-r.r/r0r1r2r6r7r8c!Cs|st}|tvrtd|tft|| ||d\}}|dur |nt}|r;t|jdkr0tdt d|| St j |dt td}tdkrdtd pUt}|rdtd |||d <t|d }|duru|j|||d }|dur}|}|tkr| rtj| }n|rtjj|g|dR}ntjj|dg|dR}td|d||j||||d}|d}dtjvri|ddi}ni|ddi}|jdur|j|d<t t!|t!| d}|j"|||d|durdn|||d}|s |dur t d|d|j"||dd|durdn|||d}t#|||| | d}t$|||||dd||||d| || d d!|vrt%||}tj&|rStd"ntd#|d$ztj'tj||dd%Wnt(yt d&|d'tj|d!Yn|wnz|t)kr| rtj| }n|rtjj|g|dR}ntjj|d(g|dR}td)|t*|\}}|pt+}t t!|t!| d}|rd|nd}td*|d+t,||||||||| | d, }t-||||||||||| || d- |j.|d.| } | S)/Nz'Invalid repo type: %s, only support: %s)r/r)r4rzCannot find the requested files in the cached path and outgoing traffic has been disabled. To enable look-ups and downloads online, set 'local_files_only' to False.z6We can not confirm the cached file is for revision: %s)r*)z user-agentzsnapshot-identifierr<ZINTRA_CLOUD_ACCELERATION_REGIONz6Intra-cloud acceleration enabled for downloading from zx-aliyun-region-id)r7)r3r4r7r@modelszDownloading Model from z to directory: )r(r,r8ZRevisionZCI_TESTzsnapshot-ci-testTrueZSnapshotcached_model_revision)r.r0TF)r'r(root recursiveZ use_cookiesheadersr8zroot='zF' returned no model files, falling back to root=None for full listing.r.r-r0r1)r4r(r, pre_filteredr2r8r6.z3Target directory already exists, skipping creation.zCreating symbolic link [z].)target_is_directoryzFailed to create symbolic link z for Zdatasetsz"Downloading Dataset to directory: zFetching file list (root: z)...)r7 root_pathr.r-r0r1) cachetemporary_cache_dirr3api dataset_name namespacerYr(r,r2r8r6)Z revision_info)/rrrrrlen cached_filesrBloggerwarningZget_root_locationrZget_user_agentstruuiduuid4rrCgetenvrZ!_get_internal_acceleration_domaininfoZget_endpoint_for_readZ get_cookiesrGabspathrHsplitprintZget_valid_revision_detailrDrVr!_normalize_patternsZget_model_filesfilter_files_by_patterns_download_file_listsr"existssymlinkOSErrorrr#r _iter_dataset_file_pages_pipeline_download_datasetZsave_model_version)!r3r4r(r)r*r+r,r-r.r/r0r1r2r6r7r8r`r_rKrYZ region_id_api directoryZrevision_detailZsnapshot_headerZextracted_root repo_filesZmasked_directorygroup_or_ownernamer^file_page_iterZcache_root_pathrNrNrOrJsf                     rJr@c  sd vr td d dd\} } j| |  d\} tg f dd}||}|sO|dkrOtd|d |d}|S) awFetch and filter dataset repo files with pagination and server-side prefix filtering. Applies per-page pattern filtering to minimize memory usage. Falls back to root_path='/' if the extracted prefix yields no results. Args: _api: HubApi instance. repo_id: Dataset repo identifier (owner/name). revision: Git revision. endpoint: API endpoint URL. token: Authentication token. root_path: Server-side directory prefix filter. allow_file_pattern: Include patterns for client-side filtering. ignore_file_pattern: Exclude patterns for client-side filtering. allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). page_size: Number of files per API page request. Returns: List of filtered file entry dicts. r@Invalid repo_id: '', expected format 'owner/name'rrbrcr8r7c sd}g} zj |d| d }Wnty3}ztd|d|WYd}~|Sd}~ww|s9 |SrJt|d}||n |dd |Dt|kr] |S|d7}q) zFFetch all pages with the given root_path, applying per-page filtering.rT r3r(r^rX page_number page_sizer8r7Zdataset_hub_id#Error fetching dataset files (page ): NrZcss"|] }|ddkr|VqdS)r treeNrE.0frNrNrO .szAfetch_repo_files.._paginate_and_filter..)get_dataset_files Exceptionrferrorrqextendrd)effective_root_pathrrz dataset_filese page_filtered rxZ_hub_idr.r0r8Z has_patternsr-r1rr3r(r7rNrO_paginate_and_filter sV      z.fetch_repo_files.._paginate_and_filterz root_path='F' returned no results, falling back to root_path='/' for full listing.)rrnget_dataset_id_and_typeanyrfrg)rxr3r(r8r7r^r.r-r0r1r_owner _dataset_name_rrzrNrrOfetch_repo_filess*"  "/ rpatterncCs&zt|WdStyYdSw)NTF)recompile BaseException)rrNrNrO_is_valid_regexDs   rpatternscCs*t|tr|g}|durdd|D}|S)NcSs"g|] }|ds |n|dqS)r@*)endswith)ritemrNrNrO Psz'_normalize_patterns..) isinstancerh)rrNrNrOrpLs rpcCs0|durg}|D] }t|r||q|SdSN)rappend)rZregex_patternsrrNrNrO_get_valid_regex_patternVs rrZrzc s t|}t|}t|}t|}t|}g}|D]uddkr!qzM|r1tfdd|Dr1Wq|r@tfdd|Dr@Wq|rOtfdd|DrOWq|r^tfdd|Ds^Wq|rmtfdd|DsmWqWnty}z td |WYd }~qd }~ww|q|S) aFilter repo file entries by include/exclude patterns. Skips 'tree' type entries. Applies fnmatch and regex pattern matching. Returns only file entries that pass all filter criteria. Args: repo_files: List of file entry dicts with 'Type', 'Path', 'Name' keys. allow_file_pattern: Include patterns (fnmatch). Files must match at least one. ignore_file_pattern: Exclude patterns (fnmatch). Matching files are skipped. allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). Returns: List of file entries that pass all filters. r rc3 |] }td|VqdSrNfnmatchrp repo_filerNrOr  z+filter_files_by_patterns..c3rrrrrrNrOrrc3s$|] }t|dduVqdSNameNrsearchrrrNrOrs  c3rrrrrrNrOrrc3rrrrrrNrOrrzInvalid file pattern: %sN)rprrrrfrgr)rzr.r-r0r1ignore_regex_patternfilteredrrNrrOrqasL   rqc  #sd vr td d dd\} } j| |  d\} tg f dd}z8d}||D]}d }|Vq@|s`|dkrftd |d |dD]}|VqZWtd SWtd SWtd Stw) aGenerator that yields filtered file pages from a dataset repo. Each yield is a non-empty list of file-entry dicts for one API page. Applies per-page pattern filtering to minimize memory usage. Falls back to root_path='/' if the extracted prefix yields no results. Args: _api: HubApi instance. repo_id: Dataset repo identifier (owner/name). revision: Git revision. endpoint: API endpoint URL. token: Authentication token. root_path: Server-side directory prefix filter. allow_file_pattern: Include patterns (fnmatch). ignore_file_pattern: Exclude patterns (fnmatch). allow_patterns: Additional include patterns (HF-compatible). ignore_patterns: Additional exclude patterns (HF-compatible). page_size: Number of files per API page request. Yields: List[dict]: Non-empty list of filtered file entries per page. r@r~rrrc 3sd}d} zj |d| d }Wnty4}ztd|d|WYd}~dSd}~ww|s9dSrEt|d}nd d |D}|t|7}|rW|Vtd |d |d dddt|krmdS|d7}q)z2Yield filtered file pages for the given root_path.rrTrrrNrZcSsg|] }|ddkr|qS)r rrrrNrNrOrszE_iter_dataset_file_pages.._paginate_pages..z Fetched z matching files (z pages)...)endflush)rrrfrrqrdro)rr total_foundrrrrrNrO_paginate_pagessd    z1_iter_dataset_file_pages.._paginate_pagesFTz root_path='rN)rrnrrro)rxr3r(r8r7r^r.r-r0r1rrrrrZ yielded_anypagerNrrOrvs@#  "7     rvc sd} d}gtfddtddddft| dQ}|D]F}|D]A}| d7} | _||rB|d7}dq&|j|d |||| d }|j t |||||| d| d }| |ffd d q&q"Wdn1sswYWdn1swYrddD}t tdddd|D| |t}td| d|d|drdtdnddrttd| ddS)a_Pipeline consumer: download dataset files as pages are yielded. Consumes the page iterator from _iter_dataset_file_pages, submitting each file to a thread pool for concurrent download. Uses tqdm for real-time progress and thread-safe error collection. Args: file_page_iter: Iterator yielding List[dict] file pages. cache: ModelFileSystemCache instance for dedup. temporary_cache_dir: Temp staging directory. repo_id: Dataset repo identifier. api: HubApi instance. dataset_name: Dataset name component. namespace: Owner/namespace component. headers: HTTP request headers. revision: Git revision. cookies: HTTP cookies. max_workers: Thread pool concurrency. endpoint: API endpoint URL. progress_callbacks: Optional progress callback list. rc szQz|Wn:tyA}z.||fWdn1s#wYtd|ddd|WYd}~n d}~wwWddSWddSdw)z8Done callback: update progress bar and collect failures.NzDownload failed for r?z: r)resultrrrfdebugrEupdate)futurerexc) failed_itemslockpbarrNrO_on_doneHs z,_pipeline_download_dataset.._on_donez filesF)totalunitdisable)r2rr file_namerbrcr(r8 disable_tqdmr6cs ||SrrN)rrf)rrNrOxs z,_pipeline_download_dataset..NcSs.g|]\}}t|tr|ddnt|qS)rr)rdictrErhrrrrNrNrOr~sz._pipeline_download_dataset.. file(s) failed to download:  cs|]}d|VqdSz - NrNrrNrNrOrz-_pipeline_download_dataset..zDownload complete: z files found, z cached, z downloadedz, z failedrr\# file(s) failed to download out of ) threadingLockr rrrefreshrsrget_dataset_file_urlZsubmitrZadd_done_callbackrfrrdrHror)r}r_r`r3rarbrcrYr(r,r2r8r6rZ total_cachedexecutorZ page_filesrurlr failed_pathsZ downloadedrN)rrrrrOrws$    (  rwr_r`rar|r{r[c s|r&g}|D]rtjd}td|dq|qnt|}t| } t| } t| } t| }g}|D]ddkrGq>zm|rWt fdd|DrWWq>| rft fdd| DrfWq>|rut fd d|DruWq>| dur| rt fd d | DsWq>| dur| rt fd d | DsWq>rtjd}td|dWq>Wnt y}z t d |WYd}~q>d}~ww|q>t |ddd f dd}t |dkrRtdt |d||}g}t|tr3t |dkr3|\}}|r3dd|D}tt |dddd |Dtdt |d d|rTtt |dt |ddSdS) NrzFile z8 already in cache with identical hash, skip downloading!r rcg|] }td|qSrrrrrrNrOrz(_download_file_lists..crrrrrrNrOrrcs g|] }t|dduqSrrrrrNrOrsc3rrrrrrNrOrrz'_download_file_lists..c3rrrrrrNrOrrz The file pattern is invalid : %sFT)r2rZfault_tolerantc sp tkrt|d d}n tkr j|d d}n td dtt|| dddS)Nr)r' file_pathr(r8rr:z, supported types: Fr)rrrrrrr)rr) rar_r,r8r{rYr|r6r3r4r(r`rNrO_download_single_files8 z3_download_file_lists.._download_single_filerzGot z files, start to download ...cSs*g|]\}}t|tr|dnt|qSr)rrrhrrNrNrOrsrrcsrrrNrrNrNrOrrzFinish downloading z files for repo ''rr\)rsrCrGbasenamerfrrrprrrrgrrdrlrtuplerrHr)rzr_r`r3rar|r{rYr4r(r,r-r.r0r1r2r8r6r[Zfiltered_repo_filesrrrrZdownload_resultrrrrN) rar_r,r8r{rYr|r6rr3r4r(r`rOrrs            $  rr)IrrCrrriconcurrent.futuresr contextlibrhttp.cookiejarrpathlibrtypingrrr r r Z tqdm.autor Zmodelscope.utils.constantr rrrrrZmodelscope.utils.file_utilsrZmodelscope.utils.loggerrZmodelscope.utils.thread_utilsrrarrcallbackr constantsrerrorsrrZ file_downloadrrrZ utils.cachingr Z utils.utilsr!r"r#r$r%rfZDEFAULT_DATASET_PAGE_SIZErhboolintrPrSrJrrrprrrqrvrwrrrNrNrNrOs                    }       g        W k      H  {