o 0j@*@sdZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZdd lmZdd lm Z e!eeZ"ee Z#ej$d d de#De%dZ&ee%dBej'dd(e"dee"dfZ)eddZ*e*j+dgdddddddeddddddf dee%dBej,ddfdedededee&dBej'd dfd!ed"e)d#ee-ej'd$d%d&dfd'ee-ej'd(d)dfd*ee-ej'd+d,d-dfd.ed/ed0dfd1d2Z.e*j+d3gd4d 5 dYd6ee%ej,d7dfd!ed/ed0dfd8d9Z/e*j+d:d;dd?Z0e*j+d@gdAd   dZd6ee%ej,d=dfdBee%dBej'dCdDdfdEee%dBej'dFdfd/ed0df dGdHZ1e*j+dIdJdKgd d[dIee%ej,dLdfd/ed0dfdMdNZ2e*j+dOgdPd   d\d6ee%ej,d=dfdQee-ej'dRdSdfdTee-ej'dUdVdfd/ed0df dWdXZ3dS)]aContains commands to interact with datasets on the Hugging Face Hub. Usage: # list datasets on the Hub hf datasets ls # list datasets with a search query hf datasets ls --search "code" # get info about a dataset hf datasets info HuggingFaceFW/fineweb N) Annotatedget_args)execute_raw_sql_query)CLIErrorRepositoryNotFoundErrorRevisionNotFoundError) DatasetSort_TExpandDatasetProperty_T) DatasetCard) REPO_LIST_DEFAULT_LIMIT AuthorOpt FilterOptLimitOpt RevisionOpt SearchOptTokenOptapi_object_to_dict get_hf_apimake_expand_properties_parser typer_factory)list_repo_files_cmd)outDatasetSortEnumcCsi|]}||qSr).0srr]/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/huggingface_hub/cli/datasets.py 8sr)typezComma-separated properties to return. When used, only the listed properties (and id) are returned. Example: '--expand=downloads,likes,tags'. Valid: z, .)helpcallbackz"Interact with datasets on the Hub.)r!z list | ls)zhf datasets lsz*hf datasets ls --sort downloads --limit 10zhf datasets ls --search "code"z*hf datasets ls --filter benchmark:officialz$hf datasets ls HuggingFaceFW/finewebz'hf datasets ls HuggingFaceFW/fineweb -Rz.hf datasets ls HuggingFaceFW/fineweb --tree -h)ZexamplesFrepo_idzVDataset ID (e.g. `username/repo-name`) to list files from. If omitted, lists datasets.searchauthorfiltersortz Sort results.limitexpandhuman_readablez--human-readablez-hz=Show sizes in human readable format (only for listing files).as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).revisiontokenreturnc  Cs |durE|dur td|durtd|durtd|dur(td|tkr1td|dur:tdt|d||| | | d S|rLtd | rStd |rZtd | durctd t| d} |rm|jnd} dd| j|||| ||dD}t|dS)zList datasets on the Hub, or files in a dataset repo. When called with no argument, lists datasets on the Hub. When called with a dataset ID, lists files in that dataset repo. Nz'Cannot use --search when listing files.z'Cannot use --author when listing files.z'Cannot use --filter when listing files.z%Cannot use --sort when listing files.z&Cannot use --limit when listing files.z'Cannot use --expand when listing files.Zdataset)r#Z repo_typer*r+r,r-r.z(Cannot use --tree when listing datasets.z-Cannot use --recursive when listing datasets.z2Cannot use --human-readable when listing datasets.z,Cannot use --revision when listing datasets.r.cSg|]}t|qSrr)r dataset_inforrr szdatasets_ls..)r&r%r$r'r(r)) typerZ BadParameterr rrvalueZ list_datasetsrtable)r#r$r%r&r'r(r)r*r+r,r-r.apisort_keyresultsrrr datasets_lsGsV.            r; leaderboard)z4hf datasets leaderboard SWE-bench/SWE-bench_VerifiedzLhf datasets leaderboard SWE-bench/SWE-bench_Verified --limit 5 --format jsonzIhf datasets ls --filter benchmark:official # list available leaderboards dataset_idz?The benchmark dataset ID (e.g. `SWE-bench/SWE-bench_Verified`).cCsvt|d}|j|d}dd|d|D}tj|gddddd d td |r9td |d jddSdS)zList model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards.r0)r#cSr1rr2rentryrrrr4sz(datasets_leaderboard..N)rankmodel_idr6sourcerBright)rAr6)headersid_keyZ alignmentszPUse 'hf datasets ls --filter benchmark:official' to list available leaderboards.zUse 'hf models info rz' to get details about a model.)rZget_dataset_leaderboardrr7hintrB)r>r(r.r8r<r:rrrdatasets_leaderboards   rHinfoz&hf datasets info HuggingFaceFW/finewebz9hf datasets info my-dataset --expand downloads,likes,tagsz+The dataset ID (e.g. `username/repo-name`).c Cst|d}z |j|||d}Wn,ty$}z td|d|d}~wty;}z td|d|d|d}~wwt|dS) z$Get info about a dataset on the Hub.r0)r#r-r)z Dataset 'z ' not found.Nz Revision 'z' not found on 'z'.)rr3rrrrdict)r>r-r)r.r8rIerrr datasets_infos rLZparquet)z(hf datasets parquet cfahlgren1/hub-statsz8hf datasets parquet cfahlgren1/hub-stats --subset modelsz6hf datasets parquet cfahlgren1/hub-stats --split trainz6hf datasets parquet cfahlgren1/hub-stats --format jsonsubsetz--subsetz(Filter parquet entries by subset/config.splitz Filter parquet entries by split.csPt|d}|j||d}fdd|D}dd|D}tj|gdddd S) z/List parquet file URLs available for a dataset.r0)r#configcs"g|] }dus |jkr|qSNrNr?rQrrr4s"z$datasets_parquet..cSs"g|] }|j|j|j|jdqS)rMrNurlsize)rOrNrSrTr?rrrr4srRrS)rErFN)rZlist_dataset_parquet_filesrr7)r>rMrNr.r8entriesfilteredr:rrQrdatasets_parquets rWsqlzhf datasets sql "SELECT COUNT(*) AS rows FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet')"zhf datasets sql "SELECT * FROM read_parquet('https://huggingface.co/api/datasets/cfahlgren1/hub-stats/parquet/models/train/0.parquet') LIMIT 5" --format jsonzRaw SQL query to execute.c CsFzt||d}Wnty}ztt||d}~wwt|dS)zAExecute a raw SQL query with DuckDB against dataset parquet URLs.)Z sql_queryr.N)r ImportErrorrstrrr7)rXr.resultrKrrr datasets_sqls r\card)z&hf datasets card HuggingFaceFW/finewebz1hf datasets card HuggingFaceFW/fineweb --metadataz?hf datasets card HuggingFaceFW/fineweb --metadata --format jsonz-hf datasets card HuggingFaceFW/fineweb --textmetadataz --metadataz'Output only the metadata from the card.textz--textz(Output only the text body (no metadata).cCsl|r|rtdtj||d}|rt|jdS|r%t|jdSt|jt d|ddS)z7Get the dataset card (README) for a dataset on the Hub.z---metadata and --text are mutually exclusive.r0zUse `hf datasets card z/ --metadata` to extract only the card metadata.N) rr loadrrJdatato_dictr_contentrG)r>r^r_r.r]rrr datasets_cards rd)r=N)NNNrP)FFN)4__doc__enumtypingrrr5Zhuggingface_hub._dataset_viewerrZhuggingface_hub.errorsrrrZhuggingface_hub.hf_apirr Zhuggingface_hub.repocardr Z _cli_utilsr r rrrrrrrrrZ _file_listingr_outputrsortedZ_EXPAND_PROPERTIESZ _SORT_OPTIONSEnumrZrOptionjoinZ ExpandOptZ datasets_clicommandZArgumentboolr;rHrLrWr\rdrrrrsV    4         R