o "j@sdZddlZddlZddlmZgZdZdZdZ dZ dZ d Z d Z d Zd Zd ZddZddZedddddddZedddddddZedddddddZedddddd"ddZedddddd d!ZdS)#a WMT14 dataset. The original WMT14 dataset is too large and a small set of data for set is provided. This module will download dataset from http://paddlepaddle.bj.bcebos.com/demo/wmt_shrinked_data/wmt14.tgz and parse training set and test set into paddle reader creators. N) deprecatedzJhttp://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/data/dev+test.tgzZ 7d7897317ddd8ba0ae5c5fa7248d3ff5z/http://paddlemodels.bj.bcebos.com/wmt/wmt14.tgzZ 0791583d57d5beb693b9414c5b36798cz1http://paddlemodels.bj.bcebos.com/wmt%2Fwmt14.tgzZ 0cb4a5366189b6acba876491c8724fa3zzzcCsdd}tj|dd>}dd|D}t|dksJ|||d|}d d|D}t|dks4J|||d|}||fWdS1sLwYdS) NcSs8i}t|D]\}}||kr|||<q|S|S)N) enumeratestripdecode)fdsizeZout_dict line_countliner U/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/dataset/wmt14.py __to_dict2s z!__read_to_dict..__to_dictrmodecSg|] }|jdr|jqS)zsrc.dictnameendswith.0Z each_itemr r r <  z"__read_to_dict..rcSr)ztrg.dictrrr r r rCr)tarfileopenlen extractfile)tar_file dict_sizer fnamessrc_dicttrg_dictr r r __read_to_dict1s $r$csfdd}|S)Nc 3s&t\tjddz}fdd|D}|D]e}||D]]}|}|d}t|dkr6q"|d}|}fddtg|t gD}|d }|} fd d| D} t|d ksjt| d krkq"| t g} tg| } || | fVq"qWddS1swYdS) Nrrcsg|] }|jr|jqSr rr) file_namer r rQrz2reader_creator..reader.. rrcg|]}|tqSr getUNK_IDXrw)r"r r r^s rcr'r r(r+)r#r r resP) r$rrrrrsplitrSTARTEND) r r!rr Z line_splitZsrc_seqZ src_wordsZsrc_idsZtrg_seqZ trg_wordsZtrg_idsZ trg_ids_nextrr%r)r"r#r readerNs8   "zreader_creator..readerr )rr%rr2r r1r reader_creatorMs!r3z2.0.0zpaddle.text.datasets.WMT14rz>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasoncCttjjtdtd|S)a WMT14 training set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Training reader creator :rtype: callable wmt14z train/trainr3paddledatasetcommondownload URL_TRAIN MD5_TRAINrr r r trainr r@cCr6)z WMT14 test set creator. It returns a reader creator, each sample in the reader is source language word ID sequence, target language word ID sequence and next word ID sequence. :return: Test reader creator :rtype: callable r7z test/testr8r?r r r testrArBcCr6)Nr7zgen/genr8r?r r r gens rCTcCsPtjjtdt}t||\}}|r$dd|D}dd|D}||fS)Nr7cSi|]\}}||qSr r rkvr r r zget_dict..cSrDr r rEr r r rHrI)r9r:r;r<r=r>r$items)rreverserr"r#r r r get_dicts  rLcCs(tjjtdttjjtdtdS)Nr7)r9r:r;r<r=r> URL_MODEL MD5_MODELr r r r fetchsrO)T)__doc__rZpaddle.dataset.commonr9Z paddle.utilsr__all__Z URL_DEV_TESTZ MD5_DEV_TESTr=r>rMrNr/r0ZUNKr*r$r3r@rBrCrLrOr r r r sj   %