o *j`@s$ddlmZmZmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddZ GdddeZ eeZddZd d ZGd d d eZGd ddeZGdddeZddZddZddZ zddlmZWn ey|ddZYnweddZddZGdddeZ dS) )absolute_importdivisionprint_functionunicode_literalsNc Cs8ddddddddd }|D] \}}|||}q|S) N-'zn'tz'mz don'tz'sz'vez're)z - z ' z n'tz 'mz do notz 'sz 'vez 're)itemsreplace)stringZ replace_mpkvr m/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/preprocessors/nlp/space/tokenizer.py clean_strings rc@s>eZdZgdfddZddZddZdd Zgfd d Zd S) TokenizerBertcsj|_|dkrsddd_|D]}|jvr&|dvr&dtjdj|<qdd jD_fd d |D}d _jtfd d|D7_t|jd_jD]}|jj vsiJd|dqYtjj _ dS|dkrddi_dd jD_fdd |D}t j |d}t j |d}t|||d_t|_tj_ dSt)Nrz [unused0]z [unused1])z[BOS]z[EOS])[PAD][UNK]z[unused]cSi|]\}}||qSr r .0r r r r r .z&Tokenizer.__init__..cg|] }j||qSr spec_convert_dictgetrtokselfr r 2sz&Tokenizer.__init__..rz[SEP]rz[CLS]z[MASK]c3s|] }|jvr|VqdSNspecial_tokensrxr r r 7s  z%Tokenizer.__init__..) never_splitzspecial token 'z' is not in the vocabularyZGPT2rzcSrr r rr r rrArcsg|] }|jvr|qSr )rrr r rr"Es  z vocab.jsonz merges.txtr%)tokenizer_typerlenrspec_revert_dictr&tuple BertTokenizer _tokenizervocab vocab_sizeospathjoin GPT2Tokenizer num_specials ValueError)r!Z vocab_pathr&r+tokenr vocab_file merges_filer r r__init__!sT      zTokenizer.__init__cCs |j|Sr$)r0tokenizer!textr r rr=Rs zTokenizer.tokenizecs`jdkrfdd|D}j|}|Sfdd|D}j|}fdd|D}|S)Nrcrr rrr r rr"Wz3Tokenizer.convert_tokens_to_ids..crr rrr r rr"[r@csg|] }|jjqSr r7r2rir r rr"])r+r0convert_tokens_to_ids)r!tokensidsr r rrEUs   zTokenizer.convert_tokens_to_idscs`jdkrj|}fdd|D}|Sfdd|D}j|}fdd|D}|S)Nrcrr r-rrr r rr"cr@z3Tokenizer.convert_ids_to_tokens..csg|] }|jjqSr rArBr r rr"frDcrr rHrr r rr"hr@)r+r0convert_ids_to_tokens)r!rGrFr r rrI`s   zTokenizer.convert_ids_to_tokenscs|}tdkrtfdd|D}jdkr'd|dd}nd|}tfdd|Dd }t|}|S) Nrcsg|]}|vr|qSr r r) ignore_tokensr rr"oz$Tokenizer.decode..r z ##csg|]}jj|qSr )r0 byte_decoderrcr r rr"t utf-8) rIr,setr+r5r bytearraydecoder)r!rGrJrFr r )rJr!rrUks     zTokenizer.decodeN)__name__ __module__ __qualname__r<r=rErIrUr r r rrs 1  rcCsjt}d}t|ddd} |}|sn |}|||<|d7}qWd|S1s.wY|S)z*Loads a vocabulary file into a dictionary.rrrRencodingTN) collections OrderedDictopenreadlinestrip)r:r1indexreaderr9r r r load_vocabs  rdcCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)rasplit)r?rFr r rwhitespace_tokenizes rfc@s:eZdZdZ    d ddZddZd d Zd d ZdS)r/z?Runs end-to-end tokenization: punctuation splitting + wordpieceTNr#cCstj|s td|t||_tdd|j D|_ ||_ |r,t ||d|_ t|jd|_|dur<||_dStd|_dS)aConstructs a BertTokenizer. Args: vocab_file: Path to a one-wordpiece-per-line vocabulary file do_lower_case: Whether to lower case the input Only has an effect when do_wordpiece_only=False do_basic_tokenize: Whether to do basic tokenization before wordpiece. max_len: An artificial maximum length to truncate tokenized sequences to; Effective maximum length is always the minimum of this value (if specified) and the underlying BERT model's sequence length. never_split: List of tokens which will never be split during tokenization. Only has an effect when do_wordpiece_only=False zCan't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`cSsg|]\}}||fqSr r )rrrGr r rr"rQz*BertTokenizer.__init__.. do_lower_caser*)r1NmB)r3r4isfiler8formatrdr1r]r^r ids_to_tokensdo_basic_tokenizeBasicTokenizerbasic_tokenizerWordpieceTokenizerwordpiece_tokenizerintmax_len)r!r:rhrsrmr*r r rr<s     zBertTokenizer.__init__cCsLg}|jr|j|D]}|j|D]}||qq |S|j|}|Sr$)rmror=rqappend)r!r? split_tokensr9Z sub_tokenr r rr=s  zBertTokenizer.tokenizecCsHg}|D] }||j|qt||jkr"tdt||j|S)z7Converts a sequence of tokens into ids using the vocab.zToken indices sequence length is longer than the specified maximum sequence length for this BERT model ({} > {}). Running this sequence through BERT will result in indexing errors)rtr1r,rsloggerwarningrk)r!rFrGr9r r rrEs z#BertTokenizer.convert_tokens_to_idscCs"g}|D] }||j|q|S)z?Converts a sequence of ids in wordpiece tokens using the vocab.)rtrl)r!rGrFrCr r rrIsz#BertTokenizer.convert_ids_to_tokens)TNTr#)rVrWrX__doc__r<r=rErIr r r rr/s $ r/c@sNeZdZdZ  dddZddZdd Zd d Zd d ZddZ ddZ dS)rnzDRuns basic tokenization (punctuation splitting, lower casing, etc.).Tr#cCs||_||_dS)znConstructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input. Nrg)r!rhr*r r rr<s zBasicTokenizer.__init__cCsn||}||}t|}g}|D]}|jr%||jvr%|}||}|||qtd |}|S)zTokenizes a piece of text.rL) _clean_text_tokenize_chinese_charsrfrhr*lower_run_strip_accentsextend_run_split_on_puncr5)r!r?Z orig_tokensrur9 output_tokensr r rr=s   zBasicTokenizer.tokenizecCsBtd|}g}|D]}t|}|dkrq ||q d|S)z$Strips accents from a piece of text.ZNFDZMnrM) unicodedata normalizecategoryrtr5)r!r?outputcharcatr r rr|s    z!BasicTokenizer._run_strip_accentscCs||jvr|gSt|}d}d}g}|t|krC||}t|r)||gd}n|r0|gd}|d||d7}|t|ksdd|DS)z&Splits punctuation on a piece of text.rTFr\cSsg|]}d|qS)rM)r5r'r r rr"-z5BasicTokenizer._run_split_on_punc..)r*listr,_is_punctuationrt)r!r?charsrCZstart_new_wordrrr r rr~s$      z!BasicTokenizer._run_split_on_punccCsTg}|D] }t|}||r|d|||dq||qd|S)z)Adds whitespace around any CJK character.rLrM)ord_is_chinese_charrtr5r!r?rrcpr r rrz/s      z&BasicTokenizer._tokenize_chinese_charscCs|dko|dk}|p|dko|dk}|p|dko|dk}|p%|dko%|dk}|p/|d ko/|d k}|p9|d ko9|d k}|pC|d koC|dk}|pM|dkoM|dk}|rRdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr )r!rtmpr r rr<s zBasicTokenizer._is_chinese_charcCsXg}|D]"}t|}|dks|dkst|rqt|r!|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rirLrM)r _is_control_is_whitespacertr5rr r rrySs   zBasicTokenizer._clean_textN)Tr#) rVrWrXrxr<r=r|r~rzrryr r r rrns    rnc@s"eZdZdZd ddZddZdS) rpzRuns WordPiece tokenization.rdcCs||_||_||_dSr$)r1 unk_tokenmax_input_chars_per_word)r!r1rrr r rr<ds zWordpieceTokenizer.__init__c Csg}t|D]m}t|}t||jkr||jqd}d}g}|t|kret|}d} ||krQd|||} |dkrAd| } | |jvrI| } n|d8}||ks0| durXd}n || |}|t|ks&|rn||jq||q|S)aTokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example: >>> input = "unaffable" >>> output = ["un", "##aff", "##able"] Args: text: A single token or whitespace separated tokens. This should have already been passed through `BasicTokenizer`. Returns: A list of wordpiece tokens. FrNrMz##r\T) rfrr,rrtrr5r1r}) r!r?rr9rZis_badstartZ sub_tokensendZ cur_substrsubstrr r rr=is>       zWordpieceTokenizer.tokenizeN)rr)rVrWrXrxr<r=r r r rrpas  rpcCs>|dks|dks|dks|dkrdSt|}|dkrdSdS)z1Checks whether `chars` is a whitespace character.rL   TZsF)rrrrr r rrs  rcCs8|dks |dks |dkrdSt|}|drdSdS)z.Checks whether `chars` is a control character.rrrFCT)rr startswithrr r rrs   rcCsxt|}|dko |dk}|p|dko|dk}|p|dko|dk}|p)|dko)|dk}|r.d St|}|d r:d Sd S) z2Checks whether `chars` is a punctuation character.!/:@[`{~TPF)rrrr)rrrrr r rrs  r) lru_cachecCsddS)NcSs|Sr$r )funcr r rszlru_cache..r r r r rrsrcstjddkr tnttttdtddtttdtddtttdtd d}|d d }d}td D]}||vrT|||d ||d7}q>fd d |D}tt ||S)a: Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. r!~r\¡¬®ÿNcsg|]}|qSr r )rnZ_chrr rr"sz$bytes_to_unicode..) sys version_infounichrchrrrangerrtdictzip)bscsrbr rrbytes_to_unicodes.        rcCs6t}|d}|ddD] }|||f|}q |S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rr\N)rSadd)wordpairsZ prev_charrr r r get_pairss rc@sbeZdZdZ   dddZddZdd Zd d Zd d ZddZ dddZ ddZ ddZ dS)r6zF GPT-2 BPE tokenizer. Peculiarities: - Byte-level BPE r NcCs|dur|ntd|_tt|dd|_dd|jD|_||_t |_ dd|j D|_ t|dd ddd }d d |D}tt|tt||_i|_td |_i|_i|_||dS) NrirRrZcSrr r rr r rrrz*GPT2Tokenizer.__init__..cSrr r rr r rrrrr\rcSsg|]}t|qSr )r.re)rmerger r rr"rKz*GPT2Tokenizer.__init__..zJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+)rrrsjsonloadr_encoderrdecodererrorsr byte_encoderrNreadrerrrr, bpe_rankscacherecompilepatr&special_tokens_decoderset_special_tokens)r!r:r;rr&rsZbpe_dataZ bpe_mergesr r rr<s zGPT2Tokenizer.__init__cCst|jt|jSr$)r,rr&r r r r__len__)szGPT2Tokenizer.__len__cs\|s i_i_dStfddt|D_ddjD_tdjdS)z Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabulary in the order of the `special_tokens` list. Nc3s&|]\}}|tj|fVqdSr$)r,r)rrCrr r rr)5sz3GPT2Tokenizer.set_special_tokens..cSrr r rr r rr7rz4GPT2Tokenizer.set_special_tokens..zSpecial tokens {})r&rr enumeraterrvinfork)r!r&r r rr,s  z GPT2Tokenizer.set_special_tokensc sX|jvr j|St|}t|}|s|S t|fddd}|jvr'ny|\}}g}d}|t|krz|||} |||| | }WntyZ|||dYn4w|||kr}|t|dkr}||d|kr}| |||d7}n | |||d7}|t|ks5t|}|}t|dkrnt|}qd |}|j|<|S) NTcsj|tdS)Ninf)rrfloat)pairr r rrHsz#GPT2Tokenizer.bpe..)keyrr\rrL) rr.rminrr,rbr} Exceptionrtr5) r!r9rrZbigramfirstsecondZnew_wordrCjr r rbpe=sV            zGPT2Tokenizer.bpecs^g}tj|D]#}dfdd|D}|dkrq |dd|dDq |S)z Tokenize a string. rMc3s,|]}t|jvrjt|VqdSr$)rr)rrr r rr)ls  z)GPT2Tokenizer.tokenize..css|]}|VqdSr$r )rZ bpe_tokenr r rr)ps rL)rfindallrr5r}rre)r!r?Z bpe_tokensr9r r rr=hs  zGPT2Tokenizer.tokenizecCsg}t|t}tjddkot|t}|s|r(||jvr!|j|S|j|dS|D]}||jvr:||j|q*||j|dq*t ||j krXt d t ||j |S)z9 Converts a sequence of tokens into ids using the vocab. rrzToken indices sequence length is longer than the specified maximum sequence length for this OpenAI GPT model ({} > {}). Running this sequence through the model will result in indexing errors) isinstancestrrrunicoder&rrrtr,rsrvrwrk)r!rFrGZpython_version_3Zpython_version_2r9r r rrEts$    z#GPT2Tokenizer.convert_tokens_to_idsFcCsBg}|D]}||jvr|s||j|q||j|q|S)z9Converts a sequence of ids in BPE tokens using the vocab.)rrtr)r!rGZskip_special_tokensrFrCr r rrIs z#GPT2Tokenizer.convert_ids_to_tokenscCs|||Sr$)rEr=r>r r rencodeszGPT2Tokenizer.encodecs>dfdd|D}tfdd|Djdjd}|S)NrMcg|]}j|qSr )r)rr9r r rr"rz(GPT2Tokenizer.decode..crr )rNrOr r rr"rrR)r)r5rTrUr)r!rFr?r r rrUs zGPT2Tokenizer.decode)r NN)F) rVrWrXrxr<rrrr=rErIrrUr r r rr6 s +  r6)! __future__rrrrr]loggingr3rrrregexrrobjectr getLoggerrVrvrdrfr/rnrprrr functoolsr ImportErrorrrr6r r r rs<i  Fv<