o *jG@sdZddlZddlZddlZddlmZddlZddlZddl Z eddZ eddZ dd Z d d Z d d ZGdddeZdddZdS)z CLIP Tokenizer.N) lru_cachecCstjtjtjtdS)Nzbpe_simple_vocab_16e6.txt.gz)ospathjoindirnameabspath__file__r r m/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/multi_modal/gemm/tokenizer.py default_bpesr cCstttdtddtttdtddtttdtdd}|dd}d }td D]}||vrI|||d ||d7}q3d d |D}tt||S) a: Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. And avoids mapping to whitespace/control characters the bpe code barfs on. !~¡¬®ÿNrcSsg|]}t|qSr )chr).0nr r r 1z$bytes_to_unicode..)listrangeordappenddictzip)bscsrbr r r bytes_to_unicodes,       r"cCs6t}|d}|ddD] }|||f|}q |S)zReturn set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairsZ prev_charcharr r r get_pairs5s r(cCs"t|}tt|}|SN)ftfyZfix_texthtmlunescapestriptextr r r basic_cleanAs r0cCstdd|}|}|S)Nz\s+ )resubr-r.r r r whitespace_cleanGsr4c@s8eZdZefdefddZddZddZdd Zd S) SimpleTokenizerbpe_pathcCst|_dd|jD|_t|dd}|dd}dd|D}t t }|d d|D}|D] }| d |q;| d d gtt|tt||_d d|jD|_tt|tt||_d d d|_tdtj|_dS)NcSi|]\}}||qSr r rkvr r r Qz,SimpleTokenizer.__init__..utf-8 ricSsg|]}t|qSr )tuplesplit)rmerger r r rTsz,SimpleTokenizer.__init__..cSsg|]}|dqS)r )rr:r r r rVr<|startoftext|> <|endoftext|>cSr7r r r8r r r r;[r<)rDrEz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)r" byte_encoderitems byte_decodergzipopenreaddecoder@rvaluesrrextendrrrlenencoderdecoder bpe_rankscacher2compile IGNORECASEpat)selfr6ZmergesZvocabrAr r r __init__Os(  zSimpleTokenizer.__init__c s|jvr j|St|dd|ddf}t|}|s#|dSg} t|fddd}|jvr6n|\}}g}d} | t|krz||| } ||| | | } Wn tyw} z| | ||| dWYd} ~ n8d} ~ ww|| |kr| t|dkr|| d|kr| ||| d 7} n | || | d7} | t|ksDt|}|}t|dkrnt|}q&t|d krt |dd |}|j|<|S) NrBTcsj|tdS)Ninf)rRgetfloat)pairrWr r qsz%SimpleTokenizer.bpe..)keyrrdr1) rSr?r(minrRrOindexrN Exceptionrprintr) rWtokenr%r&Z error_listZbigramfirstsecondZnew_wordijerrr r^r bpees`               zSimpleTokenizer.bpecsng}tt|}tj|D]#}dfdd|dD}|fdd | dDq|S)NrCc3|]}j|VqdSr))rF)rr!r^r r z)SimpleTokenizer.encode..r=c3rnr))rP)rZ bpe_tokenr^r r rorpr1) r4r0lowerr2findallrVrencoderNrmr@)rWr/Z bpe_tokensrgr r^r rss zSimpleTokenizer.encodecsDdfdd|D}tfdd|Djddddd }|S) NrCcg|]}j|qSr )rQ)rrgr^r r rr<z*SimpleTokenizer.decode..crtr )rH)rcr^r r rr<r=replace)errorsrBr1)r bytearrayrLrv)rWtokensr/r r^r rLszSimpleTokenizer.decodeN) __name__ __module__ __qualname__r strrXrmrsrLr r r r r5Ms / r5MTcst|tr|g}jdjdfdd|D}tjt||tjd}t|D]0\}}t||krO|rC|d|}|d<n td||d |t |||dt|f<q,|S) a Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input string or a list of input strings to tokenize context_length : int The context length to use; all CLIP models use 77 as the context length truncate: bool Whether to truncate the text in case its encoding is longer than the context length Returns ------- A two-dimensional tensor containing the resulting tokens, shape = [number of input strings, context_length]. We return LongTensor when torch version is <1.8.0, since older index_select requires indices to be long. rDrEcs"g|] }g|gqSr )rs)rr/Z eot_tokenZ sot_token tokenizerr r rsz!clip_tokenize..)ZdtypeNrYzInput z is too long for context length ) isinstancer}rPtorchZzerosrOint enumerate RuntimeErrorZtensor)rZtextsZcontext_lengthtruncateZ all_tokensresultrjryr rr clip_tokenizes$      r)r~T)__doc__rIr+r functoolsrr*regexr2rr r"r(r0r4objectr5rr r r r s"    X