o 0jO@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z ddlmZddlmZmZmZmZmZmZmZddlZddlmZddlmZmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+d d lm,Z,m-Z-d d l.m/Z/ed rddl0m1Z1ddl2m3Z3m4Z4ddl5m6Z6gdZ7ed e GdddZ8ddZ9Gddde:Z;GdddZddZ?d d!Z@d"d#ZAd$d%ZBGd&d'd'ZCGd(d)d)eCe'e;d*ZDd+dZ?d,d-ZEd.d/ZFd0d1ZGd2d3ZHd4d5ZIdS)6N) OrderedDict)asdict dataclass) lru_cache)AnyDictListLiteralOptionalTupleUnion)logging)class_requires_depsis_dep_available) CHAT_TEMPLATE_CONFIG_NAME AddedToken BatchEncoding EncodedInputEncodedInputPairPaddingStrategyPreTokenizedInputPreTokenizedInputPairPretrainedTokenizerBase TensorType TextInput TextInputPairTruncationStrategy)convert_to_dict_messagefn_args_to_dict)VocabZJinja2)Template) TemplateErrorTemplateSyntaxError)ImmutableSandboxedEnvironment) ChatTemplateTrieChatTemplateMixinPretrainedTokenizerInitTrackerMetac @seZdZUdZeeedfed<dZeedfed<dZ eed<e e dddZ d ifd eeee eeffd ed e eefdeefd dZd ifded ed e eeeefffddZifd e eeeeffde eeeefffddZifd e eeeeffdefddZifdeeeeefd e eeeeffdefddZede fddZedefddZdS)r&N conversationsystemqueryreturnr"cCs*dd}tdddd}||jd<||S)NcSst|N)r#)messager1z/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/models/common/tokenizer/tokenizer_utils.pyraise_exceptionKsz=ChatTemplate._compile_jinja_template..raise_exceptionT)Z trim_blocksZ lstrip_blocksZkeep_trailing_newliner3)r%globalsZ from_string) chat_templater3Z jinja_envr1r1r2_compile_jinja_templateHs   z$ChatTemplate._compile_jinja_templaterconversation_dataindex context_datacCs|jdur tdt|ttfr$t|dksJd|d|d|d}||g}|jD]}||}||}| |q.|S)z Args: conversation_data (list[str]): the conversation data which must be two parts index (int): the index of current conversation Returns: list[str]: the rendered conversation data NzaThe template for multi-turns is invalid, please check `conversation` filed in your chat-template.zUEach round/turn of conversation must be two participants, eg: [user-query, bot-query]rr)userZbotr8) r+ ValueError isinstancelisttuplelenupdater6renderappend)selfr7r8r9Zone_turn_conversationr+templateresultr1r1r2render_conversationTs&      z ChatTemplate.render_conversationcCs0|jdur|S||j}|jd||d|S)N)r-r8r1)r-r6rB)rDr-r8r9rEr1r1r2 render_queryzs  zChatTemplate.render_querycCs|dd|d<|S)z'init the context data for chat-templateZ is_trainingF)get)rDr9r1r1r2_init_context_dataszChatTemplate._init_context_datacCs*|jdurdS||j}|jdi|S)Nr1)r,r6rB)rDr9rEr1r1r2 render_systems  zChatTemplate.render_system conversationsc Cst|tr |gg}|j|d}t||d<t|ddD]\}}|dk|d<d|d<|d |j|||d 7}qt|dtsLt|dd ksLtd t|dd krbt d |dd d||j |ddt|d |d 7}|S)zrender the conversations by chat-template Args: conversations (list[list[str]]): the conversations of use and bot Returns: str: the result of conversation r9lengthNris_firstFis_lastrKr8r9rzxThe length of last conversation must be one, eg: [[user-query, bot-answer], [user-query, bot-answer], ..., [user-query]]zWThe last conversation is not a single-round, chat-template will skip the conversation: ) r=strrLr@ enumeratejoinrGr>r<rwarningrH)rDrMr9Z final_queryr8r+r1r1r2__call__s4      zChatTemplate.__call__configcCs|di|S)Nr1r1)clsrYr1r1r2 from_dictszChatTemplate.from_dictfilecCsBt|ddd }t|}Wdn1swY||S)Nrutf-8encoding)openjsonloadr[)rZr\frYr1r1r2 from_files  zChatTemplate.from_file)r.r")__name__ __module__ __qualname__r+r rrT__annotations__r,r- staticmethodrr6rintrrGrHrJrLrX classmethodr[rer1r1r1r2r&AsZ     '   &  ,r&c s:|dkrtjdsjjjdrSt\}}}}}}tj\}}} }}}fdddDrddl} j dr\t d t | j jrRjnd d nt d t | j jrijnd d t | j jrtrtfd d} | Stfdd} | SS)z Since there are some monkey patches for forward of PretrainedModel, such as model compression, we make these patches compatible with the latest forward method. forwardZStaticFunctioncs g|] }|vr|vr|qSr1r1).0arg)patch_spec_args spec_argsr1r2 s z)adapt_stale_fwd_patch..)Zoutput_hidden_statesZoutput_attentions return_dictrNZ paddlenlpzThe `forward` method of z{ is patched and the patch might be based on an old oversion which missing some arguments compared with the latest, such as zm. We automatically add compatibility on the patch for these arguments, and maybe the patch should be updated.zs is patched and the patch might be conflict with patches made by paddlenlp which seems have more arguments such as cs,D]}||dqg|Ri|Sr/popargskwargsro)new_argsrDvaluer1r2wrap_fwdsz'adapt_stale_fwd_patch..wrap_fwdcs$D]}||dq|i|Sr/rtrv)ryrzr1r2r{s)typerfendswithrm __class__inspectgetfullargspecpaddlerg startswithrrWr=nnZLayer isfunction functoolswraps) rDnamerzZpatch_spec_varargsZpatch_spec_varkwZpatch_spec_defaults_Z spec_varargsZ spec_varkwZ spec_defaultsrr{r1)ryrprDrqrzr2adapt_stale_fwd_patchsT      rcs:eZdZdZfddZed ddZfddZZS) r*a This metaclass wraps the `__init__` method of a class to add `init_config` attribute for instances of that class, and `init_config` use a dict to track the initial configuration. If the class has `_pre_init` or `_post_init` method, it would be hooked before or after `__init__` and called as `_pre_init(self, init_fn, init_args)` or `_post_init(self, init_fn, init_args)`. Since InitTrackerMeta would be used as metaclass for pretrained model classes, which always are Layer and `type(Layer)` is not `type`, thus use `type(Layer)` rather than `type` as base class for it to avoid inheritance metaclass conflicts. cs^|j}d|vr t|ddnd}d|vrt|ddnd}t||||_tt||||dS)N__init__ _pre_initZ _post_init)rgetattrr*init_and_track_confsuper)rZrbasesattrs init_func pre_init_funcpost_init_funcr~r1r2r#szInitTrackerMeta.__init__Ncstfdd}|S)aK wraps `init_func` which is `__init__` method of a class to add `init_config` attribute for instances of that class. Args: init_func (callable): It should be the `__init__` method of a class. warning: `self` always is the class type of down-stream model, eg: BertForTokenClassification pre_init_func (callable, optional): If provided, it would be hooked after `init_func` and called as `pre_init_func(self, init_func, *init_args, **init_args)`. Default None. post_init_func (callable, optional): If provided, it would be hooked after `init_func` and called as `post_init_func(self, init_func, *init_args, **init_args)`. Default None. Returns: function: the wrapped function cspr|g|Ri||g|Ri|r'|g|Ri|||_|r0||d<|jj|d<dS)NZ init_argsZ init_class)Z init_configr~rf)rDrwrxrrrr1r2__impl__Dsz5InitTrackerMeta.init_and_track_conf..__impl__)rr)rrrrr1rr2r1sz#InitTrackerMeta.init_and_track_confcst|||}tt|||Sr/)rrr* __setattr__)rDrrzrr1r2rUs zInitTrackerMeta.__setattr__)NN) rfrgrh__doc__rrjrr __classcell__r1r1rr2r*s   #r*c@sDeZdZdZddZdefddZdedeefd d Zd d Z d S)r'z Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie cCs i|_dSr/datarDr1r1r2r`s z Trie.__init__wordcCsD|sdS|j}|D]}||vr||pi||<||}q d|d<dS)uy Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` NrrKr)rDrrefcharr1r1r2addcs  zTrie.addtextr.cCst}dg}d}t|D]\}}|r||krq t}d}|D]\} } d| vr|D]V\} } | | kr6nM| | krC|d} |d}n|} |}| t|krQ|| nd}d| vr]| } | }| }|| vr| |} | d7} d| vrs| } | }| }| t|krzn|| }|| vsaq,|| ||d}n|| vr| |} | || <q || q |ri}n|D]} || =q||kr||jvr|j|||<q |D]\} } d| vrt|}|| ||nq|||S)a\ Will look for the words added to the trie within `text`. Output is the original string splitted along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` rFrKrNT) rrUsetitemsr@rCrrcut_text)rDrZstatesoffsetsskipcurrentZ current_char to_removeresetstartZ trie_pointerZ lookstartZlooktrie_pointerZlookahead_indexendZ next_charr1r1r2splitsz!            z Trie.splitcCsX|t|g}d}|D]}||krtdq ||krq |||||}q |S)NrzbThere was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it anyway.)rCr@rerror)rDrrtokensrrr1r1r2rsz Trie.cut_textN) rfrgrhrrrTrrrrr1r1r1r2r'Zs r' token_list new_tokencCs8t||}|t|kr|||krdS|||dS)zm Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. N)bisect bisect_leftr@insert)rrZ insertion_idxr1r1r2!_insert_one_token_to_ordered_list5s rcC8|dks |dks |dkrdSt|}|drdSdSz.Checks whether `chars` is a control character.   FCT unicodedatacategoryrrcatr1r1r2 _is_controlB   rcCst|}d|krdks>nd|krdks>nd|kr$dks>nd|kr/dks>nd |kr;d kr@d Sd Sd Sd S) z4Check whether `chars` is a non-normalized character.iiiPikiX3i3i$i$i2i2TFordrcpr1r1r2_is_nonnormalized_charNsrcCsnt|}d|krdks3nd|krdks3nd|kr$dks3nd|kr0dkr5d Sd Sd Sd S) zr<)rDr+r9r-r1r1r2rs   z-ChatTemplateMixin._apply_chat_template_paddlec Cs"t|tr d|dg}n)t|tr5t|dksJdt|dtr't|}nt|dtr1|}ntdz|jjd d|i|j d|i}W|St yt t|D]'}||d}t|trvd }|D]}| d d kru|d }nqf|||d<qU|jjd d|i|j d|i}Y|Sw) Nr;Zrolecontentrz!empty conversation is not allowedrmessagesrrrKr|rr1) r=rTr>r@rdictr<r5rBspecial_tokens_map TypeErrorrangerI) rDr+rrMr-irZ new_contentpartr1r1r2rsP          z&ChatTemplateMixin._apply_chat_templaterMcKsV|jstdt|jtr|dd}|j|||d}|St|jtr)|||}|S)aEncodes conversation to pairs of token ids. Turn 0: bos + system + sep + user bot + eos Turn t: sep + bot + query bot + eos Args: conversation (List[List[str]]): the conversation of data context_data (Dict[str, Any]): the context data of conversation Returns: List[list[int], list[int]]: the pair of input_ids and target_ids rrTr)r5r<r=r"ru_encode_chat_inputsr&_encode_chat_inputs_paddle)rDrMr9rxrr-r1r1r2encode_chat_inputss   z$ChatTemplateMixin.encode_chat_inputsc Cs|j|}i}|jjr|j|}|j|ddd|d<g}t|D]8\}}|dk|d<|t|dk|d<|jj|||d \}} |j|ddd} |j| ddd} || | gq#||d <|S) NF)r input_idsr,rrQrrRrSrM) r5rJr,rLencoderUr@rGrC) rDrMr9rFr,conversation_idsr8r+Z user_inputZ bot_outputZuser_idsZbot_idsr1r1r2rs&     z,ChatTemplateMixin._encode_chat_inputs_paddler,c Csi}|r"z |jjd|ddWnty!}ztd|d}~wwg}g}|D]} d| ddd| ddg} || || q(g} |D]A} |rP|g| n| } |jjd| d d |j}|ri|g| dgn| dg}|jjd||d |j}|t|d}| |qG||| }t|t| ksJd t|d t| d g}t t|D]}||j ||| |gd d ddq||d<|S)Nr,r)rz*System is not supported in this tokenizer.r;rZ assistantrFrrzGet non_learnable_parts len: z, but ans len: .)rpaddingrrMr1) r5rB Exceptionr<extendrCrr@_extract_non_learnable_partsrZ batch_encode)rDrMr9r,rrFeZconversation_dict origin_msgroundZ round_roleZansconvZroundiZ roundi_strZ roundi_no_ansZroundi_no_ans_strZ ans_roundinon_learnable_partsrrr1r1r2rsl        z%ChatTemplateMixin._encode_chat_inputsrsplit_scCsPdttj|}td||jjd|dd|j}|ddkr&||S) zJSplit the entire chat by specified words. Extract the non-learnable parts.|z(?:%s)FrrPrKNr1) rVmapreescaperr5rBrru)rDrrZ regex_patternrr1r1r2r^s z.ChatTemplateMixin._extract_non_learnable_partsc s|dd}|dd}|dd}|dd}|durd}||d<||d<||d<||d<d|d<tj|g|Ri|\}} tj| t} tj| sP|S|jdur_t d t d | | |S) N cache_dir from_hf_hubF from_aistudio subfolderrKTZreturn_tokenizer_file_dirz_Chat-template already exists in config file, it will be overwritten by chat_template.json file.z``chat_template.json` will be deprecated in the future! Please set it in `tokenizer_config.json`.) rurfrom_pretrainedospathrVrexistsr5rrWinit_chat_template) rZZpretrained_model_name_or_pathrwrxrrrr tokenizerZtokenizer_config_file_dirchat_template_filerr1r2rqs@        z!ChatTemplateMixin.from_pretrainedcCst|tr-tj|s%z t||_WdSty$td |ddwt ||_dSt|t r:t ||_dSt|trD||_dSt d|)zinit chat_tempalte by file_path or template dict data Args: chat_template (str, dict): file_path or template dict data z7The chat-template in json is not valid jinja string: {}r)linenoz"Receive error chat_template data: N)r=rTrrrr&r6r5r$formatrerr[r<)rDr5r1r1r2rs(       z$ChatTemplateMixin.init_chat_templatecCsvt|jtr9tj|t}t|ddd}tj t |j|dddWdn1s+wYt d|dSdS)Nwr^r_F) ensure_asciiindentz#Chat-template config file saved in ) r=r5r&rrrVrrarbdumprrinfo)rDZsave_directoryrrdr1r1r2save_resourcessz ChatTemplateMixin.save_resources)T)rfrgrhr5r r&rir rrTrboolrrrrrrrrrlrrrr rr1r1rr2r(sh   '   ,          @ "r(c)seZdZUdZiZeeefed<iZ eeefed<gZ e eed<e Z dZfddZdd Zed efd d Zed efd dZd eeeffddZddZ d^dee ee efded efddZddZd^ddZded e efddZddZd d!Zd"d#Z d$d%Z!d&d'Z"d^d(d)Z#d*d+Z$e% , , , ,d_d-d.Z&e%d/d0Z' d`fd1d2 Z(d3d4Z)d,d5e*j+e,j-d,d6dd,d,d,d,d,d,ddddd5fdeee.e/fd7e0eee.e/fd8ed9e*d:e,d;e0ede0ed?e0e1d@dAe0eee2fdBe0edCe0edDe0edEedFedGedHedIed e3f(dJdKZ4d5e*j+e,j-d,d6dd,d,d,d,d,d,ddd5ddd5fdLee ee e5e e.e e6e e/e e7fd8ed9e*d:e,d;e0ede0ed?e0e1d@dBe0edAe0eee2fdCe0edDe0edEedFedMedGedHedIed e3f(dNdOZ8d5e*j+e,j-d,d6d,d,d,d,d,d,ddd5ddd5fdPe ee6e9e ed,ffd8ed9e*d:e,d;e0ede0ed?e0e1d@dBe0edAe0edCe0edDe0edEedFedMedGedHedIed e3f&dQdRZ:defdSdTZ;dadedUe0e efdVdWZ<  5 5dbdXe edYedZed[ed ef d\d]Z=Z>S)cr)an Base class for all tokenizers. Inherits from [`~tokenizer_utils_base.PretrainedTokenizerBase`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). - **resource_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_resource_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **max_model_input_sizes** (`Dict[str, Optional[int]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size. - **pretrained_init_configuration** (`Dict[str, Dict[str, Any]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [`~tokenizer_utils_base.PretrainedTokenizerBase.from_pretrained`] method. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Moreover, methods common to tokenizers for tokenization, token/id conversion and encoding as model inputs are also provided here. Besides, metaclass `InitTrackerMeta` is used to create `PretrainedTokenizer`, by which subclasses can track arguments for initialization automatically and expose special tokens initialization used as attributes. added_tokens_encoderadded_tokens_decoderunique_no_split_tokensFcst|g|f|Ri|}|ddtt|jdi|i|_|j|didd|jD|_g|_ t |_ d|_ dS)z It would be hooked before `__init__` to add specials tokens (arguments of `__init__` whose name ends with `_token`) as attributes of the tokenizer instance. rDNr cSsi|]\}}|j|qSr1)r)rnvkr1r1r2 s z1PretrainedTokenizer._pre_init..Fr1) r rurr)rr rArr rr' tokens_trie_decode_use_source_tokenizer)rDZ original_initrwrx init_dictrr1r2rs  zPretrainedTokenizer._pre_initcKs|D]O\}}|dur q||jvrS|dkr9t|ttfs%Jd|dtdd|Ds2Jdt|||qt|ttfrGt|||qt d|dt |qdS) NZadditional_special_tokenszValue z is not a list or tuplecss|] }t|ttfVqdSr/)r=rTrrntr1r1r2 s zIPretrainedTokenizer._build_special_tokens_map_extended..z2One of the tokens is not a string or an AddedTokenzspecial token z- has to be either str or AddedToken but got: ) rZSPECIAL_TOKENS_ATTRIBUTESr=r>r?allsetattrrTrrr|)rDrxkeyrzr1r1r2"_build_special_tokens_map_extendeds.   z6PretrainedTokenizer._build_special_tokens_map_extendedr.cCt)zP `int`: Size of the base vocabulary (without the added tokens). NotImplementedErrorrr1r1r2 vocab_sizeszPretrainedTokenizer.vocab_sizecCsdSNFr1rr1r1r2is_fastszPretrainedTokenizer.is_fastcCs|jS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. )r rr1r1r2get_added_vocab sz#PretrainedTokenizer.get_added_vocabcCs|jt|jS)zD Size of the full vocabulary with the added tokens. )rr@r rr1r1r2__len__)szPretrainedTokenizer.__len__ new_tokensspecial_tokenscsdd|D}g}|D]N}t|tstd|dt|d|s-tdr-jr-|}|jkrY|jkrY||vrY|j vrY| |j rYt d|dq tfd d t|D}d d |D}j |j||rt|d krtj|dn*ttjt|_nt|d krtj|dn ttjt|_jt|S)aS Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens (`List[str]`or `List[AddedToken]`): Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") ```cSsg|]}t|qSr1)rT)rntokr1r1r2rrLz3PretrainedTokenizer._add_tokens..zToken z is not a string but a r do_lower_casezAdding z to the vocabularyc3s$|] \}}|t|fVqdSr/)r@)rnrr&rr1r2rcs z2PretrainedTokenizer._add_tokens..cSsi|]\}}||qSr1r1)rnrrr1r1r2rfsz3PretrainedTokenizer._add_tokens..rr)r=rTrr|hasattrr(lower unk_tokenconvert_tokens_to_idsr keysrCverboserr rrUrrAr r@rrsortedrunion _create_trie)rDr$r%Z tokens_to_addtokenZadded_tok_encoderZadded_tok_decoderr1rr2 _add_tokens/sZ            zPretrainedTokenizer._add_tokenscCsNt}|D]}t|dr|jr||jvr||q||q||_dS)Nr()r'r)r(all_special_tokensrr*r)rDrZtrier2r1r1r2r1s   z PretrainedTokenizer._create_triecKs||fS)a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. kwargs: Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. r1)rDris_split_into_wordsrxr1r1r2prepare_for_tokenizationsz,PretrainedTokenizer.prepare_for_tokenizationrcKs|d|j}tdd|jD}|j|fi|\}}t|drC|jrCdd|j|jD}dd |d d }t |d d |}|rKg}|g}n t |j}|j |}t|D]I\} } | |vr|| d } | dkrr|| dnd } | t|dkr|| dnd } t| tr| jr| r| || d<| jr| r| || d<qZg}|D]} | sq| |vr|| q||| q|S)a" Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens. Args: text (`str`): The sequence to be encoded. **kwargs (additional keyword arguments): Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: `List[str]`: The list of tokens. split_special_tokenscss&|]}t|trt||fVqdSr/)r=rrTrr1r1r2rs  z/PretrainedTokenizer.tokenize..r(cSsg|]}t|qSr1)rr)rnZs_tokr1r1r2rrsz0PretrainedTokenizer.tokenize..(rz)|z(.+?)cSs|dp |dS)Nrr)groupsr*)mr1r1r2sz.PretrainedTokenizer.tokenize..Nrr)rur7rall_special_tokens_extendedr6r)r(rr4rVrsubrrrrUrIr@r=rrstriplstriprCr _tokenize)rDrrxr7r<Zescaped_special_tokspatternZno_split_tokenrrr2Z tok_extendedleftrightZtokenized_textr1r1r2rsR         zPretrainedTokenizer.tokenizecKr)a Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. r)rDrrxr1r1r2r@szPretrainedTokenizer._tokenizecCsB|durdSt|tr||Sg}|D] }|||q|Sr/)r=rT#_convert_token_to_id_with_added_vocrC)rDridsr2r1r1r2r,s  z)PretrainedTokenizer.convert_tokens_to_idscCs*|durdS||jvr|j|S||Sr/)r _convert_token_to_idrDr2r1r1r2rD s    z7PretrainedTokenizer._convert_token_to_id_with_added_voccC |j|Sr/)vocabZ to_indicesrGr1r1r2rF z(PretrainedTokenizer._convert_token_to_idcCs d|S)z Converts a sequence of tokens (list of string) to a single string by using ``' '.join(tokens)`` . Args: tokens (list[str]): A sequence of tokens. Returns: str: Converted string. r)rV)rDrr1r1r2convert_tokens_to_strings z,PretrainedTokenizer.convert_tokens_to_stringcCst|tr"||jvr|j|}t|tr|j}|S|}|S||Sg}|D]0}t|}|r4||jvr4q&||jvrN|j|}t|trF|jn|}||q&|||q&|Sr/)r=rkr rr_convert_id_to_tokenall_special_idsrC)rDrEskip_special_tokensr2rr8r1r1r2convert_ids_to_tokens$s&       z)PretrainedTokenizer.convert_ids_to_tokenscCrHr/)rIZ to_tokens)rDr8r1r1r2rL9rJz(PretrainedTokenizer._convert_id_to_tokenNc Ks|i}tj|ddd}t|D]\}} | d} t||| <qWdn1s)wYtj|f||||d|} | S)a Instantiate an instance of `Vocab` from a file reserving all tokens by using `Vocab.from_dict`. The file contains a token per line, and the line number would be the index of corresponding token. Args: filepath (str): path of file to construct vocabulary. unk_token (str): special token for unknown token. If no need, it also could be `None`. Defaults to `None`. pad_token (str): special token for padding token. If no need, it also could be `None`. Defaults to `None`. bos_token (str): special token for bos token. If no need, it also could be `None`. Defaults to `None`. eos_token (str): special token for eos token. If no need, it also could be `None`. Defaults to `None`. **kwargs (dict): keyword arguments for `Vocab.from_dict`. Returns: Vocab: An instance of `Vocab`. r]r^r_rN)r+ pad_token bos_token eos_token)iorarUr>rkr!r[) filepathr+rPrQrRrxZ token_to_idxrdr8liner2rIr1r1r2load_vocabulary=s$ z#PretrainedTokenizer.load_vocabularycsxttr j}n tfddd}tj|ddd}|D] }||dq WddS1s5wYdS) a3 Save all tokens to a vocabulary file. The file contains a token per line, and the line number would be the index of corresponding token. Args: filepath (str): File path to be saved to. vocab (Vocab|dict): The `Vocab` or `dict` instance to be saved. cs|Sr/r1)r2rIr1r2r;vsz5PretrainedTokenizer.save_vocabulary..)rrr^r_rN)r=r!Z idx_to_tokenr/r-rSrawrite)rTrIrrdr2r1rWr2save_vocabularyis "z#PretrainedTokenizer.save_vocabularycsD|r|dur tdtj||ddSdg|rt|ndt|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer ``encode`` methods. Args: token_ids_0 (List[int]): List of ids of the first sequence. token_ids_1 (List[int], optional): List of ids of the second sequence. already_has_special_tokens (bool, optional): Whether or not the token list is already formatted with special tokens for the model. Defaults to None. Returns: results (List[int]): The list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.T) token_ids_0 token_ids_1already_has_special_tokensr)r<rget_special_tokens_maskr@)rDrZr[r\rr1r2r]{sz+PretrainedTokenizer.get_special_tokens_maskcCs$g}g}t|||r|SdS)a Returns the number of added tokens when encoding a sequence with special tokens. Args: pair (bool, optional): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Defaults to `False`. Returns: int: Number of special tokens added to sequences. N)r@ build_inputs_with_special_tokens)rDpairrZr[r1r1r2num_special_tokens_to_adds z-PretrainedTokenizer.num_special_tokens_to_addTr text_pairrpadding_strategytruncation_strategy max_lengthstrider5pad_to_multiple_of padding_side)rCrBreturn_tensorsreturn_position_idsreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthr.c sfdd}||}|dur||nd}|r |d<|d<j|fid|d|d|jd|jd |d |d | d | d | ddd| d|d| d|d|d|d|d|S)Ncst|trj|fi}|St|ttfrBt|dkrBt|dtrBr=ttjfdd|D}|S|St|ttfrXt|dkrXt|dt rX|Srbt d|dt d|d)Nrc3&|]}j|fddiVqdSr5TNrrrxrDr1r2r  zJPretrainedTokenizer._encode_plus..get_input_ids..zInput z] is not valid. Should be a string or a list/tuple of strings when `is_split_into_words=True`.zW is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers. r=rTrr,r>r?r@ itertoolschainrkr<rrr5rxrDr1r2 get_input_idss>             z7PretrainedTokenizer._encode_plus..get_input_idsrrapair_idsrr truncationrdrerfrgrhprepend_batch_axisTrirkrjrlrmrnror.)prepare_for_modelrz)rDrrarrbrcrdrer5rfrgrhrirjrkrlrmrnror.rxrz first_ids second_idsr1ryr2 _encode_pluss\%     z PretrainedTokenizer._encode_plusbatch_text_or_text_pairsrsc sfdd}g}|D]9}t|ttfs|d}}nr,t|dttfs,|d}}n|\}}||}|dur<||nd}|||fq |dkrS|durS|d<n;|rd}t|dkrht|dttfrhd}dd<dd<|rd d |Dd<d d |Dd<n d d |Dd<j|fid |d|d|d|d|d|d| d| d| d| d|d|d|d|d|d| d|}|S)Ncst|trj|fi}|St|ttfrBt|dkrBt|dtrBr=ttjfdd|D}|S|St|ttfrXt|dkrXt|dt rX|St d)Nrc3rprqrrrrsr1r2r1rtzPPretrainedTokenizer._batch_encode_plus..get_input_ids..z\Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.rurxryr1r2rz%s6           z=PretrainedTokenizer._batch_encode_plus..get_input_idsrrFTtexts text_pairscSg|]}|dqS)rr1rnrr1r1r2rr_r'z:PretrainedTokenizer._batch_encode_plus..cSr)rr1rr1r1r2rr`scSsg|]}|qSr1r1rr1r1r2rrdsrrbrcrdrerfrgrirkrjrlrmrsrnrorhr.)r=r>r?rCr@_batch_prepare_for_model)rDrrrbrcrdrer5rfrgrirhrjrkrlrmrsrnror.rxrzrZids_or_pair_idsrEr{rrZhas_pair batch_outputsr1ryr2_batch_encode_pluss            z&PretrainedTokenizer._batch_encode_plusbatch_ids_pairsc.Ks| r|stdi}g}t|D]\}\}}|dkr;|dur;| dur*d|jv} | dur3d|jv} |t||r@|jddnd}|d|\}}||}||}d}|t|kr:i}t||} | |krl|} |}!|||| }"t|"du}#|}$|||| }%|r||$|%}&||!|"}'| |!|"}(n|$|%}&|#r|!|"n|!}'dgt|!|#rdgt|"ng}(|&|d <|'|d <| r|(|d<|r|r| |!|"|d <n dgt|'|d <| |d ||| rt t t|d |d <|rt|d |d <|d |d<||d<|D]\})}*|)|vrg||)<||)|*q || t|kr-n |t| |7}|t|ks^q|rY|d||d<d|d<|ddurY|d||d<|j||fid|dtjjd|jd|d|ddd|d| ddd| d| d |d!|d"|d#dd$dd%||}|D]\})}*|)|vrg||)<||)|*qq|j||j|||| d&}|rt|| d'}|S|D])\}+},t t|,D]}-|-t|kr||+|,|-iq|,|-||-|+<qِq|S)(a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs zAsking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.rNtoken_type_idsZattention_maskT)r_roffset_mappingrZspecial_tokens_maskZ position_idsrOZseq_lenZoverflow_to_samplerrrarrrr|rdrerfrgrirkFrjrlrmrnrorhr}r.)rrdrfrgrk)Z tensor_type)r<rUZmodel_input_namesr@r`get_offset_mappingr Z(build_offset_mapping_with_special_tokensr^Z$create_token_type_ids_from_sequencesr]Z&_eventual_warn_about_too_long_sequencer>rrrCminr~r DO_NOT_PADrzpadr).rDrrrbrcrdrerfrgrirhrjrkrlrmrsrnror.rxrZbatch_outputs_listZ example_idrrZmax_len_for_pairrraZtoken_offset_mappingZtoken_pair_offset_mappingoffsetZencoded_inputsrOrEr{r_mappingZ pair_mappingrsequencerrrzrrrr1r1r2r~s             >       z,PretrainedTokenizer._batch_prepare_for_modelcCs|durdS||}dg}}t|D]S\}}t|dr;|jr;|}|jjdur:td|}d dd|D}n|jjrOtd|}d dd|D}d d d|D}||7}| |gt |q|gd }}}g} t|D]\} } | dd d kr| d d} | |j vrt|dr|jr| n| } d | vsd| vr||d dd | dd |} n,| ||dvr| t |dkr|| d|j vr|} d} nd} n ||d| |} | t | } | | | g| dkr| }qvg}t| D]@\} \} } | dkr.| d krd } n| | dd} | t | dkr&t |} n| | dd } ||| || ddfq|S)aR Returns the map of tokens and the start and end index of their start and end character. Modified from https://github.com/bojone/bert4keras/blob/master/bert4keras/tokenizers.py#L372 Args: text (str): Input text. Returns: list: The offset map of input text. NrKr(FZNFDcSg|] }t|dkr|qSZMnrrrnrr1r1r2rrDzEPretrainedTokenizer._get_bert_like_offset_mapping..cSrrrrr1r1r2rrGrcSs0g|]}t|dkst|dkst|s|qS)ri)rrrr1r1r2rrJs  rr:z##σςrrrP)rrUr)r(r*basic_tokenizerZ strip_accentsrrrVrr@r4replacer8rC)rDr split_tokensnormalized_text char_mappingrch token_mappingrchar_mapping_indexesr8r2rrr1r1r2_get_bert_like_offset_mapping.sz               z1PretrainedTokenizer._get_bert_like_offset_mappingrcCs|durdS||}t|dst|dr||S|s!||}dg}}t|D]\}}|t|7}||gt|q*|gd}}}t|dd} | rS|}g} |D]Y} | |  } | |j vrk| rk| } d| vssd | vr||d d d | d d|} n| ||dvrd } n ||d | |} | t| } | | | g| d kr| }qWg}t| D]=\}\} } | d kr|dkrd} n| |d d } |t| d krt|} n| |d d} ||| || d d fq|S) a Returns the map of tokens and the start and end index of their start and end character. Modified from https://github.com/bojone/bert4keras/blob/master/bert4keras/tokenizers.py#L372 Args: text (str): Input text. split_tokens (Optional[List[str]]): the tokens which has been split which can accelerate the operation. Returns: list: The offset map of input text. NrZwordpiece_tokenizerrKrr(FrrrPr)rr)rrUrrr@rr*rKstripr4rr8rC)rDrrrrrrrrr(rr2rrr8r1r1r2rsZ           z&PretrainedTokenizer.get_offset_mapping token_idsrNclean_up_tokenization_spacesspaces_between_special_tokensc Kst|tjr |}|dd|_|j||d}g}g}|D]&} |r(| |jvr(q| |jvr?|r9| | |g}| | q| | q|rO| | ||rWd |} nd |} |re| | } | S| S)NZuse_source_tokenizerF)rNrrK) r=npZndarraytolistrurrOrMr rCrKrVZclean_up_tokenization) rDrrNrrrxZfiltered_tokensZ sub_textsZcurrent_sub_textr2rZ clean_textr1r1r2_decodes4       zPretrainedTokenizer._decode)F)NNNNr r/)FTT)?rfrgrhrr rrTrkrir rrr'rrrrpropertyrr r!r"r#r rr3r1r6rrr@r,rDrFrKrOrLrjrVrYr]r`rrrZDO_NOT_TRUNCATErrr r rrrrrrrr rrrrrr1r1rr2r)s '    S O    +        d      z      1c^r)) metaclasscCrrrrr1r1r2rrcCsht|}|dkr |dks$|dkr|dks$|dkr|dks$|dkr&|dkr&d St|}|d r2d Sd S) z2Checks whether `chars` is a punctuation character.!/:@[`{~TPFrrrr)rrrr1r1r2_is_punctuation%s  rcCs(t|}t|ds|dvrdSdS)z8Check whether CP is the codepoint of a Symbol character.S)i0iKiTFrrr1r1r2 _is_symbol9s rcCs>|dks|dks|dks|dkrdSt|}|dkrdSdS)z; Checks whether `chars` is a whitespace character. rrrrTZsFrrr1r1r2_is_whitespaceCs  rcCs4t|tr|St|tr|ddStdt|)z Converts `text` to Unicode (if it's not already), assuming utf-8 input. Args: text (str|bytes): Text to be converted to unicode. Returns: str: converted text. r^ignorezUnsupported string type: %s)r=rTbytesdecoder<r|)rr1r1r2convert_to_unicodeQs   rcCs|}|sgS|}|S)z Runs basic whitespace cleaning and splitting on a piece of text. Args: text (str): Text to be tokenized. Returns: list(str): Token list. )rrrxr1r1r2whitespace_tokenizeas r)JrrrrSrvrbrrr collectionsr dataclassesrrrtypingrrrr r r r numpyrutilsrZ utils.depsrrZtokenizer_utils_baserrrrrrrrrrrrrrr rIr!Zjinja2r"Zjinja2.exceptionsr#r$Zjinja2.sandboxr%__all__r&rr|r*r'rTrrrrrr(r)rrrrrr1r1r1r2sr  $ <    ND\   <f