o 0jb@spddlZddlZddlmZmZmZmZmZmZm Z gdZ Gddde Z Gddde Z Gd d d eZdS) N)PretrainedTokenizer _is_control_is_punctuation _is_symbol_is_whitespaceconvert_to_unicodewhitespace_tokenize)BasicTokenizer BertTokenizerWordpieceTokenizerc@sVeZdZdZ    dddZdddZdd Zdd d Zd d ZddZ ddZ dS)r a Runs basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (bool): Whether to lowercase the input when tokenizing. Defaults to `True`. never_split (Iterable): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (bool): Whether to tokenize Chinese characters. strip_accents: (bool): Whether to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). TNcCs,|durg}||_t||_||_||_dS)zConstructs a BasicTokenizer.N) do_lower_caseset never_splittokenize_chinese_chars strip_accents)selfr rrrry/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/models/common/tokenizer/bert_tokenizer.py__init__5s   zBasicTokenizer.__init__cCst|}|r|jt|n|j}||}|jr||}t|}g}|D])}||vrF|jr>| }|j dur=| |}n|j rF| |}| | ||q&td|}|S)a9 Tokenizes a piece of text using basic tokenizer. Args: text (str): A piece of text. never_split (List[str]): List of token not to split. Returns: list(str): A list of tokens. Examples: .. code-block:: from paddlenlp.transformers import BasicTokenizer basictokenizer = BasicTokenizer() tokens = basictokenizer.tokenize('He was a puppeteer') ''' ['he', 'was', 'a', 'puppeteer'] ''' F )rrunionr _clean_textr_tokenize_chinese_charsr r lowerr_run_strip_accentsextend_run_split_on_puncjoin)rtextrZ orig_tokens split_tokenstoken output_tokensrrrtokenizeDs,     zBasicTokenizer.tokenizecCsBtd|}g}|D]}t|}|dkrq ||q d|S)z6 Strips accents from a piece of text. ZNFDZMn) unicodedata normalizecategoryappendr)rroutputcharcatrrrrrs    z!BasicTokenizer._run_strip_accentscCs|dur ||vr |gSt|}d}d}g}|t|krJ||}t|s't|r0||gd}n|r7|gd}|d||d7}|t|ksdd|DS) z8 Splits punctuation on a piece of text. NrTFrcSsg|]}d|qS)r$)r).0xrrr sz5BasicTokenizer._run_split_on_punc..)listlenrrr()rrrcharsiZstart_new_wordr)r*rrrrs$     z!BasicTokenizer._run_split_on_punccCsTg}|D] }t|}||r|d|||dq||qd|S)z; Adds whitespace around any CJK character. rr$)ord_is_chinese_charr(rrrr)r*cprrrrs      z&BasicTokenizer._tokenize_chinese_charscCs|dkr|dks@|dkr|dks@|dkr|dks@|dkr |dks@|d kr(|d ks@|d kr0|d ks@|d kr8|dks@|dkrB|dkrBdSdS)zH Checks whether CP is the codepoint of a CJK character. iNii4iMiiߦii?i@ii iiiiiTFr)rr7rrrr5s zBasicTokenizer._is_chinese_charcCsXg}|D]"}t|}|dks|dkst|rqt|r!|dq||qd|S)zT Performs invalid character removal and whitespace cleanup on text. rirr$)r4rrr(rr6rrrrs   zBasicTokenizer._clean_text)TNTNN) __name__ __module__ __qualname____doc__rr#rrrr5rrrrrr #s .  r c@s"eZdZdZdddZddZdS) r a Runs WordPiece tokenization. Args: vocab (Vocab|dict): Vocab of the word piece tokenizer. unk_token (str): A specific token to replace all unknown tokens. max_input_chars_per_word (int): If a word's length is more than max_input_chars_per_word, it will be dealt as unknown word. Defaults to 100. dcCs||_||_||_dSr8)vocab unk_tokenmax_input_chars_per_word)rr>r?r@rrrrs zWordpieceTokenizer.__init__c Csg}t|D]m}t|}t||jkr||jqd}d}g}|t|kret|}d} ||krQd|||} |dkrAd| } | |jvrI| } n|d8}||ks0| durXd}n || |}|t|ks&|rn||jq||q|S)a Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. Args: text: A single token or whitespace separated tokens. This should have already been passed through `BasicTokenizer`. Returns: list (str): A list of wordpiece tokens. Examples: .. code-block:: from paddlenlp.transformers import BertTokenizer, WordpieceTokenizer berttokenizer = BertTokenizer.from_pretrained('bert-base-uncased') vocab = berttokenizer.vocab unk_token = berttokenizer.unk_token wordpiecetokenizer = WordpieceTokenizer(vocab,unk_token) inputs = wordpiecetokenizer.tokenize("unaffable") print(inputs) ''' ["un", "##aff", "##able"] ''' FrNr$z##rT) r r0r1r@r(r?rr>r) rrr"r!r2Zis_badstartZ sub_tokensendZ cur_substrsubstrrrrr#s>       zWordpieceTokenizer.tokenizeN)r=)r9r:r;r<rr#rrrrr s  r c@s>eZdZdZddiZdidddddd d d d d ddddddddddddddddddddddddd diiZidd!d"idd!d"idd!d#id d!d#id d!d"idd!d#idd!d#idd!d#idd!d#idd!d#idd!d#idd!d"idd!d"idd!d"idd!d"idd!d"idd!d"id d!d"iiZidd$dd$dd$d d$d d$dd$dd$dd$dd$dd$dd$dd$dd$dd$dd$dd$dd$d d$iZd%Z " " & ' ( ) * + " &dBd,d-Z e d.d/Z d0d1Z d2d3Z d4d5ZdCd6d7ZdDd8d9Z &dDd:d;ZdDdd?Zd@dAZd&S)Fr a2 Constructs a BERT tokenizer. It uses a basic tokenizer to do punctuation splitting, lower casing and so on, and follows a WordPiece tokenizer to tokenize as subwords. Args: vocab_file (str): The vocabulary file path (ends with '.txt') required to instantiate a `WordpieceTokenizer`. do_lower_case (bool, optional): Whether to lowercase the input when tokenizing. Defaults to `True`. do_basic_tokenize (bool, optional): Whether to use a basic tokenizer before a WordPiece tokenizer. Defaults to `True`. never_split (Iterable, optional): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True`. Defaults to `None`. unk_token (str, optional): A special token representing the *unknown (out-of-vocabulary)* token. An unknown token is set to be `unk_token` inorder to be converted to an ID. Defaults to "[UNK]". sep_token (str, optional): A special token separating two different sentences in the same input. Defaults to "[SEP]". pad_token (str, optional): A special token used to make arrays of tokens the same size for batching purposes. Defaults to "[PAD]". cls_token (str, optional): A special token used for sequence classification. It is the last token of the sequence when built with special tokens. Defaults to "[CLS]". mask_token (str, optional): A special token representing a masked token. This is the token used in the masked language modeling task which the model tries to predict the original unmasked ones. Defaults to "[MASK]". tokenize_chinese_chars (bool, optional): Whether to tokenize Chinese characters. Defaults to `True`. strip_accents: (bool, optional): Whether to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). Defaults to `None`. Examples: .. code-block:: from paddlenlp.transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') inputs = tokenizer('He was a puppeteer') print(inputs) ''' {'input_ids': [101, 2002, 2001, 1037, 13997, 11510, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0]} ''' vocab_filez vocab.txtzbert-base-uncasedzIhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-base-uncased-vocab.txtzbert-large-uncasedzJhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-large-uncased-vocab.txtzbert-base-casedzGhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-base-cased-vocab.txtzbert-large-casedzHhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-large-cased-vocab.txtzbert-base-multilingual-uncasedzVhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-base-multilingual-uncased-vocab.txtzbert-base-multilingual-casedzThttps://bj.bcebos.com/paddle-hapi/models/bert/bert-base-multilingual-cased-vocab.txtzbert-base-chinesezIhttps://bj.bcebos.com/paddle-hapi/models/bert/bert-base-chinese-vocab.txtzbert-wwm-chinesezRhttp://bj.bcebos.com/paddlenlp/models/transformers/bert/bert-wwm-chinese-vocab.txtzbert-wwm-ext-chinesezVhttp://bj.bcebos.com/paddlenlp/models/transformers/bert/bert-wwm-ext-chinese-vocab.txtzmacbert-large-chinesezmacbert-base-chinesezsimbert-base-chinesezEhttps://bj.bcebos.com/paddlenlp/models/transformers/simbert/vocab.txtzuer/chinese-roberta-basezQhttps://bj.bcebos.com/paddlenlp/models/transformers/uer/chinese_roberta_vocab.txtzuer/chinese-roberta-mediumzuer/chinese-roberta-6l-768hzuer/chinese-roberta-smallzuer/chinese-roberta-minizuer/chinese-roberta-tinyr TFirightN[UNK][SEP][PAD][CLS][MASK]c Ks`tj|s td|||_|j||d|_||_|r&t ||| | d|_ t |j|d|_ dS)NzCan't find a vocabulary file at path '{}'. To load the vocabulary from a pretrained model please use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`)r?)r rrr)r>r?) ospathisfile ValueErrorformatr Zload_vocabularyr>do_basic_tokenizer basic_tokenizerr wordpiece_tokenizer) rrDr rPrr?Z sep_tokenZ pad_tokenZ cls_tokenZ mask_tokenrrkwargsrrrrs(  zBertTokenizer.__init__cCs t|jS)zk Return the size of vocabulary. Returns: int: The size of vocabulary. )r1r>rrrr vocab_sizes zBertTokenizer.vocab_sizecCst|jjfi|jSr8)dictr>Z token_to_idxZadded_tokens_encoderrTrrr get_vocabszBertTokenizer.get_vocabcCs^g}|jr'|jj||jdD]}||jjvr||q||j|7}q|S|j|}|S)z End-to-end tokenization for BERT models. Args: text (str): The text to be tokenized. Returns: list: A list of string representing converted tokens. )r)rPrQr#Zall_special_tokensrr(rR)rrr r!rrr _tokenizes    zBertTokenizer._tokenizecCsd|dd}|S)ac Converts a sequence of tokens (list of string) to a single string. Since the usage of WordPiece introducing `##` to concat subwords, also removes `##` when converting. Args: tokens (list): A list of string representing tokens to be converted. Returns: str: Converted string from tokens. Examples: .. code-block:: from paddlenlp.transformers import BertTokenizer berttokenizer = BertTokenizer.from_pretrained('bert-base-uncased') tokens = berttokenizer.tokenize('He was a puppeteer') ''' ['he', 'was', 'a', 'puppet', '##eer'] ''' strings = tokenizer.convert_tokens_to_string(tokens) ''' he was a puppeteer ''' rz ##r$)rreplacestrip)rtokensZ out_stringrrrconvert_tokens_to_stringsz&BertTokenizer.convert_tokens_to_stringcCs$g}g}t|||r|SdS)ad Returns the number of added tokens when encoding a sequence with special tokens. Args: pair(bool): Whether the input is a sequence pair or a single sequence. Defaults to `False` and the input is a single sequence. Returns: int: Number of tokens added to sequences. N)r1 build_inputs_with_special_tokens)rpair token_ids_0 token_ids_1rrrnum_special_tokens_to_add s z'BertTokenizer.num_special_tokens_to_addcCs@|dur|jg||jgS|jg}|jg}|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: ``[CLS] X [SEP]`` - pair of sequences: ``[CLS] A [SEP] B [SEP]`` Args: token_ids_0 (List[int]): List of IDs to which the special tokens will be added. token_ids_1 (List[int], optional): Optional second list of IDs for sequence pairs. Defaults to None. Returns: List[int]: List of input_id with the appropriate special tokens. N) cls_token_id sep_token_id)rr_r`_cls_seprrrr]s z.BertTokenizer.build_inputs_with_special_tokenscCs2|dur dg|dgSdg|dg|dgS)a Build offset map from a pair of offset map by concatenating and adding offsets of special tokens. A BERT offset_mapping has the following format: - single sequence: ``(0,0) X (0,0)`` - pair of sequences: ``(0,0) A (0,0) B (0,0)`` Args: offset_mapping_ids_0 (List[tuple]): List of wordpiece offsets to which the special tokens will be added. offset_mapping_ids_1 (List[tuple], optional): Optional second list of wordpiece offsets for offset mapping pairs. Defaults to None. Returns: List[tuple]: A list of wordpiece offsets with the appropriate offsets of special tokens. N)rrr)rZoffset_mapping_0Zoffset_mapping_1rrr(build_offset_mapping_with_special_tokens8sz6BertTokenizer.build_offset_mapping_with_special_tokenscCsV|jg}|jg}|durt|||dgSt|||dgt||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: :: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (List[int]): A list of `inputs_ids` for the first sequence. token_ids_1 (List[int], optional): Optional second list of IDs for sequence pairs. Defaults to None. Returns: List[int]: List of token_type_id according to the given sequence(s). Nrr)rcrbr1)rr_r`rerdrrr$create_token_type_ids_from_sequencesQs (z2BertTokenizer.create_token_type_ids_from_sequencescsz|r|dur tdttfdd|S|dur0dgdgt|dgdgt|dgSdgdgt|dgS)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer ``encode`` methods. Args: token_ids_0 (List[int]): A list of `inputs_ids` for the first sequence. token_ids_1 (List[int], optional): Optional second list of IDs for sequence pairs. Defaults to None. already_has_special_tokens (bool, optional): Whether or not the token list is already formatted with special tokens for the model. Defaults to None. Returns: List[int]: The list of integers either be 0 or 1: 1 for a special token, 0 for a sequence token. NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.cs|jvrdSdS)Nrr)Zall_special_ids)r.rTrrsz7BertTokenizer.get_special_tokens_mask..rr)rNr0mapr1)rr_r`Zalready_has_special_tokensrrTrget_special_tokens_maskls.z%BertTokenizer.get_special_tokens_maskcCs|jj||jS)z=Converts an index (integer) in a token (str) using the vocab.)r>Z _idx_to_tokengetr?)rindexrrr_convert_id_to_tokensz"BertTokenizer._convert_id_to_token) TTNrFrGrHrIrJTN)Fr8)NF)r9r:r;r<Zresource_files_namesZpretrained_resource_files_mapZpretrained_init_configurationZmax_model_input_sizesZ padding_siderpropertyrUrWrXr\rar]rfrgrjrmrrrrr %s9                &      !r )rKr%Ztokenizer_utilsrrrrrrr __all__objectr r r rrrrs$ /T