o 0j)b@sDddlZddlZddlZddlZddlZddlZGdddeZdS)Nc@seZdZdZ        d.ddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ e ddZe ddZd/ddZeddZe    d0ddZe       d1d d!Ze    d0d"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-ZdS)2Vocaba The class used to convert between tokens and ids. It also includes some store/load functions. Args: counter (collections.Counter, optional): A Counter instance describes the tokens and their frequencies. Its keys will be indexed according to the order of frequency sorting to construct mapping relationship. If None, `token_to_idx` must be provided as the mapping relationship. Default: None. max_size (int, optional): Max size of vocab, not including special tokens. Default: None. min_freq (int, optional): Ignore tokens whose frequencies are less than `min_freq`. Default: 1. token_to_idx (dict, optional): A dict specifies the mapping relationship between tokens and indices to be used. If provided, adjust the tokens and indices mapping according to it. If None, counter must be provided. Default: None. unk_token (str, optional): Special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): Special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): Special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): Special token for eos token. If no need, it lso could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Nc  sdfd|fd|fd|ff} | D]\} } | | | <qg} | }t|}|D]}|ds3td|| |}|durD|| vrD| |q%|dur|sOJd| D] }||vs^Jd|qQ|_d d |D_rjt fd d _j |n7d d t | D_t _j ddjD || |||r|rfdd j_| _| D] \}}|drtdt|rtd||t||qdS)N unk_token pad_token bos_token eos_token_tokenzc{} is invalid. Only keyword arguments that end in '_token' are supported to declare special tokens.z4token_to_idx should not be None when counter is Nonez{} is not in token_to_idxcSsi|]\}}||qSr ).0tokenidxr r p/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/models/common/tokenizer/vocab.py gsz"Vocab.__init__..csSNr r ) unk_indexr r jsz Vocab.__init__..cSsi|]\}}||qSr r )r r special_tokenr r r rms css|] \}}||fVqdSrr )r r r r r r qs  z!Vocab.__init__..cs jSr _token_to_idxr )selfrr r rxs_zIt is not allowed to use identifiers starting with underscore. In Python identifier names beginning with underscore are internal.zJvocab.{} already exists. Please choose a different identifier for token {})keyssortedendswith ValueErrorformatappendritems _idx_to_token collections defaultdictupdate enumerate_index_counter_keys+_sort_index_according_to_user_specificationdefault_factory_identifiers_to_tokens startswithhasattrsetattr)rcountermax_sizemin_freq token_to_idxrrrrkwargsZcombsnamevaluespecial_tokensZ special_iterZspecial_token_namer identifierr r )rrrr __init__9sx             zVocab.__init__cCst|ddd}|jddddt|}|durdn|t|}|D]1\}}||ks4t|j|kr7dS||vrV||jtt|jdgd<t|j|j |<q%dS) NcS|dS)Nrr xr r r rz+Vocab._index_counter_keys..)keycSr5)Nrr r6r r r rr8T)r9reverser) rrsortsetlenrmaxlistrr)rr+r2r,r-Z token_freqsr freqr r r r$s  zVocab._index_counter_keyscCst||jstdtt|t|kr tdt|dks3t|t|jkr=td t|j| D]"\}}|j|}|j |}||j|<||j|<||j |<||j |<qAdS)Nz`User-specified token_to_idx mapping can only contain tokens that will be part of the vocabulary.z3User-specified indices must not contain duplicates.rzUser-specified indices must not be < 0 or >= the number of tokens that will be in the vocabulary. The current vocab contains {}tokens.) r=rissubsetr.rr>valuesminr?rr idx_to_token)rr.r Znew_idxZold_idxZ ousted_tokenr r r r%s*       z1Vocab._sort_index_according_to_user_specificationc Csd}t|tttjfs|g}d}t|ttfrt|}t|tjr3t|jdkr3td t|jg}|D]+}t|t tj fsJt dt |}z ||j|Wq7tybtd |w|ri|dS|S)uc Maps the input indices to token list. Args: indices (int|list[int]|tuple[int]|numpy.ndarray): The input indice(s) for mapping. Must be an `int` or 1D `list[int]`|`tuple[int]`|`numpy.ndarray`. Returns: str|list[str]: Obtained token(s). If `indices` is an integer, it will return a str. If `indices` is a list/tuple of integers, it will return a list of str. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') tokens = vocab.to_tokens([0, 1, 2, 3]) print(tokens) # ['[PAD]', '[UNK]', '一斤三', '意面屋'] FTrzEToken indices is invalid. Expected 1D array, but received {}D array. zfThe type of `to_tokens()`'s input `indices` is not `int` which will be forcibly transferred to `int`. z4Token index {} in the provided `indices` is invalid.r) isinstancer@tuplenpZndarrayZasarrayr>shaperrintintegerwarningswarnrrKeyError)rindicesZ to_reducetokensr r r r to_tokenss6  zVocab.to_tokenscC||S)u Maps the input tokens into indices. Args: tokens (str|list[str]|tuple[str], optional): The input token(s) for mapping. Returns: int|list[int]: Obationed indice(s). If `tokens` is a str, it will return an integer. If `tokens` is a list/tuple of str, it will return a list of integers. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') tokens = vocab.to_indices(['[PAD]', '[UNK]', '一斤三', '意面屋']) print(tokens) # [0, 1, 2, 3] r rrPr r r to_indicesszVocab.to_indicescs@t|ttfs|jvrj|SjjSfdd|DS)Ncs,g|]}|jvrj|njjqSr )rr)r r rr r s  z%Vocab.__getitem__..)rFr@rGrrrSr rUr __getitem__s   zVocab.__getitem__cCs t|jSr)r>rrUr r r __len__' z Vocab.__len__cCs ||jvSrr)rr r r r __contains__*rYzVocab.__contains__cCrR)z Maps the input tokens into indices. Its function is the same as the :meth:`to_indices` method. See detail at `to_indices`. r rSr r r __call__-szVocab.__call__cC|jSr)rrUr r r rE6zVocab.idx_to_tokencCr\rrrUr r r r.;r]zVocab.token_to_idxcCsi}t|j|d<t|j|d<|j|d<|j|d<t|}|r?tj|ddd}| |Wd|S1s:wY|S) a? Summarizes some information of vocab as JSON string. If path is gaven, the JSON string will be saved into files. The JSON string and the saved file all can be used to reconstruct the :class:`Vocab` by calling :meth:`from_json` method. Args: path (str, optional): The path to save JSON string. If None, the JSON will not be saved. Default: None. Returns: str: The JSON string including information of vocab. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') json_str = vocab.to_json(path='./vocab.json') rEr.ridentifiers_to_tokenswutf-8encodingN) dictrEr.rr'jsondumpsioopenwrite)rpath vocab_dictjson_strfr r r to_json@s     z Vocab.to_jsoncCstj|r$tj|ddd }t|}Wdn1swYnt|}|d}|d}|dt }d|vrA|d=|d d||d|}|S) ap Loads :class:`Vocab` from JSON string or JSON file, which is gotten by calling :meth:`to_json` method. Args: json_str (str): JSON string or file path of JSON string. Returns: Vocab: An instance of :class:`Vocab` generated from information contained in JSON string. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') json_str = vocab.to_json(path='./vocab.json') vocab1 = Vocab.from_json(json_str) vocab2 = Vocab.from_json('./vocab.json') print(len(vocab), len(vocab1), len(vocab2)) # 1256608 1256608 1256608 rr`raNr.rr^)r+r.rr ) osriisfilerfrgrdloadloadsgetrc)clsrkrlrjr.rr^vocabr r r from_jsongs&     zVocab.from_jsonc Ks |dd|||||d|}|S)a  Builds the :class:`Vocab` from a dict. Args: token_to_idx (dict): A dict describes the mapping relationship between tokens and indices. unk_token (str, optional): The special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): The special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): The special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): The special token for eos token. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from the given dict and special tokens. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') vocab1 = Vocab.from_dict(vocab.token_to_idx) print(len(vocab), len(vocab.token_to_idx), len(vocab1)) # 1256608 1256608 1256608 N)r+r.rrrrr r )rtr.rrrrr/rur r r from_dicts2 zVocab.from_dictc Ks@t} |D]} | | qt| f|||||||d|} | S)a Builds the :class:`Vocab` according to given iterator and other information. Firstly, iterate over the `iterator` to construct a :class:`collections.Counter` and used to init the as :class:`Vocab`. Args: iterator (collections.Iterable): Iterator of tokens. Each element should be a list of tokens if wordlevel vocab is needed. max_size (int, optional): The max size of vocab, not including special tokens. Default: None. min_freq (int, optional): Ignore tokens whose frequencies are less than `min_freq`. Default: 1. token_to_idx (dict, optional): A dict specifies the mapping relationship between tokens and indices to be used. If provided, adjust the tokens and indices mapping according to it. If None, counter must be provided. Default: None. unk_token (str, optional): The special token for unknown token ''. If no need, it also could be None. Default: None. pad_token (str, optional): The special token for padding token ''. If no need, it also could be None. Default: None. bos_token (str, optional): The special token for bos token ''. If no need, it also could be None. Default: None. eos_token (str, optional): The special token for eos token ''. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from given iterator and other informations. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') vocab1 = Vocab.build_vocab([list(vocab.token_to_idx.keys())]) print(len(vocab), len(vocab1)) # 1256608 1256608 )r,r-r.rrrr)r Counterr"r) iteratorr,r-r.rrrrr/r+rPrur r r build_vocabs">   zVocab.build_vocabc Ks|i}tj|ddd}t|D]\}} | d} t||| <qWdn1s)wYtj|f||||d|} | S)a Builds the :class:`Vocab` from a file reserving all tokens by calling :meth:`Vocab.from_dict` method. The file contains a token per line, and the line index would be the index of corresponding token. Args: filepath (str): the path of file to construct vocabulary. unk_token (str, optional): special token for unknown token. If no need, it also could be None. Default: None. pad_token (str, optional): special token for padding token. If no need, it also could be None. Default: None. bos_token (str, optional): special token for bos token. If no need, it also could be None. Default: None. eos_token (str, optional): special token for eos token. If no need, it also could be None. Default: None. kwargs (dict): Keyword arguments ending with `_token`. It can be used to specify further special tokens that will be exposed as attribute of the vocabulary and associated with an index. Returns: Vocab: An instance of :class:`Vocab` generated from the given file. Example: .. code-block:: python from paddlenlp.data import Vocab # The vocab file. The sample file can be downloaded firstly. # wget https://bj.bcebos.com/paddlenlp/data/senta_word_dict.txt vocab_file_path = './senta_word_dict.txt' # Initialize the Vocab vocab = Vocab.load_vocabulary( vocab_file_path, unk_token='[UNK]', pad_token='[PAD]') print(len(vocab)) # 1256608 rnr`ra N)rrrr)rfrgr#rstriprJrrw) filepathrrrrr/r.rlindexliner rur r r load_vocabulary$s$/ zVocab.load_vocabularycCsXt|d}tt|jD] }||j|dq WddS1s%wYdS)z Save the :class:`Vocab` to a specific file. Can be reloaded by calling `load_vocabulary`. Args: filepath (str): the path of file to save vocabulary. r_r{N)rgranger>rrh)rr}rlr r r r save_vocabularybs "zVocab.save_vocabularycC|jdur |j|jS|jSr)rrrUr r r get_unk_token_idm  zVocab.get_unk_token_idcCrr)rrrUr r r get_bos_token_idtrzVocab.get_bos_token_idcCrr)rrrUr r r get_eos_token_id{rzVocab.get_eos_token_idcCrr)rrrUr r r get_pad_token_idrzVocab.get_pad_token_id)NNrNNNNNr)NNNN)NrNNNNN)__name__ __module__ __qualname____doc__r4r$r%rQrTrWrXrZr[propertyrEr.rm classmethodrvrw staticmethodrzrrrrrrr r r r rsh" U<   ' 1 < M =  r) r rfrdrorLnumpyrHobjectrr r r r s