o 0j%@sHddlZddlmZddlmZmZmZddlmZGdddeZ dS)N)copyfile)ListOptionalTuple)PretrainedTokenizerc s`eZdZgdZddiZddddddddiZiiiiiidZdZ   d0fd d Ze ddZ ddZ e de e fddZe de e fddZd1ddZddZddZddZd d!Zd"d#Z d2d$e edeefd%d&Zd2d'd(Z d3d)ee d*e ee d+edee ffd,d- Z d2d)ee d*e ee dee fd.d/ZZS)4LlamaTokenizer)Z input_idsZattention_maskZ position_ids vocab_filezsentencepiece.bpe.modelzQhttps://bj.bcebos.com/paddlenlp/models/transformers/llama/sentencepiece.bpe.model)z'__internal_testing__/micro-random-llamaz&__internal_testing__/tiny-random-llamazfacebook/llama-7bzfacebook/llama-13bzfacebook/llama-30bzfacebook/llama-65bleftTFNc  s\|durin||_tjd|||d| ||_||_||_||_|| dd|_ dS)N) bos_token eos_token unk_token from_slowT) sp_model_kwargssuper__init__r add_bos_token add_eos_tokendecode_with_prefix_spaceget_spm_processorpopsp_model) selfrrr rrrrrkwargs __class__rz/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/models/common/tokenizer/llama_tokenizer.pyr2s zLlamaTokenizer.__init__cC |jS)zReturns vocab size)rget_piece_sizerrrr vocab_sizeIs zLlamaTokenizer.vocab_sizecCs0d}|jD] }||jkr|d7}q|j|S)zc Returns the vocabulary size. added_tokens_encoder has to be added in the sp_model r)Zadded_tokens_decoderrr!r#)rZ added_sizeidrrr__len__Ns   zLlamaTokenizer.__len__returncCr N)rZbos_idr"rrr bos_token_idZ zLlamaTokenizer.bos_token_idcCr r()rZeos_idr"rrr eos_token_id^r*zLlamaTokenizer.eos_token_idc Csddl}ddlm}|jdi|j}|r||j|St|jd)}|}|j |}| }d|_ |j ||}||Wd|S1sOwY|S)Nr)sentencepiece_model_pb2rbFr)Z sentencepiecer,ZSentencePieceProcessorrLoadropenreadZ ModelProtoZ FromStringZNormalizerSpecZadd_dummy_prefixnormalizer_specZ MergeFromZSerializeToStringZLoadFromSerializedProto) rrZspmZ model_pb2 tokenizerfrmodelr1rrrrbs$      z LlamaTokenizer.get_spm_processorcs(fddtjD}|j|S)zReturns vocab as a dictcsi|]}||qSr)Zconvert_ids_to_tokens).0ir"rr wsz,LlamaTokenizer.get_vocab..)ranger#updateZadded_tokens_encoder)rZvocabrr"r get_vocabus zLlamaTokenizer.get_vocabcCs|jj|tdS)zReturns a tokenized string.)Zout_type)rencodestr)rtextrrr _tokenize{szLlamaTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)rZ piece_to_id)rtokenrrr_convert_token_to_ids z#LlamaTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)rZ id_to_piece)rindexr?rrr_convert_id_to_tokens z#LlamaTokenizer._convert_id_to_tokencCs|g}d}d}t|D])\}}||jvr,|s|dkr|d7}||j||7}d}g}q ||d}q ||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr T) enumerateZall_special_tokensrdecodeappend)rtokensZcurrent_sub_tokensZ out_stringZprev_is_specialr6r?rrrconvert_tokens_to_strings   z'LlamaTokenizer.convert_tokens_to_stringfilename_prefixcCstj|std|dtj||r|dnd|jd}tj|jtj|kr=tj|jr=t |j||fStj|jset |d}|j }| |Wd|fS1s`wY|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directory-rCrwbN)ospathisdir ValueErrorjoinresource_files_namesabspathrisfilerr/rZserialized_model_protowrite)rZsave_directoryrJZout_vocab_filefiZcontent_spiece_modelrrrsave_vocabularys2        zLlamaTokenizer.save_vocabularycCsB|jr|jg}ng}||}|dur||}|jr||jg}|Sr()rr)rr+)r token_ids_0 token_ids_1Z bos_token_idsoutputrrr build_inputs_with_special_tokenss  z/LlamaTokenizer.build_inputs_with_special_tokensrXrYalready_has_special_tokenscsh|r tj||ddS|durdgdgt|dgSdgdgt|ddgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rXrYr\Nr$r)rget_special_tokens_masklen)rrXrYr\rrrr]s0z&LlamaTokenizer.get_special_tokens_maskcCs<|jg}|durt||dgSt||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)r+r^)rrXrYZeosrrr$create_token_type_ids_from_sequencessz3LlamaTokenizer.create_token_type_ids_from_sequences)r r r TFNF)Tr()NF)__name__ __module__ __qualname__Zmodel_input_namesrRZpretrained_resource_files_mapZpretrained_init_configurationZ padding_siderpropertyr#r&rintr)r+rr:r>r@rBrIr<rrWr[rboolr]r_ __classcell__rrrrrs     !  r) rMshutilrtypingrrrZ9paddlex.inference.models.common.tokenizer.tokenizer_utilsrrrrrrs