o *j@sdZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZddlmZeZ dZ! d+ddZ"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(Gdddej#Z)Gd d!d!ej#Z*Gd"d#d#ej#Z+Gd$d%d%ej#Z,Gd&d'd'ee Z-ej.ej/e j0d(Gd)d*d*e-Z1dS),zPyTorch XLM-RoBERTa model.N)version)nn)ACT2FN)PreTrainedModel)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)Tasks) get_logger)parse_labels_in_order)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)XLMRobertaConfigrcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rdim)neinttorchZcumsumZtype_aslong) input_ids padding_idxpast_key_values_lengthmaskZincremental_indicesrk/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/xlm_roberta/backbone.py"create_position_ids_from_input_ids-s  rcs<eZdZdZfddZ     d ddZdd ZZS) XLMRobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|dt|jd|jdtj|jtjdd d |j|_tj|j|j|jd|_ dS) N)repsposition_embedding_typeabsolute position_ids)rtoken_type_idsdtypeF) persistent)super__init__r EmbeddingZ vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr#Zregister_bufferrarangeexpandzerosr%sizerrselfconfig __class__rrr,GsF  zXLMRobertaEmbeddings.__init__Nrc Cs|dur|durt||j|}n||}|dur|}n|dd}|d}|durTt|drI|jddd|f}||d|} | }n tj|tj |j j d}|dur]| |}| |} || } |jdkrt||} | | 7} || } || } | S)Nr&rr'rr)devicer$)rr&create_position_ids_from_inputs_embedsr<hasattrr'r:rr;rr%rCr/r2r#r1r3r7) r>rr'r% inputs_embedsr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr2 embeddingsr1rrrforwardjsD        zXLMRobertaEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr&rrBr)r<rr9rrrCZ unsqueezer:)r>rFrGZsequence_lengthr%rrrrDs  z;XLMRobertaEmbeddings.create_position_ids_from_inputs_embeds)NNNNr)__name__ __module__ __qualname____doc__r,rLrD __classcell__rrr@rr As $ 2r csReZdZd fdd ZdejdejfddZ      d d ejfd d ZZS)XLMRobertaSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r#r$ relative_keyrelative_key_queryr)r+r,r.num_attention_headsrE ValueErrorrattention_head_size all_head_sizerLinearquerykeyvaluer5Zattention_probs_dropout_probr7r8r#r0r-distance_embedding is_decoderr>r?r#r@rrr,s8    z XLMRobertaSelfAttention.__init__xreturncCs6|dd|j|jf}||}|ddddS)Nr&rrVr)r<rWrYviewpermute)r>rbZ new_x_shaperrrtranspose_for_scoress  z,XLMRobertaSelfAttention.transpose_for_scoresF hidden_statescCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)NrrrVrr&rTrUrBr(zbhld,lrd->bhlrzbhrd,lrd->bhlrrd) r\rgr]r^rcatr`matmulZ transposer#shapeZtensorrrCrer9r_r0tor)ZeinsummathsqrtrYrZ functionalZsoftmaxr7rf contiguousr<rZ)r>rhattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsrrrrLs               zXLMRobertaSelfAttention.forwardNNNNNNF) rMrNrOr,rTensorrgrLrQrrr@rrRs rRc8eZdZfddZdejdejdejfddZZS)XLMRobertaSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr!) r+r,rr[r.denser3r4r5r6r7r=r@rrr,O zXLMRobertaSelfOutput.__init__rh input_tensorrccC&||}||}|||}|Sryrr7r3r>rhrrrrrLV  zXLMRobertaSelfOutput.forwardrMrNrOr,rr{rLrQrrr@rr}M r}csDeZdZd fdd ZddZ      d dejfdd ZZS) XLMRobertaAttentionNcs.tt||d|_t||_t|_dS)Nr#)r+r,rRr>r}outputset pruned_headsrar@rrr,as   zXLMRobertaAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrr>rWrYrrr\r]r^rrrZunion)r>headsindexrrr prune_headshs   zXLMRobertaAttention.prune_headsFrhc Cs<||||||||}||d|} | f|dd} | S)Nrr)r>r) r>rhrqrrrsrtrurvZ self_outputsattention_outputrxrrrrL{s   zXLMRobertaAttention.forwardryrz) rMrNrOr,rrr{rLrQrrr@rr_src2eZdZfddZdejdejfddZZS)XLMRobertaIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSry) r+r,rr[r.intermediate_sizer isinstanceZ hidden_actstrrintermediate_act_fnr=r@rrr,s   zXLMRobertaIntermediate.__init__rhrccCs||}||}|Sry)rr)r>rhrrrrLs  zXLMRobertaIntermediate.forwardrrrr@rrs rcr|)XLMRobertaOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr~) r+r,rr[rr.rr3r4r5r6r7r=r@rrr,rzXLMRobertaOutput.__init__rhrrccCrryrrrrrrLrzXLMRobertaOutput.forwardrrrr@rrrrcsBeZdZfddZ      d dejfddZdd ZZS) XLMRobertaLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedr$r)r+r,chunk_size_feed_forward seq_len_dimr attentionr`add_cross_attentionrXcrossattentionr intermediaterrr=r@rrr,s   zXLMRobertaLayer.__init__NFrhc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) NrVrvrurrr&rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`ri) rr`rErXrrfeed_forward_chunkrr)r>rhrqrrrsrtrurvZself_attn_past_key_valueZself_attention_outputsrrxZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputrrrrLsp        zXLMRobertaLayer.forwardcCs||}|||}|Sry)rr)r>rZintermediate_outputrrrrrs  z"XLMRobertaLayer.feed_forward_chunkrz) rMrNrOr,rr{rLrrQrrr@rrs  Ercs@eZdZfddZ         d dejfddZZS) XLMRobertaEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)r).0_r?rr sz.XLMRobertaEncoder.__init__..F) r+r,r?rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr=r@rrr,s  zXLMRobertaEncoder.__init__NFTrhc  sb| rdnd} r dnd} r|jjrdnd} |rdnd}t|jD]o\}}| r,| |f} |dur4||nd}|dur>||nd|jre|jre|rOtdd}fdd}tj j |||||||}n ||||||}|d}|r|||df7}r| |df} |jjr| |d f} q!| r| |f} | st d d ||| | | fDSt ||| | | d S) NrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fcsfdd}|S)Ncsg|RSryr)Zinputs)modulervrurrcustom_forwardEs zPXLMRobertaEncoder.forward..create_custom_forward..custom_forwardr)rrr)rrcreate_custom_forwardCsz8XLMRobertaEncoder.forward..create_custom_forwardrr&rrVcss|] }|dur|VqdSryr)rvrrr ls z,XLMRobertaEncoder.forward..)last_hidden_statepast_key_valuesrh attentionscross_attentions) r?r enumeraterrZtrainingloggerwarningrutils checkpointtupler )r>rhrqrrrsrtrrwrvoutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrZ layer_outputsrrrrL s        zXLMRobertaEncoder.forward) NNNNNNFFTrrrr@rrs  rcr)XLMRobertaPoolercs*tt|j|j|_t|_dSry)r+r,rr[r.rZTanh activationr=r@rrr,s zXLMRobertaPooler.__init__rhrccCs(|dddf}||}||}|S)Nr)rr)r>rhZfirst_token_tensor pooled_outputrrrrLs  zXLMRobertaPooler.forwardrrrr@rr}s rcsNeZdZdZeZdZdZgZddZ d ddZ d d Z e fd d Z ZS)XLMRobertaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ZrobertaTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanZstdNg?)rrr[weightdataZnormal_r?Zinitializer_rangeZbiasZzero_r-rr3Zfill_)r>rrrr _init_weightss$      z'XLMRobertaPreTrainedModel._init_weightsFcCst|tr ||_dSdSry)rrr)r>rr^rrr_set_gradient_checkpointings  z5XLMRobertaPreTrainedModel._set_gradient_checkpointingcs:|jsfdd|jD|_fdd|jD|_dSdS)z!Remove some keys from ignore listcg|]}|vr|qSrrrkdel_keys_to_ignorerrr zCXLMRobertaPreTrainedModel.update_keys_to_ignore..crrrrrrrrrN)Ztie_word_embeddingsZ_keys_to_ignore_on_save_keys_to_ignore_on_load_missing)r>r?rrrrupdate_keys_to_ignores   z/XLMRobertaPreTrainedModel.update_keys_to_ignorec sn|dd}|dd}t||fi|}|dur%tdi|}||}n tt|jdd|i|}||_|S)ahInstantiate the model. Args: kwargs: Input args. model_dir: The model dir used to load the checkpoint and the label information. num_labels: An optional arg to tell the model how many classes to initialize. Method will call utils.parse_label_mapping if num_labels not supplied. If num_labels is not found, the model will use the default setting (2 classes). Returns: The loaded model, which is initialized by transformers.PreTrainedModel.from_pretrained model_dirNcfgZpretrained_model_name_or_pathr)popr rr+rZfrom_pretrainedr)clskwargsrrZ model_argsr?modelr@rr _instantiates   z&XLMRobertaPreTrainedModel._instantiate)F)rMrNrOrPrZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrrr classmethodrrQrrr@rrs  r)Z group_key module_namecsdeZdZdZdgZdfdd ZddZdd Zd d Z dd dZ Z S)XLMRobertaModela* The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://arxiv.org/abs/1706.03762 r%TcsDt|||_t||_t||_|rt|nd|_| dSry) r+r,r?r rKrencoderrpoolerZ post_init)r>r?Zadd_pooling_layerr@rrr,s    zXLMRobertaModel.__init__cCs|jjSryrKr/)r>rrrget_input_embeddingssz$XLMRobertaModel.get_input_embeddingscCs ||j_dSryr)r>r^rrrset_input_embeddingss z$XLMRobertaModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r>Zheads_to_prunerrrrr _prune_headsszXLMRobertaModel._prune_headsNc CsD| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durD|}n|durQ|dd}ntd|\}}|dur`|jn|j}| durp| ddj dnd}|durt j |||f|d}|durt |j d r|j jddd|f}|||}|}n t j|t j|d }|||}|jjr|dur|\}}}||f}|durt j ||d}||}nd}|||jj}|j |||||d }|j|||||| | | | | d }|d}|jdur||nd}| s||f|d dSt|||j|j|j|jdS)a+ Args: input_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `((batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `((batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `((batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer&z5You have to specify either input_ids or inputs_embedsrrV)rCr'rB)rr%r'rFr) rqrrrsrtrrwrvrrr)rZ pooler_outputrrhrr)r?rvrZuse_return_dictr`rwrXr<rCrlrZonesrErKr'r:r;rZget_extended_attention_maskZinvert_attention_maskZ get_head_maskrrrr rrhrr)r>rrqr'r%rrrFrsrtrrwrvrrrGZ batch_sizerHrCrrIrJZextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputsZsequence_outputrrrrrL s[      zXLMRobertaModel.forward)T) NNNNNNNNNNNNN) rMrNrOrPrr,rrrrLrQrrr@rrs*  r)r)2rPrnrZtorch.utils.checkpoint packagingrrZtransformers.activationsrZtransformers.modeling_utilsrZmodelscope.metainforZmodelscope.modelsrrZmodelscope.models.builderr Zmodelscope.outputsr Zmodelscope.utils.constantr Zmodelscope.utils.loggerr Zmodelscope.utils.nlp.utilsr Zmodelscope.utils.torch_utilsrrr configurationrrZ_CONFIG_FOR_DOCrModuler rRr}rrrrrrrZregister_moduleZbackboneZ xlm_robertarrrrrsH            p6_fK