o *j@sdZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZddlmZeZdZGdddej Z!Gdddej Z"Gdddej Z#Gdddej Z$Gdddej Z%Gdddej Z&Gdddej Z'Gdd d ej Z(Gd!d"d"ej Z)Gd#d$d$e eZ*ej+ej,e j-d%Gd&d'd'e*Z.dS)(z PyTorch MegatronBERT model.N)nn)ACT2FN)PreTrainedModel)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)Tasks) get_logger)parse_labels_in_order)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)MegatronBertConfigrcs<eZdZdZfddZ     d dejfddZZS) MegatronBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ t |j |_|dt|jdt|dd|_dS)N)Z padding_idx position_ids)rposition_embedding_typeabsolute)super__init__r EmbeddingZ vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutZregister_buffertorcharangeexpandgetattrrselfconfig __class__m/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/megatron_bert/backbone.pyr.s*  zMegatronBertEmbeddings.__init__Nrreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|dur5tj|tj|jjd}|dur>||}||}||} |jdkrU| |} | | 7} | | } | S)Nrrdtypedevicer) sizerr"zeroslongr0rrrrr!) r' input_idstoken_type_idsr inputs_embedspast_key_values_length input_shape seq_lengthr embeddingsrr+r+r,forwardHs.        zMegatronBertEmbeddings.forward)NNNNr) __name__ __module__ __qualname____doc__rr"Tensorr; __classcell__r+r+r)r,r+s rcsJeZdZd fdd ZdejdejfddZ      d d d ZZS) MegatronBertSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rr relative_keyrelative_key_queryr)rrrnum_attention_headshasattr ValueErrorintattention_head_size all_head_sizerLinearquerykeyvaluerZattention_probs_dropout_probr!r%rrrdistance_embedding is_decoder)r'r(rr)r+r,rrs8    z"MegatronBertSelfAttention.__init__xr-cCs6|dd|j|jf}||}|ddddS)NrrrFr)r1rGrKviewpermute)r'rSZ new_x_shaper+r+r,transpose_for_scoress  z.MegatronBertSelfAttention.transpose_for_scoresFcCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |jrv| | f}t| | dd} |j dks|j dkr| d}tj |tj |j d dd}tj |tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}| |} n|j dkrtd | |}td | |}| ||} | t|j} |dur| |} tjj| dd}||}|dur||}t|| }|dddd }| dd|jf}||}|r9||fn|f}|jrE||f}|S)NrrrFdimrrDrEr.)r/zbhld,lrd->bhlrzbhrd,lrd->bhlrrT)rNrWrOrPr"catrRmatmulZ transposerr1r#r3r0rUrQrtor/ZeinsummathsqrtrKrZ functionalZsoftmaxr!rV contiguousrL)r' hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layerZattention_scoresr9Zposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr+r+r,r;s               z!MegatronBertSelfAttention.forwardNNNNNNF) r<r=r>rr"r@rWr;rAr+r+r)r,rBps rBc$eZdZfddZddZZS)MegatronBertSelfOutputcs.tt|j|j|_t|j|_dSri) rrrrMrdenserr r!r&r)r+r,r zMegatronBertSelfOutput.__init__cC||}||}||Srirmr!)r'raZresidualr+r+r,r; s  zMegatronBertSelfOutput.forwardr<r=r>rr;rAr+r+r)r,rl rlcs:eZdZfddZddZ      d ddZZS) MegatronBertAttentioncs>ttj|j|jd|_t||_t ||_ t |_ dS)Neps) rrr LayerNormrlayer_norm_epslnrBr'rloutputset pruned_headsr&r)r+r,rs    zMegatronBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrX)lenrr'rGrKr{rrNrOrPryrmrLunion)r'headsindexr+r+r, prune_headss   z!MegatronBertAttention.prune_headsNFc CsF||}||||||||} || d|} | f| dd} | S)Nrr)rxr'ry) r'rarbrcrdrerfrgZ ln_outputsZ self_outputsattention_outputrhr+r+r,r;-s   zMegatronBertAttention.forwardrj)r<r=r>rrr;rAr+r+r)r,rss rscrk)MegatronBertIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSri) rrrrMrintermediate_sizerm isinstanceZ hidden_actstrrintermediate_act_fnr&r)r+r,rJs   z!MegatronBertIntermediate.__init__cCs||}||}|Sri)rmr)r'rar+r+r,r;Rs  z MegatronBertIntermediate.forwardrqr+r+r)r,rHs rcs8eZdZfddZdejdejdejfddZZS)MegatronBertOutputcs.tt|j|j|_t|j|_ dSri) rrrrMrrrmrr r!r&r)r+r,r[rnzMegatronBertOutput.__init__ra input_tensorr-cCrorirp)r'rarr+r+r,r;`s  zMegatronBertOutput.forward)r<r=r>rr"r@r;rAr+r+r)r,rYs rcs:eZdZfddZ      d ddZddZZS) MegatronBertLayercst|j|_d|_t||_|j|_|j|_|jr+|js&t|dt||_ t j |j |j d|_t||_t||_dS)Nrz> should be used as a decoder model if cross attention is addedrt)rrchunk_size_feed_forward seq_len_dimrs attentionrRadd_cross_attention TypeErrorcrossattentionrrvrrwrxr intermediaterryr&r)r+r,rjs    zMegatronBertLayer.__init__NFc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) NrFrgrfrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`rZ) rrRrHAttributeErrorrr feed_forward_chunkrr)r'rarbrcrdrerfrgZself_attn_past_key_valueZself_attention_outputsrrhZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputr+r+r,r;{sp        zMegatronBertLayer.forwardcCs$||}||}|||}|Sri)rxrry)r'rZ ln_outputZintermediate_outputrr+r+r,rs   z$MegatronBertLayer.feed_forward_chunkrj)r<r=r>rr;rrAr+r+r)r,rhs  Ercs8eZdZfddZ         dddZZS) MegatronBertEncodercsNt|_tfddtjD|_tjj j d|_ d|_ dS)Ncsg|]}tqSr+)r).0_r(r+r, sz0MegatronBertEncoder.__init__..rtF) rrr(rZ ModuleListrangenum_hidden_layerslayerrvrrwrxgradient_checkpointingr&r)rr,rs   zMegatronBertEncoder.__init__NFTc  sl| rdnd} r dnd} r|jjrdnd} |rdnd}t|jD]o\}}| r,| |f} |dur4||nd}|dur>||nd|jre|jre|rOtdd}fdd}tj j |||||||}n ||||||}|d}|r|||df7}r| |df} |jjr| |d f} q!| |}| r| |f} | st d d ||| | | fDSt ||| | | d S) Nr+zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fcsfdd}|S)Ncsg|RSrir+)Zinputs)modulergrfr+r,custom_forwards zRMegatronBertEncoder.forward..create_custom_forward..custom_forwardr+)rrr)rr,create_custom_forwardsz:MegatronBertEncoder.forward..create_custom_forwardrrrrFcss|] }|dur|VqdSrir+)rvr+r+r, 's z.MegatronBertEncoder.forward..)last_hidden_statepast_key_valuesra attentionscross_attentions)r(r enumeraterrZtrainingloggerwarningr"utils checkpointrxtupler )r'rarbrcrdrer use_cachergoutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrZ layer_outputsr+rr,r;s         zMegatronBertEncoder.forward) NNNNNNFFTrqr+r+r)r,rs rcrk)MegatronBertPoolercs*tt|j|j|_t|_dSri)rrrrMrrmZTanh activationr&r)r+r,r:s zMegatronBertPooler.__init__cCs(|dddf}||}||}|S)Nr)rmr)r'raZfirst_token_tensor pooled_outputr+r+r,r;?s  zMegatronBertPooler.forwardrqr+r+r)r,r8rrrcsTeZdZdZeZdZdZdgZfddZ ddZ dd d Z e fd d Z ZS)MegatronBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ZbertTrc s*tj|jfi|tt||dSri)rrZ name_or_pathr)r'r(kwargsr)r+r,rSsz$MegatronBertPreTrainedModel.__init__cCs~t|tjtjfr|jjjd|jjdnt|tj r(|j j |jj dt|tjr;|j dur=|j j dSdSdS)zInitialize the weightsg)meanZstdg?N) rrrMrweightdataZnormal_r(Zinitializer_rangervZbiasZzero_Zfill_)r'rr+r+r, _init_weightsWs  z)MegatronBertPreTrainedModel._init_weightsFcCst|tr ||_dSdSri)rrr)r'rrPr+r+r,_set_gradient_checkpointingds  z7MegatronBertPreTrainedModel._set_gradient_checkpointingc sn|dd}|dd}t||fi|}|dur%tdi|}||}n tt|jdd|i|}||_|S)ahInstantiate the model. Args: kwargs: Input args. model_dir: The model dir used to load the checkpoint and the label information. num_labels: An optional arg to tell the model how many classes to initialize. Method will call utils.parse_label_mapping if num_labels not supplied. If num_labels is not found, the model will use the default setting (2 classes). Returns: The loaded model, which is initialized by transformers.PreTrainedModel.from_pretrained model_dirNcfgZpretrained_model_name_or_pathr+)popr rrrZfrom_pretrainedr)clsrrrZ model_argsr(modelr)r+r, _instantiatehs   z(MegatronBertPreTrainedModel._instantiate)F)r<r=r>r?rZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_keys_to_ignore_on_load_missingrrr classmethodrrAr+r+r)r,rHs  r)Z group_key module_namecsdeZdZdZdfdd ZddZddZd d Z dd efd dZ Z S)MegatronBertModelaThe bare MegatronBert Model transformer outputting raw hidden-states without any specific head on top.", This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MegatronBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. TcsDt|||_t||_t||_|rt|nd|_| dSri) rrr(rr:rencoderrpoolerZ post_init)r'r(Zadd_pooling_layerr)r+r,rs    zMegatronBertModel.__init__cCs|jjSrir:r)r'r+r+r,get_input_embeddingssz&MegatronBertModel.get_input_embeddingscCs ||j_dSrir)r'rPr+r+r,set_input_embeddingss z&MegatronBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r'Zheads_to_prunerr~r+r+r, _prune_headsszMegatronBertModel._prune_headsNr-c Ks | dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durD|}n|durQ|dd}ntd|\}}|dur`|jn|j}| durp| ddj dnd}|durt j |||f|d}|durt j |t j |d }||||}|jjr|dur|\}}}||f}|durt j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}|jdur||nd}| s||f|d dSt|||j|j|j|jd S)a4 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`BertTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Others (**kwargs) some additional parameters might passed in from upstream pipeline, which not influence the results. NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrF)r0r.)r4rr5r6r7) rbrcrdrerrrgrrr)rZ pooler_outputrrarr)r(rgrZuse_return_dictrRrrIr1r0shaper"Zonesr2r3Zget_extended_attention_maskZinvert_attention_maskZ get_head_maskrr:rrr rrarr)r'r4rbr5rrcr6rdrerrrgrrrr8Z batch_sizer9r0r7Zextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputsZsequence_outputrr+r+r,r;sd     zMegatronBertModel.forward)T) NNNNNNNNNNNNN) r<r=r>r?rrrrr r;rAr+r+r)r,rs,  r)/r?r^r"Ztorch.utils.checkpointrZtransformers.activationsrZtransformers.modeling_utilsrZmodelscope.metainforZmodelscope.modelsrrZmodelscope.models.builderrZmodelscope.outputsr Zmodelscope.utils.constantr Zmodelscope.utils.loggerr Zmodelscope.utils.nlp.utilsr Zmodelscope.utils.torch_utilsr rr configurationrrZ_CONFIG_FOR_DOCModulerrBrlrsrrrrrrZregister_moduleZbackboneZ megatron_bertrr+r+r+r,sB          E7_q<