o *j~@sTdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZddlmZddl m!Z!e"ZGddde j#Z$Gdddej%j&Z'Gddde(Z)ddZ*Gdddej%j&Z+Gddde j#Z,Gddde j#Z-Gd d!d!e j#Z.Gd"d#d#e j#Z/Gd$d%d%e j#Z0Gd&d'd'e j#Z1Gd(d)d)e j#Z2Gd*d+d+e j#Z3d,d-Z4 . .d@d/d0Z5ej6j7d1d2Z8ej6j7d3d4Z9ej6j7d5d6Z:Gd7d8d8e j#Z;Gd9d:d:e j#Zej?ej@d=Gd>d?d?e=ZAdS)Az PyTorch DeBERTa-v2 model.)Sequence)OptionalTupleUnionN)nn) LayerNorm)ACT2FN)PreTrainedModel)softmax_backward_data)Models)Model TorchModel)MODELS)AttentionBackboneModelOutput)logger)Tasks)DebertaV2Configcs0eZdZfddZddZeddZZS) ContextPoolercs2tt|j|j|_t|j|_||_ dSN) super__init__rLinearZpooler_hidden_sizedense StableDropoutZpooler_dropoutdropoutconfigselfr __class__j/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/deberta_v2/backbone.pyr*s   zContextPooler.__init__cCs8|dddf}||}||}t|jj|}|SNr)rrrrZpooler_hidden_act)r hidden_statesZ context_tokenZ pooled_outputr!r!r"forward1s   zContextPooler.forwardcC|jjSr)r hidden_sizerr!r!r" output_dim;szContextPooler.output_dim)__name__ __module__ __qualname__rr%propertyr) __classcell__r!r!rr"r(s   rc@s4eZdZdZeddZeddZeddZdS) XSoftmaxa Masked Softmax which is optimized for saving memory Args: input (`torch.tensor`): The input tensor that will apply softmax. mask (`torch.IntTensor`): The mask matrix where 0 indicate that element will be ignored in the softmax calculation. dim (int): The dimension that will apply softmax Example: >>> import torch >>> from transformers.models.deberta_v2.modeling_deberta_v2 import XSoftmax >>> # Make a tensor >>> x = torch.randn([4, 20, 100]) >>> # Create a mask >>> mask = (x > 0).int() >>> # Specify the dimension to apply softmax >>> dim = -1 >>> y = XSoftmax.apply(x, mask, dim) cCsX||_|tj}||tt|jj}t ||j}| |d| ||Sr#) dimtotorchbool masked_filltensorfinfodtypeminsoftmax masked_fill_save_for_backward)rinputmaskr0rmaskoutputr!r!r"r%\s  zXSoftmax.forwardcCs$|j\}t||||j|}|ddfSr) saved_tensorsr r0)r grad_outputr?Z inputGradr!r!r"backwardhs   zXSoftmax.backwardc Csddlmm}ddlm}m}|jd||jdd}|jd|d|jdtj dtj d d ||jd d}|||||jdt t | jd } ||| |} ||| ||jdtj dtjd d S) Nr)r4r9ZCastLong)Zto_iSubConstantrr7)Zvalue_tZByte)Ztorch.onnx.symbolic_helperZonnxZsymbolic_helperZtorch.onnx.symbolic_opset9r4r9opZcast_pytorch_to_onnxr2r5Zint64r6typer7r8Zuint8) grr=r0Zsym_helpr4r9Zmask_cast_valueZr_maskr?r!r!r"symbolicos2  zXSoftmax.symbolicN)r*r+r,__doc__ staticmethodr%rBrJr!r!r!r"r/As  r/c@seZdZddZdS)DropoutContextcCsd|_d|_d|_d|_dS)NrrT)rr=scale reuse_maskr(r!r!r"rs zDropoutContext.__init__N)r*r+r,rr!r!r!r"rMs rMcCst|ts |}d}n|j}||j9}|jr|jnd}|dkr2|dur2dt|d| tj }t|tr?|jdur?||_||fS)Nrr) isinstancerMrrNrOr=r2Z empty_likeZ bernoulli_r1r3)r<Z local_contextrr=r!r!r"get_masks    rQc @sZeZdZdZeddZeddZedejj dejj de e e fd ejj fd d Zd S) XDropoutzlOptimized dropout function to save computation and memory by using mask operation instead of multiplication.cCsDt||\}}dd||_|dkr ||||d|jS|S)Ng?rr)rQrNr;r4)ctxr< local_ctxr=rr!r!r"r%s  zXDropout.forwardcCs0|jdkr|j\}||d|jdfS|dfS)Nrr)rNr@r4)rSrAr=r!r!r"rBs zXDropout.backwardrIr<rTreturncCs4ddlm}|}t|tr|j}d}|||||S)Nr)symbolic_opset12T)Z torch.onnxrVrPrMr)rIr<rTrVZ dropout_ptrainr!r!r"rJs  zXDropout.symbolicN)r*r+r,rKrLr%rBr2Z_CZGraphValuerfloatrMrJr!r!r!r"rRs   rRcsBeZdZdZfddZddZddZdd d Zd d ZZ S)rz Optimized dropout module for stabilizing the training Args: drop_prob (float): the dropout probabilities cs t||_d|_d|_dSr#)rr drop_probcount context_stack)rrZrr!r"rs  zStableDropout.__init__cCs$|jr|jdkrt||S|S)zr Call the module Args: x (`torch.tensor`): The input tensor to apply dropout r)trainingrZrRapply get_context)rxr!r!r"r%szStableDropout.forwardcCsd|_d|_dSr#)r[r\r(r!r!r" clear_contexts zStableDropout.clear_contextTrcCs2|jdurg|_d|_|jD]}||_||_qdSr#)r\r[rOrN)rrOrNcr!r!r" init_contexts  zStableDropout.init_contextcCsT|jdur'|jt|jkr|jt|j|j}|j|_|jd7_|S|jS)Nr)r\r[lenappendrMrZr)rrSr!r!r"r_s  zStableDropout.get_context)Tr) r*r+r,rKrr%rarcr_r.r!r!rr"rs  rc$eZdZfddZddZZS)DebertaV2SelfOutputcs<tt|j|j|_t|j|j|_t|j |_ dSr) rrrrr'rrlayer_norm_epsrhidden_dropout_probrrrr!r"rs zDebertaV2SelfOutput.__init__cC&||}||}|||}|Srrrrrr$Z input_tensorr!r!r"r%   zDebertaV2SelfOutput.forwardr*r+r,rr%r.r!r!rr"rgs rgcs.eZdZfddZ    dddZZS)DebertaV2Attentioncs(tt||_t||_||_dSr)rrDisentangledSelfAttentionrrgr?rrrr!r"rs    zDebertaV2Attention.__init__FNc CsJ|j||||||d}|r|\}}|dur|}|||} |r#| |fS| S)N) query_states relative_posrel_embeddings)rr?) rr$attention_maskoutput_attentionsrqrrrsZ self_output att_matrixattention_outputr!r!r"r%s   zDebertaV2Attention.forwardFNNNrnr!r!rr"ro  rocs2eZdZfddZdejdejfddZZS)DebertaV2IntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rrrrr'intermediate_sizerrPZ hidden_actstrrintermediate_act_fnrrr!r"r;s   zDebertaV2Intermediate.__init__r$rUcCs||}||}|Sr)rr})rr$r!r!r"r%Cs  zDebertaV2Intermediate.forward)r*r+r,rr2Tensorr%r.r!r!rr"rz9s rzcrf)DebertaV2OutputcsBtt|j|j|_t|j|j|_t |j |_ ||_ dSr) rrrrr{r'rrrhrrirrrrr!r"rLs   zDebertaV2Output.__init__cCrjrrkrlr!r!r"r%SrmzDebertaV2Output.forwardrnr!r!rr"rJs rcs.eZdZfddZ    dddZZS)DebertaV2Layercs,tt||_t||_t||_dSr)rrro attentionrz intermediaterr?rrr!r"r]s   zDebertaV2Layer.__init__NFc CsH|j||||||d}|r|\}}||} || |} |r"| |fS| S)Nrurqrrrs)rrr?) rr$rtrqrrrsrurwrvZintermediate_outputZ layer_outputr!r!r"r%cs   zDebertaV2Layer.forward)NNNFrnr!r!rr"r[ryrcrf) ConvLayercsxtt|dd}t|dd}t|dd|_tj|j|j||dd|d|_t|j|j |_t |j |_ ||_ dS) Nconv_kernel_sizeZ conv_groupsrconv_acttanh)paddinggroups)rrgetattrrrZConv1dr'convrrhrrirr)rrZ kernel_sizerrr!r"rs      zConvLayer.__init__c Cs||dddddd}d|}||d|dt|j | |}||}| | |}|durE|}|S| | kr`| dkr[|dd}|d}| |j}||}|S)Nrrr)rpermute contiguousr3r: unsqueezeexpandsizerrrrr1r0squeezer7) rr$Zresidual_states input_maskoutr>Zlayer_norm_inputr? output_statesr!r!r"r%s(     zConvLayer.forwardrnr!r!rr"r~s rcsNeZdZdZfddZddZddZdd d Z   dd dZZ S)DebertaV2Encoderz8Modified BertEncoder with relative position bias supportcsttfddtjD|_tdd|_|jrMtdd|_ |j dkr/j |_ tdd|_ |j d }|j d krE|j d }t |j |_d dtd d dD|_d|jvrmtj jdd|_tdd d krytnd|_d|_dS)Ncsg|]}tqSr!)r.0_rr!r" sz-DebertaV2Encoder.__init__..relative_attentionFmax_relative_positionsrrposition_bucketsrrcSsg|]}|qSr!)strip)rr`r!r!r"rs norm_rel_ebdnone| layer_normT)Zelementwise_affiner)rrrZ ModuleListrangeZnum_hidden_layerslayerrrrmax_position_embeddingsr Embeddingr'rslowersplitrrrhrrgradient_checkpointing)rr pos_ebd_sizerrr"rsF        zDebertaV2Encoder.__init__cCs2|jr|jjnd}|durd|jvr||}|S)Nr)rrsweightrr)rrsr!r!r"get_rel_embeddings z"DebertaV2Encoder.get_rel_embeddingcCsV|dkr|dd}||dd}|}|S|dkr)|d}|S)Nrrrr)r0rrbyte)rrtZextended_attention_maskr!r!r"get_attention_masks  z#DebertaV2Encoder.get_attention_maskNcCsH|jr"|dur"|dur|dn|d}t||d|j|jd}|S)Nr bucket_size max_position)rrbuild_relative_positionrr)rr$rqrrqr!r!r" get_rel_posszDebertaV2Encoder.get_rel_posTFc s|dkr |}n |ddk}||}||||}|r"dnd} r(dnd} t|tr4|d} n|} |} | } t|j D]l\}}|rL| | f} |j rg|j rgfdd}t j j||| |||| } n || |||| d} rw| \} }|dkr|jdur||| |} |dur| }t|tr|dt|j kr||dnd} n| } r| |f} qA|r| | f} |std d | | | fDSt| | | d S) Nrrrr!csfdd}|S)Ncsg|RSrr!)Zinputs)modulerur!r"custom_forwardszODebertaV2Encoder.forward..create_custom_forward..custom_forwardr!)rrru)rr"create_custom_forward sz7DebertaV2Encoder.forward..create_custom_forward)rqrrrsrurcss|] }|dur|VqdSrr!)rvr!r!r" <sz+DebertaV2Encoder.forward..Zlast_hidden_stater$ attentions)r0sumrrrrPrr enumeraterrr]r2utils checkpointrrdtupler)rr$rtoutput_hidden_statesrurqrr return_dictrZall_hidden_statesZall_attentionsZnext_kvrsriZ layer_modulerZatt_mr!rr"r%s              zDebertaV2Encoder.forward)NN)TFNNT) r*r+r,rKrrrrr%r.r!r!rr"rs % rc Cst|}|d}t||k|| k@t|d|t|}tt||tt|d||d|}t||k||||}|S)Nrr)r2signwherer5Ztype_asabsceillog)rrrrrmidZabs_posZlog_posZ bucket_posr!r!r"make_log_bucket_positionEs.   rrcCstd|}td|}|dddf|dddf}|dkr*|dkr*t|||}|tj}|d|ddf}|d}|S)af Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key bucket_size (int): the size of position bucket max_position (int): the maximum allowed absolute position Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] rN)r2arangerr1longr)Z query_sizeZkey_sizerrZq_idsZk_idsZ rel_pos_idsr!r!r"rWs     rcCs*||d|d|d|dgS)Nrrrrrr)c2p_pos query_layerrrr!r!r"c2p_dynamic_expandx rcCs*||d|d|d|dgS)Nrrrr)rr key_layerr!r!r"p2c_dynamic_expandrrcCs*||dd|d|dfS)Nrrr)Z pos_indexp2c_attrr!r!r"pos_dynamic_expandsrcsBeZdZdZfddZddZ    d dd Zd d ZZS) rpa Disentangled self-attention module Parameters: config (`DebertaV2Config`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaV2Config`] cst|j|jdkrtd|jd|jd|j|_|j|j}t|d||_|j|j|_tj |j|jdd|_ tj |j|jdd|_ tj |j|jdd|_ t|dd |_ |jdurb|jng|_t|d d |_|jrt|d d |_t|d d |_|jdkr|j|_|j|_|jdkr|j|_t|j|_|j sd|jvrtj |j|jdd|_d|jvrt |j|j|_t|j|_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()attention_head_sizeTbias share_att_keyFrrrrrc2pp2c)rrr'num_attention_heads ValueErrorrrZ all_head_sizerr query_projkey_proj value_projr pos_att_typerrrrrrri pos_dropout pos_key_projpos_query_projZattention_probs_dropout_probr)rrZ_attention_head_sizerr!r"rs\            z"DisentangledSelfAttention.__init__cCsL|dd|df}||}|ddddd|d|dS)Nrrrrr)rviewrr)rr`Zattention_headsZ new_x_shaper!r!r"transpose_for_scoress  z.DisentangledSelfAttention.transpose_for_scoresFNc Cs|dur|}||||j}||||j}||||j} d} d} d|jvr1| d7} d|jvr:| d7} ttj| dtj d| } t || ddtj| |j d} |jrm||}|||||| } | duru| | } | } | d|j| d| d} t| |d}||}t |d| d| d| }|d|j| d| ddd dd }| ddd }||}|r||fS|S) a Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.ByteTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, optional): Whether return the attention matrix. query_states (`torch.FloatTensor`, optional): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. NrrrrrFrrrrr)rrrrrrr2sqrtr5rrYbmm transposer7rrdisentangled_attention_biasrr/r^rrr)rr$rtrurqrrrsrrZ value_layerZrel_att scale_factorrNZattention_scoresZattention_probsZ context_layerZnew_context_layer_shaper!r!r"r%s&           z!DisentangledSelfAttention.forwardc Cs$|dur|d}t||d|j|jd}|dkr%|dd}n|dkr1|d}n|dkr@td||j}| |j }|d|dddfd}|j r| | ||j|d|jdd}| |||j|d|jdd} n4d |jvr| |||j|d|jdd} d |jvr| |||j|d|jdd}d} d |jvrttj| d tjd |} t|| d d} t||d|dd} tj| d | d|d|d|d gd } | | tj| | jd 7} d |jvrttj|d tjd |} |d|dkrLt|d|d|j|jd |j }|d}n|}t| |d|dd}t||d d}tj|d |d|d|d|dgd d d}| |tj| |jd 7} | S)Nrrrrrrrz2Relative position ids must be of dim 2 or 3 or 4. rrrrF)r0index)rrrrr0rrrrr1devicerrrrrepeatrrrrr2rr5rYrrclampZgatherrrr7)rrrrrrsrrZatt_spanZpos_query_layerZ pos_key_layerZscorerNZc2p_attrZr_posZp2c_posrr!r!r"r.s              z5DisentangledSelfAttention.disentangled_attention_biasrx) r*r+r,rKrrr%rr.r!r!rr"rps ,  \rpcs4eZdZdZfddZ     dddZZS)DebertaV2EmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cstt|dd}t|d|j|_tj|j|j|d|_t|dd|_ |j s,d|_ n t|j |j|_ |j dkrCt|j |j|_ |j|jkrTtj|j|jdd|_t|j|j|_t|j|_||_|d t|j d dS) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFr position_ids)rr)rrrr'rrrZ vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsr embed_projrrhrrirrZregister_bufferr2rr)rrrrr!r"rs>       zDebertaV2Embeddings.__init__Nc CsN|dur |}n|dd}|d}|dur$|jddd|f}|dur3tj|tj|jjd}|dur<||}|jdurI||}nt|}|} |j rW| |7} |j j dkrf| |} | | 7} |j |j jkrr|| } || } |dur|| kr|dkr|dd}|d}|| j}| |} || } | S)Nrrr7rrrr)rrr2zerosrrrrZ zeros_likerrrrrr'rrr0rrr1r7r) r input_idstoken_type_idsrr= inputs_embeds input_shapeZ seq_lengthr embeddingsrr!r!r"r%sB             zDebertaV2Embeddings.forward)NNNNN)r*r+r,rKrr%r.r!r!rr"rs !rcsZeZdZdZeZdZdgZdgZdZ fddZ dd Z dd d Z e fd dZZS)DebertaV2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ZdebertarrTc s*tj|jfi|tt||dSr)rrZ name_or_pathr rrkwargsrr!r"rsz!DebertaV2PreTrainedModel.__init__cCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rA|jjjd|jjd|j durC|jj|j  dSdSdS)zInitialize the weights.g)meanZstdN) rPrrrdataZnormal_rZinitializer_rangerZzero_rr)rrr!r!r" _init_weightss    z&DebertaV2PreTrainedModel._init_weightsFcCst|tr ||_dSdSr)rPrr)rrvaluer!r!r"_set_gradient_checkpointings  z4DebertaV2PreTrainedModel._set_gradient_checkpointingc sD|dd}|durtdi|}||}|Stt|j|d}|S)N model_dir)Zpretrained_model_name_or_pathr!)poprrr Zfrom_pretrained)clsr rZ ponet_configmodelrr!r" _instantiate s z%DebertaV2PreTrainedModel._instantiate)F)r*r+r,rKrZ config_classZbase_model_prefixZ_keys_to_ignore_on_load_missingZ"_keys_to_ignore_on_load_unexpectedZsupports_gradient_checkpointingrrr classmethodrr.r!r!rr"r s  r ) module_namecseZdZdZfddZddZddZdd Z dd ee j d ee j d ee j dee j dee j dee dee dee de e effddZZS)DebertaV2Modela The bare DeBERTa_v2 Model transformer outputting raw hidden-states without any specific head on top. The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (`DebertaV2Config`): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. c s8t|t||_t||_d|_||_|dSr#) rrrr rencoderz_stepsrZ post_initr rr!r"r-s    zDebertaV2Model.__init__cCr&rr rr(r!r!r"get_input_embeddings7sz#DebertaV2Model.get_input_embeddingscCs ||j_dSrr)rZnew_embeddingsr!r!r"set_input_embeddings:s z#DebertaV2Model.set_input_embeddingscCstd)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel z7The prune function is not implemented in DeBERTa model.)NotImplementedError)rZheads_to_pruner!r!r" _prune_heads=szDebertaV2Model._prune_headsNrrtrrrrurrrUc  s|dur|njj}|dur|njj}|dur|njj}|dur*|dur*td|dur3|} n|dur@|dd} ntd|durK|jn|j} |durYtj| | d}|durftj | tj | d}j |||||d} j | |d||d } | d } j d kr| d }fd d tj D}| d}j }j |}j | }|d dD]}|||d|||d}| |q| d}|s|f| |rd nddSt||r| jnd| jdS)u5 Args: input_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`): Indices of input sequence tokens in the vocabulary. attention_mask (`torch.FloatTensor` of shape `('batch_size, sequence_length')`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. token_type_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. position_ids (`torch.LongTensor` of shape `('batch_size, sequence_length')`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `('batch_size, sequence_length', hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a dataclass instead of a plain tuple. Returns: Returns `modelscope.outputs.AttentionBackboneModelOutput` Examples: >>> from modelscope.models import Model >>> from modelscope.preprocessors import Preprocessor >>> model = Model.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite', task='backbone') >>> preprocessor = Preprocessor.from_pretrained('damo/nlp_debertav2_fill-mask_chinese-lite') >>> print(model(**preprocessor('这是个测试'))) NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)rr)rrrr=rT)rrurrrcsg|]}jjdqSr)rrrr(r!r"rsz*DebertaV2Model.forward..Frrr)rrurZuse_return_dictrrrr2Zonesrrr rrrrrrrerr$r)rrrtrrrrurrrrZembedding_outputZencoder_outputsZencoded_layersr$ZlayersrqrsZrel_posrZsequence_outputr!r(r"r%Es9       zDebertaV2Model.forward)NNNNNNNN)r*r+r,rKrrrr rr2r~r3rrrr%r.r!r!rr"rsB     r)rr)BrKcollections.abcrtypingrrrr2Ztorch.utils.checkpointrZtorch.nnrZtransformers.activationsrZtransformers.modeling_utilsr Ztransformers.pytorch_utilsr Zmodelscope.metainfor Zmodelscope.modelsr r Zmodelscope.models.builderrZmodelscope.outputsrZmodelscope.utilsrloggingZmodelscope.utils.constantr configurationrZ get_loggerModulerZautogradFunctionr/objectrMrQrRrrgrorzrrrrrrZjitscriptrrrrprr Zregister_moduleZbackboneZ deberta_v2rr!r!r!r"s`            H )2&#) !   |Y0