o *j@s:dZddlZddlZddlZddlmmZddlmZ ddl m Z Gdddejj Z Gdddejj ZGd d d ejj Zejjd d Zd dZGdddejj ZGdddejj ZGdddejj ZddZddZGdddejj ZGdddejj ZGdddejj ZGdd d ejj ZdS)!z Transformer.N)FusedLayerNorm)mpucs&eZdZfddZdddZZS)PositionalEmbeddingcs>tt|||_ddtd|d|}|d|dS)Ni'@inv_freq)superr__init__ hidden_sizetorcharangeZregister_buffer)selfr r __class__m/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/mglm/model/transformer.pyr s zPositionalEmbedding.__init__NcCsft||j}tj||gdd}|dur(|dddddf|ddS|dddddfS)Ndim)r Zgerrcatsincosexpand)rZpos_seqZbszZ sinusoid_inpZpos_embrrrforward$s  zPositionalEmbedding.forwardN)__name__ __module__ __qualname__r r __classcell__rrrrrs  rcs4eZdZdZ d fdd ZddZddZZS) ParallelCrossAttentionz.Parallel cross-attention layer for TransformerNcstt||dur |}t}t|||_t|||_t|||_tj ||d|d|_ tj |d|dd|d|_ t j ||_tj||d|d|_t j ||_tjrftjjt_tjjt_dSdS)NF gather_output init_methodZstrider"r#Tinput_is_parallelr#)r r r rget_model_parallel_world_sizedividehidden_size_per_partitionhidden_size_per_attention_head!num_attention_heads_per_partitionColumnParallelLinearquery key_valuer nnDropoutattention_dropoutRowParallelLineardenseoutput_dropout deepspeed checkpointing is_configuredget_cuda_rng_tracker checkpoint)rr num_attention_headsattention_dropout_proboutput_dropout_probr#output_layer_init_method world_sizerrrr 1sH   zParallelCrossAttention.__init__cC6|dd|j|jf}|j|}|ddddSz_Transpose a 3D tensor [b, s, np*hn] into a 4D tensor with size [b, np, s, hn]. Nrrr$rsizer,r+viewpermutertensorZnew_tensor_shaperrr_transpose_for_scores` z,ParallelCrossAttention._transpose_for_scorescCs(||}||}t|d\}}||}||} ||} t|| dd} | t |j } |durCt | |dd|} tj j dd| } t || } Wdn1sbwYt| | } | dddd } | dd|jf}| j|} || }||}|S) Nr$rg@?rrrrB)r.r/rsplit_tensor_along_last_dimrIr matmul transposemathsqrtr+mulr0Softmaxr9forkr2rF contiguousrDr*rEr4r5)r hidden_statesencoder_states cross_maskmixed_query_layer mixed_x_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputrrrrjs>             zParallelCrossAttention.forwardr)rrr__doc__r rIrrrrrrr .s / r csReZdZdZ    d fdd ZddZedd d Z    dd d ZZ S)ParallelSelfAttentionaParallel self-attention layer for GPT2. Self-attention layer takes input with size [b, s, h] where b is the batch size, s is the sequence length, and h is the hidden size and creates output of the same size. Arguments: hidden_size: total hidden size of the layer (h). num_attention_heads: number of attention heads (n). Note that we require n to be divisible by number of GPUs used to parallelize the model. Also, we require hidden size to be divisible by n. attention_dropout_prob: dropout probability for the attention scores. init_method: weight initialization. output_layer_init_method: output layer initialization. If None, use `init_method`. We use the following notation: h: hidden_size n: num_attention_heads p: number of partitions np: n/p hp: h/p hn: h/n b: batch size s: sequence length NFrLc stt|||_|dur|}t} t|| |_t|||_t|| |_ ||_ | |_ tj |d|dd|d|_ |rHtj ||d|d|_tj||_tj||d|d|_tj||_tjrqtjjt_tjjt_dSdS)NrBFr%r!Tr&)r rfr performerrr(r)r*r+r,relative_encodingattention_scaler-query_key_valuerelativer r0r1r2r3r4r5r6r7r8r9r:) rr r;r<r=r#r>rhrgrir?rrrr sP   zParallelSelfAttention.__init__cCr@rArCrGrrrrIrJz+ParallelSelfAttention._transpose_for_scorescCstjg|dd|ddR|j|jd}tj||gdd}|jg|dd|dd|dR}|ddddddf|}|rvt|d|df}|t ||d|dddddddf}|S)NrKrdevicedtyperrr) r ZzerosrDrmrnrrEZview_asonestril)xZ zero_triuZzero_padZx_paddedrorrr _rel_shifts&4 4z ParallelSelfAttention._rel_shiftcCs\|d}|dur||}t|d\} } } n!t||fd} || }t|d\} } } | dd| df} || } || }|| }|jr||}||}| | d}t || dd}| | d}t || dd}| |}||}|t |j}n/|jdkrt | t |j| ddt |j|j}nt | | ddt |j}t||}|jdkr|jdddd}||8}||j9}|d d|}tjjdd |}t ||}Wdn1swYt ||}|dd dd}|dd|jf}|j|}||}||}|S) NrrBrrKrLT)rZkeepdimrgrr$)rDrjrrMr rrIrhrk unsqueezerNrOrrrPrQr+rirRmaxr0rSr9rTr2rFrUr*rEr4r5)rrV ltor_maskposition_embeddingsr_w_biasr_r_biasmem query_lengthrZrYr[r\rr]r^r_Zrelative_layerZ rw_head_qZac_scoreZ rr_head_qZbd_scorer`Zmax_attention_scoresrarbrcrdrrrrs                     zParallelSelfAttention.forwardNFFrLFNNNN) rrrrer rI staticmethodrrrrrrrrrfs 6 rfcCs*d|dtd|dd||S)zOpenAI's gelu implementation.g?rLgQ63E?gHm?)r tanhrqrrr gelu_implfs  rcCst|Sr)rrrrrgelunsrcs,eZdZdZ dfdd ZddZZS) ParallelMLPaMLP for GPT2. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform gelu transformation, and project the state back into h hidden dimension. At the end, dropout is also applied. Arguments: hidden_size: The hidden size of the self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. output_layer_init_method: output layer initialization. If None, use `init_method`. Ncs\tt||dur |}tj|d|d|d|_tjd||d|d|_tj ||_ dS)NFr!Tr&) r rr rr- dense_h_to_4hr3 dense_4h_to_hr r0r1dropout)rr r=r#r>rrrr s zParallelMLP.__init__cCs*||}t|}||}||}|Sr)rrrr)rrVZintermediate_parallelrdrrrrs   zParallelMLP.forwardrrrrrer rrrrrrrrs rcs0eZdZdZ dfdd Z dddZZS)ParallelDecoderLayerA single layer transformer for GPT2. We use the following notation: h: hidden size n: number of attention heads b: batch size s: sequence length Transformore layer takes input with size [b, s, h] and returns an output of the same size. Arguments: hidden_size: The hidden size of the self attention. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. output_layer_init_method: output layers (attention output and mlp output) initialization. If None, use `init_method`. Ncstt||dur |}t||d|_t||||||d|_t||d|_t||||||d|_ t||d|_ t ||||d|_ dS)Nepsr>) r rr LayerNorminput_layernormrfself_attentionpost_self_layernormr cross_attentionpost_attention_layernormrmlp)rr r;r<r=layernorm_epsilonr#r>rrrr s@   zParallelDecoderLayer.__init__c Cs^||}|||}||}||}||||} || } || }||} | | } | Sr)rrrrrr) rrVrWrurXlayernorm_outputZself_attention_outputZself_layernorm_inputZself_layernorm_outputattention_outputlayernorm_input mlp_outputrdrrrrs    zParallelDecoderLayer.forwardrrrrrrrs #4rcs<eZdZdZ    d fdd Z    d ddZZS) ParallelTransformerLayerrNFrLc shtt||dur |}t||d|_t|||||||| | d |_t||d|_t||||d|_ dS)Nrr>rhrgrir) r rr rrrf attentionrrr) rr r;r<r=rr#r>rhrgrirrrr 0s0   z!ParallelTransformerLayer.__init__c Cs\||}|dur||nd}|||||||}||} || }||} | | } | Sr)rrrr) rrVrurvrwrxryrrrrrdrrrrZs   z ParallelTransformerLayer.forwardr{r}rrrrrrs#-rcsfdd}|S)z!Init method based on N(0, sigma).ctjjj|ddSNrmeanstdr r0initnormal_rHsigmarrinit_zz#unscaled_init_method..init_r)rrrrrunscaled_init_methodws rcs"|td|fdd}|S)z3Init method based on N(0, sigma/sqrt(2*num_layers).rcrrrrrrrrrz!scaled_init_method..init_)rPrQ)r num_layersrrrrscaled_init_methods rcsPeZdZdZ         dfdd Z  dd d Zdd dZZS)GPT2ParallelTransformeraGPT-2 transformer. This module takes input from embedding layer and it's output can be used directly by a logit layer. It consists of L (num-layers) blocks of: layer norm self attention residual connection layer norm mlp residual connection followed by a final layer norm. Arguments: num_layers: Number of transformer layers. hidden_size: The hidden size of the self attention. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. checkpoint_activations: if True, checkpoint activations. checkpoint_num_layers: number of layers to checkpoint. This is basically the chunk size in checkpoitning. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method_std: standard deviation of the init method which has the form N(0, std). use_scaled_init_for_output_weights: If True use 1/sqrt(2*num_layers) scaling for the output weights ( output of self attention and mlp). rh㈵>{Gz?TFrLc stt||_| |_| |_||_ |_ |_ r rJd| r(t |t j ||_ |_||_ rt|_t}t|_t||_t j t |j|j|_d|j_t j t |j|j|_d|j_t |j|jWdn1swYn7|rt j |d|_t j |d|_t j j j!|jj"ddnt j ||_t j j j!|jj"dd f ddt j #fddt$|D|_%t&d |_'t(j)*rt(j)j+t_+t(j)j,t_,dSdS) NTrrrc s> rttdStt d S)Nrr)rrrr) r<rir init_method_stdrr;r=r>rgrhuse_decoder_layerrr get_layers, z3GPT2ParallelTransformer.__init__..get_layercsg|]}qSrr).0_)rrr sz4GPT2ParallelTransformer.__init__..r)-r rr r checkpoint_activationscheckpoint_num_layersmax_memory_lengthrgrrr r0r1embedding_dropoutrhblock_position_encodingrrvrr(r)r+r, ParameterZTensorrwZmodel_parallelrxZno_gradZzero_Z Embeddingblock_position_embeddingsrrweightZ ModuleListrangelayersrfinal_layernormr6r7r8r9r:)rrr r;Zmax_sequence_lengthrZembedding_dropout_probr<r=rrrrZ"use_scaled_init_for_output_weightsrhrrgrrir?r) r<rirr rrr;r=r>rgrhrrr s         z GPT2ParallelTransformer.__init__Ncsdd\}|r|ddnd} || } t|dkp(t|k} jr:s2Jd| dks:Jd| rXrB|n|} dfdd } jsW| || | d}n|dddddd| | df}jrtj| dd d jjd } |} |}n)j r|dddf|dddf}} |}|j r |}| fd d j dksˆrȈgngfdd}jr"d}tj}j}||kr!js|gn||g}jr||jjg7}|r |||||7}tj||||g|R||7}||ksnEtjD]?\}}js3|gn||g}jrE||jjg7}|rL||nd}||d|ij dks^req'}j dksur}j|d|fS)Nr$rrzFattention_mask should be a scalar to indicate the separation position.zDo not support transformer-xl.csd||f}t|}rd|dddd|f<n)|dd}tj||j|jddd}||ddk}|| d |d}|dkr^|dd}tj ||f|fdd}| d}|S)Nrrrrlr$r) Znew_onesr rprr rmrnrEZ masked_fillrsZ expand_asr)Z seq_lengthsep memory_lengthmZidsmask) batch_sizerV is_scalarrrbuild_mask_matrix2s2  z:GPT2ParallelTransformer.forward..build_mask_matrix)rrgrlcsr|S|Sr)detach)Z_hidden_states) detach_memoryrr check_detachesz5GPT2ParallelTransformer.forward..check_detachcsfdd}|S)Ncsj}|d|dd}}jr#|dd|dd}}n |dd|dd}}t|D]%\}}|r>||nd}||g|Rd|i}jdksRrY|q4|S)Nrrrry)rrh enumeraterappend)ZinputsZlayers_Zx_Zmems_ilayerZmem_i_)rend mem_layers return_memoryrstartrrcustom_forwardqszGGPT2ParallelTransformer.forward..custom..custom_forwardr)rrr)rrrr)rrrcustomosz/GPT2ParallelTransformer.forward..customry)r)r)rDr Znumelrgitemrhr rmrnrvrrrrrlenrrrrwrxrr:rrr update_mems)rrVZ position_idsattention_maskZ memory_statesrWrrrzrZ key_lengthZis_seprrZposition_sequencervZblock_position_idsrrlr chunk_lengthargsrrZmem_irdr)rrrrVrrrrrrs              zGPT2ParallelTransformer.forwardc Cs|r |ddnd}|dd}||}|st|j|}g}tt|D]2}||kr=|||dd| dfq&|tj||dd| |df||fddq&|S)Nrrr)rDminrrrrr r) rZhiddensZmemsrrrzZnew_memory_lengthZnew_memsrrrrrs" " z#GPT2ParallelTransformer.update_mems) rrrTFFFFrL)NNFTr|)rrrrer rrrrrrrrs&-t rcs8eZdZdZdejffdd ZddZddZZ S) BertParallelSelfAttentionaParallel self-attention layer for BERT. Self-attention layer takes input with size [b, s, h] where b is the batch size, s is the sequence length, and h is the hidden size and creates output of the same size. Arguments: hidden_size: total hidden size of the layer (h). num_attention_heads: number of attention heads (n). Note that we require n to be divisible by number of GPUs used to parallelize the model. Also, we require hidden size be divisible by n. dropout_prob: dropout probability for the attention scores. output_parallel: If true, no all-gather is done on the output and the output values will be per partition. We use the following notation: h: hidden_size n: num_attention_heads p: number of partitions np: n/p hp: h/p hn: h/n b: batch size s: sequence length Fcstt|||_||_||_||_t}t |||_ t |||_ t |||_ tj |d|dd|d|_tj||_tjrQtjjt_tjjt_dSdS)NrBFr%)r rr r r; dropout_proboutput_parallelrr(r)r*r+r,r-rjr r0r1rr6r7r8r9r:)rr r;rrr#r?rrrr s2  z"BertParallelSelfAttention.__init__cCr@rArCrGrrrrIrJz/BertParallelSelfAttention._transpose_for_scorescCs||}t|d\}}}||}||}||} tt|j} t|| | dd| } | |7} tj j dd| } t || } Wdn1sWwYt| | } | dddd} | dd|jf}| j|} |jr| }|St| }|S)NrBrrKrrr$r)rjrrMrIrPrQr+r rNrOr0rSr9rTrrFrUrDr*rErZ!gather_from_model_parallel_region)rrVrrZrYr[r\r]r^r_Z norm_factorr`rarbrcrdrrrrs<        z!BertParallelSelfAttention.forward) rrrrerxavier_normal_r rIrrrrrrrs# rcs2eZdZdZddejffdd ZddZZS)BertParallelTransformerOutputz[The output layer used after self attention and intermediate parts of transformer layer.g-q=FcsBtt|tj||||d|_tj||_ t ||d|_ dS)Nr&r) r rr rr3r4r r0r1rr layernorm)rZ input_sizeZ output_sizerrr'r#rrrr 7sz&BertParallelTransformerOutput.__init__cCs*||}||}||}||}|Sr)r4rr)rrVZ input_tensorrrrrrHs   z%BertParallelTransformerOutput.forward rrrrerrr rrrrrrr3srcs.eZdZdZejffdd ZddZZS)BertParallelTransformerLayera_A single layer transformer for Bert. We use the following notation: h: hidden size n: number of attention heads b: batch size s: sequence length Transformore layer takes input with size [b, s, h] and returns an output of the same size. Arguments: hidden_size: The hidden size of the self attention. intermediate_size: size of the intermediate state after self attention. In both BERT and GPT this is set to be 4 times the hidden size. num_attention_heads: number of attention head in the self attention. attention_dropout_prob: dropout probability of the attention score in self attention. output_dropout_prob: dropout probability for the outputs after self attention and final output. intermediate_activation_fn: activation function for output of intermediate. layernorm_epsilon: epsilon used in layernorm to avoid division by zero. init_method: initialization method used for the weights. Note that all biases are initialized to zero and layernorm weight are initialized to one. c sltt|t|||d|d|_t||||d|d|_tj||d|d|_ ||_ t||||d|d|_ dS)NT)rr#)rr'r#Fr!) r rr rrr self_outputrr- intermediateintermediate_activation_fnrd) rr Zintermediate_sizer;r<r=rrr#rrrr ps>  z%BertParallelTransformerLayer.__init__cCs<|||}|||}||}||}|||}|Sr)rrrrrd)rrVrZattention_output_parallelZattention_self_outputZintermediate_output_parallelZ layer_outputrrrrs z$BertParallelTransformerLayer.forwardrrrrrrPs '*r)rerPr6r Z torch.nn.initr0rZ#apex.normalization.fused_layer_normrrZ megatron_utilrModulerr rfZjitscriptrrrrrrrrrrrrrrrs2   lM 5ld  5v