o "j@sddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZmZd d Zd dZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZdS)N)core)VarDesc)no_grad)convert_np_dtype_to_dtype_)in_dynamic_mode) functional)Layer)Constant)_convert_attention_mask_convert_param_attr_to_listcCsX|durdSd|_ts*tj}tj}d||j_d||j_dSdS)NT) Zis_distributedrpaddleZstaticZdefault_startup_programZ current_blockZdefault_main_programZ_find_var_recursivename)varZ startup_blockZ main_blockrk/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/incubate/nn/layer/fused_transformer.py_set_var_distributed src Cst|s|St|tjurt|}|jrHt |}t |j |dddd}t }||krE|td}|n|}n|}|durs||jkrstjjj|jd|j|d}Wdn1smwYn|}|}|}|} || |S)Ng333333?F)place)dtype)r Zis_floating_pointtyperZVarTyperrZ is_gpu_placerZ size_of_dtypenpprodshapegpu_memory_availableZ_copy_toZCPUPlacevalueZ get_tensor_clearrbaseZ frameworkZ_dygraph_place_guardcastZ_share_data_with) trZ size_dtypeZwaiting_alloc_memoryrZt_usedZt_castedZnew_tZ dst_tensorZ src_tensorrrr _to_dtype.s2      r cs<eZdZdZ     d fdd ZddZd d ZZS) !FusedBiasDropoutResidualLayerNorma Applies fused_bias_dropout_residual_layer_norm operation. Parameters: embed_dim (int): The expected feature size in the input and output. dropout_rate (float, optional): The dropout probability used on attention weights to drop some attention targets for the dropout after attention. 0 for no dropout. Default 0.5. bias_attr (ParamAttr|bool, optional): To specify the bias parameter property. Default: None, which means the default bias parameter property is used. If it is set to False, this layer will not have trainable bias parameter. See usage for details in :code:`ParamAttr`. epsilon (float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> # input: [batch_size, seq_len, embed_dim] >>> x = paddle.rand((2, 4, 128)) >>> # residual: [batch_size, seq_len, embed_dim] >>> residual = paddle.rand((2, 4, 128)) >>> fused_bias_dropout_residual_ln = paddle.incubate.nn.FusedBiasDropoutResidualLayerNorm(128) >>> output = fused_bias_dropout_residual_ln(x, residual) >>> print(output.shape) [2, 4, 128] ?Nh㈵>cst|dksJd||j|_||_||_||_|j|g|j|jdd|_ |j|j|gt ddd|_ |j|j|gdd|_ ||_ ||_||_dS) Nr6Expected embed_dim to be greater than 0, but received Trattrris_bias?rr&rdefault_initializerr&rr')super__init___helperget_default_dtype_dtypeZ _bias_attrZ _weight_attr embed_dimcreate_parameter linear_biasr ln_scaleln_bias dropout_rate_epsilonr )selfr2r7 weight_attr bias_attrepsilonr  __class__rrr.ts6    z*FusedBiasDropoutResidualLayerNorm.__init__c Cs0tj|||j|j|j|j|j|jd|jd }|S)aU Applies fused_bias_dropout_residual_layer_norm operation. Parameters: x (Tensor): The input tensor. It is a tensor with shape `[batch_size, seq_len, embed_dim]`. The data type should be float32 or float64. residual (Tensor, optional): The residual tensor. It is a tensor with shape `[batch_size, value_length, vdim]`. The data type should be float32 or float64. Returns: Tensor|tuple: It is a tensor that has the same shape and data type as `x`. upscale_in_train) xresidualZbiasr5r6r7 ln_epsilontrainingmoder ) incubate_fZ&fused_bias_dropout_residual_layer_normr4r5r6r7r8rCr )r9r@rAoutrrrforwards z)FusedBiasDropoutResidualLayerNorm.forwardcCs4|jr d|jnd}d|j|j|j|j|j|S)N, name=zAembed_dim={}, seq_len={}, dropout_rate={}, epsilon={}, dtype={}{})r formatr2Zseq_lenr7r8r1r9Zname_strrrr extra_reprsz,FusedBiasDropoutResidualLayerNorm.extra_repr)r"NNr#N)__name__ __module__ __qualname____doc__r.rGrL __classcell__rrr=rr!Ss#%r!csbeZdZdZ                   dfdd Zdd d Zd d ZddZZS)FusedMultiHeadAttentiona Attention mapps queries and a set of key-value pairs to outputs, and Multi-Head Attention performs multiple parallel attention to jointly attending to information from different representation subspaces. Please refer to `Attention Is All You Need `_ for more details. Parameters: embed_dim (int): The expected feature size in the input and output. num_heads (int): The number of heads in multi-head attention. dropout_rate (float, optional): The dropout probability used on attention weights to drop some attention targets for the dropout after attention. 0 for no dropout. Default 0.5. attn_dropout_rate (float, optional): The dropout probability used on attention weights to drop some attention targets for the dropout in attention. 0 for no dropout. Default 0.5. kdim (int, optional): The feature size in key. If None, assumed equal to `embed_dim`. Default None. vdim (int, optional): The feature size in value. If None, assumed equal to `embed_dim`. Default None. normalize_before (bool, optional): Indicate whether it is pre_layer_norm (True) or post_layer_norm architecture (False). Default False. need_weights (bool, optional): Indicate whether to return the attention weights. Now, only False is supported. Default False. qkv_weight_attr(ParamAttr, optional): To specify the weight parameter property for QKV projection computation. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. qkv_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for QKV projection computation. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. linear_weight_attr(ParamAttr, optional): To specify the weight parameter property for linear projection computation. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. linear_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for linear projection computation. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. pre_ln_scale_attr(ParamAttr, optional): To specify the weight parameter property for pre_layer_norm computation. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. pre_ln_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for pre_layer_norm computation. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ln_scale_attr(ParamAttr, optional): To specify the weight parameter property for post_layer_norm computation. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ln_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for post_layer_norm computation. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. epsilon (float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. nranks (int, optional): Distributed tensor model parallel nranks. Default is 1, means not using tensor parallel. ring_id (int, optional): For distributed tensor model parallel. Default is -1, means not using tensor parallel. transpose_qkv_wb (bool, optional): Support input qkv matmul weight shape as [hidden_size, 3 * hidden_size] and qkv matmul bias shape as [3 * hidden_size]. Will transpose the weight to [3, num_head, head_dim, hidden_size] and transpose bias to [3, num_head, hidden_size] in the fused_attention_op. Only support for GPU for now. The default value is False, which is not do transpose to qkv_w and qkv_b. name (str, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> # input: [batch_size, sequence_length, embed_dim] >>> query = paddle.rand((2, 4, 128)) >>> # self attention mask: [batch_size, num_heads, query_len, query_len] >>> attn_mask = paddle.rand((2, 2, 4, 4)) >>> multi_head_attn = paddle.incubate.nn.FusedMultiHeadAttention(128, 2) >>> output = multi_head_attn(query, None, None, attn_mask=attn_mask) >>> print(output.shape) [2, 4, 128] r"NFr#rcs6t|dksJd||dksJd|||_|j|_||_||_||_||_ |||_ ||_ ||_ ||_ |j ||ksIJd|dusQJd||dksYJ|||_ ||_|jrx|d|j |j g}d|j |j g}nd|j |j |g}d|j |j g}|j|| |jdd|_|j|| |jd d|_|j|j |j |g| |jdd|_|j|g| |jd d|_|d kr|d ksJt|jt|jt|j|r|j| |gtd d d|_|j||gd d|_d|_d|_nd|_d|_|j||gtd d d|_|j||gd d|_||_||_||_dS)Nrr$2Expected nhead to be greater than 0, but received (embed_dim must be divisible by num_headsFz&Only support need_weight is False now.r%TrrSr(r)r*r,)r-r.normalize_beforer/r0r1r8_ring_idr2 num_headshead_dimkdimvdim need_weightstranspose_qkv_wbr3 qkv_weightqkv_bias linear_weightr4rr pre_ln_scale pre_ln_biasr5r6r7attn_dropout_rater )r9r2rYr7rdr[r\rWr]qkv_weight_attr qkv_bias_attrlinear_weight_attrlinear_bias_attrpre_ln_scale_attrpre_ln_bias_attr ln_scale_attr ln_bias_attrr<nranksring_idr^r Zqkv_wight_shapeZqkv_bias_shaper=rrr.s           z FusedMultiHeadAttention.__init__cCs|dur t||j}tjdid|d|jd|jd|jd|jd|jd|j d |j d |j d |j d |j d |d|d|jd|jd|j d|jd|jd|jd|jd|j}|S)a Applies multi-head attention to map queries and a set of key-value pairs to outputs. Parameters: query (Tensor): The queries for multi-head attention. It is a tensor with shape `[batch_size, query_length, embed_dim]`. The data type should be float32 or float64. key (Tensor, optional): The keys for multi-head attention. It is a tensor with shape `[batch_size, key_length, kdim]`. The data type should be float32 or float64. If None, use `query` as `key`. Default None. value (Tensor, optional): The values for multi-head attention. It is a tensor with shape `[batch_size, value_length, vdim]`. The data type should be float32 or float64. If None, use `query` as `value`. Default None. attn_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (MultiHeadAttention.Cache|MultiHeadAttention.StaticCache, optional): Now, only None is supported. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type as `query`, representing attention output. Nr@r_rapre_layer_normrbrcr5r6Zpre_ln_epsilonr`r4Zcache_kv attn_maskr7rdrBrCrnrYr^r r)r rrEZfused_multi_head_attentionr_rarWrbrcr5r6r8r`r4r7rdrCrXrYr^r )r9querykeyrrpcacherFrrrrGs\"       zFusedMultiHeadAttention.forwardc CsH|jr d|jnd}d|j|j|j|j|j|j|j|j |j |j | S)NrHrIzembed_dim={}, num_heads={}, dropout_rate={}, attn_dropout_rate={}, epsilon={}, kdim={}, vdim={}, normalize_before={}, need_weights={}, dtype={}{}) r rJr2rYr7rdr8r[r\rWr]r1rKrrrrLsz"FusedMultiHeadAttention.extra_reprc Cg}|jr|t|j|t|jn|t|j|t|j|jD]'\}}t||vr6q+|durRt t ||}Wdn1sMwYq+||_ dSN) rWappendidrbrcr5r6 _parametersitemsrr r1r9rZlayer_norm_params_idrrparamZ param_appliedrrr _amp_decorate   z%FusedMultiHeadAttention._amp_decorate)r"r"NNFFNNNNNNNNr#rrSFN)NNNN rMrNrOrPr.rGrLr|rQrrr=rrRs2U }?rRcs\eZdZdZ                dfd d Zdd d Zd dZddZZS)FusedFeedForwarda Parameters: d_model (int): The expected feature size in the input and output. dim_feedforward (int): The hidden layer size. dropout_rate (float, optional): The dropout probability used in pre-process and post-precess. Default 0.1 epsilon (float, optional): he small value added to the variance to prevent division by zero. Default: 1e-05. activation (str, optional): The activation function. Default relu. act_dropout_rate (float, optional): The dropout probability after activition. If None, use the value of `dropout_rate`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into, preprocessing or postprocessing. Default False linear1_weight_attr(ParamAttr, optional): To specify the weight parameter property for FFN first linear. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. linear1_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for FFN first linear. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. linear2_weight_attr(ParamAttr, optional): To specify the weight parameter property for FFN second linear. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. linear2_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for FFN second linear. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ln1_scale_attr(ParamAttr, optional): To specify the weight parameter property for FFN pre_layer_norm. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ln1_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for FFN pre_layer_norm. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ln2_scale_attr(ParamAttr, optional): To specify the weight parameter property for FFN post_layer_norm. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ln2_bias_attr(ParamAttr|bool, optional): To specify the bias parameter property for FFN layer_norm. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. nranks (int, optional): Distributed tensor model parallel nranks. Default is 1, means not using tensor parallel. ring_id (int, optional): For distributed tensor model parallel. Default is -1, means not using tensor parallel. name (str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> from paddle.incubate.nn import FusedFeedForward >>> paddle.device.set_device('gpu') >>> fused_feedforward_layer = FusedFeedForward(8, 8) >>> x = paddle.rand((1, 8, 8)) >>> out = fused_feedforward_layer(x) >>> print(out.shape) [1, 8, 8] 皙?r#reluNFrrScst|dksJd||dksJd||j|_||_||dks,J||}||_||_|dur<|n||_ ||_ ||_ ||_ ||_ |j||g||jdd|_|j|g| |jdd|_|j||g| |jdd|_|j|g| |jdd|_|dkr|dksJt|jt|jt|j|r|j|g| dtd d |_|j|g| dd |_d|_d|_nd|_d|_|j|g|dtd d |_|j|g|dd |_||_dS) Nr4Expected d_model to be greater than 0, but received >Expected dim_feedforward to be greater than 0, but received {}Fr%TrrSr(rr&r'r+rr&r')r-r.rJr/r0r1_d_model_dim_feedforward _dropout_rate_act_dropout_rate _act_method_normalize_beforer8rXr3_linear1_weight _linear1_bias_linear2_weight _linear2_biasrr _ln1_scale _ln1_bias _ln2_scale _ln2_biasr )r9d_modeldim_feedforwardr7r< activationact_dropout_raterWlinear1_weight_attrlinear1_bias_attrlinear2_weight_attrlinear2_bias_attrZln1_scale_attrZ ln1_bias_attrZln2_scale_attrZ ln2_bias_attrrmrnr r=rrr.4s         zFusedFeedForward.__init__cCsTtj||j|j|j|j|j|j|j|j |j |j |j |j |j |j|j|j|jd}|S)N) Z dropout1_rateZ dropout2_raterZ ln1_epsilonZ ln2_epsilonrorCrnr )rEZfused_feedforwardrrrrrrrrrrrr8rrCrXr )r9srcrsrFrrrrGs*zFusedFeedForward.forwardc Cs@|jr d|jnd}d|j|j|j|j|j|j|j|j | S)NrHrIzd_model={}, dim_feedforward={}, dropout_rate={}, epsilon={}, activation={}, act_dropout_rate={}, normalize_before={}, dtype={}{}) r rJrrrr8rrrr1rKrrrrLszFusedFeedForward.extra_reprc Crtru) rrvrwrrrrrxryrr r1rzrrrr|r}zFusedFeedForward._amp_decorate)rr#rNFNNNNNNNNrrSNrur~rrr=rrs,A krcs:eZdZdZ       d fdd Zd dd ZZS) FusedTransformerEncoderLayera FusedTransformerEncoderLayer is composed of two sub-layers which are self (multi-head) attention and feedforward network. Before and after each sub-layer, pre-process and post-precess would be applied on the input and output accordingly. If `normalize_before` is True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Parameters: d_model (int): The expected feature size in the input and output. nhead (int): The number of heads in multi-head attention(MHA). dim_feedforward (int): The hidden layer size in the feedforward network(FFN). dropout_rate (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.1 activation (str, optional): The activation function in the feedforward network. Default relu. attn_dropout_rate (float, optional): The dropout probability used in MHA to drop some attention target. If None, use the value of `dropout`. Default None act_dropout_rate (float, optional): The dropout probability used after FFN activition. If None, use the value of `dropout`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default False weight_attr(ParamAttr|list|tuple, optional): To specify the weight parameter property. If it is a list/tuple, `weight_attr[0]` would be used as `weight_attr` for MHA, and `weight_attr[1]` would be used as `weight_attr` for linear in FFN. Otherwise, MHA and FFN both use it as `weight_attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr` . bias_attr (ParamAttr|list|tuple|bool, optional): To specify the bias parameter property. If it is a list/tuple, `bias_attr[0]` would be used as `bias_attr` for MHA, and `bias_attr[1]` would be used as `bias_attr` for linear in FFN. Otherwise, MHA and FFN both use it as `bias_attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Default: None, which means the default bias parameter property is used. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> from paddle.incubate.nn import FusedTransformerEncoderLayer >>> paddle.device.set_device('gpu') >>> # encoder input: [batch_size, src_len, d_model] >>> enc_input = paddle.rand((2, 4, 128)) >>> # self attention mask: [batch_size, n_head, src_len, src_len] >>> attn_mask = paddle.rand((2, 2, 4, 4)) >>> encoder_layer = FusedTransformerEncoderLayer(128, 2, 512) >>> enc_output = encoder_layer(enc_input, attn_mask) >>> print(enc_output.shape) [2, 4, 128] rrNFc s"t|_|jd|jddt|dks!Jd||dks,Jd||dks7Jd||dur=|n|}|durE|n|}||_t| d} t| d} t|||||j| d| d| d| d| d| d| d| dd |_t ||||||j| d | d | d | d d |_ dS) Nr9r>rrrTz`_ , and see `TransformerEncoder` and `TransformerDecoder` for more details. Users can configurate the model architecture with corresponding parameters. Note the usage of `normalize_before` representing where to apply layer normalization (in pre-process or post-precess of multi-head attention or FFN), and some transformer like models are different on this, such as `BERT `_ and `GPT2 `_ . The default architecture here places layer normalization in post-process and applies another layer normalization on the output of last encoder/decoder layer. Parameters: d_model (int, optional): The expected feature size in the encoder/decoder input and output. Default 512 nhead (int, optional): The number of heads in multi-head attention(MHA). Default 8 num_encoder_layers (int, optional): The number of layers in encoder. Default 6 num_decoder_layers (int, optional): The number of layers in decoder. Default 6 dim_feedforward (int, optional): The hidden layer size in the feedforward network(FFN). Default 2048 dropout (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.1 activation (str, optional): The activation function in the feedforward network. Default relu. attn_dropout (float, optional): The dropout probability used in MHA to drop some attention target. If None, use the value of `dropout`. Default None act_dropout (float, optional): The dropout probability used after FFN activition. If None, use the value of `dropout`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default False weight_attr(ParamAttr|list|tuple, optional): To specify the weight parameter property. If it is a list/tuple, the length of `weight_attr` could be 1, 2 or 3. If it is 3, `weight_attr[0]` would be used as `weight_attr` for self attention, `weight_attr[1]` would be used as `weight_attr` for cross attention of `TransformerDecoder`, and `weight_attr[2]` would be used as `weight_attr` for linear in FFN. If it is 2, `weight_attr[0]` would be used as `weight_attr` both for self attention and cross attntion and `weight_attr[1]` would be used as `weight_attr` for linear in FFN. If it is 1, `weight_attr[0]` would be used as `weight_attr` for self attention, cross attention and linear in FFN. Otherwise, the three sub-layers all uses it as `weight_attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr` . bias_attr (ParamAttr|list|tuple|bool, optional): To specify the bias parameter property. If it is a list/tuple, the length of `bias_attr` could be 1, 2 or 3. If it is 3, `bias_attr[0]` would be used as `bias_attr` for self attention, `bias_attr[1]` would be used as `bias_attr` for cross attention of `TransformerDecoder`, and `bias_attr[2]` would be used as `bias_attr` for linear in FFN. If it is 2, `bias_attr[0]` would be used as `bias_attr` both for self attention and cross attntion and `bias_attr[1]` would be used as `bias_attr` for linear in FFN. If it is 1, `bias_attr[0]` would be used as `bias_attr` for self attention, cross attention and linear in FFN. Otherwise, the three sub-layers all uses it as `bias_attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Default: None,which means the default bias parameter property is used. custom_encoder (Layer, optional): If custom encoder is provided, use it as the encoder. Default None custom_decoder (Layer, optional): If custom decoder is provided, use it as the decoder. Default None Examples: .. code-block:: python >>> import paddle >>> from paddle.nn import Transformer >>> # src: [batch_size, tgt_len, d_model] >>> enc_input = paddle.rand((2, 4, 128)) >>> # tgt: [batch_size, src_len, d_model] >>> dec_input = paddle.rand((2, 6, 128)) >>> # src_mask: [batch_size, n_head, src_len, src_len] >>> enc_self_attn_mask = paddle.rand((2, 2, 4, 4)) >>> # tgt_mask: [batch_size, n_head, tgt_len, tgt_len] >>> dec_self_attn_mask = paddle.rand((2, 2, 6, 6)) >>> # memory_mask: [batch_size, n_head, tgt_len, src_len] >>> cross_attn_mask = paddle.rand((2, 2, 6, 4)) >>> transformer = Transformer(128, 2, 4, 4, 512) >>> output = transformer(enc_input, ... dec_input, ... enc_self_attn_mask, ... dec_self_attn_mask, ... cross_attn_mask) >>> print(output.shape) [2, 6, 128] rrNFcsttru)r-r.NotImplementedError)r9rrZnum_encoder_layersZnum_decoder_layersrZdropoutrZ attn_dropoutZ act_dropoutrWr:r;Zcustom_encoderZcustom_decoderr=rrr.s zFusedTransformer.__init__cCstru)r)r9rZtgtrZtgt_maskZ memory_maskrrrrGszFusedTransformer.forward)rrrrrrrNNFNNNN)NNNrrrr=rrs$arcsdeZdZdZ                     dfd d Z      dd d ZZS)FusedMultiTransformeral, FusedMultiTransformer is composed of multi transformer layers which contains two sub-layers which are self (multi-head) attention and feedforward network. The function of one transformer layer is consistent with the following pseudo code: .. code-block:: python >>> # doctest: +SKIP('This is not an example') >>> if pre_layer_norm: ... out = layer_norm(x) ... out = qkv_linear(out) + qkv_bias ... else: ... out = qkv_linear(x) + qkv_bias >>> out = transpose(out, perm=[2, 0, 3, 1, 4]) >>> # extract q, k and v from out. >>> q = out[0:1, ::] >>> k = out[1:2, ::] >>> v = out[2:3, ::] >>> out = q * k^t >>> out = attn_mask + out >>> out = softmax(out) >>> out = dropout(out) >>> out = out * v >>> out = transpose(out, perm=[0, 2, 1, 3]) >>> out = linear(out) >>> if pre_layer_norm: ... out = x + dropout(out + bias) ... else: ... out = layer_norm(x + dropout(out + bias)) >>> residual = out; >>> if pre_layer_norm: ... out = ffn_layer_norm(out) >>> out = ffn1_linear(out) >>> out = dropout(activation(out + ffn1_bias)) >>> out = ffn2_linear(out) >>> out = residual + dropout(out + ffn2_bias) >>> if not pre_layer_norm: ... out = ffn_layer_norm(out) Parameters: embed_dim (int): The expected feature size in the input and output. num_heads (int): The number of heads in multi-head attention(MHA). dim_feedforward (int): The hidden layer size in the feedforward network(FFN). dropout_rate (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.0 activation (str, optional): The activation function in the feedforward network. Default "gelu". normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default True ln_scale_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for Attention layer_norm. For Attention layer_norm weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ln_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for Attention layer_norm. For Attention layer_norm bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. qkv_weight_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for Attention qkv computation. For Attention qkv weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. qkv_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for Attention qkv computation. For Attention qkv bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. linear_weight_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for Attention linear. For Attention linear weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. linear_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for Attention linear computation. For Attention linear bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ffn_ln_scale_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for FFN layer_norm. For FFN layer_norm weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ffn_ln_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for FFN layer_norm. For FFN layer_norm bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ffn1_weight_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for FFN first linear. For FFN first linear weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ffn1_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for FFN first linear. For FFN first linear bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. ffn2_weight_attrs(ParamAttr|list|tuple, optional): To specify the weight parameter property for FFN second linear. For FFN second linear weight, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr`. ffn2_bias_attrs(ParamAttr|list|tuple|bool, optional): To specify the bias parameter property for FFN second linear. For FFN second linear bias, if it is a list/tuple, `attrs[0]` would be used as `attr` for transformer layer 0, and `attrs[1]` would be used as `attr` for transformer layer 1, etc. Otherwise, all layers both use it as `attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. Default: None, which means the default bias parameter property is used. See usage for details in :code:`ParamAttr`. epsilon (float, optional): Small float value added to denominator of the layer_norm to avoid dividing by zero. Default: 1e-05. num_layers (int, optional): The number of layers of the transformer. If `qkv_weight_attrs` is a list or tuple, the number of layers is obtained from `qkv_weight_attrs`. num_layers only takes effect when `qkv_weight_attrs` is not a list or tuple. Default: -1. nranks (int, optional): Distributed tensor model parallel nranks. Default is 1, means not using mp. trans_qkvw (bool, optional): Whether to transpose for weights of qkv. If true, the shape eights of qkv should be [3, num_head, dim_head, dim_embed]. Otherwise the shape of weights of qkv should be [dim_embed, 3, num_head, dim_head]. Default: True. ring_id (int, optional): For distributed tensor model parallel. Default is -1, means not using mp. name (str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> from paddle.incubate.nn import FusedMultiTransformer >>> paddle.device.set_device('gpu') >>> # encoder input: [batch_size, src_len, d_model] >>> enc_input = paddle.rand((2, 4, 128)) >>> # self attention mask: [batch_size, 1, src_len, src_len] >>> attn_mask = paddle.rand((2, 1, 4, 4)) >>> encoder_layers = FusedMultiTransformer(128, 2, 512, num_layers=1) >>> enc_output = encoder_layers(enc_input, attn_mask) >>> print(enc_output.shape) [2, 4, 128] geluTNr#rSrc3st|dksJd||dksJd||dks&Jd|||_|j|_||_||_||_ ||_ ||_ |||_ |j ||ksNJd|dkrX|dksXJ||dks`J||dkshJ||}||}||_ t| ttfr~t| dksJgg|_|_gg|_|_gg|_|_gg|_|_gg|_|_gg|_|_fdd }tD]<}|||}|||}|| |}|| |}|| |}|| |} || |}!|||}"|||}#|||}$|||}%|||}&|j||gt d d d }'|j||gd d}(|j|rd||j |gn|d||j g||jdd})|jd||j g||jd d}*|j||j |g||jdd}+|j|g| |jd d},|j|g|!dt d d}-|j|g|"d d}.|j||g|#|jdd}/|j|g|$|jd d}0|j||g|%|jdd}1|j|g|&|jd d}2|dkrt!|)t!|*t!|/t!|0t!|+t!|1|j"|'|j"|(|j"|)|j"|*|j"|+|j"|,|j"|-|j"|.|j"|/|j"|0|j"|1|j"|2q||_#||_$||_%dS)Nrr$rTrrUrrScs*t|ttfrt|ksJ||S|Sru) isinstancelisttuplelen)attrsidx num_layersrrget_attrsz0FusedMultiTransformer.__init__..get_attrr(r)r*Tr,rVFr%rr)&r-r.rJrWr/r0r1r8 _trans_qkvwrXr2rYrZrrrrr ln_scales ln_biases qkv_weights qkv_biaseslinear_weights linear_biases ffn_ln_scales ffn_ln_biases ffn1_weights ffn1_biases ffn2_weights ffn2_biasesranger3r rrvr7rr )3r9r2rYrr7rrWZln_scale_attrsZ ln_bias_attrsZqkv_weight_attrsZqkv_bias_attrsZlinear_weight_attrsZlinear_bias_attrsZffn_ln_scale_attrsZffn_ln_bias_attrsZffn1_weight_attrsZffn1_bias_attrsZffn2_weight_attrsZffn2_bias_attrsr<rrm trans_qkvwrnr rirkrlrerfrgrhZffn_ln_scale_attrZffn_ln_bias_attrZffn1_weight_attrZffn1_bias_attrZffn2_weight_attrZffn2_bias_attrr5r6r_r`rar4Z ffn_ln_scaleZ ffn_ln_biasZ ffn1_weightZ ffn1_biasZ ffn2_weightZ ffn2_biasr=rrr.s                                      zFusedMultiTransformer.__init__rc Cs|durt|t|jksJtj||j|j|j|j|j|j|j |j |j |j |j |jf id|jd|jd|d|d|d|d|d |d |jd |d |jd |jddd|jd|jd|j} | S)am Applies multi transformer layers on the input. Parameters: src (Tensor): The input of Transformer layers. It is a tensor with shape `[batch_size, sequence_length, d_model]`. The data type should be float16 or float32. attn_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape `[batch_size, 1, sequence_length, sequence_length]`. It can be None when nothing wanted or needed to be prevented attention to. Default None. caches (list(Tensor)|tuple(Tensor), optional): The cache structure tensors for the inference generation model. It is only used for inference and should be None for training. The shape is `[2, batch_size, num_head, max_seq_len, head_dim]`. Default None. pre_caches (list(Tensor)|tuple(Tensor), optional): The prefix caches for the generation model. The shape is `[2, bsz, num\_head, cache\_len, head\_dim]`. Default None. rotary_embs (Tensor optional): The RoPE embs for the rotary computation. The shape is `[2, bsz, 1, seq\_len, head\_dim]`. Default None. rotary_emb_dims (int, optional): The rotary_emb_dims of rotary computation, and it is 0 when rotary_embs is None, 1 when rotary_embs is not None and pos_extra_ids is None, 2 when rotary_embs and pos_extra_ids are both not None. Default 0. seq_lens (Tensor optional): The sequence lengths of this batch. The shape is `[bsz]`. Default None. time_step (Tensor, optional): The time step tensor for the generation model. Which used in decode stage, to represent the time step, that is, the real seq_len of CacheKV. The shape is `[1]`, must be in CPUPlace. Default None. Returns: Tensor|tuple: If `caches` is None, return a tensor that has the same shape and data type with `src`, representing the output of Transformer layers. If `caches` is not None, return the tuple (output, caches), which output is the output of Transformer layers, caches is inplace with input `caches`. Nror<Z cache_kvs pre_caches rotary_embs time_stepseq_lensrpr7rotary_emb_dimsrrCrDr?rrnr )rrrEZfused_multi_transformerrrrrrrrrrrrrWr8r7rrCrrXr ) r9rrpcachesrrrrrrFrrrrGgsd/zFusedMultiTransformer.forward)rrTNNNNNNNNNNNNr#rSrTrSN)NNNNrNNrrrr=rrsD)Fr)numpyrr Z paddle.baserZpaddle.base.corerZpaddle.base.dygraphrZpaddle.base.frameworkrZpaddle.frameworkrZpaddle.incubate.nnrrEZ paddle.nnrZpaddle.nn.initializerr Zpaddle.nn.layer.transformerr r rr r!rRrrrrrrrrs.         %q4c2x