o #jK@sddlZddlmmZddlmZmZddlmZddl m Z da da da e dddZd d Z   dd d ZddZddZ  ddddddddZ      dddZ    dddZdS)N)_C_opsin_dynamic_mode) LayerHelper)signature_safe_contextmanagerFTccsBt}t}t}|a|a|az dVW|a|a|adS|a|a|aw)z With the sdp_kernel context manager, different algorithm implementations can be selected for scaled_dot_product_attention. N) g_enable_mathg_enable_mem_efficientg_enable_flash)Z enable_mathZ enable_flashZenable_mem_efficientZoriginal_enable_mathZoriginal_enable_flashZoriginal_enable_mem_efficientr e/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/nn/functional/flash_attention.py sdp_kernelsr cCs*t|d}d|_tj|dd}d|_|S)NgT)Zdiagonal)paddleZ full_likeZ stop_gradientZtriu)xmaskr r r get_triangle_upper_mask1s rc Cs|jd}t|gd}t|gd}t|gd}tj||d|dd}|s1t|} nt} d| vrGt|} || }t|} ntj |} |dkrZtj | ||dd } t| |} t| gd} | |rn| fSd fS) z This is a basic implementation of scaled dot product attention composed of combinations of fundamental components. )rr gT)ryZ transpose_yZxpurZupscale_in_train)trainingmodeN) shaper Z transposematmulFsoftmax get_devicerZincubateZ softmax_mask_fuse_upper_triangledropout) querykeyvalue dropout_ratecausalreturn_softmaxrhead_dimproductweightsplaceroutr r r _math_attention9s,     r)cCs|dkrdSdS)N flash_attn mem_efficientr )r$r r r _select_sdp_cudadsr-cCst}tdurd|vrdSt|Stdur"tdur"tdur"tdtdur6tdur0tdur0dSd|vr6dStdurBtdurBt|StdurHdSdS) z There are currently three different implementation options available for scaled dot product attention, and the chosen approach depends on whether it is determined by the sdp_kernel configuration or specified through input values. NZgpumathFz@No available backend for scaled_dot_product_attention was found.Tr+r,)r rrr-rrAssertionError)r$r'r r r _select_sdpks*r0)fixed_seed_offsetrng_namernamec Cs6|jd} t| } | dkrwtr,t||||d|||| | \} } }}| |r)| fSdfStdit}|jdd}||}||}|t j }|t j }||||d}||||d}|j d|||||| |dd ||rt|fSdfS| d krd d l m}||||d|d|d }|dfSt|||||||dS)a4 The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API is only support inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. key(Tensor): The key tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. value(Tensor): The value tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. dropout(float): The dropout ratio. causal(bool): Whether enable causal mode. return_softmax(bool): Whether to return softmax. fixed_seed_offset(Tensor, optional): With fixed seed, offset for dropout mask. training(bool): Whether it is in the training phase. rng_name(str): The name to select Generator. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: out(Tensor): The attention tensor. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax(Tensor): The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((1, 128, 2, 16)) >>> output = paddle.nn.functional.flash_attention.flash_attention(q, q, q, 0.9, False, False) >>> print(output) (Tensor(shape=[1, 128, 2, 16], dtype=float32, place=Place(cpu), stop_gradient=True, [[[[0.34992966, 0.34456208, 0.45826620, ..., 0.39883569, 0.42132431, 0.39157745], [0.76687670, 0.65837246, 0.69117945, ..., 0.82817286, 0.76690865, 0.71485823]], ..., [[0.71662450, 0.57275224, 0.57053083, ..., 0.48108247, 0.53336465, 0.54540104], [0.59137970, 0.51350880, 0.50449550, ..., 0.38860250, 0.40526697, 0.60541755]]]]), None) rr+NqZinput_param_name)r5kvr2r(r softmax_lse seed_offsetrr"r#is_testr3typeinputsoutputsattrsr,r)memory_efficient_attention)Z attn_biaspscaler)r!r"r#rr+)rr0rrr+rlocals input_dtype"create_variable_for_type_inferencer float32int64 append_opZ-paddle.incubate.nn.memory_efficient_attentionrCr))rrr rr"r#r2r3rr4r$Z sdp_func_nameresult_attentionresult_softmax_helperdtyper(rr:r;r@rArCoutputr r r flash_attentions N          rScCstr"t|||||| d||||| | | | \}}|| r|fSdfStd it}|jdd}||}||}|tj}|tj }|||||| d}||||d}|j d||||||| | | | dd|| ro|fSdfS) ag The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API is only support inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. key(Tensor): The key tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. value(Tensor): The value tensor in the Attention module. 3-D tensor with shape: [total_seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. cu_seqlens_q(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index query. cu_seqlens_k(Tensor): The cumulative sequence lengths of the sequences in the batch, used to index key and value. max_seqlen_q(int): Maximum sequence length of query in the batch. max_seqlen_k(int): Maximum sequence length of key/value in the batch. scale(float): The scaling of QK^T before applying softmax. dropout(float): The dropout ratio. causal(bool): Whether enable causal mode. return_softmax(bool): Whether to return softmax. fixed_seed_offset(Tensor, optional): With fixed seed, offset for dropout mask. rng_name(str): The name to select Generator. training(bool): Whether it is in the training phase. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: out(Tensor): The attention tensor. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. softmax(Tensor): The softmax tensor. None if return_softmax is False. Examples: .. code-block:: python >>> import paddle >>> paddle.seed(2023) >>> q = paddle.rand((2, 128, 8, 16), dtype='float16') >>> cu = paddle.arange(0, 384, 128, dtype='int32') >>> qq = paddle.reshape(q, [256, 8, 16]) >>> output = paddle.nn.functional.flash_attention.flash_attn_unpadded(qq, qq, qq, cu, cu, 128, 128, 0.25, 0.0, False, False) Nflash_attn_unpaddedr5r6)r5r7r8 cu_seqlens_q cu_seqlens_kr2r9) max_seqlen_q max_seqlen_krErr"r#r=r3r>)rT) rrrTrrGrHrIr rJrKrL)rrr rUrVrWrXrErr"r#r2r3rr4rMrNrPrQr(rr:r;r@rAr r r rT*snN      rTc Cs|durt|||||\}} |Str.d} d} d} t|||| |||| | | \}} } } |Std it} | jdd}| |}| |}| tj }| tj }||||d}||||d }| j d||||d| dd d |S) a The equation is: .. math:: result=softmax(\frac{ Q * K^T }{\sqrt{d}}) * V where : ``Q``, ``K``, and ``V`` represent the three input parameters of the attention module. The dimensions of the three parameters are the same. ``d`` represents the size of the last dimension of the three parameters. Warning: This API only supports inputs with dtype float16 and bfloat16. Args: query(Tensor): The query tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. key(Tensor): The key tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. value(Tensor): The value tensor in the Attention module. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float61 or bfloat16. attn_mask(Tensor,optional): A float mask of the same type as query, key, value that is added to the attention score. dropout_p(float): The dropout ratio. is_causal(bool): Whether enable causal mode. training(bool): Whether it is in the training phase. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Returns: out(Tensor): The attention tensor. 4-D tensor with shape: [batch_size, seq_len, num_heads, head_dim]. The dtype can be float16 or bfloat16. Examples: .. code-block:: python >>> # doctest: +SKIP('bfloat need V100 compile') >>> import paddle >>> q = paddle.rand((1, 128, 2, 16), dtype=paddle.bfloat16) >>> output = paddle.nn.functional.scaled_dot_product_attention(q, q, q, None, 0.9, False) >>> print(output) >>> # doctest: -SKIP N)NFr1r+r5r6)r5r7r8 attn_maskr9r<r>rF) rSrrr+rrGrHrIr rJrKrL)rrr rYZ dropout_pZ is_causalrr4r(rOr2r#r3rPrQrr:r;r@rAr r r scaled_dot_product_attentionsh>      rZ)FTT)rFFT)rFF)rFFNr1TN)NrFTN)r Zpaddle.nn.functionalnnZ functionalrrrZpaddle.base.layer_helperrZpaddle.base.wrapped_decoratorrrrrr rr)r-r0rSrTrZr r r r sV     +' &