o #j)@sddlmZmZddlmZddlmZddlmZddl m Z m Z m Z ddZ dd d Zdd dZ dddZ dddZd S))_C_opsversion) check_dtype)convert_np_dtype_to_dtype_)get_device_capability) LayerHelperin_dynamic_modein_dynamic_or_pir_modecCs>t}|dur|dkrt\}}t|d|}|Std)NFalse zmPaddle is not compiled with CUDA, we cannot get SMVersion from device, please try to compile Paddle with CUDA)rcudarint ValueError)Z cuda_versionmajorminorarchra/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/nn/quant/quantized_linear.py_get_arch_infos rweight_only_int8NcCs|durt}|dks|dks|dks|dksJd|dtr)t|||Sd}t|fit}|d }|d }|j|d |i||d ||d d||fS)a Quantization function for weight_only and llm.int8's weight. Args: x (Tensor): The input Tensor to be quantized, the data type is float16 or bfloat16. algo (str): The algo that is x will be apply, must be one of 'weight_only_int8', 'weight_only_int4' and 'llm.int8', default: 'weight_only_int8'. arch (int): The compute arch for target device. For example, A100 is 80, v100 is 70, if you do not assign arch, we will get arch from your device, default: None. Returns: out (Tensor): The Tensor which is the quantitative results, the data type is int8, the shape is transposition of x. scale (Tensor): The scale Tensor which is the scale of pre-channel, the data type is float32. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_quantize >>> paddle.seed(2023) >>> x = paddle.rand(shape=[64, 32], dtype=paddle.float16) >>> out, scale = weight_quantize(x, algo='weight_only_int8') >>> print(out.shape) [32, 64] >>> print(scale.shape) [32] NFPVK>Currently weight_quantize only support SM70/75/80/86. but got  weight_quantizeint8floatx)outscale)algortypeinputsZoutputsattrs)rrrrrlocals"create_variable_for_type_inference append_op)rr"rr$helperr r!rrrr's$"   rfloat16cCsvt|dddgdt|}trt||||Sd}t|fit}||}|j|||dd|i||dd|S) a Dequantization function for weight_only and llm.int8's weight. Args: x (Tensor): The input Tensor to be dequantized, the data type is int8. scale (Tensor): The scale Tensor which is the output of weight_quantize, the data type is float32. algo (str): The algo that is x will be apply, must be one of 'weight_only_int8', 'weight_only_int4' and 'llm.int8', default: 'weight_only_int8'. out_dtype (str|np.dtype): The output Tensor's data type, must be one of 'float16' and 'bfloat16', default: 'float16'. Returns: out (Tensor): The Tensor which is the dequantitative results, the data type is float16 or bfloat16, the shape is transposition of x. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_quantize, weight_dequantize >>> paddle.seed(2023) >>> x = paddle.rand(shape=[64, 32], dtype=paddle.float16) >>> out, scale = weight_quantize(x, algo='weight_only_int8') >>> x_dequant = weight_dequantize(out, scale) out_dtyper+Zbfloat16weight_dequantize)rr!r )r"r,r#) rrrrr-rr'r(r))rr!r"r,r$r*r rrrr-[s   r-rc Cs|durt}|dks|dks|dks|dksJd|dtr.t||||||}|St|dd d gd d }t|fit}|j} |g|g|gd } |durW|g| d <||d} || }|j || d|i| d|S)a Applies matrix multiplication of two tensors and then bias addition if provided. This method requires CUDA version >= 11.2. Args: x (Tensor): The first input Tensor to be multiplied, the data type is float16 or bfloat16. weight (Tensor): The second input Tensor to be multiplied. Its rank must be 2. bias (Tensor|None): The input bias Tensor. If it is None, no bias addition would be performed. Otherwise, The bias is added to the matrix multiplication result. weight_scale (Tensor|None): The input scale Tensor Provided to weight for dequantization. Its rank must be 1. weight_dtype(str): The dtype of weight Tensor, must be one of 'int8', 'int4', Defaulted to 'int8'. arch (int): The compute arch for target device. For example, A100 is 80, v100 is 70, if you do not assign arch, we will get arch from your device, default: None. Returns: Tensor: the output Tensor, the data type is the same as that of x. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import weight_only_linear >>> x = paddle.cast(paddle.randn([1, 2, 64]), dtype='float16') >>> weight = paddle.cast(paddle.randint(0, 127, [32, 64]), dtype='int8') >>> scale = paddle.randn([32], dtype='float32') >>> bias = paddle.cast(paddle.randn([32]), dtype='float16') >>> if paddle.device.cuda.get_device_capability()[0] >= 8: ... out = weight_only_linear(x, weight, bias=bias, weight_scale=scale, weight_dtype='int8') ... print(out.shape) [1, 2, 32] Nrrrrrr weight_dtyperZint4weight_only_linearrweight weight_scalebias)r.rr r#) rrrr/rrr'dtyper(r)) rr1r3r2r.rr r$r*r4r%r&rrrr/s>""      r/@c Cstrt|||||}|Sd}t|fit}|j}|g|g|gd} |r,|g| d<d|i} ||}|j|| d|i| d|S)a Applies matrix multiplication of two tensors and then bias addition if provided. This method requires CUDA version >= 11.2. Args: x (Tensor): the first input Tensor to be multiplied, the data type is float16 or bfloat16. weight (Tensor): the second input Tensor to be multiplied. Its rank must be 2. bias (Tensor|None): the input bias Tensor. If it is None, no bias addition would be performed. Otherwise, the bias is added to the matrix multiplication result. weight_scale (Tensor|None): the input scale Tensor Provided to weight for dequantization. Its rank must be 1. threshold(float): The min value of outlier in activation, outlier's channel will be apply multiply with x.dtype. Returns: Tensor: the output Tensor, the data type is the same as that of x. Examples: .. code-block:: python >>> # doctest: +SKIP('No testing required') >>> import paddle >>> from paddle.nn.quant import llm_int8_linear >>> x = paddle.cast(paddle.randn([1, 2, 64]), dtype='float16') >>> weight = paddle.cast(paddle.randint(0, 127, [32, 64]), dtype='int8') >>> scale = paddle.randn([32], dtype='float32') >>> bias = paddle.cast(paddle.randn([32]), dtype='float16') >>> if paddle.device.cuda.get_device_capability()[0] >= 8: ... out = llm_int8_linear(x, weight, bias=bias, weight_scale=scale, threshold=6.0) ... print(out.shape) [1, 2, 32] llm_int8_linearr0r3 thresholdr r#)r rr6rr'r4r(r)) rr1r3r2r7r r$r*r4r%r&rrrr6s*&  r6)rN)rr+)NNrN)NNr5)ZpaddlerrZpaddle.base.data_feederrZpaddle.base.frameworkrZpaddle.device.cudarZpaddle.frameworkrrr rrr-r/r6rrrrs    4/ M