o 0j6@sddlZddlmZddlZddlmmZddlmZmZGdddej Z Gdddej Z Gdd d ej Z Gd d d ej Z Gd d d ej ZGdddej ZGdddej ZGdddej ZGdddeZide dedddfde de de de dd ifd!ed"ed#e d$ejd%ejd&ejd'ejd(ejd)ed*ed+ejZeeZd,d-ZedZedZedZedZed#Z ed)Z!ed"Z"ed!Z#dS).N) OrderedDict)Tensornnc@"eZdZdZdedefddZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 inputreturnc Cs6d|dttdtj|dt|dS)N??@Hm?g@)paddletanhmathsqrtpipowselfrry/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/models/common/transformers/activations.pyforwardszNewGELUActivation.forwardN__name__ __module__ __qualname____doc__rrrrrrrrcsLeZdZdZd deffdd Zdedefdd Zdedefd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 Fuse_gelu_pythoncs(t|r |j|_dStjj|_dSN)super__init__ _gelu_pythonactr functionalgelu)rr __class__rrr"3s  zGELUActivation.__init__rrcCs |ddt|tdS)Nr r r )r erfrrrrrrr#:s zGELUActivation._gelu_pythoncCs ||Sr )r$rrrrr= zGELUActivation.forward)F) rrrrboolr"rr#r __classcell__rrr'rr+s rc@r)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcCs*d|dt|ddd||S)Nr r g3E?r )r rrrrrrFszFastGELUActivation.forwardNrrrrrr-Ar-c@r)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrcCs|td|S)NgZd;?)FsigmoidrrrrrVszQuickGELUActivation.forwardNrrrrrr/Qr.r/cs<eZdZdZdedeffdd Zdedefdd ZZS) ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr!r"r3r4)rr3r4r'rrr"gs   zClippedGELUActivation.__init__xrcCstt||j|jSr )r Zclipr&r3r4)rr7rrrroszClippedGELUActivation.forward) rrrrfloatr"rrr,rrr'rr2Zs r2c@r)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rrcC t|Sr )r0silurrrrr|r*zSiLUActivation.forwardNrrrrrr9ssr9c@r)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish rrcCr:r )r0mishrrrrrr*zMishActivation.forwardNrrrrrr<rr<c@r)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrcCs|Sr rrrrrrszLinearActivation.forwardNrrrrrr>r.r>cseZdZfddZZS)ClassInstantiercs4t|}t|tr |n|if\}}|di|S)Nr)r! __getitem__ isinstancetuple)rkeycontentclskwargsr'rrr@s zClassInstantier.__getitem__)rrrr@r,rrr'rr?sr?r&Zgelu_10i )r3r4 gelu_fastgelu_newZgelu_pytorch_tanh gelu_pythonrTZlinearr= quick_geluZ hardswishZpreluZreluZrelu6r1r;ZswishrcCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)Zactivation_stringrrrget_activations rP)$r collectionsrr Zpaddle.nn.functionalrr%r0rZLayerrrr-r/r2r9r<r>r?Z HardswishZPReLUZReLUZReLU6ZSigmoidZTanhZACT2CLSrLrPrJrIr&rHrKr;r=Z linear_actrrrrsv