o *j*3@shddlZddlZddlmZddlmmZGdddejZGdddejZ GdddejZ dS)Ncs@eZdZ     d fdd ZddZd d Zd d ZZS) SelfAttention'Ncstt|ddg|_|dur$|}||jvs$Jdg|jR||_||_||_||_||_ t t t |_ |_ |_t|jD],}|j t j|||dd|j t j|||dd|jt j|||ddqIt j||dd|_t jdd|_t jd d |_dS) a The basic (multi-head) Attention 'cell' containing the learnable parameters of Q, K and V :param int input_size: Feature input size of Q, K, V. :param int output_size: Feature -hidden- size of Q, K, V. :param int freq: The frequency of the sinusoidal positional encoding. :param int heads: Number of heads for the attention module. :param str | None pos_enc: The type of the positional encoding [supported: Absolute, Relative]. absoluterelativeNzSupported encodings: F) in_features out_featuresZbiasdim?p)superr__init__Zpermitted_encodingslower input_size output_sizeheadspos_encfreqnn ModuleListWkWqWvrangeappendLinearoutZSoftmaxsoftmaxDropoutdrop)selfrrrrr_ __class__q/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/cv/video_summarization/pgl_sum.pyr sP   zSelfAttention.__init__cCs|j}|j}tjddt|D|jjjd}tjddt|dD|jjjd}||j dd}|j |j ddd}| |j dd}tj |||jjjd}t ||d||||d|f<t||d||||d|df<|S) a.Calculate the sinusoidal positional encoding based on the absolute position of each considered frame. Based on 'Attention is all you need' paper (https://arxiv.org/abs/1706.03762) :param int T: Number of frames contained in Q, K and V :return: Tensor with shape [T, T] cSg|]}|qSr(r(.0kr(r(r) Iz5SelfAttention.getAbsolutePosition..devicecSr*r(r(r+r(r(r)r.Kr/rrr )rrtorchtensorrr weightr1reshapeshaperepeat_interleaverepeatzerossincos)r$TrdposiAPr(r(r)getAbsolutePosition?s&*z!SelfAttention.getAbsolutePositionc Cs|j}d|}|d }tjddt|D|jjjd}tjddt|D|jjjd}||jdd}|j |jddd}| |jdd}|||}tj |||jjjd}tjd dt|dD|jjjd} t |d d d| f||d d d| f|d d d| f||d d d| f<t |d d d| df||d d d| df|d d d| df||d d d| df<|S) a+Calculate the sinusoidal positional encoding based on the relative position of each considered frame. r_pos calculations as here: https://theaisummer.com/positional-embeddings/ :param int T: Number of frames contained in Q, K and V :return: Tensor with shape [T, T] r2rcSr*r(r(r+r(r(r)r.cr/z5SelfAttention.getRelativePosition..r0cSr*r(r(r+r(r(r)r.dr/rr cSr*r(r(r+r(r(r)r.or/N)rr3r4rr r5r1r6r7r8r9r:r;r<) r$r=rr>Zmin_rposr@jZr_posRPidxr(r(r)getRelativePositionXs, "" B6z!SelfAttention.getRelativePositionc Csg}t|jD]`}|j||}|j||}|j||}t||dd}|jdurR|jdkr@|j |j dd}||}n|jdkrR|j |j dd} || }| |} | | } t| |} || q|tj|dd} | | fS)a Compute the weighted frame features, based on either the global or local (multi-head) attention mechanism. :param torch.tensor x: Frame features with shape [T, input_size] :return: A tuple of: y: Weighted features based on the attention weights, with shape [T, input_size] att_weights : The attention weights (before dropout), with shape [T, T] rrNr)r=rr )rrrrrr3matmulZ transposerrBr7rFr!r#rr catclone) r$xZoutputsheadKQVZenergiesrArDZ att_weightsZ _att_weightsyr(r(r)forwardxs&         zSelfAttention.forward)rrrrN)__name__ __module__ __qualname__rrBrFrP __classcell__r(r(r&r)r s2 rc4eZdZ       d fdd ZddZZS) MultiAttentionrrNrc stt|t|||||d|_||_|jdur>|jdks"Jdt|_t |jD]}|j t|||||ddq,gd|_ ||_ |j durb|j |_ |j |j vsdJdg|j RdSdS)a} Class wrapping the MultiAttention part of PGL-SUM; its key modules and parameters. :param int input_size: The expected input feature size. :param int output_size: The hidden feature size of the attention mechanisms. :param int freq: The frequency of the sinusoidal positional encoding. :param None | str pos_enc: The selected positional encoding [absolute, relative]. :param None | int num_segments: The selected number of segments to split the videos. :param int heads: The selected number of global heads. :param None | str fusion: The selected type of feature fusion. )rrrrrNr2znum_segments must be None or 2+)addmultavgmaxzFusion method must be: )rrVrr attention num_segmentsrrlocal_attentionrrZpermitted_fusionsfusionr) r$rrrrr]rr_r%r&r(r)rs:     "zMultiAttention.__init__c CsT||\}}|jdur|jdurt|jd|j}t|jD]}||}|d|}|||}|j||\} } tj ||| ddd|||<tj | ddd} |jdkrf|||| 7<q!|jdkrv|||| 9<q!|jdkr|||| 7<|||d<q!|jd krt ||| | |||<q!||fS) a Compute the weighted frame features, based on the global and locals (multi-head) attention mechanisms. :param torch.Tensor x: Tensor with shape [T, input_size] containing the frame features. :return: A tuple of: weighted_value: Tensor with shape [T, input_size] containing the weighted frame features. attn_weights: Tensor with shape [T, T] containing the attention weights. Nrrr2)rr rXrYrZr[) r\r]r_mathceilr7rr^F normalizerIr3r[) r$rJweighted_value attn_weightsZ segment_sizesegmentZleft_posZ right_posZlocal_xZweighted_local_valueZattn_local_weightsr(r(r)rPs@       zMultiAttention.forwardrrrNNrNrQrRrSrrPrTr(r(r&r)rVs/rVcrU) PGL_SUMrrNrc stt|t|||||||d|_tj||d|_tj|jjdd|_ tj dd|_ tj |dd|_ tj |jjdd|_t|_t|_dS) al Class wrapping the PGL-SUM model; its key modules and parameters. :param int input_size: The expected input feature size. :param int output_size: The hidden feature size of the attention mechanisms. :param int freq: The frequency of the sinusoidal positional encoding. :param None | str pos_enc: The selected positional encoding [absolute, relative]. :param None | int num_segments: The selected number of segments to split the videos. :param int heads: The selected number of global heads. :param None | str fusion: The selected type of feature fusion. )rrrrr]rr_)rr rr rgư>)Znormalized_shapeepsN)rrirrVr\rrlinear_1r linear_2r"r#Z LayerNormnorm_y norm_linearZReLUreluZSigmoidsigmoid)r$rrrrr]rr_r&r(r)rs. zPGL_SUM.__init__cCs|d|jd}|}||\}}||}||}||}||}||}||}||}||}| |}| dd}||fS)a Produce frames importance scores from the frame features, using the PGL-SUM model. :param torch.Tensor frame_features: Tensor of shape [T, input_size] containing the frame features produced by using the pool5 layer of GoogleNet. :return: A tuple of: y: Tensor with shape [1, T] containing the frames importance scores in [0, 1]. attn_weights: Tensor with shape [T, T] containing the attention weights. r r) r6r7r\r#rmrkrornrlrpview)r$Zframe_featuresZresidualrdrerOr(r(r)rPs          zPGL_SUM.forwardrgrhr(r(r&r)ris(ri) r`r3Ztorch.nnrZtorch.nn.functionalZ functionalrbModulerrVrir(r(r(r)s X