o *j4@s ddlZddlmZddlZddlmZddlmmZ ddl m Z ddl m Z ddlmZddlmZddlmZmZddlmZdd lmZmZejeje jd Gd d d e ZGd dde ZGddde ZGddde Z Gddde Z!Gddde Z"dS)N)Models) TorchModel)MODELS)Config) ModelFileTasks) load_clip)get_state_dictset_seed) module_namecsBeZdZdZdeffdd Zd ddZdd Zd d d ZZ S) VoPa The implementation of 'VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval'. This model is dynamically initialized with the following parts: - clip: the upstream pre-trained backbone model (CLIP in this code) - pool_frames: the frames pooling method - visual_prompt_learner: visual prompt - ImageEncoder: get image encoder - TextPromptLearner: text prompt - TextEncoder: get text encoder model_dircstt|t|d}t|d}t|tj}t|j |_ t |d|_ t t|j jjj|j _t t|j jj|j _t|j j|j |_t|j |j |_t|j |j |_t|j |j |_t|j |j |_|t ||!dt"j#d<t$|j j%dS)zl Initialize a VoP Model Args: model_dir: model id or path, zVoP_msrvtt9k.pthz ViT-B-32.pt)namefalseZTOKENIZERS_PARALLELISMN)&superr __init__ospjoinrZ CONFIGURATIONr from_fileZ hyperparamconfigr Zcliplistrangevisual transformerlayers vpt_layers tpt_layersBaselinePooling pooling_type pool_framesVisualPromptLearnervisual_prompt_learner ImageEncoder image_encoderTextPromptLearnertext_prompt_learner TextEncoder text_encoderZload_state_dictr evalosenvironr seed)selfrargskwargsZ model_pathZ clip_archZ config_path __class__i/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/cv/vop_retrieval/model.pyr!s.    z VoP.__init__FcCsv|jd}|dd|jj|jj}|}|||}||jddd}|||jjd}|d|}|r9||fS|S)z Get video Features Args: videos: the dim is [1, 12, 3, 224, 224] return_all_frames: default False rTdimZkeepdimN) shapereshaper input_resr"r$norm num_framesr )r-Zvideosreturn_all_frames batch_size video_datavisual_promptsvideo_featuresvideo_features_pooledr2r2r3get_video_featuresFs   zVoP.get_video_featurescCs*|}|||}||jddd}|S)zh Get Text Features Args: text_data: the dim is [1, 69] r4Tr6)r&r(r;)r- text_data text_prompts text_featuresr2r2r3get_text_featuresas  zVoP.get_text_featuresc Cs|djd}|d}|d}|dd|jj|jj}|}|||}|}|||} | | jddd} ||jddd}|||jj d}| | |} |rW| || fS| | fS)z Dynamic Forward Function of VoP Args: data: the input data return_all_frames: default False Zvideortextr4r5Tr6) r8r9rr:r"r$r&r(r;r<r ) r-datar=r>rDr?r@rArErFrBr2r2r3forwardos.    z VoP.forward)F) __name__ __module__ __qualname____doc__strrrCrGrJ __classcell__r2r2r0r3r s  %r cs0eZdZdZfddZddZddZZS)rz( Redefined Pooling Function cs&tt||dkr|j|_dSt)NZavg)rrr _avg_pooling pooling_funcNotImplementedError)r-rrr0r2r3rs zBaselinePooling.__init__cCs|jdd}|S)a$ Pooling mean of frames Args: text_embeds: the input text embedding which is None here. video_embeds: the input video embedding with [1, 12, 512]. Returns: video_embeds_pooled: num_vids x embed_dim rr7)mean)r- text_embeds video_embedsZvideo_embeds_pooledr2r2r3rQs zBaselinePooling._avg_poolingcCs |||SN)rR)r-rVrWr2r2r3rJs zBaselinePooling.forward)rKrLrMrNrrQrJrPr2r2r0r3rs  rc(eZdZdZfddZddZZS)r!a The implementation of visual prompt. This module is used to define the learnable prompt parameters: the number of tokens is 8, the prompt dimension is 768, and the initialization weight std used is 0.02. csdtt||j}|jjjjd}|j}t j t |j d|||d}t jj|ddt ||_dS)Nrrdtype{Gz?Zstd)rr!r vp_token_numrln_postweightr8r[torchemptylenrnninitnormal_ Parameterr@)r- clip_modelrr^Zvp_dimr[r@r0r2r3rszVisualPromptLearner.__init__cCs |j}|SrX)r@)r-Zvpr2r2r3rJszVisualPromptLearner.forwardrKrLrMrNrrJrPr2r2r0r3r!s  r!crY)r%a The implementation of visual prompt. This module is used to define the learnable prompt parameters: the number of tokens is 4, the prompt dimension is 512, and the initialization weight std used is 0.02. cstt||j}|j}|dkr|dksJ|jjjd}|j}t j t |j ||||d}t jj|ddt ||_||_||_dS)NrrZr\r])rr%rtp_prefix_token_numtp_suffix_token_numln_finalr`r8r[rarbrcrrdrerfrgrE)r-rhrrjrkZtp_dimr[rEr0r2r3rs   zTextPromptLearner.__init__cCs<|jddd|jddf|jdd|jdddffSrX)rErj)r-r2r2r3rJszTextPromptLearner.forwardrir2r2r0r3r%s r%crY)r#z The implementation of image encoder. This module is used to obtain the features of each frame of the video. csvtt|||_|j|_|j|_|j|_|jj|_|jj |_ |jj |_ |jj |_ |jj |_ |jj |_ |jj|_dSrX)rr#rrrr^r<rconv1class_embeddingpositional_embeddingln_prerr_projr-rhrr0r2r3rs      zImageEncoder.__init__c Cs|jd}||}|||jdd}|ddd}|j|j}tj|d|jd|j|j d}||}tj ||gdd}||j |j}t |j jD]}||jvr|j|}||ddddddf|dd}tj |ddddddf||ddddddfgdd}|dkr||}|ddd}|j j||}|ddd}|d|jvrtj |ddddddf|ddd|jdddfgdd}qM||dddddf}|jdur||j}|S)a  The forward function of image encoder. Args: visual_prompts: the visual prompt, dim is [12, 1, 8, 768] x: the input data, dim is [12, 3, 224, 224] Returns: x: the output data, dim is [12, 512] rrr4)r[devicerTN)r8rmr9permuterntor[raZzerosrtcatrorrrrindexrepeatrp resblocksr^r_rq) r-r@xr>Zx_1Zx_2i_layeri_promptZ cur_layer_vpr2r2r3rJs@   @ <  zImageEncoder.forwardrir2r2r0r3r#s r#crY)r'z The implementation of text encoder. This module is used to obtain the features of each word of the sentence. csvtt||j|_|j|_|j|_|j|_|j|_|j|_|j |_ d|j vs*J|j |_ |j |_ |j |j |_ dS)Nr) rr'rrtoken_embeddingrorltext_projectionr[rrjrk tp_token_numrrr0r2r3r9szTextEncoder.__init__c Cs8|||j}|jd}|\}}t|jjD]}||jvr|j|}|j dkr^|||dddddf |dd} t j |ddddddf| |ddddddfgdd}|j dkr|||dddddf |dd} t j |ddddddf| |ddddddfgdd}|dkr||j|j}|ddd}|jj||}|ddd}|d|jvr|ddddddf} |ddd|j d|j ddf} |ddddddf} t j | | | gdd}|}q|||j}|t |jd|jdd|jf|j}|S)a The forward function of text encoder. Args: text_prompts: the text prompt, dim is 2 x [12, 4, 512] text: the input data, dim is [1, 69] Returns: x: the output data, dim is [1, 512] rrNr4rTrs)r~typer[r8rrrrrxrjexpandrarwrkrorurzrlZarangeZargmaxrr)r-rErHr{r>Z prompt_prefixZ prompt_suffixr|r}Zcur_layer_tp_prefixZcur_layer_tp_suffixZtemp_1Ztemp_2Ztemp_3tempr2r2r3rJHsp     6  4zTextEncoder.forwardrir2r2r0r3r'3s r')#r*Zos.pathpathrraZtorch.nnrdZtorch.nn.functionalZ functionalFZmodelscope.metainforZ'modelscope.models.base.base_torch_modelrZmodelscope.models.builderrZmodelscope.utils.configrZmodelscope.utils.constantrrZbackboner Z basic_utilsr r Zregister_moduleZ vop_retrievalZvop_retrieval_modelr rr!r%r#r'r2r2r2r3s*       }"G