o *j@shddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z Gd d d e ZdS) )AnyDictN)Image) transforms) load_image)ModeKeys)OfaBasePreprocessorcseZdZdZejffdd Zdeee fdeee ffddZ deee fdeee ffdd Z deee fdeee ffd d Z Z S) &OfaVisualQuestionAnsweringPreprocessorz5 OFA preprocessor for question answer tasks. c sftt|j|||g|Ri|tddtj|j|jftjjdt tj |j |j dg|_ dS)zpreprocess the data Args: cfg(modelscope.utils.config.ConfigDict) : model config model_dir (str): model path, mode: preprocessor mode (model mode) cSs |dS)NRGB)convert)imagerw/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/preprocessors/ofa/visual_question_answering.py#s zAOfaVisualQuestionAnsweringPreprocessor.__init__..) interpolation)meanstdN)superr __init__rZComposeZResizeZpatch_image_sizeZInterpolationModeZBICUBICZToTensorZ Normalizerrpatch_resize_transform)selfcfgZ model_dirmodeargskwargs __class__rrrs$   z/OfaVisualQuestionAnsweringPreprocessor.__init__datareturncCs |jtjkr ||S||S)N)rrZTRAIN_build_train_sample_build_infer_sample)rrrrr__call__+s   z/OfaVisualQuestionAnsweringPreprocessor.__call__c Cs||}|jd|dddd}|jdkr,t|j|g}t|dd|jg}n<|jdkrGt|d |g}t|dd|jg}n!|jd krft|d dd |g}t|dd|jg}nt|j j |dt | d<||d <||d <|j durt t |t |jf}t |t |d}tt |t |dt |D]}|jg|||} |j | } d||| <q||d<|S)a& Building training samples. step 1. Preprocess the data using the logic of `_build_infer_sample` and make sure the label data in the result. step 2. Preprocessing the label data to generate `target` and `prev_output_token`. - add blank in the front out label data and tokenize it as `target` item. - if `prompt_type` is `None`, add the bos token as previous output tokens, add eos tokens as target items. - if `prompt_type` is `src`, concatenate source text input with target item as previous output tokens, remove the bos token and add eos token as target items. - if `prompt_type` is `prev_output`, just like the `prompt_type` is src, the difference is that it will remove the eos token in source text input in this setting. - padding the source item as final target item. step 3. Add constraint mask. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text` and `label`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])`, decoder prompt, label, target previous output tokens and constraint mask. z {}labelF)Zadd_bosZadd_eosnonerNsrcsource prev_outputZprev_output_tokenstargetTconstraint_mask)r! tokenize_textformat prompt_typetorchcatbos_itemZeos_itemNotImplementedError tokenizerZ pad_token_idlenZconstraint_trieZzerosZtgt_dictboolrangeZbostolistZget_next_layer) rrsampleZtgt_itemZprev_output_itemZ target_itemr*Z start_idxiZconstraint_prefix_tokenZconstraint_nodesrrrr 1sJ     z:OfaVisualQuestionAnsweringPreprocessor._build_train_samplecCs|||jd}||}||jd}|||j}|ds&|dn|}|d|}|jdkr9|j}n|jdkrA|}n|jdkrM|dd }nt ||t d g|d }d |jvro|jd |vro||jd |d <|S)a< Building inference samples. step 1. Preprocessing image input for model's image input. - get pillow image from data. - do some transforms to the pillow image, such as resize, normalize etc. step 2. Preprocessing the text input for model's text input. - add blank in the front of input text. - tokenize the result above as source text input. step 3. Calculating the decoder prompt. - if `prompt_type` is `None`, using bos token. - if `prompt_type` is `src`, using source text input - if `prompt_type` is `prev_output`, using source text input without eos token. step 4. Whether or not to add label data which refer to an answer to the question in this task. Args: data (`Dict[str, Any]`): Input data, should contains the key of `image` `text`. Return: A dict object, contains source text input, patch images, patch masks with `Tensor([True])`, decoder prompt and label. r text? r$r%r'Nr(T)r& patch_imageZ patch_maskdecoder_promptZanswerr#) Z get_img_pilZ column_maprZ pre_questionZmax_src_lengthendswithr+r-r0r1r.Ztensor)rrr r<r9Zinputsr=r7rrrr!os*     z:OfaVisualQuestionAnsweringPreprocessor._build_infer_sample)__name__ __module__ __qualname____doc__rZ INFERENCErrstrrr"r r! __classcell__rrrrr s""*>r )typingrrr.ZPILrZ torchvisionrZmodelscope.preprocessors.imagerZmodelscope.utils.constantrbaser r rrrrs