o 0j@sddlZddlZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZd d lmZd dlmZddl m!Z!ej"edGddde!Z#dS)N)AnyDictListOptionalTupleUnion)logging)pipeline_requires_extra) custom_open)ImageBatchSampler) ReadImage) HPIConfigPaddlePredictorOption) benchmark)BaseChat)LayoutParsingResult)PP_ChatOCR_PipelineZiec-seZdZdZdgZdddddddddedeed eed eeeefd ee d e d ee eeefe fde ddffddZ deddfddZdeddfddZdeddfddZdedefddZ                    dede eeeejeejfdee dee dee dee d ee eefd!ee d"ee eeeefefd#eed$eed%eed&eed'eed(eed)eed*eed+eed,eed-eed.eed/eedef,d0d1Zd2ed3eddfd4d5Zd6edeefd7d8Zd9eedeeeeffd:d;Z < =  dfd2ed>ed?ed@e dAedef dBdCZ dgdDed3edAeddfdEdFZ dgd6edAedefdGdHZ!dIe eeefdeefdJdKZ"dLe#dMedIedNedOeddf dPdQZ$dAedRe dDedIeedSed>edefdTdUZ%   <            dhdIe eeefd2eedRe dDed>edVedWedXedYedZed[ed\ed]ed^ed_ed`edAedef$dadbZ&didcddZ'Z(S)jPP_ChatOCRv3_PipelinezPP-ChatOCR PipelinezPP-ChatOCRv3-docNFT)deviceengine engine_config pp_optionuse_hpip hpi_configinitial_predictorconfigrrrrrrrreturnc stjd ||||||d| |d|_||_|dd|_d|_d|_d|_|r9| || || |t dd|_ tdd |_d |_dS) aInitializes the PP-ChatOCRv3 pipeline. Args: config (Dict): Configuration dictionary containing various settings. device (Optional[str], optional): The device to use for prediction. Defaults to `None`. engine (Optional[str], optional): Inference engine. Defaults to `None`. engine_config (Optional[Dict[str, Any]], optional): Engine-specific config. Defaults to `None`. pp_option (Optional[PaddlePredictorOption], optional): Paddle predictor options. Defaults to `None`. use_hpip (bool, optional): Whether to use HPIP. Defaults to `False`. hpi_config (Optional[Union[Dict[str, Any], HPIConfig]], optional): HPIP configuration. Defaults to `None`. initial_predictor (bool, optional): Whether to initialize the predictor. Defaults to `True`. )rrrrrr pipeline_nameuse_layout_parserTNr)Z batch_sizeZBGR)formati)super__init__r!rgetr"layout_parsing_pipelinechat_bot retrieverinintial_visual_predictorinintial_chat_predictorinintial_retriever_predictorr Z batch_samplerrZ img_readertable_structure_len_max) selfrrrrrrrrkwargs __class__r$s/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/pipelines/pp_chatocr/pipeline_v3.pyr&*s.       zPP_ChatOCRv3_Pipeline.__init__cCs<|dd|_|jr|didddi}|||_dS)a Initializes the visual predictor with the given configuration. Args: config (dict): The configuration dictionary containing the necessary parameters for initializing the predictor. Returns: None r"TZ SubPipelinesZ LayoutParserZpipeline_config_errorz)config error for layout_parsing_pipeline!N)r'r"Zcreate_pipeliner()r/rZlayout_parsing_configr$r$r3r+bs   z/PP_ChatOCRv3_Pipeline.inintial_visual_predictorcCs2ddlm}|didddi}|||_dS)a  Initializes the retriever predictor with the given configuration. Args: config (dict): The configuration dictionary containing the necessary parameters for initializing the predictor. Returns: None rcreate_retriever SubModulesZ LLM_RetrieverZretriever_config_errorzconfig error for llm retriever!N)r5r'r*)r/rr5retriever_configr$r$r3r-vs z2PP_ChatOCRv3_Pipeline.inintial_retriever_predictorcCsddlm}|didddi}|||_ddlm}|didid d d i}|||_|didid d d i}|||_dS)a Initializes the chat predictor with the given configuration. Args: config (dict): The configuration dictionary containing the necessary parameters for initializing the predictor. Returns: None rcreate_chat_botr6ZLLM_ChatZchat_bot_config_errorzconfig error for llm chat bot!)create_prompt_engineeringZPromptEngneeringZKIE_CommonTextZpe_config_errorzconfig error for text_pe!Z KIE_Tablezconfig error for table_pe!N)r7r:r'r)r;text_petable_pe)r/rr:chat_bot_configr;Ztext_pe_configZtable_pe_configr$r$r3r,s0       z-PP_ChatOCRv3_Pipeline.inintial_chat_predictorlayout_parsing_resultcCsi}|d}tt|D],}||d}||d}|dvrq d|}||vr-|||<q ||d|7<q |d}g} g} |D]} | | dd | d d } | | qCi} || d <| | d <| | d<| S)z Decodes the visual result from the layout parsing result. Args: layout_parsing_result (LayoutParsingResult): The result of layout parsing. Returns: dict: The decoded visual information. parsing_res_listZ block_labelZ block_content)tableZformulaz words in z table_res_listZ pred_html Ztable_ocr_predZ rec_textsnormal_text_dicttable_text_listtable_html_list)rangelenappendjoin)r/r?rDr@ZpnolabelcontentkeyrBrErFZ table_resZsingle_table_text visual_infor$r$r3decode_visual_results.      z*PP_ChatOCRv3_Pipeline.decode_visual_resultinputuse_doc_orientation_classifyuse_doc_unwarpinguse_seal_recognitionuse_table_recognitionlayout_threshold layout_nmslayout_unclip_ratiolayout_merge_bboxes_modetext_det_limit_side_lentext_det_limit_typetext_det_threshtext_det_box_threshtext_det_unclip_ratiotext_rec_score_threshseal_det_limit_side_lenseal_det_limit_typeseal_det_threshseal_det_box_threshseal_det_unclip_ratioseal_rec_score_threshcks|jdkrtdddiV|jdur td||j|jj|fid|d|d|d |d |d |d |d | d| d| d| d| d|d|d|d|d|d|d|d|D]}||}||d}|VqddS)a& This function takes an input image or a list of images and performs various visual prediction tasks such as document orientation classification, document unwarping, general OCR, seal recognition, and table recognition based on the provided flags. Args: input (Union[str, list[str], np.ndarray, list[np.ndarray]]): Input image path, list of image paths, numpy array of an image, or list of numpy arrays. use_doc_orientation_classify (bool): Flag to use document orientation classification. use_doc_unwarping (bool): Flag to use document unwarping. use_seal_recognition (bool): Flag to use seal recognition. use_table_recognition (bool): Flag to use table recognition. layout_threshold (Optional[float]): The threshold value to filter out low-confidence predictions. Default is None. layout_nms (Optional[bool], optional): Whether to use layout-aware NMS. Defaults to `False`. layout_unclip_ratio (Optional[Union[float, Tuple[float, float]]], optional): The ratio of unclipping the bounding box. Defaults to `None`. If it's a single number, then both width and height are used. If it's a tuple of two numbers, then they are used separately for width and height respectively. If it's None, then no unclipping will be performed. layout_merge_bboxes_mode (Optional[str], optional): The mode for merging bounding boxes. Defaults to `None`. text_det_limit_side_len (Optional[int]): Maximum side length for text detection. text_det_limit_type (Optional[str]): Type of limit to apply for text detection. text_det_thresh (Optional[float]): Threshold for text detection. text_det_box_thresh (Optional[float]): Threshold for text detection boxes. text_det_unclip_ratio (Optional[float]): Ratio for unclipping text detection boxes. text_rec_score_thresh (Optional[float]): Score threshold for text recognition. seal_det_limit_side_len (Optional[int]): Maximum side length for seal detection. seal_det_limit_type (Optional[str]): Type of limit to apply for seal detection. seal_det_thresh (Optional[float]): Threshold for seal detection. seal_det_box_thresh (Optional[float]): Threshold for seal detection boxes. seal_det_unclip_ratio (Optional[float]): Ratio for unclipping seal detection boxes. seal_rec_score_thresh (Optional[float]): Score threshold for seal recognition. **kwargs: Additional keyword arguments. Returns: dict: A dictionary containing the layout parsing result and visual information. Fz1The models for layout parser are not initialized.errorNzGThe layout parsing pipeline is not initialized, will initialize it now.rQrRrSrTrUrVrWrXrYrZr[r\r]r^rbr_r`rarcrd)r?rN) r"r rer(warningr+rpredictrO)r/rPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdr0r?rNZvisual_predict_resr$r$r3visual_predictst ?          z$PP_ChatOCRv3_Pipeline.visual_predictrN save_pathcCst|ts |g}n|}tj|}tj|st|t|d}|t j |dddWddS1s9wYdS)a- Save the visual info list to the specified file path. Args: visual_info (dict): The visual info result, which can be a single object or a list of objects. save_path (str): The file path to save the visual info list. Returns: None wF ensure_ascii N) isinstancelistospathdirnameexistsmakedirsr writejsondumps)r/rNrivisual_info_list directoryfoutr$r$r3save_visual_info_list@s      z+PP_ChatOCRv3_Pipeline.save_visual_info_list data_pathcCsBt|d}|}t|}Wd|S1swY|S)z Loads visual info list from a JSON file. Args: data_path (str): The path to the JSON file containing visual info. Returns: list[dict]: A list of dict objects parsed from the JSON file. rN)r readlinervloads)r/r|findatarxr$r$r3load_visual_info_listXs   z+PP_ChatOCRv3_Pipeline.load_visual_info_listrxc Cstg}g}g}|D],}|d}|D] }||dd||<q|d}|d} |||||| q|||fS)aW Merge visual info lists. Args: visual_info_list (list[dict]): A list of visual info results. Returns: tuple[list, list, list]: A tuple containing four lists, one for normal text dicts, one for table text lists, one for table HTML lists. rDrmr7rErF)replacerIextend) r/rxall_normal_text_listall_table_text_listall_table_html_listZsingle_visual_inforDrMrErFr$r$r3merge_visual_info_listgs     z,PP_ChatOCRv3_Pipeline.merge_visual_info_list ,min_characters block_sizeflag_save_bytes_vectorr8cCsft|ts |g}n|}|durddlm}||}n|jdur*td||j|j}| |} | \} } } i} g}t | D]\}}| D]\}}||d|dg7}qGq?t | | D]\}}t |||jkrr|d|g7}q]d|}d | d <t ||krd | d <|j| d <|| d <|j||d| d<|r|| d| d<d| d <| Sd| d <|| d<| S)a Build a vector representation from visual information. Args: visual_info (dict): The visual information input, can be a single instance or a list of instances. min_characters (int): The minimum number of characters required for text processing, defaults to 3500. block_size (int): The size of each chunk to split the text into. flag_save_bytes_vector (bool): Whether to save the vector as bytes, defaults to `False`. retriever_config (dict): The configuration for the retriever, defaults to `None`. Returns: dict: A dictionary containing the vector info and a flag indicating if the text is too short. Nrr48The retriever is not initialized,will initialize it now.：rmutable：r7Frflag_too_short_text model_namer)rvectorT)rnror7r5r*r rfr-rr enumerateitemsziprHr.rJrZgenerate_vector_databaseencode_vector_store_to_bytes)r/rNrrrr8rxr5r*all_visual_inforrr vector_info all_itemsirDtypetext table_html table_textZ all_text_strr$r$r3 build_vectors\          z"PP_ChatOCRv3_Pipeline.build_vectorrc Cstj|}tj|st||dur ddlm}||}n|jdur0t d| |j |j}t |}|dsM|dsM||d|d<d|d<t|d}|tj|d d d WddS1sjwYdS) Nrr4rrrrTrjFrkrm)rprqrrrsrtr7r5r*r rfr-rcopydeepcopyrr rurvrw) r/rrir8ryr5r*Zvector_info_datarzr$r$r3 save_vectors6          z!PP_ChatOCRv3_Pipeline.save_vectorcCsd}|durddlm}||}n|jdur!td||j|j}t|dG}|}t |}d|vs?d|vs?d|vrOt dd d iWdS|drh| |d|d<d |d<Wd|SWd|S1sswY|S) Nrr4rr}rrrzInvalid vector info.rez%Invalid vector info when load vector!F) r7r5r*r rfr-ropenr~rvrredecode_vector_store_from_bytes)r/r|r8rr5r*rrr$r$r3 load_vectors>           z!PP_ChatOCRv3_Pipeline.load_vectorkey_listcCsL|dkrgSt|tr |St|tr$tdd|}|ddd}|SgS)z Formats the key list. Args: key_list (str|list[str]): A string or a list of strings representing the keys. Returns: list[str]: A list of formatted keys. r7z [\t\n\r\f\v]u，,)rnrostrresubrsplit)r/rr$r$r3 format_keys   z PP_ChatOCRv3_Pipeline.format_keyr)prompt final_resultsfailed_resultsc Cs||}|d}|d}|dur"d|vr|g|d<n|d||dur2td||jfdS||}|D]\} } | |vrP| |vrP|| | || <q;dS)a Generate and merge chat results into the final results dictionary. Args: prompt (str): The input prompt for the chat bot. key_list (list): A list of keys to track which results to merge. final_results (dict): The dictionary to store the final merged results. failed_results (list): A list of failed results to avoid merging. Returns: None rLZreasoning_contentNz.chat bot error: [prompt:] %s [result:] %s )Zgenerate_chat_resultsrIr reZ ERROR_MASSAGEZfix_llm_result_formatrremove) r/r)rrrrZ llm_resultZllm_result_contentZllm_result_reasoning_contentrMvaluer$r$r3generate_and_merge_chat_results's*    z5PP_ChatOCRv3_Pipeline.generate_and_merge_chat_resultsuse_vector_retrievalrcCsD|rq|durq|durddlm}||}n|jdur%td||j|j}dd|D} |d} |ds`|d |jksLJd |d d |jd |d rU|| } |j | | d|d} | St | dkrmd | } | Sd} | Sg} t |D]\} }| D]\}}| |d|dg7} qqwd | } t | |krtd| S)a} Retrieve related normal text based on vector retrieval or all normal text list. Args: retriever_config (dict): Configuration for the retriever. use_vector_retrieval (bool): Whether to use vector retrieval. vector_info (dict): Dictionary containing vector information. key_list (list[str]): List of keys to generate question keys. all_normal_text_list (list): List of normal text. min_characters (int): The minimum number of characters required for text processing, defaults to 3500. Returns: str: Related normal text. Nrr4rcSsg|]}|qSr$r$).0rMr$r$r3 {szAPP_ChatOCRv3_Pipeline.get_related_normal_text..rrrzThe vector model name (z+) does not match the retriever model name (z&). Please check your retriever config.r2)Ztopkrrr7rrmzMThe input text content is too long, the large language model may truncate it.)r7r5r*r rfr-rrrZsimilarity_retrievalrHrJrr)r/r8rrrrrr5r*Zquestion_key_listr related_textrrrDrrr$r$r3get_related_normal_textUsL           z-PP_ChatOCRv3_Pipeline.get_related_normal_texttext_task_descriptiontext_output_formattext_rules_strtext_few_shot_demo_text_content!text_few_shot_demo_key_value_listtable_task_descriptiontable_output_formattable_rules_str table_few_shot_demo_text_content"table_few_shot_demo_key_value_listr>c Csn||}t|dkrddiSt|ts|g}n|}|jdur*td||j|dur9ddl m }||}n|j}| |}|\}}}i}gd}t|dkrt ||D]/\}}t|||j kr|fD]}t|dkr|jj||| | | ||d }||||||qgqWt|dkr|||||||}t|dkr|jj|||||| | d }||||||d|iS) a Generates chat results based on the provided key list and visual information. Args: key_list (Union[str, list[str]]): A single key or a list of keys to extract information. visual_info (dict): The visual information result. use_vector_retrieval (bool): Whether to use vector retrieval. vector_info (dict): The vector information for retrieval. min_characters (int): The minimum number of characters required. text_task_description (str): The description of the text task. text_output_format (str): The output format for text results. text_rules_str (str): The rules for generating text results. text_few_shot_demo_text_content (str): The text content for few-shot demos. text_few_shot_demo_key_value_list (str): The key-value list for few-shot demos. table_task_description (str): The description of the table task. table_output_format (str): The output format for table results. table_rules_str (str): The rules for generating table results. table_few_shot_demo_text_content (str): The text content for table few-shot demos. table_few_shot_demo_key_value_list (str): The key-value list for table few-shot demos. chat_bot_config(dict): The parameters for LLM chatbot, including api_type, api_key... refer to config file for more details. retriever_config (dict): The parameters for LLM retriever, including api_type, api_key... refer to config file for more details. Returns: dict: A dictionary containing the chat results. rZchat_resu Error:输入的key_list无效！Nz;The LLM chat bot is not initialized,will initialize it now.rr9)u大模型调用失败u未知u未找到关键信息Noner7)Ztask_descriptionZ output_formatZ rules_strZfew_shot_demo_text_contentZfew_shot_demo_key_value_list)rrHrnror)r rfr,rr7r:rrr.r=Zgenerate_promptrrr<) r/rrNrrrrrrrrrrrrrr>r8rxr:r)rrrrrrrrZ table_inforrr$r$r3chats -              zPP_ChatOCRv3_Pipeline.chatcOstddS)NzPP-ChatOCRv3-doc Pipeline do not support to call `predict()` directly! Please invoke `visual_predict`, `build_vector`, `chat` sequentially to obtain the result.)r re)r/argsr0r$r$r3rgszPP_ChatOCRv3_Pipeline.predict)NNNNNNNNNNNNNNNNNNNN)rrFN)N)TNrNNNNNNNNNNNN)r N))__name__ __module__ __qualname____doc__entitiesrrrrrboolrrr&dictr+r-r,rrOrnpZndarrayfloatrintrhr{rrorrrrrrrrrrg __classcell__r$r$r1r3r#s   8)*      h   M !"! . F      ~r)$rrvrprtypingrrrrrrnumpyrutilsr Z utils.depsr Zutils.file_interfacer Zcommon.batch_samplerr Z common.readerrmodelsrrZutils.benchmarkrZcomponents.chat_serverrZlayout_parsing.resultrZ pipeline_baserZ time_methodsrr$r$r$r3s&