o 0jR0@sDddlZddZddZddZdd Zd d Zd d ZddZdS)NcCs||dkrdS|dkr||dnd}|dt|kr"||dnd}|s,|r.dS|s6|r8dS|dvr>dSdS)zH Check if the given character is a sentence ending punctuation. .Fr) r   "'u”u’)u】u」u》T)lenisdigitisalpha)textiprevnextrt/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/pipelines/pp_doctranslation/utils.py_is_sentence_dots  rcCs<t|d}gd}t|t|D]}|||vrP|d}|t|kr9||dvr9|d7}|t|kr9||dvs)|t|krOt|d||krO||fSq||dkrt||r|d}|t|kr{||dvr{|d7}|t|kr{||dvsk|t|krt|d||kr||fSqt|ddD]w}|||vr|d}|t|kr||dvr|d7}|t|kr||dvst|d||kr||fSq||dkrt||r|d}|t|kr||dvr|d7}|t|kr||dvst|d||kr||fSqt|t|t|t|fS) z Find the position to split the text into two chunks. Args: text (str): The original text to be split. chunk_size (int): The maximum size of each chunk. Returns: int: The index where the text should be split. )ru。;u；!u！?u？rz Nrr)r rangermin)r chunk_sizecenterZ split_charsrjrrr_find_split_pos)sJ         rc Csz|}t||kr||St||\}}|d|}||d}|||}|r/t|||}|r7t|||} ||| S)af Split the text recursively and translate each chunk. Args: text (str): The original text to be split. chunk_size (int): The maximum size of each chunk. translate_func (callable): A function that translates a single chunk of text. results (list): A list to store the translated chunks. Returns: None N)stripr rsplit_text_recursive) rrtranslate_funcZ split_posZend_whitespaceleftright whitespaceZ left_textZ right_textrrrr!Zs       r!c Cs|d}|dds|ddrB|d}|dds'|ddr+|dnd}|r8d|ddnd|dd}nd}d}|}t|||}d d |dD} d| } |ri|d| d|n| } || dS) a{ Translate a code block and append the result to the results list. Args: code_block (str): The code block to be translated. chunk_size (int): The maximum size of each chunk. translate_func (callable): A function that translates a single chunk of text. results (list): A list to store the translated chunks. Returns: None rr```~~~rrrNcSs,g|]}|ds|ds|qS)r&r')r startswith).0linerrr s  z(translate_code_block..)r splitr(joinr!append) Z code_blockrr"resultslinesheaderfooterZ code_contentZtranslated_code_linesZfiltered_code_linesZtranslated_coderesultrrrtranslate_code_blockxs,    * r4c&Csddl}ddlm}|ddkr3|ddkr3|d|dkr3t||kr3||}||dS||d}t}g} g} |jddd D]*} | d d g} | rpt | |vrp| } | ri| | | | | t | qF|}d}|t| krg}g}d}|t| kr|t| ||kr|| ||| ||t| |7}|d 7}|t| kr|t| ||ksd }| |}||}||}t||D]\}}|||d}|jD] }|||qq|t| ks{g}|jddd D]} | d d gs| r|| qd}t|}||krg}g}td}||krs|| }t||krHt|||} ||| |d 7}q#d|d}!|t|!|krXn|||||!|t|!7}|d 7}||ks(|s||d  }d|d}!||d g}|!g}|rdd |d}||}||d}"|"d}#t||#D] \}$}%|$|% q||ks|t|dS)a{ Translate a HTML block and append the result to the results list. Args: html_block (str): The HTML block to be translated. chunk_size (int): The maximum size of each chunk. translate_func (callable): A function that translates a single chunk of text. results (list): A list to store the translated chunks. Returns: None rN BeautifulSoup<> html.parserT)string recursivetdthrZ__TD__z
    z
  1. z
    2. z
      rz
    Zli)copybs4r6countr r.setZfind_allZ find_parentidZdecode_contentsr addr-r,zipclearcontentsdeepcopyr!Z replace_withstr)&Z html_blockrr"r/r?r6Z translatedsoupZtd_seenZtd_batch_nodesZtd_batch_textsnodeZ parent_tdZtd_textZ batch_sizerZ batch_nodesZ batch_textsZcurrent_length placeholderZ batch_textZtranslated_batchZtranslated_linesZtd_noder*fragchildZ text_nodesidxtotalZli_textsZ node_textZtranslated_textZli_strZ trans_soupZtranslated_lisZ orig_nodeZli_tagrrrtranslate_html_blocks                           #rQcCsddlm}||d}g}g}d}d}d}|jD].}t|dr@|jdur@t|} ||} || || || 7}|d7}q|t|7}qg} t| |} d} t | D]4\} }|\}}| t |kr|| |vr| || || }| d7} d |f| | <| t |kr|| |vshqT| S) z. Split the original text into chunks. rr5r:z<>rnameNrhtml) r@r6rGhasattrrRrIformatr.split_and_append_text enumerater replace)rr6rJZ html_blocksZhtml_placeholdersZplaceholder_fmtZtext_after_placeholderindexelemZhtml_strrLZ splited_blockZ current_indexrOblock_contentrrrsplit_original_textss@             r^cCs|rltdtj}d}||D]5}||kr:|||}td|}|D]}|r9|d|fq*|d|f| }q|t |krl||d}td|}|D]}|rk|d|fq\|S)a Split the text and append the result to the result list. Args: result (list): The current result list. text_content (str): The text content to be processed. Returns: list: The updated result list after processing the text content. z(```.*?\n.*?```|~~~.*?\n.*?~~~)rz\n{2,}rcodeN) r recompileDOTALLfinditerstartr,r.groupendr )r3Z text_contentZ code_patternZlast_posmZnon_codeZ paragraphsprrrrVHs*       rV)r`rrr!r4rQr^rVrrrrs1+w .