o 0j9@sddlZddlZddlmZmZddlmZddlmZddl m Z m Z ddl m Z e dr:dd lmZdd lmZe d rJdd lmZdd lmZe ddd Gdddee dZdS)N)ABCabstractmethod)List)logging)class_requires_depsis_dep_available)AutoRegisterABCMetaClassZ langchainDocument)RecursiveCharacterTextSplitterzlangchain-community) vectorstores)FAISSzlangchain-text-splittersc seZdZdZdZdZfddZeddZedd Z d e fd d Z d e d e fddZ ded e fddZde d efddZdgdfdee dedee d dfddZddd e fddZde d dfd d!Z " # $d*d%ee ddd&ed'ed(ed e f d)d Z ZS)+ BaseRetrieverzBase RetrieverTZPADDLEX_VECTOR_STOREcstd|_d|_dS)z*Initializes an instance of base retriever.N)super__init__ model_name embeddingself __class__v/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddlex/inference/pipelines/components/retriever/base.pyr(s  zBaseRetriever.__init__cCtd)z Declaration of an abstract method. Subclasses are expected to provide a concrete implementation of generate_vector_database. zCThe method `generate_vector_database` has not been implemented yet.NotImplementedErrorrrrrgenerate_vector_database.z&BaseRetriever.generate_vector_databasecCr)z Declaration of an abstract method. Subclasses are expected to provide a concrete implementation of similarity_retrieval. z?The method `similarity_retrieval` has not been implemented yet.rrrrrsimilarity_retrieval8rz"BaseRetriever.similarity_retrievalreturncCs|jS)zt Get the model name used for generating vectors. Returns: str: The model name. )rrrrrget_model_nameBszBaseRetriever.get_model_namescCs ||jS)z Check if the given string starts with the vector store prefix. Args: s (str): The input string to check. Returns: bool: True if the string starts with the vector store prefix, False otherwise. ) startswithVECTOR_STORE_PREFIX)rr"rrris_vector_storeKs zBaseRetriever.is_vector_storevector_store_bytescCs|jt|dS)z Encode the vector store bytes into a base64 string prefixed with a specific prefix. Args: vector_store_bytes (bytes): The bytes to encode. Returns: str: The encoded string with the prefix. ascii)r$base64 b64encodedecode)rr&rrrencode_vector_storeWs z!BaseRetriever.encode_vector_storevector_store_strcCst|t|jdS)a Decodes the vector store string by removing the prefix and decoding the base64 encoded string. Args: vector_store_str (str): The vector store string with a prefix. Returns: bytes: The decoded vector store data. N)r( b64decodelenr$)rr,rrrdecode_vector_storees z!BaseRetriever.decode_vector_storei,)  u。z text_list block_size separatorsrcCs\t|d|d}|d|}dd|D}z tj||jd}W|Sty-d}Y|Sw)a Generates a vector database from a list of texts. Args: text_list (list[str]): A list of texts to generate the vector database from. block_size (int): The size of each chunk to split the text into. separators (list[str]): A list of separators to use when splitting the text. Returns: FAISS: The generated vector database. Raises: ValueError: If an unsupported API type is configured. ) chunk_sizeZ chunk_overlapr5r0cSsg|]}t|dqS)Z page_contentr ).0textrrr sz:BaseRetriever.generate_vector_database..)Z documentsrN)r Z split_textjoinrZfrom_documentsr ValueError)rr3r4r5Z text_splitterZtextsZ all_splits vectorstorerrrrqs r>cCs$|dur |j}|S||}|S)z Encode the vector store serialized to bytes. Args: vectorstore (FAISS): The vector store to be serialized and encoded. Returns: str: The encoded vector store. N)r$r+Zserialize_to_bytes)rr>rrrencode_vector_store_to_bytess  z*BaseRetriever.encode_vector_store_to_bytescCsJ||s td||}|dkrtddStjj||jdd}|S)a Decode a vector store from bytes according to the specified API type. Args: vectorstore (str): The serialized vector store string. Returns: FAISS: Deserialized vector store object. Raises: ValueError: If the retrieved vector store is not for PaddleX or if an unsupported API type is specified. z-The retrieved vectorstore is not for PaddleX.z5The retrieved vectorstore is empty,will empty vector.NT)Z embeddingsZallow_dangerous_deserialization) r%r=r/rwarningr rZdeserialize_from_bytesr)rr>Zvectorrrrdecode_vector_store_from_bytess   z,BaseRetriever.decode_vector_store_from_bytes? query_text_list sleep_timetopkmin_charactersc Csd}|dur|S|D]?}|}t||j||d} dd| D} t| ddd} | ddd D]\} } | d krHt|t| |krDn|| 7}q0q |S) a Retrieve similar contexts based on a list of query texts. Args: query_text_list (list[str]): A list of query texts to search for similar contexts. vectorstore (FAISS): The vector store where to perform the similarity search. sleep_time (float): The time to sleep between each query, in seconds. Default is 0.5. topk (int): The number of results to retrieve per query. Default is 2. min_characters (int): The minimum number of characters required for text processing, defaults to 3500. Returns: str: A concatenated string of all unique contexts found. r2N)kcSsg|] \}}|j|fqSrr8)r9Zdocumentscorerrrr;sz6BaseRetriever.similarity_retrieval..cSs|dS)Nr)xrrrsz4BaseRetriever.similarity_retrieval..)keyg)timesleepZ'similarity_search_with_relevance_scoressortedr.) rrFr>rGrHrIZall_CZ query_textZQUESTIONdocscontextr:rKrrrrs  )rCrDrE)__name__ __module__ __qualname____doc__Z_BaseRetriever__is_baser$rrrrstrr!boolr%bytesr+r/rintr?rBfloat __classcell__rrrrr sT       #"r) metaclass)r(rQabcrrtypingrZ paddlex.utilsrZ utils.depsrrZutils.subclass_registerr Zlangchain_core.documentsr Zlangchain_text_splittersr Zlangchain_communityr Z langchain_community.vectorstoresrrrrrrs