o j%D@s"dZddlmZdZdgZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZddlmZmZmZmZmZmZddlmZmZdd lmZmZm Z m!Z!dd l"m#Z#erodd l$m%Z%dd lm&Z&dd l'm(Z(m)Z)m*Z*dZ+e e e,e,fe,e,gdfZ-GdddeeZ.Gddde Z/dS)zCUse the HTMLParser library to parse HTML files that aren't too bad.) annotationsMITHTMLParserTreeBuilder) HTMLParser) AnyCallablecastDictIterableListOptional TYPE_CHECKINGTupleTypeUnion) AttributeDictCDataComment DeclarationDoctypeProcessingInstruction)EntitySubstitution UnicodeDammit)DetectsXMLParsedAsHTMLHTMLHTMLTreeBuilderSTRICTParserRejectedMarkup) BeautifulSoup)NavigableString) _Encoding _Encodings _RawMarkupz html.parserNc@seZdZUdZded<dZded< edd6ddZd ed <ded<ded<d7ddZd8ddZ d9d:dd Z d9d;d"d#Z dd/d0Zd. F)handle_empty_elementN)handle_starttag handle_endtag)r5r<r=r6r6r7handle_startendtags z*BeautifulSoupHTMLParser.handle_startendtagTr?boolc Cs|}|D]3\}}|durd}||vr5|j}||jkrq|d|jfvr)|||<qtt|}||||q|||<q|jjjrF| \}} nd}} |jj |dd||| d} | durl| j rl|rl|j |dd|j ||jdurx||dSdS)zHandle an opening tag, e.g. '' :param handle_empty_element: True if this tag is known to be an empty-element tag (i.e. there is not expected to be any closing tag). N) sourceline sourceposF)check_already_closed)r2r+r)r'r_DuplicateAttributeHandlerr,r1Zstore_line_numbersgetposr@Zis_empty_elementrAr4appendZ_root_tag_nameZ_root_tag_encountered) r5r<r=r?Z attr_dictkeyvalueZon_duperErFZtagObjr6r6r7r@s2         z'BeautifulSoupHTMLParser.handle_starttagrGcCs.|r||jvr|j|dS|j|dS)zHandle a closing tag, e.g. '' :param tag: A tag name. :param check_already_closed: True if this tag is expected to be the closing portion of an empty-element tag, e.g. ''. N)r4remover,rA)r5r<rGr6r6r7rAs z%BeautifulSoupHTMLParser.handle_endtagdatacCs|j|dS)z4Handle some textual data that shows up between tags.N)r, handle_datar5rNr6r6r7rOsz#BeautifulSoupHTMLParser.handle_datanamecCsh|drt|dd}n|drt|dd}nt|}t|\}}|r-d|j_||dS)zHandle a numeric character reference by converting it to the corresponding Unicode character and treating it as textual data. :param name: Character number, possibly in hexadecimal. xXTN) startswithintlstriprZnumeric_character_referencer,contains_replacement_charactersrO)r5rQZ real_namerNZreplacement_addedr6r6r7handle_charrefs  z&BeautifulSoupHTMLParser.handle_charrefcCs0tj|}|dur |}nd|}||dS)zHandle a named entity reference by converting it to the corresponding Unicode character(s) and treating it as textual data. :param name: Name of the entity reference. Nz&%s)rZHTML_ENTITY_TO_CHARACTERgetrO)r5rQ characterrNr6r6r7handle_entityrefs z(BeautifulSoupHTMLParser.handle_entityrefcCs&|j|j||jtdS)zOHandle an HTML comment. :param data: The text of the comment. N)r,endDatarOrrPr6r6r7handle_comment s  z&BeautifulSoupHTMLParser.handle_commentdeclcCs6|j|tdd}|j||jtdS)zYHandle a DOCTYPE declaration. :param data: The text of the declaration. zDOCTYPE N)r,r]lenrOr)r5r_r6r6r7 handle_decls  z#BeautifulSoupHTMLParser.handle_declcCsN|drt}|tdd}nt}|j|j||j|dS)z{Handle a declaration of unknown type -- probably a CDATA block. :param data: The text of the declaration. zCDATA[N)upperrUrr`rr,r]rO)r5rNclsr6r6r7 unknown_decl s  z$BeautifulSoupHTMLParser.unknown_declcCs0|j|j||||jtdS)z\Handle a processing instruction. :param data: The text of the instruction. N)r,r]rOZ_document_might_be_xmlrrPr6r6r7 handle_pi/s   z!BeautifulSoupHTMLParser.handle_piN)r,rr-rr+r.r/r)r8r&r9r:)r<r&r=r>r9r:)T)r<r&r=r>r?rCr9r:)r<r&rGrCr9r:)rNr&r9r:)rQr&r9r:)r_r&r9r:)__name__ __module__ __qualname__r'__annotations__r)r3r;rBr@rArOrYr\r^rardrer6r6r6r7r$=s*     >     r$cseZdZUdZdZded<dZded<eZded<ee e gZ d ed <d ed <dZ ded <  d&d'fdd Z   d(d)dd Zefd*d$d%ZZS)+rzA Beautiful soup `bs4.builder.TreeBuilder` that uses the :py:class:`html.parser.HTMLParser` parser, found in the Python standard library. FrCis_xmlT picklabler&NAMEz Iterable[str]featuresz$Tuple[Iterable[Any], Dict[str, Any]] parser_argsTRACKS_LINE_NUMBERSNOptional[Iterable[Any]] parser_kwargsOptional[Dict[str, Any]]r/rc spt}dD]}||vr||}|||<qtt|jdi||p#g}|p'i}||d|d<||f|_dS)aConstructor. :param parser_args: Positional arguments to pass into the BeautifulSoupHTMLParser constructor, once it's invoked. :param parser_kwargs: Keyword arguments to pass into the BeautifulSoupHTMLParser constructor, once it's invoked. :param kwargs: Keyword arguments for the superclass constructor. r*Fconvert_charrefsNr6)dictpopsuperrr3updatern)r5rnrqr/Zextra_parser_kwargsargrL __class__r6r7r3Ks  zHTMLParserTreeBuilder.__init__markupr#user_specified_encodingOptional[_Encoding]document_declared_encodingexclude_encodingsOptional[_Encodings]r9DIterable[Tuple[str, Optional[_Encoding], Optional[_Encoding], bool]]ccst|tr|dddfVdSg}|r||g}|r!||t|||d|d}|jdur3td|j|j|j|jfVdS)a2Run any preliminary steps necessary to make incoming markup acceptable to the parser. :param markup: Some markup -- probably a bytestring. :param user_specified_encoding: The user asked to try this encoding. :param document_declared_encoding: The markup itself claims to be in this encoding. :param exclude_encodings: The user asked _not_ to try any of these encodings. :yield: A series of 4-tuples: (markup, encoding, declared encoding, has undergone character replacement) Each 4-tuple represents a strategy for parsing the document. This TreeBuilder uses Unicode, Dammit to convert the markup into Unicode, so the ``markup`` element of the tuple will always be a string. NFT)known_definite_encodingsuser_encodingsZis_htmlrzPCould not convert input to Unicode, and html.parser will not accept bytestrings.) isinstancer&rJrZunicode_markuprZoriginal_encodingZdeclared_html_encodingrX)r5r{r|r~rrrZdammitr6r6r7prepare_markupis4     z$HTMLParserTreeBuilder.prepare_markup _parser_classtype[BeautifulSoupHTMLParser]r:c Cs|j\}}t|ts J|jdusJ||jg|Ri|}z |||Wnty:}zt|d}~wwg|_dS)z :param markup: The markup to feed into the parser. :param _parser_class: An HTMLParser subclass to use. This is only intended for use in unit tests. N) rnrr&r,feedcloseAssertionErrorrr4)r5r{rr-r/parserer6r6r7rs    zHTMLParserTreeBuilder.feed)NN)rnrprqrrr/r)NNN) r{r#r|r}r~r}rrr9r)r{r#rrr9r:)rfrgrh__doc__rjrirk HTMLPARSERrlrrrmror3rr$r __classcell__r6r6ryr7r:s     ! H)0r __future__r __license____all__ html.parserrtypingrrrr r r r r rrrZ bs4.elementrrrrrrZ bs4.dammitrrZ bs4.builderrrrrZbs4.exceptionsrZbs4rr Z bs4._typingr!r"r#rr&rHr$rr6r6r6r7s(  4    ~