o *Îjiã@sþdZddlmZmZmZmZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlmmZddlmZddlmZddlmZddlmZdd „Zd d „Zd d „Ze e¡Ze ej ¡ddddddddœZ!dZ"dZ#dZ$dd„Z%dd„Z&dd„Z'e&ejjj(e'dœZ)Gd d!„d!e*ƒZ+zdd"l,m-Z.Wne/yµe0d#ƒGd$d%„d%ej1ƒZ.YnwGd&d'„d'ej1ƒZ2Gd(d)„d)ej1ƒZ3Gd*d+„d+ej1ƒZ4Gd,d-„d-ej1ƒZ5Gd.d/„d/ej1ƒZ6Gd0d1„d1ej1ƒZ7Gd2d3„d3ej1ƒZ8Gd4d5„d5ej1ƒZ9Gd6d7„d7ej1ƒZ:Gd8d9„d9ej1ƒZ;Gd:d;„d;ej1ƒZd?„d?ej1ƒZ>Gd@dA„dAej1ƒZ?GdBdC„dCej1ƒZ@GdDdE„dEe@ƒZAGdFdG„dGe@ƒZBGdHdI„dIe@ƒZCGdJdK„dKe@ƒZDGdLdM„dMe@ƒZEGdNdO„dOe@ƒZFGdPdQ„dQe@ƒZGGdRdS„dSe@ƒZHdS)TzPyTorch BERT model.é)Úabsolute_importÚdivisionÚprint_functionÚunicode_literalsN)Ú cached_path)Úmpu)Únn)ÚCrossEntropyLosscs‡‡fdd„}|S)Ncótjjj|ˆˆdS©N©ÚmeanÚstd©ÚtorchrÚinitÚnormal_©Ztensorr ©úo/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/mglm/model/modeling_bert.pyÚinit_)óz!normal_init_method..init_r)r rrrr rÚnormal_init_method'srcs$ˆt d|¡‰‡‡fdd„}|S)z3Init method based on N(0, sigma/sqrt(2*num_layers).ç@cr r rrr rrr3rz!scaled_init_method..init_)ÚmathÚsqrt)r rÚ num_layersrrr rÚscaled_init_method/srcCs*| d¡}| d¡}||}| d¡}|S)Néé)Ú unsqueeze)Úattention_maskZattention_mask_b1sZattention_mask_bs1Zattention_mask_bssÚextended_attention_maskrrrÚbert_extended_attention_mask9s   r#z#/root/data/bert-base-uncased.tar.gzzMhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gzzJhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gzzKhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased.tar.gzzYhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased.tar.gzzWhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased.tar.gzzLhttps://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz)zbert-base-uncasedzbert-large-uncasedzbert-base-casedzbert-large-casedzbert-base-multilingual-uncasedzbert-base-multilingual-casedzbert-base-chinesezbert_config.jsonzpytorch_model.binz model.ckptc Cs:zddl}ddl}ddl}Wn tytdƒ‚wtj |¡}td |¡ƒ|j   |¡}g}g}|D]\} } td | | ¡ƒ|j   || ¡} |  | ¡|  | ¡q4t ||ƒD]Ã\} } |  d¡} tdd„| Dƒƒrttd  d | ¡¡ƒqW|} | D]Y} | d | ¡r‡| d | ¡}n| g}|dd ks–|dd krœt| dƒ} n%|ddks¨|ddkr®t| dƒ} n|ddkrºt| dƒ} nt| |dƒ} t|ƒdkrÑt|dƒ}| |} qx| dd…dkràt| dƒ} n | d kré| | ¡} z | j| jksòJ‚Wnty }z |j| j| jf7_‚d}~wwtd | ¡ƒt | ¡| _qW|S)z, Load tf checkpoints in a pytorch model rNzšLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z(Converting TensorFlow checkpoint from {}z"Loading TF weight {} with shape {}ú/css|]}|dvVqdS))Zadam_vZadam_mNr)Ú.0ÚnrrrÚ |s€z*load_tf_weights_in_bert..z Skipping {}z [A-Za-z]+_\d+z_(\d+)ZkernelÚgammaÚweightZ output_biasÚbetaÚbiasZoutput_weightsrriõÿÿÿZ _embeddingszInitialize PyTorch weight {})ÚreÚnumpyZ tensorflowÚ ImportErrorÚprintÚosÚpathÚabspathÚformatÚtrainZlist_variablesZ load_variableÚappendÚzipÚsplitÚanyÚjoinÚ fullmatchÚgetattrÚlenÚintÚ transposeÚshapeÚAssertionErrorÚargsrZ from_numpyÚdata)ÚmodelZtf_checkpoint_pathr,ÚnpÚtfZtf_pathZ init_varsÚnamesZarraysÚnamer?ÚarrayÚpointerZm_nameÚlÚnumÚerrrÚload_tf_weights_in_bert_sn  ÿû             €  €þrMcCs |ddt |t d¡¡S)zûImplementation of the gelu activation function. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) gà?çð?r)rÚerfrr©ÚxrrrÚgelužs rRcCs|t |¡S©N)rZsigmoidrPrrrÚswish¦srT)rRÚrelurTc@sheZdZdZ          dd d „Zedd„ƒZedd„ƒZdd„Zdd„Z dd„Z dS)Ú BertConfigzEConfiguration class to store the configuration of a `BertModel`. éé é rRçš™™™™™¹?érç{®Gáz”?Fçê-™—q=cCsÜt|tƒr3t|ddd}t | ¡¡}Wdƒn1swY| ¡D] \}}||j|<q'dSt|tƒrj||_ ||_ ||_ ||_ ||_ ||_||_||_| |_| |_| |_| |_| |_||_||_||_dStdƒ‚)a¨Constructs BertConfig. Args: vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `BertModel`. hidden_size: Size of the encoder layers and the pooler layer. num_hidden_layers: Number of hidden layers in the Transformer encoder. num_attention_heads: Number of attention heads for each attention layer in the Transformer encoder. intermediate_size: The size of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act: The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu" and "swish" are supported. hidden_dropout_prob: The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob: The dropout ratio for the attention probabilities. max_position_embeddings: The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). type_vocab_size: The vocabulary size of the `token_type_ids` passed into `BertModel`. initializer_range: The stdev of the truncated_normal_initializer for initializing all weight matrices. Úrúutf-8©ÚencodingNzhFirst argument must be either a vocabulary size (int)or the path to a pretrained model config file (str))Ú isinstanceÚstrÚopenÚjsonÚloadsÚreadÚitemsÚ__dict__r=Ú vocab_sizeÚ hidden_sizeÚnum_hidden_layersÚnum_attention_headsÚ hidden_actÚintermediate_sizeÚhidden_dropout_probÚattention_probs_dropout_probÚmax_position_embeddingsÚtype_vocab_sizeÚinitializer_rangeÚ deep_initÚfp32_layernormÚfp32_embeddingÚlayernorm_epsilonÚfp32_tokentypesÚ ValueError)ÚselfÚvocab_size_or_config_json_filerkrlrmrornrprqrrrsrtrurvrwryrxÚreaderZ json_configÚkeyÚvaluerrrÚ__init__±s> )þý ÿ  ÿzBertConfig.__init__cCs*tdd}| ¡D] \}}||j|<q |S)zAConstructs a `BertConfig` from a Python dictionary of parameters.éÿÿÿÿ)r|)rVrhri)ÚclsZ json_objectÚconfigr~rrrrÚ from_dict÷s  zBertConfig.from_dictcCsFt|ddd }| ¡}Wdƒn1swY| t |¡¡S)z9Constructs a `BertConfig` from a json file of parameters.r^r_r`N)rdrgr„rerf)r‚Z json_filer}ÚtextrrrÚfrom_json_fileÿs ÿzBertConfig.from_json_filecCs t| ¡ƒSrS)rcÚto_json_string©r{rrrÚ__repr__s zBertConfig.__repr__cCst |j¡}|S)z0Serializes this instance to a Python dictionary.)ÚcopyÚdeepcopyri©r{ÚoutputrrrÚto_dict s zBertConfig.to_dictcCstj| ¡ddddS)z*Serializes this instance to a JSON string.rT)ÚindentÚ sort_keysÚ )reÚdumpsrŽrˆrrrr‡szBertConfig.to_json_stringN)rWrXrXrYrRrZrZr[rr\FFFFr]) Ú__name__Ú __module__Ú __qualname__Ú__doc__r€Ú classmethodr„r†r‰rŽr‡rrrrrV­s2 ðF   rV)ÚFusedLayerNormzYBetter speed can be achieved with apex installed from https://www.github.com/nvidia/apex.cs&eZdZd‡fdd„ Zdd„Z‡ZS)Ú BertLayerNormr]cs<tt|ƒ ¡t t |¡¡|_t t |¡¡|_ ||_ dS)z[Construct a layernorm module in the TF style (epsilon inside the square root). N) Úsuperr™r€rÚ ParameterrZonesr)Zzerosr+Úvariance_epsilon)r{rkÚeps©Ú __class__rrr€s zBertLayerNorm.__init__cCsN|jddd}|| d¡jddd}||t ||j¡}|j||jS)NrT)Zkeepdimr)r Úpowrrrœr)r+)r{rQÚuÚsrrrÚforward$szBertLayerNorm.forward)r]©r“r”r•r€r£Ú __classcell__rrržrr™sr™cs*eZdZdZ‡fdd„Zddd„Z‡ZS)ÚBertEmbeddingszLConstruct the embeddings from word, position and token_type embeddings. cs€tt|ƒ ¡t |j|j¡|_t |j|j¡|_ t |j |j¡|_ |j |_ |j |_ |j|_t|j|jd|_t |j¡|_dS©N©r)ršr¦r€rÚ EmbeddingrjrkÚword_embeddingsrrÚposition_embeddingsrsÚtoken_type_embeddingsrvrwryr™rxÚ LayerNormÚDropoutrpÚdropout©r{rƒržrrr€/s"ÿÿÿÿzBertEmbeddings.__init__Nc Cs@| d¡}tj|tj|jd}| d¡ |¡}|dur t |¡}| |¡}|  |¡}|  |¡}|j sc|||}|j rB|j sB| ¡}| ¡} |j rM| ¡}| |¡}|j rb|j r]| ¡}n<| | ¡}n6| ¡| ¡| ¡}|j ry|j sy| ¡}| ¡} |j r„| ¡}| |¡}|j r™|j r”| ¡}n| | ¡}| |¡}|S)Nr)ÚdtypeÚdevicer)ÚsizerZarangeÚlongr²r Z expand_asÚ zeros_likerªr«r¬ryrwrvÚhalfÚtypeÚfloatr­r¯) r{Ú input_idsÚtoken_type_idsZ seq_lengthZ position_idsZwords_embeddingsr«r¬Ú embeddingsÚ previous_typerrrr£EsL  ÿ         €ÿ     zBertEmbeddings.forwardrS©r“r”r•r–r€r£r¥rrržrr¦+s r¦cs,eZdZ‡fdd„Zdd„Zdd„Z‡ZS)ÚBertSelfAttentioncs¢tt|ƒ ¡|j|jdkrtd|j|jfƒ‚|j|_t|j|jƒ|_|j|j|_t   |j|j¡|_ t   |j|j¡|_ t   |j|j¡|_ t  |j¡|_dS)NrzLThe hidden size (%d) is not a multiple of the number of attention heads (%d))ršr¾r€rkrmrzr=Úattention_head_sizeÚ all_head_sizerÚLinearÚqueryr~rr®rqr¯r°ržrrr€rs  þÿÿzBertSelfAttention.__init__cCs6| ¡dd…|j|jf}|j|Ž}| dddd¡S)Nrrrré)r³rmr¿ÚviewÚpermute)r{rQZ new_x_shaperrrÚtranspose_for_scores„s ÿ z&BertSelfAttention.transpose_for_scorescCsÐ| |¡}| |¡}| |¡}| |¡}| |¡}| |¡}t || dd¡¡} | t |j ¡} | |} t j dd| ƒ} |  | ¡} |   ¡} t | |¡} |  dddd¡ ¡} |  ¡dd…|jf} | j| Ž} | S)Nréþÿÿÿ©ÚdimrrrrÃ)rÂr~rrÆrÚmatmulr>rrr¿rZSoftmaxr¯r·rÅÚ contiguousr³rÀrÄ)r{Ú hidden_statesr!Zmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZattention_probsr¼Z context_layerZnew_context_layer_shaperrrr£Šs.       ÿÿ  ÿ zBertSelfAttention.forward)r“r”r•r€rÆr£r¥rrržrr¾ps r¾có$eZdZ‡fdd„Zdd„Z‡ZS)ÚBertSelfOutputcs‚tt|ƒ ¡t|dƒr|jrtd|j|jd}ntd|jd}t j |j |j dd|_ |j |_ t|j |jd|_t  |j¡|_dS©Nruç)r rrr T©r+r¨)ršrÎr€ÚhasattrrurrtrlrrrÁrkÚdenservr™rxr­r®rpr¯©r{rƒZ init_methodržrrr€­ó$ýÿ ÿ ÿzBertSelfOutput.__init__cCóP| |¡}| |¡}||}| ¡}|jr| ¡}| |¡}|jr&| |¡}|SrS©rÓr¯r·rvr¸r­©r{rÌÚ input_tensorZln_inputr¼rrrr£Åó    zBertSelfOutput.forwardr¤rrržrrΫó rÎcrÍ)Ú BertAttentioncs&tt|ƒ ¡t|ƒ|_t|ƒ|_dSrS)ršrÜr€r¾r{rÎrr°ržrrr€Ôs zBertAttention.__init__cCs| ||¡}| ||¡}|SrSrŒ)r{rÙr!Z self_outputÚattention_outputrrrr£às  zBertAttention.forwardr¤rrržrrÜÒs  rÜcrÍ)ÚBertIntermediatecsLtt|ƒ ¡tj|j|jdd|_t|j t ƒr t |j |_ dS|j |_ dS)NTrÑ) ršrÞr€rrÁrkrorÓrbrnrcÚACT2FNÚintermediate_act_fnr°ržrrr€ès ÿ ÿÿzBertIntermediate.__init__cCs| |¡}| |¡}|SrS)rÓrà©r{rÌrrrr£÷s  zBertIntermediate.forwardr¤rrržrrÞæs rÞcrÍ)Ú BertOutputcs‚tt|ƒ ¡t|dƒr|jrtd|j|jd}ntd|jd}t j |j |j dd|_ |j|_t|j |jd|_t  |j¡|_dSrÏ)ršrâr€rÒrurrtrlrrrÁrorkrÓrvr™rxr­r®rpr¯rÔržrrr€ÿrÕzBertOutput.__init__cCrÖrSr×rØrrrr£rÚzBertOutput.forwardr¤rrržrrâýrÛrâcrÍ)Ú BertLayercs0tt|ƒ ¡t|ƒ|_t|ƒ|_t|ƒ|_dSrS) ršrãr€rÜÚ attentionrÞÚ intermediaterârr°ržrrr€&s  zBertLayer.__init__cCs&| ||¡}| |¡}| ||¡}|SrS)rärår)r{rÌr!rÝZintermediate_outputZ layer_outputrrrr£,s   zBertLayer.forwardr¤rrržrrã$ó rãcs*eZdZ‡fdd„Z  ddd„Z‡ZS)Ú BertEncodercs2tt|ƒ ¡t ‡fdd„tˆjƒDƒ¡|_dS)Ncsg|]}tˆƒ‘qSr)rã)r%Ú_©rƒrrÚ :sz(BertEncoder.__init__..)ršrçr€rZ ModuleListÚrangerlÚlayerr°ržrérr€5s ÿzBertEncoder.__init__TFc sžg}‡fdd„}|r.d}tˆjƒ}d} ||kr-t |||| ƒ||d¡}|| 7}||ksntˆjƒD]\} } | ||ƒ}|rC| |¡q3|rH|rM| |¡|S)Ncs‡‡‡fdd„}|S)Ncs2ˆjˆˆ…}|d}|D] }|||dƒ}q |S)Nrr)rì)ÚinputsZlayersZx_rì)Úendr{ÚstartrrÚcustom_forwardNs z;BertEncoder.forward..custom..custom_forwardr)rïrîrðrˆ)rîrïrÚcustomLsz#BertEncoder.forward..customrr)r<rìrÚ checkpointÚ enumerater5) r{rÌr!Úoutput_all_encoded_layersÚcheckpoint_activationsZall_encoder_layersrñrJrÚ chunk_lengthÚiZ layer_modulerrˆrr£Es,   þü€  € zBertEncoder.forward)TFr¤rrržrrç3s  ürçcrÍ)Ú BertPoolercs.tt|ƒ ¡t |j|j¡|_t ¡|_dSrS) ršrør€rrÁrkrÓZTanhÚ activationr°ržrrr€oszBertPooler.__init__cCs(|dd…df}| |¡}| |¡}|S)Nr)rÓrù)r{rÌZfirst_token_tensorÚ pooled_outputrrrr£ts  zBertPooler.forwardr¤rrržrrømó røcrÍ)ÚBertPredictionHeadTransformcs\tt|ƒ ¡t |j|j¡|_t|jt ƒrt |jn|j|_ t |j|j d|_|j|_dSr§)ršrür€rrÁrkrÓrbrnrcrßÚtransform_act_fnr™rxr­rvr°ržrrr€s  ÿÿÿ z$BertPredictionHeadTransform.__init__cCsH| |¡}| |¡}| ¡}|jr| ¡}| |¡}|jr"| |¡}|SrS)rÓrýr·rvr¸r­)r{rÌr¼rrrr£ˆs    z#BertPredictionHeadTransform.forwardr¤rrržrrü}s  rücrÍ)ÚBertLMPredictionHeadcsbttˆƒ ¡t|ƒˆ_tj| d¡| d¡ddˆ_|j ˆ_ |j ˆ_ ‡fdd„}|ˆ_ dˆ_ dS)NrrFrÑcsˆjr| ¡S|SrS)rwr¶rrˆrrÚconvert_to_type¦sz6BertLMPredictionHead.__init__..convert_to_type) ršrþr€rüÚ transformrrÁr³ÚdecoderrwrvÚtype_converterÚ converted)r{rƒÚbert_model_embedding_weightsrÿržrˆrr€–s ý  zBertLMPredictionHead.__init__cCsR|jsd|_|jr|j ¡|jr|jj ¡| | |¡¡}| |¡|j }|S)NT) rrwrr¶rvr­r¸rrr+rárrrr£¯s  zBertLMPredictionHead.forwardr¤rrržrrþ”s rþcrÍ)ÚBertOnlyMLMHeadcstt|ƒ ¡t||ƒ|_dSrS)ršrr€rþÚ predictions©r{rƒrržrrr€Ás ÿzBertOnlyMLMHead.__init__cCó| |¡}|SrS)r)r{Úsequence_outputÚprediction_scoresrrrr£Æó zBertOnlyMLMHead.forwardr¤rrržrr¿rûrcrÍ)ÚBertOnlyNSPHeadcs"tt|ƒ ¡t |jd¡|_dS©Nr)ršr r€rrÁrkÚseq_relationshipr°ržrrr€ÍszBertOnlyNSPHead.__init__cCrrS)r)r{rúÚseq_relationship_scorerrrr£Ñr zBertOnlyNSPHead.forwardr¤rrržrr Ës r crÍ)ÚBertPreTrainingHeadscs.tt|ƒ ¡t||ƒ|_t |jd¡|_dSr ) ršrr€rþrrrÁrkrrržrrr€Øs ÿzBertPreTrainingHeads.__init__cCs@| |¡}|j ¡D] }|durq | |¡}q | |¡}||fSrS)rrÚ parametersÚtype_as)r{r rúr Úprrrrr£Þs   zBertPreTrainingHeads.forwardr¤rrržrrÖrærcsBeZdZdZ‡fdd„Zdd„Ze      d d d „ƒZ‡ZS) ÚPreTrainedBertModelz† An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. cs:tt|ƒ ¡t|tƒstd |jj|jj¡ƒ‚||_ dS)Nz¶Parameter config in `{}(config)` should be an instance of class `BertConfig`. To create a model from a Google pretrained model use `model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`) ršrr€rbrVrzr3rŸr“rƒ)r{rƒríÚkwargsržrrr€ís  ÿý zPreTrainedBertModel.__init__cCs|t|tjtjfƒr|jjjd|jjdnt|t ƒr'|j j  ¡|jj  d¡t|tjƒr:|j dur<|j j  ¡dSdSdS)z! Initialize the weights. rÐr rNN) rbrrÁr©r)rBrrƒrtr™r+Zzero_Zfill_)r{ÚmodulerrrÚinit_bert_weights÷sÿ  ÿz%PreTrainedBertModel.init_bert_weightsNFr]c sž|tvr t|} n|} zt| |d} Wnty,t d |d t ¡¡| ¡¡YdSw| | kr:t d | ¡¡n t d | | ¡¡d} t j   | ¡rN| } n*t   ¡} t d | | ¡¡t | d¡ }| | ¡Wdƒn1sqwY| } t j  | t¡}t |¡}||_||_||_||_t d  |¡¡||g|¢Ri| ¤Ž}ˆdur³t j  | t¡}t |¡‰g}g}ˆ ¡D]$}d}d |vrÉ| d d ¡}d |vrÓ| d d ¡}|rß| |¡| |¡q»t||ƒD] \}}ˆ |¡ˆ|<qåg‰g‰g‰tˆddƒ‰ˆ  ¡‰ˆdur ˆˆ_!d‡‡‡‡‡‡fdd„ ‰ˆ|t"|dƒrdnddt#ˆƒdkr4t$d |j%j&ˆ¡ƒt#ˆƒdkrEt$d |j%j&ˆ¡ƒ| rMt' (| ¡|S)aæ Instantiate a PreTrainedBertModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained model file if needed. Params: pretrained_model_name: either: - a str with the name of a pre-trained model to load selected in the list of: . `bert-base-uncased` . `bert-large-uncased` . `bert-base-cased` . `bert-large-cased` . `bert-base-multilingual-uncased` . `bert-base-multilingual-cased` . `bert-base-chinese` - a path or url to a pretrained model archive containing: . `bert_config.json` a configuration file for the model . `pytorch_model.bin` a PyTorch dump of a BertForPreTraining instance cache_dir: an optional path to a folder in which the pre-trained models will be cached. state_dict: an optional state dictionary (collections.OrderedDict object) to use instead of Google pre-trained models *inputs, **kwargs: additional input for the specific Bert class (ex: num_labels for BertForSequenceClassification) )Ú cache_dirz“Model name '{}' was not found in model name list ({}). We assumed '{}' was a path or url but couldn't find any file associated to this path or url.z, Nzloading archive file {}z(loading archive file {} from cache at {}z)extracting archive file {} to temp dir {}zr:gzzModel config {}r(r)r*r+Ú _metadataÚc shˆdurin ˆ |dd…i¡}| ˆ||dˆˆˆ¡|j ¡D]\}}|dur1ˆ|||dƒq dS)NrTÚ.)ÚgetZ_load_from_state_dictZ_modulesrh)rÚprefixZlocal_metadatarGÚchild©Ú error_msgsÚloadÚmetadataZ missing_keysÚ state_dictZunexpected_keysrrr!ls ÿ þ€þz1PreTrainedBertModel.from_pretrained..loadÚbertzbert.)rrz7Weights of {} not initialized from pretrained model: {}z0Weights from pretrained model not used in {}: {})r))ÚPRETRAINED_MODEL_ARCHIVE_MAPrÚFileNotFoundErrorÚloggerÚerrorr3r9ÚkeysÚinfor0r1ÚisdirÚtempfileÚmkdtempÚtarfilerdÚ extractallÚ CONFIG_NAMErVr†rvrwrxryÚ WEIGHTS_NAMErr!Úreplacer5r6Úpopr;rŠrrÒr<r/rŸr“ÚshutilÚrmtree)r‚Zpretrained_model_namer#rrvrwrxryrírZ archive_fileZresolved_archive_fileÚtempdirZserialization_dirÚarchiveÚ config_filerƒrCZ weights_pathZold_keysZnew_keysr~Znew_keyÚold_keyrrrÚfrom_pretrainedsž!  ÿ  ýýø ÿ ÿ ÿ       €    ÿÿ z#PreTrainedBertModel.from_pretrained)NNFFr]F) r“r”r•r–r€rr—r:r¥rrržrrès  ùrcs2eZdZdZ‡fdd„Z    d dd„Z‡ZS) Ú BertModela BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configuration to build a new model Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `output_all_encoded_layers`: boolean which controls the content of the `encoded_layers` output as described below. Default: `True`. Outputs: Tuple of (encoded_layers, pooled_output) `encoded_layers`: controlled by `output_all_encoded_layers` argument: - `output_all_encoded_layers=True`: outputs a list of the full sequences of encoded-hidden-states at the end of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size], - `output_all_encoded_layers=False`: outputs only the full sequence of hidden-states corresponding to the last attention block of shape [batch_size, sequence_length, hidden_size], `pooled_output`: a torch.FloatTensor of size [batch_size, hidden_size] which is the output of a classifier pretrained on top of the hidden state associated to the first character of the input (`CLF`) to train on the Next-Sentence task (see BERT's paper). Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = modeling.BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = modeling.BertModel(config=config) >>> all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask) cs>tt|ƒ |¡t|ƒ|_t|ƒ|_t|ƒ|_|  |j ¡dSrS) ršr;r€r¦r»rçÚencoderrøÚpoolerÚapplyrr°ržrrr€®s    zBertModel.__init__NTFc CsÄ|dur t |¡}|durt |¡}| d¡ d¡}|jt|j ¡ƒjd}d|d}|  ||¡}|j||||d}|d} |j  ¡D] } | durKqD|   | ¡} |  | ¡} |rZ|r^|d}|| fS)Nrr)r±rNgˆÃÀ©rôrõr) rZ ones_likerµr ÚtoÚnextr<rr±r»r=r) r{r¹rºr!rôrõr"Zembedding_outputZencoded_layersr rrúrrrr£µs4  ÿ  ü  zBertModel.forward)NNTFr½rrržrr;ƒó * ûr;có4eZdZdZ‡fdd„Z     ddd„Z‡ZS) ÚBertForPreTrainingaz BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language modeling head, and - the next sentence classification head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `masked_lm_labels`: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size] `next_sentence_label`: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence. Outputs: if `masked_lm_labels` and `next_sentence_label` are not `None`: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss. if `masked_lm_labels` or `next_sentence_label` is `None`: Outputs a tuple comprising - the masked language modeling logits of shape [batch_size, sequence_length, vocab_size], and - the next sentence classification logits of shape [batch_size, 2]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForPreTraining(config) >>> masked_lm_logits_scores, seq_relationship_logits = model(input_ids, token_type_ids, input_mask) có>tt|ƒ |¡t|ƒ|_t||jjjjƒ|_ |  |j ¡dSrS) ršrDr€r;r$rr»rªr)r‚r>rr°ržrrr€s   ÿzBertForPreTraining.__init__NFcCs’|j|||d|d\}}| ||¡\} } |durE|durEtdd} | |  d|jj¡ ¡| d¡ƒ} | |  dd¡ ¡| d¡ƒ} | | }|S| | fS©NFr?r©Z ignore_indexr)r$r‚r rÄrƒrjr¸)r{r¹rºr!Úmasked_lm_labelsÚnext_sentence_labelrõr rúr rÚloss_fctÚmasked_lm_lossÚnext_sentence_lossÚ total_lossrrrr£s. ûÿ þþzBertForPreTraining.forward©NNNNFr½rrržrrDâs 0 úrDcó2eZdZdZ‡fdd„Z    ddd„Z‡ZS) ÚBertForMaskedLMa BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `masked_lm_labels`: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size] Outputs: if `masked_lm_labels` is not `None`: Outputs the masked language modeling loss. if `masked_lm_labels` is `None`: Outputs the masked language modeling logits of shape [batch_size, sequence_length, vocab_size]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForMaskedLM(config) >>> masked_lm_logits_scores = model(input_ids, token_type_ids, input_mask) crErS) ršrPr€r;r$rr»rªr)r‚r>rr°ržrrr€as   ÿzBertForMaskedLM.__init__NFc CsX|j|||d|d\}}| |¡}|dur*tdd} | | d|jj¡| d¡ƒ} | S|S)NFr?rrG)r$r‚r rÄrƒrj) r{r¹rºr!rHrõr rèr rJrKrrrr£hs  û  þzBertForMaskedLM.forward©NNNFr½rrržrrP8s ( ûrPcrO) ÚBertForNextSentencePredictiona7 BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params: config: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `next_sentence_label`: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence. Outputs: if `next_sentence_label` is not `None`: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss. if `next_sentence_label` is `None`: Outputs the next sentence classification logits of shape [batch_size, 2]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForNextSentencePrediction(config) >>> seq_relationship_logits = model(input_ids, token_type_ids, input_mask) cs4tt|ƒ |¡t|ƒ|_t|ƒ|_| |j¡dSrS) ršrRr€r;r$r r‚r>rr°ržrrr€ªs  z&BertForNextSentencePrediction.__init__NFc CsT|j|||d|d\}}| |¡}|dur(tdd} | | dd¡| d¡ƒ} | S|SrF)r$r‚r rÄ) r{r¹rºr!rIrõrèrúrrJrLrrrr£°s  û   þz%BertForNextSentencePrediction.forwardrQr½rrržrrR€s )ûrRcó4eZdZdZd ‡fdd„ Z    d dd„Z‡ZS) ÚBertForSequenceClassificationaäBERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_labels`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_labels]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_labels = 2 >>> model = BertForSequenceClassification(config, num_labels) >>> logits = model(input_ids, token_type_ids, input_mask) rcóNtt|ƒ |¡||_t|ƒ|_t |j¡|_ t  |j |¡|_ |  |j¡dSrS)ršrTr€Ú num_labelsr;r$rr®rpr¯rÁrkÚ classifierr>r©r{rƒrVržrrr€ôs  z&BertForSequenceClassification.__init__NFc Cs\|j|||d|d\}}| |¡}| |¡}|dur,tƒ} | | d|j¡| d¡ƒ} | S|S©NFr?r)r$r¯rWr rÄrV) r{r¹rºr!ÚlabelsrõrèrúÚlogitsrJÚlossrrrr£üs û  z%BertForSequenceClassification.forward©rrQr½rrržrrTÈs+ ûrTcrO) ÚBertForMultipleChoicea] BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_choices`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_choices]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]], [[12, 16, 42], [14, 28, 57]]]) >>> input_mask = torch.LongTensor([[[1, 1, 1], [1, 1, 0]],[[1,1,0], [1, 0, 0]]]) >>> token_type_ids = torch.LongTensor([[[0, 0, 1], [0, 1, 0]],[[0, 1, 1], [0, 0, 1]]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_choices = 2 >>> model = BertForMultipleChoice(config, num_choices) >>> logits = model(input_ids, token_type_ids, input_mask) csHtt|ƒ |¡t|ƒ|_t |j¡|_t  |j d¡|_ |  |j ¡dS)Nr)ršr^r€r;r$rr®rpr¯rÁrkrWr>rr°ržrrr€>s  zBertForMultipleChoice.__init__NFcCs |jdd…\}}| d| d¡¡}| d| d¡¡} | d| d¡¡} |j|| | d|d\} } | | ¡} | | ¡} |  d|¡}|durNtƒ}|||ƒ}|S|S)NrrFr?)r?Zreshaper³r$r¯rWr )r{r¹rºr!rZrõZ batch_sizeZ num_choicesZflat_input_idsZflat_token_type_idsZflat_attention_maskrèrúr[Zreshaped_logitsrJr\rrrr£Es.ÿÿ û    zBertForMultipleChoice.forwardrQr½rrržrr^rBr^crS) ÚBertForTokenClassificationa BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of the last layer. Params: `config`: a BertConfig class instance with the configuration to build a new model. `num_labels`: the number of classes for the classifier. Default = 2. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `labels`: labels for the classification output: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_labels]. Outputs: if `labels` is not `None`: Outputs the CrossEntropy classification loss of the output with the labels. if `labels` is `None`: Outputs the classification logits of shape [batch_size, sequence_length, num_labels]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> num_labels = 2 >>> model = BertForTokenClassification(config, num_labels) >>> logits = model(input_ids, token_type_ids, input_mask) rcrUrS)ršr_r€rVr;r$rr®rpr¯rÁrkrWr>rrXržrrr€s   z#BertForTokenClassification.__init__NFc Csˆ|j|||d|d\}}t ¡ ¡ | |¡}Wdƒn1s"wY| |¡}|durBtƒ} | | d|j¡| d¡ƒ} | S|SrY) r$rZget_cuda_rng_trackerÚforkr¯rWr rÄrV) r{r¹rºr!rZrõr rèr[rJr\rrrr£Ÿs  û ÿ z"BertForTokenClassification.forwardr]rQr½rrržrr_cs+ûr_crC) ÚBertForQuestionAnsweringa BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence output that computes start_logits and end_logits Params: `config`: a BertConfig class instance with the configuration to build a new model. Inputs: `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts `extract_features.py`, `run_classifier.py` and `run_squad.py`) `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to a `sentence B` token (see BERT paper for more details). `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max input sequence length in the current batch. It's the mask that we typically use for attention when a batch has varying length sentences. `start_positions`: position of the first token for the labeled span: torch.LongTensor of shape [batch_size]. Positions are clamped to the length of the sequence and position outside of the sequence are not taken into account for computing the loss. `end_positions`: position of the last token for the labeled span: torch.LongTensor of shape [batch_size]. Positions are clamped to the length of the sequence and position outside of the sequence are not taken into account for computing the loss. Outputs: if `start_positions` and `end_positions` are not `None`: Outputs the total_loss which is the sum of the CrossEntropy loss for the start and end token positions. if `start_positions` or `end_positions` is `None`: Outputs a tuple of start_logits, end_logits which are the logits respectively for the start and end position tokens of shape [batch_size, sequence_length]. Examples: >>> # Already been converted into WordPiece token ids >>> input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]]) >>> input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]]) >>> token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]]) >>> config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768, >>> num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072) >>> model = BertForQuestionAnswering(config) >>> start_logits, end_logits = model(input_ids, token_type_ids, input_mask) cs:tt|ƒ |¡t|ƒ|_t |jd¡|_|  |j ¡dSr ) ršrar€r;r$rrÁrkÚ qa_outputsr>rr°ržrrr€ås  z!BertForQuestionAnswering.__init__NFcCsä|j|||d|d\}}| |¡} | jddd\} } |  d¡} |  d¡} |durn|durnt| ¡ƒdkr9| d¡}t| ¡ƒdkrF| d¡}|  d¡} | d| ¡| d| ¡t| d} | | |ƒ}| | |ƒ}||d}|S| | fS) NFr?rrrÈrrGr)r$rbr7Zsqueezer<r³Zclamp_r )r{r¹rºr!Zstart_positionsZ end_positionsrõr rèr[Z start_logitsZ end_logitsZ ignored_indexrJZ start_lossZend_lossrMrrrr£õs2 û            z BertForQuestionAnswering.forwardrNr½rrržrra·s -úra)Ir–Ú __future__rrrrrŠÚloggingrr0r4r.r,rerZtorch.nn.functionalrZ functionalÚFZdata_utils.file_utilsrZ megatron_utilrZtorch.nnr rrr#Ú getLoggerr“r'ÚsetLevelÚINFOr%r0r1ZTF_WEIGHTS_NAMErMrRrTrUrßÚobjectrVZ#apex.normalization.fused_layer_normr˜r™r.r/ÚModuler¦r¾rÎrÜrÞrârãrçrørürþrr rrr;rDrPrRrTr^r_rarrrrÚsŠ      ò?f ÿûE;'':+  _VHHKPT