o *j@sPddlZddlmZddlmZeZd ddZ  d ddZ  d d d ZdS) N)nn) get_loggercCs|jd}tjdd}tj|tjdd|tjdd|ddtj|tjdd|tjdd|dd}|durKt||t||}|St||}|S)a Calculate two logits' the KL div value symmetrically. :param logits1: The first logit. :param logits2: The second logit. :param attention_mask: An optional attention_mask which is used to mask some element out. This is usually useful in token_classification tasks. If the shape of logits is [N1, N2, ... Nn, D], the shape of attention_mask should be [N1, N2, ... Nn] :return: The mean loss. none)Z reduction)dimN)shaperZ KLDivLosstorchsumZ LogSoftmaxZSoftmaxmean)Zlogits1Zlogits2attention_maskZ labels_numZKLDivlossr k/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/structbert/adv_utils.py_symmetric_kl_divs4 rh㈵>cKsj|dur|nd|}||j|d|}|dd|vr&|dd|vr,dn|d} |d} | s8d} d|vrA|d|di|d|i} | j} t|| | } tj | |dj}|j dd t d d j d d d d}t t|}|rtd|S||d}|||}t |||}t|||}|di|d|i} | j}t||| }||S)a4 Calculate the adv loss of the model. :param embedding: Original sentence embedding :param model: The model, or the forward function(including decoder/classifier), accept kwargs as input, output logits :param ori_logits: The original logits outputted from the model function :param ori_loss: The original loss :param adv_grad_factor: This factor will be multiplied by the KL loss grad and then the result will be added to the original embedding. More details please check:https://arxiv.org/abs/1908.04577 The range of this value always be 1e-3~1e-7 :param adv_bound: adv_bound is used to cut the top and the bottom bound of the produced embedding. If not proveded, 2 * sigma will be used as the adv_bound factor :param sigma: The std factor used to produce a 0 mean normal distribution. If adv_bound not proveded, 2 * sigma will be used as the adv_bound factor :param kwargs: the input param used in model function :return: The original loss adds the adv loss Nr input_ids inputs_embedswith_attention_maskFr Tinfrkeepdimprz'Nan occurred when calculating adv loss.gư>r datanewsizeZnormal_popZlogitsrrZautogradZgradZnormfloatmaxanyisnanloggerwarningmin) embeddingmodelZ ori_logitsori_lossadv_grad_factor adv_boundsigmakwargs embedding_1rr outputsZ v1_logitsr emb_grad emb_grad_normis_nan embedding_2Z adv_logitsadv_lossr r rcompute_adv_loss3sN         r5cKsP|dur|nd|}||j|d|} |dd|vr&|d|d i|d| i} | j\} } t|| t|| } | d} tj | | dj}|j ddt ddj d dd d}t t|}|rptd |S||}| ||}t | ||}t| ||}|d i|d|i} | j\}}t||t||}||S) a Calculate the adv loss of the model. This function is used in the pair logits scenario. :param embedding: Original sentence embedding :param model: The model, or the forward function(including decoder/classifier), accept kwargs as input, output logits :param start_logits: The original start logits outputted from the model function :param end_logits: The original end logits outputted from the model function :param ori_loss: The original loss :param adv_grad_factor: This factor will be multiplied by the KL loss grad and then the result will be added to the original embedding. More details please check:https://arxiv.org/abs/1908.04577 The range of this value always be 1e-3~1e-7 :param adv_bound: adv_bound is used to cut the top and the bottom bound of the produced embedding. If not proveded, 2 * sigma will be used as the adv_bound factor :param sigma: The std factor used to produce a 0 mean normal distribution. If adv_bound not proveded, 2 * sigma will be used as the adv_bound factor :param kwargs: the input param used in model function :return: The original loss adds the adv loss NrrrrTrrrrz,Nan occurred when calculating pair adv loss.r r)r'r(Z start_logitsZ end_logitsr)r*r+r,r-r.r/Zv1_logits_startZ v1_logits_endr r0r1r2r3Zadv_logits_startZadv_logits_endr4r r rcompute_adv_loss_pairosX       r6)N)Nr)rrZmodelscope.utils.loggerrr$rr5r6r r r rs     B