o *j#@sddlZddlmZddlZddlZddlmZddl mm Z ddl m Z mZGdddejZd/ddZd d Zd d Zd dZddZGdddejZGdddejZGdddejeZGdddejZGdddejZGdddeZGdddejZdd ZGd!d"d"ejZGd#d$d$ejZ Gd%d&d&ejZ!Gd'd(d(e!Z"Gd)d*d*ejZ#Gd+d,d,e Z$Gd-d.d.eZ%dS)0N)abstractmethod)PretrainedConfigPreTrainedModelcseZdZfddZZS) GroupNormcstt|||jSN)superrforwardfloattypedtypeselfx __class__t/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/multi_modal/guided_diffusion/unet.pyrszGroupNorm.forward)__name__ __module__ __qualname__r __classcell__rrrrrsr'cCs|d}tt| tjd|tjd|j|jd}|dddf|d}tj t |t |gdd}|drRtj |t |ddddfgdd}|S) aY Create sinusoidal timestep embeddings. :param timesteps: a 1-D Tensor of N indices, one per batch element. These may be fractional. :param dim: the dimension of the output. :param max_period: controls the minimum frequency of the embeddings. :return: an [N x dim] Tensor of positional embeddings. r)startendr )deviceNdim) thexpmathlogZarangefloat32torr catcossinZ zeros_like) timestepsrZ max_periodhalfZfreqsargsZ embeddingrrrtimestep_embeddings r,cCLt|tjtjtjfr"|jj|j_|jdur$|jj|j_dSdSdS)z/ Convert primitive modules to float16. N) isinstancennConv1dConv2dConv3dweightdatar*biasZllrrrconvert_module_to_f16+  r7cCr-)zP Convert primitive modules to float32, undoing convert_module_to_f16(). N) r.r/r0r1r2r3r4r r5r6rrrconvert_module_to_f325r8r9cOsV|dkr tj|i|S|dkrtj|i|S|dkr$tj|i|Std|)z4 Create a 1D, 2D, or 3D convolution module. rrzunsupported dimensions: )r/r0r1r2 ValueError)dimsr+kwargsrrrconv_nd?sr>cCs4|rt|t|}tj|t|g|RS||S)a Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pass. :param func: the function to evaluate. :param inputs: the argument sequence to pass to `func`. :param params: a sequence of parameters `func` depends on but does not explicitly take as arguments. :param flag: if False, disable gradient checkpointing. )tupleZCheckpointFunctionapplylen)funcZinputsparamsflagr+rrr checkpointLs rEc s>eZdZdZ d dedededeffdd Zd d ZZS) AttentionPool2dzS Adapted from CLIP: https://github.com/openai/CLIP/blob/main/clip/model.py N spacial_dim embed_dimnum_heads_channels output_dimcspttt||dd|d|_td|d|d|_td||p'|d|_ |||_ t |j |_ dS)Nrr?r:) r__init__r/ Parameterr Zrandnpositional_embeddingr>qkv_projc_proj num_heads QKVAttention attention)r rGrHrIrJrrrrLbs  zAttentionPool2d.__init__cCs|j^}}}|||d}tj|jddd|gdd}||jdddddf|j}||}| |}| |}|dddddfS)NrT)rZkeepdimrr) shapereshaper r&meanrNr%r rOrSrP)r rbcZ_spatialrrrrqs $   zAttentionPool2d.forwardr)rrr__doc__intrLrrrrrrrF]s rFc@seZdZdZeddZdS) TimestepBlockzT Any module where forward() takes timestep embeddings as a second argument. cCsdS)zJ Apply the module to `x` given `emb` timestep embeddings. Nrr rembrrrrszTimestepBlock.forwardN)rrrrYrrrrrrr[|sr[c@seZdZdZddZdS)TimestepEmbedSequentialzt A sequential module that passes timestep embeddings to the children that support it as an extra input. cCs,|D]}t|tr|||}q||}q|Sr)r.r[)r rr]layerrrrrs    zTimestepEmbedSequential.forwardN)rrrrYrrrrrr^s r^c*eZdZdZdfdd ZddZZS) UpsampleaB An upsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then upsampling occurs in the inner-two dimensions. rNcsJt||_|p ||_||_||_|r#t||j|jddd|_dSdS)Nr:rpadding)rrLchannels out_channelsuse_convr<r>conv)r rdrfr<rerrrrLs   zUpsample.__init__cCst|jd|jks J|jdkr(tj||jd|jdd|jddfdd}ntj|ddd}|jr8||}|S)Nrr:rnearestmode)Z scale_factorrk)rTrdr<F interpolaterfrgr rrrrs $ zUpsample.forwardrNrrrrYrLrrrrrrras  racr`) DownsampleaE A downsampling layer with an optional convolution. :param channels: channels in the inputs and outputs. :param use_conv: a bool determining if a convolution is applied. :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then downsampling occurs in the inner-two dimensions. rNcs|t||_|p ||_||_||_|dkrdnd}|r,t||j|jd|dd|_dS|j|jks4Jtj ||d|_dS)Nr:r)rrrr)striderc)Z kernel_sizerq) rrLrdrerfr<r>opr/Z AvgPool2d)r rdrfr<rerqrrrrLs"   zDownsample.__init__cCs|jd|jks J||S)Nr)rTrdrrr rrrrs zDownsample.forwardrnrorrrrrps rpcs@eZdZdZ       d fdd ZddZd d ZZS) ResBlocka A residual block that can optionally change the number of channels. :param channels: the number of input channels. :param emb_channels: the number of timestep embedding channels. :param dropout: the rate of dropout. :param out_channels: if specified, the number of out channels. :param use_conv: if True and out_channels is specified, use a spatial convolution instead of a smaller 1x1 convolution to change the channels in the skip connection. :param dims: determines if the signal is 1D, 2D, or 3D. :param use_checkpoint: if True, use gradient checkpointing on this module. :param up: if True, use this block for upsampling. :param down: if True, use this block for downsampling. NFrc st||_||_||_|p||_||_||_||_t t d|t t |||jddd|_| p4| |_| rGt|d||_t|d||_n| rXt|d||_t|d||_nt |_|_t t t ||rod|jn|j|_t t d|jt t j|dt ||j|jddd|_t j|jdj|j|krt |_dS|rt |||jddd|_dSt |||jd|_dS) N r:rrbFr)pr)rrLrd emb_channelsdropoutrerfuse_checkpointuse_scale_shift_normr/ SequentialrSiLUr> in_layersupdownrah_updx_updrpZIdentityLinear emb_layersZDropout out_layersinitzeros_r3skip_connection) r rdrvrwrerfryr<rxupdownrrrrLs\          zResBlock.__init__cCst|j||f||jS)a  Apply the block to a Tensor, conditioned on a timestep embedding. :param x: an [N x C x ...] Tensor of features. :param emb: an [N x emb_channels] Tensor of timestep embeddings. :return: an [N x C x ...] Tensor of outputs. rE_forward parametersrxr\rrrr,szResBlock.forwardc Cs|jr#|jdd|jd}}||}||}||}||}n||}|||j}t|jt|jkrI|d}t|jt|jks;|j rr|j d|j dd}}t j |ddd\} } ||d| | }||}n ||}| |}| ||S)Nr).Nrrrr)r}r|r~rrr r rArTryrr chunkr) r rr]Zin_restZin_convhZemb_outZout_normZout_restscaleshiftrrrr7s&      zResBlock._forward)NFFrFFFrrrrYrLrrrrrrrrssA rscs:eZdZdZ    d fdd ZddZd d ZZS) AttentionBlocka An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted to the N-d case. https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66. rrFcst||_|dkr||_n||dks Jd|d||||_||_td||_td||dd|_|rAt |j|_ nt |j|_ td||d|_ t j|j jdS)Nrrzq,k,v channels z' is not divisible by num_head_channels rtrr:)rrLrdrQrxrnormr>qkvrRrSQKVAttentionLegacyproj_outr/rrr3)r rdrQnum_head_channelsrxuse_new_attention_orderrrrrLVs    zAttentionBlock.__init__cCst|j|f||jSrrr rrrruszAttentionBlock.forwardcCsV|j^}}}|||d}|||}||}||}||j||g|RS)Nr)rTrUrrrSr)r rrWrXspatialrrrrrrys   zAttentionBlock._forward)rrFFrrrrrrNs rcCsL|dj^}}}tt|}d||d|}|jt|g7_dS)a( A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.profile( model, inputs=(inputs, timestamps), custom_ops={QKVAttention: QKVAttention.count_flops}, ) rrN)rTrZnpprodZ total_opsr Z DoubleTensor)model_xyrWrXrZ num_spatialZ matmul_opsrrrcount_flops_attns rc4eZdZdZfddZddZeddZZS)rzi A module which performs QKV attention. Matches legacy QKVAttention + input/output heads shaping ct||_dSrrrLn_headsr rrrrrL  zQKVAttentionLegacy.__init__c Cs|j\}}}|d|jdksJ|d|j}|||j|d|j|dd\}}}dtt|} td|| || } tj| dd | j } td| |} | |d|S)z Apply QKV attention. :param qkv: an [N x (H * 3 * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r:rrr bct,bcs->btsr bts,bcs->bct) rTrrUsplitr"sqrtr einsumsoftmaxr r r r rbswidthlengthchqkvrr3arrrrs  zQKVAttentionLegacy.forwardcC t|||Srrrrrrrr count_flops zQKVAttentionLegacy.count_flops rrrrYrLr staticmethodrrrrrrrs  rcr)rRzP A module which performs QKV attention and splits in a different order. crrrrrrrrLrzQKVAttention.__init__c Cs|j\}}}|d|jdksJ|d|j}|jddd\}}}dtt|} td|| ||j|||| ||j||} tj| dd | j } td| | ||j||} | |d|S)z Apply QKV attention. :param qkv: an [N x (3 * H * C) x T] tensor of Qs, Ks, and Vs. :return: an [N x (H * C) x T] tensor after attention. r:rrrrrr) rTrrr"rr rviewrr r r rUrrrrrs zQKVAttention.forwardcCrrrrrrrrrzQKVAttention.count_flopsrrrrrrRs  rRcsVeZdZdZ           dfd d Zd d ZddZdddZZS) UNetModela The full UNet model with attention and timestep embedding. :param in_channels: channels in the input Tensor. :param model_channels: base channel count for the model. :param out_channels: channels in the output Tensor. :param num_res_blocks: number of residual blocks per downsample. :param attention_resolutions: a collection of downsample rates at which attention will take place. May be a set, list, or tuple. For example, if this contains 4, then at 4x downsampling, attention will be used. :param dropout: the dropout probability. :param channel_mult: channel multiplier for each level of the UNet. :param conv_resample: if True, use learned convolutions for upsampling and downsampling. :param dims: determines if the signal is 1D, 2D, or 3D. :param num_classes: if specified (as an int), then this model will be class-conditional with `num_classes` classes. :param use_checkpoint: use gradient checkpointing to reduce memory usage. :param num_heads: the number of attention heads in each attention layer. :param num_head_channels: if specified, ignore num_heads and instead use a fixed channel width per attention head. :param num_heads_upsample: works with num_heads to set a different number of heads for upsampling. Deprecated. :param use_scale_shift_norm: use a FiLM-like conditioning mechanism. :param resblock_updown: use residual blocks for up/downsampling. :param use_new_attention_order: use a different attention pattern for potentially increased efficiency. rrrrhTrNFrrc st|dkr |}||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | r1t jnt j|_||_||_||_|d}tt||tt|||_|j dur`t| ||_t|d|}}ttt| ||dddg|_||_ |g}d}t!|D]~\}}t"|D]<}t#|||t||| | |dg}t||}||vr|$t%|| |||d|j$t||j |7_ |$|q|t&|dkr|}|j$t|rt#||||| | |d d nt'|| | |d |}|$||d 9}|j |7_ qtt#|||| | |d t%|| |||dt#|||| | |d |_(|j |7_ tg|_)t*t!|dddD]s\}}t"|dD]g}|+}t#||||t||| | |dg}t||}||vrt|$t%|| |||d|r||kr|}|$|rt#||||| | |d dnt,|| | |d |d }|j)$t||j |7_ qEq;tt-d|tt| ||ddd|_.tj/0|j.dj1dS)Nrrhrr:rrbrer<rxryrxrQrrTrer<rxryrr<rerr<rxry)rer<rxryrrt)2rrL image_size in_channelsmodel_channelsrenum_res_blocksattention_resolutionsrw channel_mult conv_resample num_classesrxr float16r$r rQrnum_heads_upsampler/rzrr{ time_embedZ Embedding label_embrZ ModuleListr^r> input_blocks _feature_size enumeraterangersappendrrArp middle_block output_blockslistpopraroutrrr3) r rrrrerrrwrrr<rrxuse_fp16rQrrryresblock_updownrtime_embed_dimrZinput_chinput_block_chansdslevelmult_layersout_chiZichrrrrLsJ                 )zUNetModel.__init__cC(|jt|jt|jtdSz< Convert the torso of the model to float16. N)rr@r7rrr rrrconvert_to_fp16  zUNetModel.convert_to_fp16cCrz< Convert the torso of the model to float32. N)rr@r9rrrrrrconvert_to_fp32rzUNetModel.convert_to_fp32cCs|du|jduks Jdg}|t||j}|jdur/|j|jdfks(J|||}||j}|jD] }|||}| |q8| ||}|j D]}t j ||gdd}|||}qN||j}||S)a Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :param y: an [N] Tensor of labels, if class-conditional. :return: an [N x C x ...] Tensor of outputs. NzrrrrrsrrrArprpoolrZAdaptiveAvgPool2dZFlattenrrrr3rFZReLUNotImplementedError)r rrrrerrrwrrr<rxrrQrrryrrrrrrrrrrrrrrrrLs                     zEncoderUNetModel.__init__cC|jt|jtdSr)rr@r7rrrrrr z EncoderUNetModel.convert_to_fp16cCrr)rr@r9rrrrrrrz EncoderUNetModel.convert_to_fp32cCs|t||j}g}||j}|jD]}|||}|jdr.|||jj ddq| ||}|jdrT|||jj ddt j |dd}| |S||j}| |S)z Apply the model to an input batch. :param x: an [N x C x ...] Tensor of inputs. :param timesteps: a 1-D batch of timesteps. :return: an [N x K] Tensor of outputs. r)rr:rr)Zaxis)rr,rr r rr startswithrrVrr r&r)r rr)r]resultsrrrrrrs"          zEncoderUNetModel.forward) rrTrFFrrrFFFrrrrrrrs($rcsDeZdZdddddgdddd d d d d dd d d ffdd ZZS) UNetConfigir:r)rt@g)rKrrrrrhrhNFTrhrrc s|||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||_tjdi|dS)Nr)rrrrerrrwrrrxrrQrrryrrrrL)r rrrrerrrwrrrxrrQrrryrrr=rrrrLs$zUNetConfig.__init__)rrrrLrrrrrrs&rcs2eZdZeZfddZdddZddZZS) HFUNetModelcst|tdid|jd|jd|jd|jd|jd|jd|j d|j d |j d |j d |j d |jd |jd|jd|jd|jd|j|_dS)NrrrrerrrwrrrxrrQrrryrrr)rrLrrrrrerrrwrrrxrrQrrryrrr)r configrrrrLsH       zHFUNetModel.__init__NcCs|j|||Sr)rr)r rr)rrrrr szHFUNetModel.forwardcCs.|jjt|jjt|jjtdSr)rrr@r7rrrrrrrszHFUNetModel.convert_to_fp16r) rrrrZ config_classrLrrrrrrrrs   r)r)&r"abcrnumpyrZtorchr Ztorch.nnr/Ztorch.nn.functionalZ functionalrlZ transformersrrrr,r7r9r>rEModulerFr[rzr^rarprsrrrrRrrrrrrrrrs>       !"t4"$ T)