o )j @sddlZddlZddlZddlmZmZmZddlZddlm Z ddlm Z ddl m Z m Z de de de fdd Zde de de fd d ZGd d d eZGdddej jZeeefZGdddejjZddZde defddZGdddejjZGdddej jZdddede jfddZGd d!d!ej jZ dVde d"ed#ed$e!de f d%d&Z"Gd'd(d(ejjZ#d)d*Z$Gd+d,d,ejjZ% - .dWde d/ed0ed1ed2e&f d3d4Z'de de fd5d6Z(Gd7d8d8ej jZ)Gd9d:d:e jZ*Gd;d<dd>ej jZ,Gd?d@d@ej jZ-GdAdBdBejjZ.GdCdDdDej jZ/GdEdFdFej jZ0de fdGdHZ1de fdIdJZ2de fdKdLZ3de fdMdNZ4GdOdPdPej jZ5de dQede fdRdSZ6e7dTkr{e89ej:e;dUe[final x] are map to [initial y], [final y] respectively. cGst|dks Jt|t|dkr"t|dtr"t|dj|_ndd|D|_|jD]\}}t|ttfs>Jt|t|ttfsKJt|q-tt|jdD]!}|j|dd|j|dksvJ||j||j|dfqUdS)NrcSs g|] \}}t|t|fqSr)float.0rr rrr I z,PiecewiseLinear.__init__..) len isinstancerlistpairsrinttyperange)selfargsrr irrr__init__Ds" zPiecewiseLinear.__init__cCsdt|jdddS)NzPiecewiseLinear(r))strr"r&rrr__str__UszPiecewiseLinear.__str__cCs||jddkr|jddS||jddkr |jddS|jd\}}tdt|jD]&}|j|\}}||krP||krP|||||||S||}}q/J)Nrrr*)r"r%r)r&rZcur_xZcur_yr(Znext_xZnext_yrrr__call__Ys zPiecewiseLinear.__call__cstfdd|jDS)Ncsg|] \}}||fqSrrralpharrrisz+PiecewiseLinear.__mul__..)rr")r&r1rr0r__mul__hszPiecewiseLinear.__mul__csPtttfrtfdd|jDS|\}tddt|jjDS)Ncs g|] }|d|dfqSrrr)rprrrrmrz+PiecewiseLinear.__add__..cSs(g|]\}}|d|d|dfqSr3rrspZxprrrros)r rr#rr"get_common_basiszipr&rsrr5r__add__ks  zPiecewiseLinear.__add__cCsHt|ttfr td|f}|j|dd\}}tddt|j|jDS)NrTinclude_crossingscS*g|]\}}|dt|d|dfqSr3)r r6rrrrvz'PiecewiseLinear.max..r rr#rr8r9r"r:rrrr rs   zPiecewiseLinear.maxcCsNt|ts t|trtd|f}|j|dd\}}tddt|j|jDS)NrTr=cSr?r3)minr6rrrr}r@z'PiecewiseLinear.min..rAr:rrrrBys   zPiecewiseLinear.mincCs |j|jkSr )r")r&otherrrr__eq__s zPiecewiseLinear.__eq__Fr4r>csrtts JtttddjDddjD}fdd|D}fdd|D}|rg}tt|dD]M}||||k}||d||dk} || krt||||} t||d||d} | | | } ||| ||d||} | | q=t|dkrtt||}fdd|D}fdd|D}tt ||tt ||fS) aH Returns (self_mod, p_mod) which are equivalent piecewise linear functions to self and p, but with the same x values. p: the other piecewise linear function include_crossings: if true, include in the x values positions where the functions indicate by this and p cross. cSsg|]\}}|qSrr)rr_rrrrz4PiecewiseLinear.get_common_basis..cg|]}|qSrrrrr-rrrrFcrGrrrHr4rrrrFrrcrGrrrHr-rrrrFcrGrrrHrIrrrrF) r rr$sortedsetr"r%rrappendr9)r&r4r>Zx_valsZy_vals1Zy_vals2Z extra_x_valsr(Z_compare_results1Z_compare_results2Zdiff_curZ diff_nextposZ extra_x_valr)r4r&rr8s2 "      z PiecewiseLinear.get_common_basisN)F)__name__ __module__ __qualname____doc__r)r.r/r2r<r rBrDboolr8rrrrr=s rcsReZdZdZdddeffddZdefdd Zd d Zd d Z ddZ Z S)ScheduledFloataH This object is a torch.nn.Module only because we want it to show up in [top_level module].modules(); it does not have a working forward() function. You are supposed to cast it to float, as in, float(parent_module.whatever), and use it as something like a dropout prob. It is a floating point value whose value changes depending on the batch count of the training loop. It is a piecewise linear function where you specify the (x,y) pairs in sorted order on x; x corresponds to the batch index. For batch-index values before the first x or after the last x, we just use the first or last y value. Example: self.dropout = ScheduledFloat((0.0, 0.2), (4000.0, 0.0), default=0.0) `default` is used when self.batch_count is not set or not in training mode or in torch.jit scripting mode. defaultrVcs*td|_d|_||_t||_dSr )superr) batch_countnamerVrschedule)r&rVr' __class__rrr)s zScheduledFloat.__init__r cCs"d|jdt|jjddS)Nz batch_count=z , schedule=rr*)rXr,rZr"r-rrr extra_reprs zScheduledFloat.extra_reprcCsn|j}|dus|jrtjstjrt|jS||j}t dkr5t d|j d|jd||S)Ng-C6*?zScheduledFloat: name=z, batch_count=z, ans=) rXtrainingr rr is_tracingrrVrZrandomlogginginforY)r&rXansrrr __float__s   zScheduledFloat.__float__cCsDt|ts t|trt|j||jdSt|j|j|j|jdSNrU)r rr#rSrZrVr&rrrrr<s zScheduledFloat.__add__cCsJt|ts t|trt|j||jdSt|j|jt|j|jdSre)r rr#rSrZr rVrfrrrr s   zScheduledFloat.max) rNrOrPrQrr)r,r]rdr<r __classcell__rrr[rrSs rSc@s8eZdZdZededefddZedefddZd S) SoftmaxFunctionz Tries to handle half-precision derivatives in a randomized way that should be more accurate for training than the default behavior. rdimcCs<|j|d}tr|tj}|||j|_||_|S)Nri) softmaxr is_autocast_enabledtofloat16save_for_backwarddtypeZx_dtyperi)ctxrrircrrrforwards   zSoftmaxFunction.forwardans_gradcCs||j\}tjjjdd(|tj}|tj}||}|||j|jdd}|dfWdS1s7wYdS)NFenabledTriZkeepdim) saved_tensorsr cudaampautocastrmfloat32sumri)rqrsrcx_gradrrrbackwards  $zSoftmaxFunction.backwardN) rNrOrPrQ staticmethodrr#rrr~rrrrrhs  rhcCs@|j|dd\}}||||j|dd}|||S)NTrv)r Zsub_Zexp_r|Zdiv_)tensorriZmax_valsrEZsum_exprrrinplace_softmax s    rricCs0|jr tjs tjrt||St||Sr ) requires_gradr rrr_rrhapply)rrirrrrk(s  rkc @sHeZdZededededededef ddZed edefd d Zd S) BiasNormFunctionrbias log_scale channel_dimstore_output_for_backpropr c Cs|jdksJ|dkr||j}||_||_t|d|jD]}|d}qtj||d|dd}|d|}||} ||rF| n|| | | | S)Nrrr*Trv) ndimrrr% unsqueezer meanrrodetach) rqrrrrrrE_x_bias_squarescalesrcrrrrr9s&   zBiasNormFunction.forwardrsc Cs|j\}}}}|jr||}n|}|}d|_d|_d|_t'tj||d|jdd}|d|}||}|j |dWdn1sLwY|j |j |j ddfS)NTrrvrZgradient) rwrrrr enable_gradrrrr~gradflatten) rqrsZans_or_xrrrrrrcrrrr~Us"  zBiasNormFunction.backwardN) rNrOrPrrr#rRrrr~rrrrr2s" rcs\eZdZdZ     ddeded ed ed ed ed dffdd Zded efddZ Z S)BiasNorma This is intended to be a simpler, and hopefully cheaper, replacement for LayerNorm. The observation this is based on, is that Transformer-type networks, especially with pre-norm, sometimes seem to set one of the feature dimensions to a large constant value (e.g. 50), which "defeats" the LayerNorm because the output magnitude is then not strongly dependent on the other (useful) features. Presumably the weight and bias of the LayerNorm are required to allow it to do this. Instead, we give the BiasNorm a trainable bias that it can use when computing the scale for normalization. We also give it a (scalar) trainable scale on the output. Args: num_channels: the number of channels, e.g. 512. channel_dim: the axis/dimension corresponding to the channel, interpreted as an offset from the input's ndim if negative. This is NOT the num_channels; it should typically be one of {-2, -1, 0, 1, 2, 3}. log_scale: the initial log-scale that we multiply the output by; this is learnable. log_scale_min: FloatLike, minimum allowed value of log_scale log_scale_max: FloatLike, maximum allowed value of log_scale store_output_for_backprop: only possibly affects memory use; recommend to set to True if you think the output of this module is more likely than the input of this module to be required to be stored for the backprop. r*??F num_channelsrr log_scale_min log_scale_maxrr Ncs^tt|||_||_tt||_ tt |j ddd|_ ||_ ||_||_dS)Nrg-C6?)rZstd)rWrr)rrnn Parameterr rremptyZnormal_rrrr)r&rrrrrrr[rrr)s  zBiasNorm.__init__rcCs|j|j|jks JtjstjrM|j}|dkr!||j7}|j}t |d|jD]}| d}q,tj ||d|dd}|d|j }||St|j t|jt|j|jd}t||j||j|jS) Nrrr*rTrvr)rBr r^)shaperrr rrr_rrr%rrrrlimit_param_valuerrrr^rrr)r&rrrrErrrrrrrrs.   zBiasNorm.forward)r*rrrF) rNrOrPrQr#rrRr)rrrrgrrr[rrjs.!rr) initial_scalercOstj|i|}t/|jdd|9<|jdur3tjj|jd|d|Wd|SWd|S1s>wY|S)aT Behaves like a constructor of a modified version of nn.Linear that gives an easy way to set the default initial parameter scale. Args: Accepts the standard args and kwargs that nn.Linear accepts e.g. in_features, out_features, bias=False. initial_scale: you can override this if you want to increase or decrease the initial magnitude of the module's output (affects the initialization of weight_scale and bias_scale). Another option, if you want to do something like this, is to re-initialize the parameters. N皙皙?)rLinearr no_gradweightrinituniform_)rr'kwargsrcrrr ScaledLinears    rc s|eZdZdZ  ddedededeffdd Zdd ed ed efddZ d efddZ d eded e eeffddZ Z S)ChunkCausalDepthwiseConv1da? Behaves like a depthwise 1d convolution, except that it is causal in a chunkwise way, as if we had a block-triangular attention mask. The chunk size is provided at test time (it should probably be kept in sync with the attention mask). This has a little more than twice the parameters of a conventional depthwise conv1d module: we implement it by having one depthwise convolution, of half the width, that is causal (via right-padding); and one depthwise convolution that is applied only within chunks, that we multiply by a scaling factor which depends on the position within the chunk. Args: Accepts the standard args and kwargs that nn.Linear accepts e.g. in_features, out_features, bias=False. initial_scale: you can override this if you want to increase or decrease the initial magnitude of the module's output (affects the initialization of weight_scale and bias_scale). Another option, if you want to do something like this, is to re-initialize the parameters. rTchannels kernel_sizerrcst|ddks J|dd}tj||||ddd|_tj|||||d|d|_ttd|||_ ||_ t :|jj dd|9<|jj dd|9<|rqtjj |jjd|d|WddSWddS1s|wYdS)NrrrT) in_channels out_channelsgroupsrpaddingrrr)rWr)rZConv1d causal_convchunkwise_convrr zeroschunkwise_conv_scalerrrrrr)r&rrrrZhalf_kernel_sizer[rrr)sD     "z#ChunkCausalDepthwiseConv1d.__init__r*r chunk_sizer c Cs |j\}}}|jd}|dks||kr|}| |}tjj|||f}||dd||f}|j|||fks;J|d|df} | jd|} | ||| |} | dddd|| ||} | | } | |} | | } | || ||dddd} | ||| |dd|f} | |S)zForward function. Args: x: a Tensor of shape (batch_size, channels, seq_len) chunk_size: the chunk size, in frames; does not have to divide seq_len exactly. rr.Nr) rrr r functionalpadrZreshapeZpermuter_get_chunk_scale) r&rr batch_sizerseq_lenleft_padZ right_padx_causalx_chunkZ num_chunks chunk_scalerrrrrs<         z"ChunkCausalDepthwiseConv1d.forwardcCs|jd}|jd}||jkr%|ddd|f}|dd| df}n'||j}|jd}tj|||j|jd}tj||fdd}tj||fdd}d||S)zxReturns tensor of shape (num_channels, chunk_size) that will be used to scale the output of self.chunkwise_conv.rrN)devicerpr*rjr)rrrr rrrpcat)r&rZ left_edgeZ right_edgetrrrrrrAs       z+ChunkCausalDepthwiseConv1d._get_chunk_scalecachec Cs|j\}}}|jd}|jd|ksJ|jd|ftj||gdd}|d| df}||}|j|||fks      "zSwooshRFunction.forwardrcCs,|j\}d}d}|||d|}||S)Nrrrr)rqrrrrrrrr~ms zSwooshRFunction.backwardNrrrrrrBs #rc@r)SwooshRrr cCrzReturn Swoosh-R activation.rTrrrr) r rrr_rrprrrrrrrrrryrzSwooshR.forwardNrrrrrrwrrc@r) SwooshROnnxrr cCrrrrrrrrrrzSwooshROnnx.forwardNrrrrrrrrcCsF|d}d||j}t|tdk||}|d|dS)Nrrinfrrrlogrmrpr rrrx_offsetlog_sumrrrSwooshLForwardrcCs^|d}d||j}t|tdk||}ddd|}|d|d|fS)zo https://k2-fsa.github.io/k2/python_api/api.html#swoosh-l-forward-and-deriv :param x: :return: rrrq= ףp?rrrrrrrZderivrrrSwooshLForwardAndDeriv r cCsF|d}d||j}t|tdk||}|d|dS)NrrrrrrrrrSwooshRForwardr rcCs^|d}d||j}t|tdk||}ddd|}|d|d|fS)zo https://k2-fsa.github.io/k2/python_api/api.html#swoosh-r-forward-and-deriv :param x: :return: rrr rrrrr rrrSwooshRForwardAndDerivr rcs\eZdZdZ     ddeded ed ed ed eed e ffdd Z de fddZ Z S)ActivationDropoutAndLineara2 This merges an activation function followed by dropout and then a nn.Linear module; it does so in a memory efficient way so that it only stores the input to the whole module. If activation == SwooshL and dropout_shared_dim != None, this will be equivalent to: nn.Sequential(SwooshL(), Dropout3(dropout_p, shared_dim=dropout_shared_dim), ScaledLinear(in_channels, out_channels, bias=bias, initial_scale=initial_scale)) If dropout_shared_dim is None, the dropout would be equivalent to Dropout2(dropout_p). Note: Dropout3 will be more memory efficient as the dropout mask is smaller. Args: in_channels: number of input channels, e.g. 256 out_channels: number of output channels, e.g. 256 bias: if true, have a bias activation: the activation function, for now just support SwooshL. dropout_p: the dropout probability or schedule (happens after nonlinearity). dropout_shared_dim: the dimension, if any, across which the dropout mask is shared (e.g. the time dimension). If None, this may be less memory efficient if there are modules before this one that cache the input for their backprop (e.g. Balancer or Whiten). TrrTr*rrrr activation dropout_pdropout_shared_dimrc sFtt||||d}|j|_|d|j||_||_||_dS)N)rrr) rWr)rrZregister_parameterrrrr) r&rrrrrrrZ linear_moduler[rrr)s  z#ActivationDropoutAndLinear.__init__rcCsftjs tjs |js1|jdkrt|}n|jdkr!t|}nJ|jtjj ||j |j SdS)Nrr) r rrr_r^rrrrrZlinearrrrfrrrrrs     z"ActivationDropoutAndLinear.forward)TrrTr*r)rNrOrPrQr#rRr,rrrr)rrrrgrrr[rrs.rrcCsn||jdkr|ddd|ddddfSt|j}||d|d<tj||j|jd}tj||fddS)zb :param x: (b, c, t, f) :param num_channels: :return: x: (b, num_channels, t, f) rNrrj)rr!r rrprr)rrrrrrrconvert_num_channels s   rrrr )rT)=ramathr`typingrrrr Ztorch.nnrrZtorch.cuda.amprrrrobjectrModulerSrrZautogradFunctionrhrr#rkrrrrrr,rrrrrRrrrrrrrrrrrr rrrrrN getLoggersetLevelINFOZset_num_threadsZset_num_interop_threadsrrrrs   s < 8O) !    65 O