o j@sddlZddlmZddlmZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZmZdd lmZd d lmZGd d d eZddZGdddZGdddeZdS)N) defaultdict)Enum)_C_ops _legacy_C_ops)core) check_type) to_variable)_dygraph_tracer dygraph_only)in_dynamic_mode)amp_global_statec@seZdZdZdZdZdS)OptimizerStaterr N)__name__ __module__ __qualname__INITUNSCALEDSTEPPEDrrW/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/amp/grad_scaler.pyrsrcCs dtjiS)Nstate)rrrrrr_refresh_optimizer_state%s rc@seZdZdZe       d/dd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.S)0 AmpScalera AmpScaler is used for Auto-Mixed-Precision training/inferring in imperative mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has seventeen methods `scale()`, `unscale_()`, `minimize()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling. Commonly, it is used together with `amp_guard` to achieve Auto-Mixed-Precision in imperative mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 2**15. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 1000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Returns: An AmpScaler object. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) T@@?r c CsZt}|s td|r$|js$|js$|js$td|jd}||_|jr|dks2Jd|dks:Jd||_ ||_ ||_ ||_ ||_ d|_d|_||_ttdgtj|_ttdgtj|_ttdgtj|_ttdgtj|_ttdgtj|_tt|j gtj|_d|_tt|_ dSdS)Nz;current_tracer is None, maybe it is not in imperative mode.zqAmpScaler can only be enabled on CUDAPlace, XPUPlace and CustomPlace, current place is %s, so it makes no effect.F?zThe incr_ratio must be > 1.0.zThe decr_ratio must be < 1.0.r)!r ValueErrorZ_expected_placeZ is_gpu_placeZ is_xpu_placeZis_custom_placewarningswarn_enable_init_loss_scaling _incr_ratio _decr_ratio_incr_every_n_steps_decr_every_n_nan_or_inf _incr_count _decr_count_use_dynamic_loss_scalingrnparrayastypeZbool_ _found_inf_temp_found_inf_value_false_temp_found_inf_fp16_temp_found_inf_bf16_temp_found_inf_fp32float32_scale_cache_founf_infrr_optimizer_states) selfenableinit_loss_scaling incr_ratio decr_ratioincr_every_n_stepsdecr_every_n_nan_or_infuse_dynamic_loss_scalingZtracerrrr__init__Zsb zAmpScaler.__init__cCs\t|dtjjd|jr$tjdkr$|jr$d|_d|_t dtj|js)|S||j S)al Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`AmpScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The Tensor to scale. Returns: The scaled Tensor or original Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) varzAmpScaler.scale()Zfloat16Fz^It is not recommended to use dynamic loss scaling for %s, so GradScaler is disable by default.) rreagerZTensorr#r Z amp_dtyper+r!r"r5r8rArrrscales   zAmpScaler.scalecOs|js |j|i|S|jt|}|dtjur||d\}}t|dr?|d|j |j|i|\}}| d|_ n|j rFd|_ n |j|i|\}}d|_ |j rZ| tt|_||fS)a This function is similar as `Optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `Optimizer.minimize()`. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, ... parameters=model.parameters() ... ) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) r)NN_set_auxiliary_var found_infTF)r#minimizer7idrr_unscalehasattrrEr/_get_auxiliary_varr6r+_updaterr)r8 optimizerargskwargsoptimizer_stateZ optimize_opsZ params_gradsrrrrGs$$   zAmpScaler.minimizec Cs|jsdS|jt|}|dtjurtd|dtjur"tdt|ddr~t|j dt r~g}g}g}g}|j D]A}|dD]:}| dur{| | | j tjjjkrb| | qA| j tjjjkrt| | qA| | qAq;n+trtj|j\}}}ndd |jD}d d |D}d d |D}d d |D}|j|_t|rt||j||jt|j|j|_t|rt||j||j t|j|j |_t|rt||j||j!t|j|j!|_tj|d<dS) a Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. NrzMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step(). _param_groupsrparamscSs g|] }|dur|qSN) _grad_ivar.0paramrrr Cs  z&AmpScaler._unscale..cS g|] }|jtjjjkr|qSr)dtyperVarDescVarTypeFP16rUrrrrXH cSrYr)rZrr[r\BF16rUrrrrXMr^cSrYr)rZrr[r\ZFP32rUrrrrXRr^)"r#r7rHrr RuntimeErrorrgetattr isinstancerQdictrTappendrZrr[r\r]r_r rBZget_grads_listsZ_parameter_listr0r/lenrZcheck_finite_and_unscaler5r1rZ bitwise_orr2r3) r8rMrPZ param_gradsZparam_grads_fp16Zparam_grads_bf16Zparam_grads_fp32grouprWrrrrI s      zAmpScaler._unscalecCs|jsdS|jr5d|_|jd|_|j|jkr3tdt|jt|jt|j |j|j |_d|_dSd|_|jd|_|j|j krN|j|j |_d|_dS)z+ Updates the loss_scaling. Nrr z:Found inf or nan, current scale is: {}, decrease to: {}*{}) r#r6r)r*r(printformatfloatr5r&r'r%r8rrrrLys,    zAmpScaler._updatecC|jS)z Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. )r#rjrrr is_enablezAmpScaler.is_enablecCrk)z Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamicly return true. )r+rjrrris_use_dynamic_loss_scalingrmz%AmpScaler.is_use_dynamic_loss_scalingcCrk)z Return the initial loss scaling factor. Reurns: float: the initial loss scaling factor. )r$rjrrrget_init_loss_scalingrmzAmpScaler.get_init_loss_scalingcCs&||_tt|jgtj|_dS)z Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(int): The new_init_loss_scaling used to update initial loss scaling factor.s N)r$rr,r-r.r4r5r8Znew_init_loss_scalingrrrset_init_loss_scalings zAmpScaler.set_init_loss_scalingcCrk)z Return the multiplier to use when increasing the loss scaling. Reurns: float: the multiplier to use when increasing the loss scaling. r%rjrrrget_incr_ratiormzAmpScaler.get_incr_ratiocCs|dksJd||_dS)a  Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. rz!The new_incr_ratio must be > 1.0.Nrrr8Znew_incr_ratiorrrset_incr_ratio zAmpScaler.set_incr_ratiocCrk)z Get the less-than-one-multiplier to use when decreasing the loss scaling. Reurns: float: the less-than-one-multiplier to use when decreasing the loss scaling. r&rjrrrget_decr_ratiormzAmpScaler.get_decr_ratiocCs|dksJd||_dS)a) Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. rz!The new_decr_ratio must be < 1.0.Nrwr8Znew_decr_ratiorrrset_decr_ratiorvzAmpScaler.set_decr_ratiocCrk)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Reurns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. r'rjrrrget_incr_every_n_stepsrmz AmpScaler.get_incr_every_n_stepscC ||_dS)a^ Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Nr{r8Znew_incr_every_n_stepsrrrset_incr_every_n_steps z AmpScaler.set_incr_every_n_stepscCrk)a  Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Reurns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. r(rjrrrget_decr_every_n_nan_or_infrmz%AmpScaler.get_decr_every_n_nan_or_infcCr})au Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Nrr8Znew_decr_every_n_nan_or_infrrrset_decr_every_n_nan_or_infrz%AmpScaler.set_decr_every_n_nan_or_infc Cs4|jr|j|j|j|j|j|j|j|j dSiS)a Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Reurns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. )rDr;r<r=r> incr_count decr_countr?) r#r5numpyr%r&r'r(r)r*r+rjrrr state_dicts zAmpScaler.state_dictcCs|jsdSt|dkrtd|dd|_tt|jgtj|_ |d|_ |d|_ |d|_ |d|_ |d |_|d |_|d |_dS) z Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `AmpScaler.state_dict()`. NrzdThe input state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rDr;r<r=r>rrr?)r#rer`r$rr,r-r.r4r5r%r&r'r(r)r*r+r8rrrrload_state_dict%s"       zAmpScaler.load_state_dictN)Trrrrr T)rrr__doc__r r@rDrGrIrLrlrnrorqrsrurxrzr|rrrrrrrrrr)s:0 =1Bn            rcs eZdZdZ       d0fdd Zfd d Zfd d ZddZddZfddZ fddZ fddZ fddZ fddZ fddZfddZfd d!Zfd"d#Zfd$d%Zfd&d'Zfd(d)Zfd*d+Zfd,d-Zfd.d/ZZS)1 GradScalerak GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods `scale()`, `unscale_()`, `minimize()`, `step()`, `update()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling, it equal to `step()` + `update()`. `step()` is similar as `optimizer.step()`, which performs parameters updating. `update` is used to update the loss_scaling. Commonly, it is used together with `paddle.amp.auto_cast` to achieve Auto-Mixed-Precision in dynamic graph mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 65536.0. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 2000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 1. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Returns: An GradScaler object. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() T@rrr c st|||||||dSrS)superr@)r8r9r:r;r<r=r>r? __class__rrr@vs zGradScaler.__init__c t|S)aJ Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rrDrCrrrrDs zGradScaler.scalecstj|g|Ri|S)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rrG)r8rMrNrOrrrrGs"zGradScaler.minimizecCs|js|S|jt|}|dtjurtd|dtjur%||t |dr<| d|j || d|_ n|j rCd|_ n|d|_ tj|d<|jsYtt|_dSdS)at This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() rz7step() has already been called since the last update().rErFTFN)r#stepr7rHrrr`rrIrJrEr/rKr6r+rr)r8rMrPrrrrs*   zGradScaler.stepcCs&|jsdS|jr|tt|_dS)a Updates the loss_scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() N)r#r+rLrrr7rjrrrupdate s  zGradScaler.updatecr)aE Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.unscale_(optimizer) # unscale the parameter >>> scaler.step(optimizer) >>> scaler.update() >>> optimizer.clear_grad() )rrI)r8rMrrrunscale_*s !zGradScaler.unscale_c tS)a Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> enable = scaler.is_enable() >>> print(enable) True )rrlrjrrrrlM zGradScaler.is_enablecr)av Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamicly return true. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() >>> print(use_dynamic_loss_scaling) True )rrnrjrrrrnhrz&GradScaler.is_use_dynamic_loss_scalingcr)a% Return the initial loss scaling factor. Reurns: float: the initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> init_loss_scaling = scaler.get_init_loss_scaling() >>> print(init_loss_scaling) 1024 )rrorjrrrrorz GradScaler.get_init_loss_scalingct|dS)a Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_init_loss_scaling()) 1024 >>> new_init_loss_scaling = 1000 >>> scaler.set_init_loss_scaling(new_init_loss_scaling) >>> print(scaler.get_init_loss_scaling()) 1000 N)rrqrprrrrqz GradScaler.set_init_loss_scalingcr)a= Return the multiplier to use when increasing the loss scaling. Reurns: float: the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_ratio = scaler.get_incr_ratio() >>> print(incr_ratio) 2.0 )rrsrjrrrrsrzGradScaler.get_incr_ratiocr)a Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_ratio()) 2.0 >>> new_incr_ratio = 3.0 >>> scaler.set_incr_ratio(new_incr_ratio) >>> print(scaler.get_incr_ratio()) 3.0 N)rrurtrrrrurzGradScaler.set_incr_ratiocr)aV Get the less-than-one-multiplier to use when decreasing the loss scaling. Reurns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_ratio = scaler.get_decr_ratio() >>> print(decr_ratio) 0.5 )rrxrjrrrrxrzGradScaler.get_decr_ratiocr)a7 Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_ratio()) 0.5 >>> new_decr_ratio = 0.1 >>> scaler.set_decr_ratio(new_decr_ratio) >>> print(scaler.get_decr_ratio()) 0.1 N)rrzryrrrrzrzGradScaler.set_decr_ratiocr)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Reurns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_every_n_steps = scaler.get_incr_every_n_steps() >>> print(incr_every_n_steps) 1000 )rr|rjrrrr|.rz!GradScaler.get_incr_every_n_stepscr)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_every_n_steps()) 1000 >>> new_incr_every_n_steps = 2000 >>> scaler.set_incr_every_n_steps(new_incr_every_n_steps) >>> print(scaler.get_incr_every_n_steps()) 2000 N)rrr~rrrrIrz!GradScaler.set_incr_every_n_stepscr)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Reurns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() >>> print(decr_every_n_nan_or_inf) 2 )rrrjrrrrgrz&GradScaler.get_decr_every_n_nan_or_infcr)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_every_n_nan_or_inf()) 2 >>> new_decr_every_n_nan_or_inf = 3 >>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) >>> print(scaler.get_decr_every_n_nan_or_inf()) 3 N)rrrrrrrrz&GradScaler.set_decr_every_n_nan_or_infcr)a. Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() )rrrjrrrrs "zGradScaler.state_dictcr)a; Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() >>> scaler.load_state_dict(scaler_state) N)rrrrrrrszGradScaler.load_state_dict)Trrrrr T)rrrrr@rDrGrrrrlrnrorqrsrurxrzr|rrrrr __classcell__rrrrrBs:5  $< #            $r)r! collectionsrenumrrr,ZpaddlerrZ paddle.baserZpaddle.base.data_feederrZpaddle.base.dygraphrZpaddle.base.frameworkr r Zpaddle.frameworkr Z auto_castr rrrrrrrrs&