o "j+@sddlZddlmZddlmZddlmZddlmZddlm Z dd l m Z m Z m Z gZd d Zd d ZGdddeZddZdS)N) framework)PyLayer)core)get_rng_state_tracker)utils)check_recompute_necessarydetach_variableswith_rng_state_trackerc Csz|j}|j}|dkr |St|}|dksJd||dks'Jd|||}||}||}||}|||S)Nrrzcan't recompute zero elementzHThe capacity of the activation ({}) cannot be divisible by mp_degree({}))nranksrankpaddleZnumelformatZflatten_) tensormp_group mp_degreemp_rankZ tensor_numeldataZ part_sizestartendrt/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/distributed/fleet/recompute/recompute_hybrid.py_split_activations   rcCs`|j}|j}|dkr |St|j}|d|j9<t||j}|j| |}| |S)Nrr) r r listshaperemptydtypeZ process_groupZ all_gathercudawait)rrrrZ tensor_shapeouttaskrrr_merge_activation6s r"c@s(eZdZdZeddZeddZdS)_HPRecomputeFunctionaz Compared with paddle.distributed.fleet.utils.recompute, there are the following differences: 1. In order to support PipeLineParallel, the input of recompute is modified to ensure that the input can be tuple type. 2. Offload support for activation 3. Support MP segmentation of activation to further reduce cuda memory 4. Adapt to the random state of MP cOs,||_||_t|_t|_||_||_ ||_ g|_ g|_ g|_ g}t} dtvsIdtvsI| ddtjvsIJd| dt} | jtjjkrVdnd|_| jtjjkrdd |_n| jtjjtjjfvrsd |_ntd | j| j|_| \|_ |_!t"||i|} Wdn1swYt#|D]U\} } t$| r| j%}|r|j &| j't(| )|*}|r|+n|} n|r| +n| } || _%|&| |j &| |j &dt,r|r|-| q|j &| q|j.|t$| r|| g7}| S|| 7}t/| S) Nzgpu:zxpu::rz2Recompute with RNG is not support current device: .FTO2O1zunsupported amp level: )0 run_functionkwargsrZ get_rng_state fwd_rng_staterZget_states_trackerfwd_rng_state_trackerroffload partitioninputstensor_indices tensor_shapesZ get_devicesplitZdeviceZget_all_custom_device_typer_dygraph_tracerZ _amp_levelrZAmpLevelZO0is_fw_autocastr& amp_levelr' ValueErrorZ _amp_dtype amp_dtypeZ_get_amp_op_listamp_white_listamp_black_listZno_grad enumerate is_tensor stop_gradientappendrrdetachclonecpuZin_dynamic_modeZmark_non_differentiableZsave_for_backwardtuple)ctxr( all_outputsrr,r-argsr)Z tensor_inputsZ cur_devicetraceroutputsiargstaterrrforwardNst                z_HPRecomputeFunction.forwardc GsBtjjt|j}|j}|j}t|}tj j }t |D]2\}}|j rE||j} t|||j||||<| ||_|jrO|||n||||<q#t} d| _t|j|jJ|jrtjj|j|j|j|j |j!dt"t#|} |j$| i|j%} Wdn1swYnt"t#|} |j$| i|j%} Wdn1swYt&| t'j(j)r| f} t*| t*|ksJg} g}t+t*| D]}t&| |t'j(j)r| |js| ,| ||,||qt*| dkrt-dtj./| |t#dd| D}|WdS1swYdS)NT)enableZcustom_white_listZcustom_black_listlevelrrzInone of output has stop_gradient=False, this recompute() is not necessarycss&|]}t|tjjr|VqdS)N) isinstancereagerTensorZ _grad_ivar).0inprrr s  z0_HPRecomputeFunction.backward..)0rbaseZdygraphguardrr.r/r0Z saved_tensor distributedZ ParallelEnv device_idr9r-r;r"rr=Zreshape_r,rrr2 _has_gradr r*r+r3ampZ auto_castr7r8r4r6r r@r(r)rLrrMrNlenranger< RuntimeErrorZautogradbackward)rArCr.r/r0ZtensorsrUrFidxrHrDZdetached_inputsrEZforward_outputs_with_gradZbackward_inputsZgradsrrrr[s         &z_HPRecomputeFunction.backwardN)__name__ __module__ __qualname____doc__ staticmethodrIr[rrrrr#Es  _r#c Os|dd}|dusJd|dd}|dd}tjr#t|g}tj|||||g|Ri|t|dkr?|dS|D]}t |rPt |sPd |_ qAt |S) ah recompute intermediate activations to save the memory in hybrid parallel scene. # NODTE(shenliang03)The current hybrid parallel recompute has limitations. # It cannot handle the following situations: # 1. The calculation output of recompute, there are tensors that do not require gradients. # 2. The forward output tensor has no gradient. This problem can be solved temporarily by detach(). # 3. Here, we only use float dtype to distinguish whether a gradient is needed in output tensor Parameters: ctx(dict): include 'mp_group', 'offload', and 'partition' keys. the key 'mp_group' (Group), represents the avtivations are splitted in which group. the key 'offload' (bool, optional, default=False), represents whether to offload to cpu. the key 'partition' (bool, optional, default=False), represents whether to split activations in the mp_group. function(paddle.nn.Layer): layer of sequence of layers that describes part of forward pass of the model whose intermediate activations will be released to save memory in forward stage and will be recomputed in backward stage for gradient calculation. *args(Tensor): inputs(tuple) to the function. **kwargs(Dict): inputs(dict) to the function. Returns: Output of function on args and kwargs. rNz8ctx must contains mp_group and mp_group can not be None.r,Fr-rrT)getrr2rVr r#applyrXrr:rZis_float_tensorr;r@) rAfunctionrCr)rr,r-rBoutputrrrrecompute_hybrids.       rf)rrZpaddle.autogradrZpaddle.frameworkrZ$meta_parallel.parallel_layers.randomrZmeta_parallel.pp_utilsrZ recomputer r r __all__rr"r#rfrrrrs      6