o "j3@sddlZddlmZddlmZddlmZddlmZm Z ddZ ddZ d d Z  dd dZ ddZ    dddZdS)N) framework) data_feeder)_get_global_group_warn_cur_rank_not_in_groupcCs2|r |j||S|j|||}|r||S)N) process_groupZ all_to_all_tensor_on_calc_streamZall_to_all_tensorwait) out_tensor in_tensorgroupsync_opuse_calc_streamtaskrs/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/distributed/communication/stream/all_to_all.py_all_to_all_tensor_in_dygraphsrcCsdt|dkr tdt|dkr|dd|D7}|r"|j||S|j|||}|r0||S)Nr*The input tensor_list should not be empty.cSsg|]}t|qSr)paddleZ empty_like).0Ztensorrrr /s z*_all_to_all_in_dygraph..)len RuntimeErrorrZall_to_all_on_calc_stream all_to_allr)Zout_tensor_listZin_tensor_listr r r r rrr_all_to_all_in_dygraph(s  rc CsLd}|durdn|j}t}tj|fit}|} t|trBt|dkr*t dt|dj dkr;t j |dd} nt j |dd} |} t|trZt|dkrStd|j| jd} t| dgdd |j|d | gid | gi||d d t|tr|stj| ddt|dj dkr|t | ddS|t | |ddSdS)Nalltoallrr)Zaxisz;The 'out_tensor_list' for all_to_all must be an empty list.)dtyper )Zfloat16Zfloat32Zfloat64Zint32Zint64rXZOut)ring_idr )typeZinputsZoutputsattrsF)r )iddistget_world_sizerZ LayerHelperlocals isinstancelistrrshaperstackconcat ValueErrorZ"create_variable_for_type_inferencerrZcheck_variable_and_dtypeZ append_oprextendZunstacksplit) out_tensor_or_tensor_listin_tensor_or_tensor_listr r r Zop_typerZnrankshelperr rrrr_all_to_all_in_static_modeAsV      r.TFcCst|rdS|s|rtd|durtd|durtdtrQ|dur)tn|}t|}t|}|rA|rAt|||||S|sM|sMt|||||Std|dusYJdt |||||S)a Scatter a tensor (or a tensor list) across devices and gather outputs to another tensor (or a tensor list, respectively). Args: out_tensor_or_tensor_list (Union[Tensor, List[Tensor]]): The output. If it is a tensor, it should be correctly-sized. If it is a list, it should be empty or contain correctly-sized tensors. Its data type should be the same as the input. in_tensor_or_tensor_list (Union[Tensor, List[Tensor]]): The input to scatter (must be specified on the source rank). If it is a tensor, it should be correctly-sized. If it is a list, it should contain correctly-sized tensors. Support float16, float32, float64, int32, int64, int8, uint8 or bool as the input data type. group (Group, optional): Communicate in which group. If none is given, use the global group as default. sync_op (bool, optional): Indicate whether the communication is sync or not. If none is given, use true as default. use_calc_stream (bool, optional): Indicate whether the communication is done on calculation stream. If none is given, use false as default. This option is designed for high performance demand, be careful to turn it on except you are clearly know its meaning. Returns: Return a task object. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> out_tensor_list = [] >>> if dist.get_rank() == 0: ... data1 = paddle.to_tensor([[1, 2, 3], [4, 5, 6]]) ... data2 = paddle.to_tensor([[7, 8, 9], [10, 11, 12]]) >>> else: ... data1 = paddle.to_tensor([[13, 14, 15], [16, 17, 18]]) ... data2 = paddle.to_tensor([[19, 20, 21], [22, 23, 24]]) >>> task = dist.stream.alltoall(out_tensor_list, [data1, data2], sync_op=False) >>> task.wait() >>> print(out_tensor_list) >>> # [[[1, 2, 3], [4, 5, 6]], [[13, 14, 15], [16, 17, 18]]] (2 GPUs, out for rank 0) >>> # [[[7, 8, 9], [10, 11, 12]], [[19, 20, 21], [22, 23, 24]]] (2 GPUs, out for rank 1) N5use_calc_stream can only be true in sync op behavior.zThe output should be specified.zThe input should be specified.z:The output and input should be both tensor or tensor list.z3Group can not be used in static graph mode for now.) rrrin_dynamic_moderrZ is_tensorrrr.)r+r,r r r Z out_is_tensorZ in_is_tensorrrrrsV.   rcst||durfddtD}|dur%fddtD}|r0|j||S|j|||}|r@||S)Ncg|] }jdqSrr%r_)r world_sizerrrz/_alltoall_single_in_dygraph..cr1r2r3r4)r r6rrrr7)r r!rangerZ all_to_all_single_on_calc_streamZall_to_all_singler)rr out_split_sizesin_split_sizesr r r r r)r rr6r_alltoall_single_in_dygraphs&    r;cCsRt|rdS|s|rtdtr%|durtn|}t|||||||Std)a Split and Scatter the splitted input tensor to the out tensor across devices. Args: out_tensor(Tensor): The output tensor. Its data type should be the same as the input. in_tensor (Tensor): The input tensor. Its data type should be float16, float32, float64, int32, int64, int8, uint8 or bool. out_split_sizes (List[int], optional): Split sizes of out_tensor for dim[0]. If not given, dim[0] of out_tensor must be divisible by group size and out_tensor will be gathered averagely from all participators. If none is given, use a empty list as default. in_split_sizes (List[int], optional): Split sizes of in_tensor for dim[0]. If not given, dim[0] of in_tensor must be divisible by group size and in_tensor will be scattered averagely to all participators. If none is given, use a empty list as default. group (Group, optional): Communicate in which group. If none is given, use the global group as default. sync_op (bool, optional): Indicate whether the communication is sync or not. If none is given, use true as default. use_calc_stream (bool, optional): Indicate whether the communication is done on calculation stream. If none is given, use false as default. This option is designed for high performance demand, be careful to turn it on except you are clearly know its meaning. Returns: Return a task object. Warning: This API only supports the dygraph mode now. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> local_rank = dist.get_rank() >>> # case 1 >>> output = paddle.empty([2], dtype="int64") >>> if local_rank == 0: ... data = paddle.to_tensor([0, 1]) >>> else: ... data = paddle.to_tensor([2, 3]) >>> task = dist.stream.alltoall_single(output, data, sync_op=False) >>> task.wait() >>> out = output.numpy() >>> print(out) >>> # [0, 2] (2 GPUs, out for rank 0) >>> # [1, 3] (2 GPUs, out for rank 1) >>> # case 2 >>> size = dist.get_world_size() >>> output = paddle.empty([(local_rank + 1) * size, size], dtype='float32') >>> if local_rank == 0: ... data = paddle.to_tensor([[0., 0.], [0., 0.], [0., 0.]]) >>> else: ... data = paddle.to_tensor([[1., 1.], [1., 1.], [1., 1.]]) >>> out_split_sizes = [local_rank + 1 for i in range(size)] >>> in_split_sizes = [i + 1 for i in range(size)] >>> task = dist.stream.alltoall_single(output, ... data, ... out_split_sizes, ... in_split_sizes, ... sync_op=False) >>> task.wait() >>> out = output.numpy() >>> print(out) >>> # [[0., 0.], [1., 1.]] (2 GPUs, out for rank 0) >>> # [[0., 0.], [0., 0.], [1., 1.], [1., 1.]] (2 GPUs, out for rank 1) Nr/zPpaddle.distributed.stream.alltoall_single is only supported in dygraph mode now.)rrrr0rr;)rr r9r:r r r rrralltoall_singles(J r<)NTF)NNNTF)rZpaddle.distributed distributedr rZ paddle.baserZ&paddle.distributed.communication.grouprrrrr.rr;r<rrrrs&   A `$