o "j @sjddlmZddlmZGdddZGdddeZGdddeZGd d d eZGd d d eZ dS) N)RNGStatesTrackerc@sPeZdZdZddZddZddZedd Zed d Z d d Z ddZ dS)StrategyGroupBasea The base class of communication group with distributed strategy. Args: list_of_ranks: A 2D-array, such as `[[0, 1, 2, 3], [4, 5, 6, 7]]`. Ranks in sublist represents they are in the same communication group. Returns: The instance of strategy group. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle.distributed as dist >>> from paddle.distributed.fleet.base.strategy_group import StrategyGroupBase >>> dist.init_parallel_env() >>> strategy_group = dist.fleet.base.strategy_group.StrategyGroupBase([[0, 1], [2, 3]]) >>> print(strategy_group.world_size) 2 cCsFtsJdt|sJdt|_||_||_t|_ dS)z5 Initialize the communication group. z6The global communication group need to be initialized.z#The list_of_ranks can not be empty.N) distZis_initializedlenZget_rank_rank_list_of_ranks _create_group_grouprrandom_states_trackerselfZ list_of_ranksr m/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/distributed/fleet/base/strategy_group.py__init__.s   zStrategyGroupBase.__init__cCs|j||dS)z3 Add random seed for current rank. N)r add)r nameseedr r radd_random_seed;sz!StrategyGroupBase.add_random_seedcC|jS)z0 Get the random states tracker. )r r r r rget_random_states_trackerAsz+StrategyGroupBase.get_random_states_trackercsDg|jD] }t|qtfddD}|r dSS)z The world size of communication group. Returns: Integer if the world_size of each group are equal, or a list of world_size if they are not equal. c3s|] }|dkVqdS)rNr ).0 world_sizeZworld_size_listr r Rs  z/StrategyGroupBase.world_size..r)rappendrall)r ranksZis_valuer rrrGs  zStrategyGroupBase.world_sizecCr)z The communication group which current rank belongs to. Returns: Group if current rank only belong to single communication group, or a list of Group if it belongs many. )r rr r rgroupWzStrategyGroupBase.groupcCs\g|_|jD]}tj|d}|j|vr|j|q|jsdSt|jdkr+|jdS|jS)Nrr) list_of_grouprr new_grouprrr)r rrr r rras    zStrategyGroupBase._create_groupcCsTd|jd}|js|dStt|jD]}|d|dt|j|d7}q|S)Nzseed: z; z No group.zGroup[z]: )Z_seedr"rangerstr)r Z debug_strir r r__repr__qs "zStrategyGroupBase.__repr__N) __name__ __module__ __qualname____doc__rrrpropertyrrrr'r r r rrs    rc eZdZdZfddZZS)DPGroupa! The communication group strategy for data parallel. Args: list_of_ranks: A 2D-array, such as `[[0, 1, 2, 3], [4, 5, 6, 7]]`. Ranks in sublist represents they are in the same communication group. Returns: The instance of data parallel strategy group. c.t|t|jtrJd|jddS)NRank z belongs to multi dp groupssuperr isinstancerlistrr  __class__r rr  zDPGroup.__init__r(r)r*r+r __classcell__r r r5rr.z r.cr-)MPGroupa# The communication group strategy for model parallel. Args: list_of_ranks: A 2D-array, such as `[[0, 1, 2, 3], [4, 5, 6, 7]]`. Ranks in sublist represents they are in the same communication group. Returns: The instance of model parallel strategy group. cr/)Nr0z belongs to multi mp groupsr1r r5r rrr7zMPGroup.__init__r8r r r5rr;r:r;cr-) ShardingGroupa) The communication group strategy for sharding parallel. Args: list_of_ranks: A 2D-array, such as `[[0, 1, 2, 3], [4, 5, 6, 7]]`. Ranks in sublist represents they are in the same communication group. Returns: The instance of sharding parallel strategy group. cr/)Nr0z! belongs to multi sharding groupsr1r r5r rrr7zShardingGroup.__init__r8r r r5rr<r:r<csLeZdZdZfddZeddZeddZedd Zd d Z Z S) PPGroupa) The communication group strategy for pipeline parallel. Args: list_of_ranks: A 2D-array, such as `[[0, 1, 2, 3], [4, 5, 6, 7]]`. Ranks in sublist represents they are in the same communication group. Returns: The instance of pipeline parallel strategy group. csht|t|jtrJd|jdd|_d|_d|_d|_ d|_ d|_ |j dkr2| dSdS)Nr0z belongs to multi pp groupsr!)r2rr3rr4r_send_next_group_send_prev_group_recv_next_group_recv_prev_group_rank_of_next_stage_rank_of_prev_stager_create_p2p_groupr r5r rrs    zPPGroup.__init__cCr)z Rank of the previous pp stage. Returns: The global rank of previous pp stage. `None` if without previous. )rCrr r rrank_of_prev_stagerzPPGroup.rank_of_prev_stagecCr)z Rank of the next pp stage. Returns: The global rank of next pp stage. `None` if without next. )rBrr r rrank_of_next_stagerzPPGroup.rank_of_next_stagecCs|j|j|j|jfS)z Communication subgroup in order to switch data with previous and next stage. Returns: Four subgroups including send/recv to/from prev/next. )r>r?r@rArr r r p2p_groupss  zPPGroup.p2p_groupsc Cs|j}|jD]X}t|D]Q\}}||d|}||d|}|j|kr+||_||_tj||gd}|j|kr<||_n|j|krD||_ tj||gd}|j|krU||_ q |j|kr]||_ q q|jrk|j rk|j rk|j stJd|jddS)Nr!r z/Error occurs while creating p2p group for rank .) rr enumeraterrBrCrr#r>rAr?r@) r ZdegreeridxrankZ next_rankZ prev_rankZ next_groupZ prev_groupr r rrDs<       zPPGroup._create_p2p_group) r(r)r*r+rr,rErFrGrDr9r r r5rr=s    r=) Zpaddle.distributed distributedrZ#paddle.distributed.fleet.layers.mpurrr.r;r<r=r r r rs  f