o "j@sBddlZddlZddlZddlmZddlmZGdddZdS)N)StrategyGroupBasec@sNeZdZdZidfddZddZddZd d Zd d Zd dZ ddZ dS)OrthogonalStrategya A hybrid of multiple distributed strategies. Strategies need to be orthogonal, means the ranks are organized like a square if there are two strategies, a cube if there aree three strategies, etc. Args: list_of_strategy(list): Stategy in the list should be represented as tuple, format as (strategy_name, degree, strategy_class). fused_strategy_dict(dict, optional): Exist strategies can be fused to new strategy. Use the name of new strategy as key, a list of strategy names you want to fuse as value. Returns: The instance of strategy. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> from paddle.distributed.fleet.base.strategy_group import DPGroup, MPGroup, PPGroup >>> from paddle.distributed.fleet.base.orthogonal_strategy import OrthogonalStrategy >>> dist.init_parallel_env() >>> strategy = OrthogonalStrategy([("dp", 2, DPGroup), ("mp", 2, MPGroup), ("pp", 2, PPGroup)], fused_strategy_dict={"check": ["mp", "pp"]}) Nc s|_|_|dur |nttt_i_i_dd|D_ dd|D_ t dj _ ddj D}fddtj|D}tt|j_t|D]\}}|d}|dj|<|} |d | j|<qXi_dS) NcSg|]}|dqS)r.0strategyrrr/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/distributed/fleet/base/orthogonal_strategy.py >sz/OrthogonalStrategy.__init__..cSr)rrrrr r AZ CoordinatecSsg|]}t|qSr)range)rdegreerrr r Gr csg|]}j|qSr) _coordinate)rcoordselfrr r Hs rr )Z_list_of_strategy_fused_strategy_dictlistr distZget_world_size_strategy_rank_list_name_to_group_dict_name_to_degree_dict_list_of_strategy_name_list_of_degree collections namedtupler_check_valid_strategy itertoolsproductdictzip_coord_to_rank_dict enumerate_calc_rank_list_name_to_fused_group_dict_create_fused_group) rZlist_of_strategyZfused_strategy_dictZstrategy_rank_listrangesZ list_of_coordidxrZ strategy_name rank_listrrr __init__2s@     zOrthogonalStrategy.__init__cCs$||jvs Jd|d|j|S)z Get strategy group with specific name. Args: name: The name of strategy group Returns: An instance of specific strategy group. Strategy group  is not created.)rrrnamerrr strategy_group[  z!OrthogonalStrategy.strategy_groupcCs$||jvs Jd|d|j|S)z Get fused strategy group with specific name. Args: name: The name of fused strategy group Returns: (StrategyGroupBase): An instance of strategy group. zFused strategy group r-)r&r.rrr fused_strategy_groupjr1z'OrthogonalStrategy.fused_strategy_groupcCs(||jvs Jd|d|j|jjS)z Get local rank in strategy group with specific name. Args: name: The name of strategy group Returns: (Integer): Local rank in specific strategy. r,r-)rrgrouprankr.rrr rank_in_strategyys z#OrthogonalStrategy.rank_in_strategycCst|jtt|jksJd|jtdd|j}|t|jks/Jdt|j||j D]}|D]}||jvsGJd|dq8q4dS)NzDefined duplicated strategies: cSs||SNr)xyrrr sz:OrthogonalStrategy._check_valid_strategy..z=There are total {} ranks, but need {} ranks in this strategy.zCan not fuse strategy z without defined previous.) lenrset functoolsreducerrformatrvalues)rZ num_of_ranksfused_strategyrrrr rs,      z(OrthogonalStrategy._check_valid_strategyc Cs|jD]}|j|}tt|j|}g}|D] }|t|j|qg}|D] }|t|j|q)g}tj |D]?}i} g} t |D] \} } | | || <qGtj |D]} t | D] \} }|| || <q]| |j |j di| qW|| q=t ||j|<qdS)Nr)rrr;r differenceappendr rrr r$r#rrr&)rr/r@Znon_fused_strategyZnon_fused_rangesrZ fused_rangesr*Znon_fused_ranksZ coord_dictranksiZnon_fused_rankZ fused_ranksZ fused_rankrrr r's8    z&OrthogonalStrategy._create_fused_groupc Csg}t|jD]\}}||krq|t|qg}tj|D](}g}t|j|D]}t|} | ||||j|j | q*||q|Sr6) r$rrBr rr rinsertr#r) rZ strategy_axisr(r)rr*rrCvalZ coord_listrrr r%s   z"OrthogonalStrategy._calc_rank_list) __name__ __module__ __qualname____doc__r+r0r2r5rr'r%rrrr rs ) r) rr<rZpaddle.distributed distributedrZ,paddle.distributed.fleet.base.strategy_grouprrrrrr s