o "j*+@sddlZddlmZddlmZddlmZddlmZddl m Z gZ GdddZ Gd d d ej d ZGd d d ej d ZdS)N)base)Executor RoleMakerBase)SGD)OptimizerWithMixedPrecisionc@seZdZdZdZdZdZdS)ModezY There are various mode for fleet, each of them is designed for different model. N)__name__ __module__ __qualname____doc__Z TRANSPILERZPSLIBZ COLLECTIVErrg/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/incubate/distributed/fleet/base.pyrs rc@seZdZdZddZddZddZdd Zd d Zd1d dZ ddZ ddZ d1ddZ ddZ ddZddZd2ddZddZd d!Zejd"d#Zejd2d$d%Zejd&d'Zejd(d)Zejd2d*d+Zej  , d3d-d.Zejd2d/d0ZdS)4Fleetz Fleet is the base class, transpiler and pslib are implementation of Fleet. Args: mode(Mode): the implementation of Fleet's mode. Returns: None cCs"d|_||_d|_d|_d|_dS)NF)_is_initialized_mode _optimizer _role_maker _executor)selfmoderrr__init__/s  zFleet.__init__cC |jS)z Check whether the node is the first instance of worker. Returns: bool: True if this is the first node of worker, False if not. )ris_first_workerrrrrr6 zFleet.is_first_workercCr)zV Get current worker index. Returns: int: node id )r worker_indexrrrrr@ zFleet.worker_indexcCr)zd Get current total worker number. Returns: int: worker numbers )r worker_numrrrrr!Ir zFleet.worker_numcCr)z Check whether the node is an instance of worker. Returns: bool: True if this is a node of worker, False if not. )r is_workerrrrrr"RrzFleet.is_workerFcC|r d|jS|jSz Get current server endpoints, such as ["127.0.0.1:1001", "127.0.0.1:1002"]. Returns: list/string: server endpoints ,)joinrZget_trainer_endpointsrZ to_stringrrrworker_endpoints\ zFleet.worker_endpointscCst|jS)zc Get current total worker number. Returns: int: server number )lenrget_pserver_endpointsrrrr server_numiszFleet.server_numcCr)zV Get current server index. Returns: int: node id )r server_indexrrrrr-rr zFleet.server_indexcCr#r$)r&rr+r'rrrserver_endpoints{r)zFleet.server_endpointscCr)z Check whether the node is an instance of server. Returns: bool: True if this is a node of server, False if not )r is_serverrrrrr/rzFleet.is_servercCr)z Check whether the node is an instance of server. Returns: bool: True if this is a node of server, False if not. )ris_xpurrrrr0rz Fleet.is_xpuc Cst|ts td|}|}t||}t||}|g|}t|D] }||d7<q&gg|}d} t|D]}|| | ||||<| ||7} q<||S)a split files before distributed training, example 1: files is [a, b, c ,d, e] and trainer_num = 2, then trainer 0 gets [a, b, c] and trainer 1 gets [d, e]. example 2: files is [a, b], and trainer_num = 3, then trainer 0 gets [a], trainer 1 gets [b], trainer 2 gets [] Args: files(list): file list need to be read. Returns: list: files belongs to this worker. z/files should be a list of file need to be read.r r) isinstancelist TypeErrorrr!r*range) rfilesZ trainer_idZtrainers remainder blocksizeblocksiZ trainer_filesbeginrrr split_filess       zFleet.split_filesNcCsXtt|_|rt|tsddlm}|rt||std||_|j d|_ dS)a< should be called only once in user's python scripts, init() will initialize RoleMaker which is used for identifying current node's role, e.g. worker, server, etc. Args: role_maker(RoleMakerBase): subclass of RoleMakerBase. Returns: None rrz/role_maker must be an instance of RoleMakerBaseTN) rrZCPUPlacerr1rZ,paddle.incubate.distributed.fleet.role_makerr3rZ generate_roler)rZ role_makerZRoleMakerBaseIncubaterrrinits    z Fleet.initcCs|j||dS)z all reduce between workers, only support array of one dim. Args: input(list|numpy.array): array of one dim output(list|numpy.array): array of one dim N)rall_reduce_worker)rinputoutputrrrr=szFleet.all_reduce_workercCs|jdS)z) barrier between workers N)rbarrier_workerrrrrr@szFleet.barrier_workercCdSNrrrrr init_workerzFleet.init_workercKrArBr)rZ model_dirkwargsrrr init_serverrDzFleet.init_servercCrArBrrrrr run_serverrDzFleet.run_servercCrArBrrrrr stop_workerrDzFleet.stop_workercCrArBrrZ optimizerZstrategyrrrdistributed_optimizerrDzFleet.distributed_optimizerTcCrArBr)rexecutordirnameZfeeded_var_namesZ target_vars main_programZexport_for_deploymentZ legacy_formatrrrsave_inference_models zFleet.save_inference_modelcCrArBr)rrKrLrMrrrsave_persistables rDzFleet.save_persistables)FrB)NTF)r r rrrrrr!r"r(r,r-r.r/r0r;r<r=r@abcabstractmethodrCrFrGrHrJrNrOrrrrr$sD         #        r) metaclassc@sXeZdZdZd ddZej    d ddZejddZej    d d d Z dS) DistributedOptimizera< DistributedOptimizer is a wrapper for paddle.base.optimizer A user should pass a paddle.base.optimizer to DistributedOptimizer minimize() function is implemented. DistributedOptimizer is the starting point for a user who wants to run distributed training. The optimized information will be stored in Fleet() instance who holds the global information about current distributed training. Args: optimizer(Optimizer): subclass of Optimizer. strategy(any): the user define config for Optimizer. Returns: None NcCs.t|tjst|tstd||_||_dS)Nz*optimizer must be an instance of Optimizer)r1r __bases__rr3rZ _strategyrIrrrr$s  zDistributedOptimizer.__init__cCdS)a First part of `minimize`, do auto-diff to append backward ops for the current program. Args: loss (Variable): loss variable to run optimizations. startup_program (Program): startup_program for initializing parameters in `parameter_list`. parameter_list (list): list of Variables to update. no_grad_set (set|None): set of Variables should be ignored. callbacks (list|None): list of callables to run when appending backward operator for one parameter. Return: list: list of (param, grad) pair, grad is the output of backward. Examples: See examples in `apply_gradients`. Nr)rZlossZstartup_programparameter_list no_grad_set callbacksrrrbackward-szDistributedOptimizer.backwardcCrU)a Second part of `minimize`, appending optimization operators for given `params_grads` pairs. Args: params_grads (list): list of (param, grad) pair to do optimization. Returns: list: A list of operators appended to the current program. Examples: .. code-block:: python >>> # doctest: +SKIP('The network is not defined.') >>> loss = network() >>> optimizer = base.optimizer.SGD(learning_rate=0.1) >>> params_grads = optimizer.backward(loss) >>> # you may append operations for params_grads here >>> # ... >>> optimizer.apply_gradients(params_grads) Nr)rZ params_gradsrrrapply_gradientsKsz$DistributedOptimizer.apply_gradientscCrU)a Add operations to minimize `loss` by updating `parameter_list`. This method combines interface `backward()` and `apply_gradients()` into one. Args: losses (Variable|Variable List): loss variable to run optimizations. scopes (Scope| Scope List): scope instance. startup_programs (Program|Program List): startup_program for initializing parameters in `parameter_list`. parameter_list (list): list of Variables to update. no_grad_set (set|None): set of Variables should be ignored. Returns: tuple: (optimize_ops, params_grads) which are, list of operators appended; and list of (param, grad) Variables pair for optimization. Nr)rZlossesZscopesZstartup_programsrVrWrrrminimizedszDistributedOptimizer.minimizerB)NNNN) r r rrrrPrQrYrZr[rrrrrSs"    rS)rPZpaddlerZpaddle.base.executorrZ(paddle.distributed.fleet.base.role_makerrZpaddle.optimizerrZpaddle.static.amp.decoratorr__all__rABCMetarrSrrrrs      n