o "jAs@sNddlZddlmZmZddlmZddlmZmZddlm Z ddl m Z m Z m Z ddlmZddlmZd d lmZmZGd d d eZGd ddeZd.ddZd/ddZd/ddZGdddeZejddddfddZd0ddZGdddeZ   d1d!d"Zd2d#d$Z d%d&Z! d/d'd(Z" d/d)d*Z# +    d3d,d-Z$dS)4N)_C_ops _legacy_C_ops)PyLayer) check_dtypecheck_variable_and_dtype) collective) LayerHelper_create_tensorin_dynamic_mode)Layer) dygraph_utils)ReduceOp_get_reduce_opc@$eZdZeddZeddZdS)c_identity_eagerc Cs&||_|r|St|ddd|jddSNuse_calc_streamTring_iduse_model_parallel)groupr c_identityid)ctxtensorrskip_c_identity_dynamicrk/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/distributed/fleet/layers/mpu/mp_ops.pyforwardszc_identity_eager.forwardcCs ttjd}|jj|||S)N _c_identity)rrSUMr process_groupall_reduce_on_calc_stream)rdyop_typerrrbackward,s zc_identity_eager.backwardN__name__ __module__ __qualname__ staticmethodrr%rrrrrs  rc@r) c_split_eagerc Cs,||_||_t|ddd|jd|d|dd S)NrTrranknranksr)rr-rc_splitr)rrrr,r-rrrr4szc_split_eager.forwardcCs@|j}|j}|d|j|d<tj||jd}|j|||S)Nrdtype)rshaper-paddleemptyr0r!Z%all_gather_into_tensor_on_calc_stream)rr#r out_shapeoutrrrr%Fszc_split_eager.backwardNr&rrrrr+3s  r+FcCs|dur |s dS|durdn|j}trt|||Sd}t|fit}|j|jd}t |dgdd|j |d|id |i|d d d d |S) a' Return a copy of the tensor, mainly used with model parallel. Args: tensor (Tensor): The input Tensor. Its data type should be float16, float32, float64, int32 or int64. group (int): The id of the process group to work on. Returns: Tensor. Nrrr/rfloat16float32float64int32int64Zuint16rXOutT)rrrtypeinputsZoutputsattrs) is_memberrr rapplyrlocals"create_variable_for_type_inferencer0r append_op)rrrrr$helperr5rrrrSs0  rc Cs|dur |s dS|durtn|}|j}tj}|j}|j}tr4t |d|ddd|d|dd Sd}t |fit }|j |j d }t|d gd d |j|d |id|i|dd||dd|S)a* Return allgather of the tensor, mainly used with model parallel. Args: tensor (Tensor): The input Tensor. Its data type should be float16, float32, float64, int32 or int64. group (int): The id of the process group to work on. Returns: Tensor. NrrTr,r-rc_concatr/rr6 _c_concatr<r=)rrrr-r,r>)rBr_get_default_groupr_get_global_envr,r-r rrHrrDrEr0rrF rrr global_rankr,r-r$rGr5rrrrI~sT   rIc Cs|dur |s dS|durdn|j}tj}|dur|n||}|dur,tjn|j}tr:t ||||Sd}t |fit }|j |jd}t|dgdd|j|d|id |i|d ||d d d |S) af Split tensor evenly among all members, mainly used with model parallel. Args: tensor (Tensor): The input Tensor. Its data type should be float16, float32, float64, int32 or int64. rank (int): The rank of the current process. group (int): The id of the process group to work on. Returns: Tensor. Nrr.r/rr6_c_splitr<r=T)rrr,r-rr>)rBrrrKr,get_group_rank world_sizer-r r+rCrrDrEr0rrFrLrrrrNs@    rNc@r)mp_allreduce_eagercCsB|j|_||_|rt|d}|j|||St|d|d|jS)N _mp_allreducerr)rrrrr!r"rZc_allreduce_sum_)rrrrroprr$rrrrs  zmp_allreduce_eager.forwardc Cs"|jr|St|ddd|jddSr)rrrr)rr#rrrr% szmp_allreduce_eager.backwardNr&rrrrrQs  rQTc Cs|dur |s dStr.|durtn|}|tjks$Jd|dt||||||S|dur4dn|j}d}t |fit }|j |j d} t |dgd||j|d |id | i||d d | S) z`[it is same as allreduce above, but it supports model parallel. And it support inplace startegy]NzUnknown parameter: .rmp_allreduce_sumr/r)r7r8r9r:Z int64uint16r<r=rrr>)rBr rrJrr rQrCrrrDrEr0rrF) rrSrrrrrr$rGr5rrrrRs@   rRc Csztr t||||Sd}t|fit}|jdd}t|dddg|||}|jd||dd|i||d d |S) aX Lookup table according to index. Args: table (Tensor): The input Tensor. Its data type should be float16, float32, float64. index (Tensor): The index to lookup table. start_index (int): The initial index for table range. name (string): The name of the api Returns: Tensor. c_embeddingtable)Zinput_param_nameinputr:r;)ZIdsWr=) start_index vocab_sizer>) r rrXrrDZ input_dtyperrErF) rYindexr\r]namer$rGr0tmprrr_c_lookup_tableMs  racs8eZdZdZ   d fdd ZddZddZZS) _Linearz Linear Ncsbt|j|_||_||_|j||g|j|jdd|_|j|g|j|jdd|_ ||_ dS)NF)r1attrr0is_biasT) super__init__Z_helperget_default_dtype_dtypeZ _weight_attr _bias_attrcreate_parameterweightbiasr_)selfZ in_featuresZ out_features weight_attr bias_attrr_ __class__rrrfqs"   z_Linear.__init__cCst||j|j|jd}|S)N)xrkrlr_)_linearrkrlr_)rmrZr5rrrrsz_Linear.forwardcCs8|jr d|jnd}d|jjd|jjd|j|S)Nz, name=z+in_features={}, out_features={}, dtype={}{}r)r_formatrkr1rh)rmZname_strrrr extra_reprsz_Linear.extra_repr)NNN)r'r(r)__doc__rfrrw __classcell__rrrprrblsrbc Cs\|dur |s dS|durdn|j}tj}|dur|n||}|dur,tjn|j}tt |j } tt |j } | d| krR| | krRt d| d| d| d| kr_t j |dd}trzt||d|d |d |d | \} } |sv| S| | fS||||d } tdit}|j|jd} |j|jd} |jd ||d| | d| d|r| | fS| S)Nrruz\Expected input_dims - 1 = label_dims or input_dims == label_dims (got input_dimsz , label_dims)rWaxisrr,r- ignore_index)rr,r-r~c_softmax_with_cross_entropyr/)ZLogitsLabel)ZSoftmaxZLossr>)r)rBrrrKr,rOrPr-lenlistr1 ValueErrorr2Z unsqueezer rrrrDrEr0rF)ZlogitslabelrZreturn_softmaxr~rrMr,r-Z input_dimsZ label_dimsZsoftmaxZlossrArGrrr_c_softmax_with_cross_entropysl    rc Cstr#t|jd}t|||dddddd tj||t|jddSt dit }|j}t|jd ks9Jd t |d gd dt |d gd d|g|gd}dddd}| |} |jd|d| i|d|dur| |} |jd| g|gdd| gidt|jdid| S| } | S)z Fuction Linear r/ transpose_XF transpose_Yalpharur|linearr z/X latitude is not supported greater than 3 now.rr)r7r8r9r0)r<Y)rrrZ matmul_v2r=r>NZelementwise_addr})r)r r r0rmatmulr Z_append_bias_in_dygraphrr1rrDrrrErF) rrrkrlr_Zpre_biasrGr0r@rAr`resrrrrssZ       rscCsN|durdSd|_tj}tj}d||j_d||j_dS)NT)is_distributedr2staticdefault_startup_program current_blockdefault_main_programZ_find_var_recursiver_)var startup_block main_blockrrr_set_var_distributedsrc  Cs|| dur | s dS| durdn| j} |dkr | rt|| d}nt|| d}tjj||||| d} tjjj}||| j |dkr@dn| j | j }t | j |dkrY| j durYt | j |s]|St|j}|d|dkrldn|9<tj}|j||j|j|jdd|jd}|dkr|jdd |id |i| d d d | j dur|| j }|S|jdd |id |i|| |d d dd |S)z Parallel Linear axis the dimension of the parameter of linear layer. axis = 0: the row dimension axis = 1: the col dimension Nrr)rnror_ruF)r1r0r? lod_levelZ persistableZis_dataneed_check_feedrUr<r=TrVr>rH)r,rr-rr)rBrrNrr2nnZLinear functionalrrkrlr_rrirr1rrrZ create_varr0r?rdescrrF)rrZnum_rowsZnum_colsr} param_attrro gather_out inner_rankr-Z split_tensorr_rrrZlinear_function linear_outr4rr5rrr_parallel_linear s~         rcCs|dur |s dS|durdn|j}td it} |} |} | | } | } | |dg}| j||| dd}|dkrFtjjj ||dd|dStj }tj }d|j|j_d|j|j_t||| |d|d }t||ddd }|S) z Parallel Embedding Nr_parallel_embeddingruF)rcr1r0rd)rkZ padding_idxsparser_T)r\r]r_)rrr)r)rBrrrDrgrjr2rr embeddingrrZ global_blockrvarsr_rrarR)rrZper_part_embeddingsZ origin_sizerrnum_partitionsr_rrrG per_part_sizer,Zvocab_start_indexr0sizerkrrZoutput_parallelr5rrrrsD   rruc Cst|ttfs Jdt|dksJdt|tsJdddg} || vs.Jd| dtr5td d d lm} | j sBJd | } | } | |} |dkr|d ksZJd |d |d ksnJd|d d||d |}t ||||| ||dd}|Sd}|d kr|d |d ksJd|d d|d|d |}||df}|j d|d krd}n.|dkr|d|d ksJd|dd|d|d|}|d |f}ntd|dt||d |d||||| |||dd }|S)aH Split the weight of the specified operation into multiple devices and do the computation in parallel. Now the following three cases are supported. Case 1: Parallel Embedding The weight of the embedding operation is a NxM matrix with N rows and M columns. With parallel embedding, the weight is split into num_partitions partitions, each of which is a matrix with (N/num_partitions + 1) rows and M column where the last row as the padding idx. Suppose we split the NxM weight into two partitons on device_0 and device_1 respectively. Then, one each device, the final weight has (N/2 + 1) rows with the index range from 0 to N/2. On device_0, all values in the input within [0, N/2 -1] keep unchanged and all other values are changed to N/2 which is the padding index and are mapped to all zeros after embedding. In the same way, on device_1, the value V in the input within [N/2, N-1] will be changed to (V - N/2), and all other values are changed to N/2 and are mapped to all zeros after embedding. Finally, the results on the two devices are sum-reduced. The Embedding put on single card is as shown below: .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_embedding_single.png :width: 800 :height: 350 :alt: single_embedding :align: center Parallel Embedding is shown as below: .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_embedding_split.png :width: 800 :alt: split_embedding :align: center Case 2: Row Parallel Linear The weight of the linear operation is a NxM matrix with N rows and M columns. With row parallel linear, the weight is split into num_partitions partitions, each of which is a matrix with N/num_partitions rows and M column. The linear layer put on single card is shown as below, the input variable is represented by X, the weight matrix is represented by W and the output vaiable is O. The linear layer on single card is simple matrix multiplication operation, O = X * W. .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_single.png :width: 800 :alt: single_linear :align: center Row Parallel Linear is shown as below. As the name suggests, Row Parallel Linear splits the weight matrix W into [[W_row1], [W_row2]] along the row. And accordingly the input is splitted along the column into [X_col1, X_col2] and multiply their respective weight matrices. Finally apply AllReduce on the output from each card to get the final output. .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_row.png :width: 800 :alt: split_row :align: center Case 3: Column Parallel Linear The weight of the linear operation is a NxM matrix with N rows and M columns. With column parallel linear, the weight is split into num_paratitions partitions, each of which is a matrix with N rows and M/num_partitions column. The linear layer put on single card has been illustrated on case 2 and Column Parallel Linear is shown as below. The Column Parallel Linear splits the weight matrix W into [W_col1, W_col2] along the column and these splitted matrices respectively multiply the input. Finally apply AllGather on the output from each card to get the final output. .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_col.png :width: 800 :alt: split_col :align: center As observed, the column parallel linear and row parallel linear can be combined to skip one ALLGATHER communication operator. Furthermore the Attention and MLP can be combined to imporve the performance as shown below. .. image:: https://githubraw.cdn.bcebos.com/PaddlePaddle/docs/develop/docs/api/paddle/distributed/img/split_col_row.png :width: 800 :alt: split_col_row :align: center Args: x (Tensor): Input tensor. It's data type should be float16, float32, float64, int32 or int64. size (list|tuple): A list or tuple with two elements indicating the shape of the weight. operation (str): The name of the operation. The supported operations are 'linear' and 'embedding'. axis (int, Optional): Indicate along which axis to split the weight. Default: 0. num_partitions (int, Optional): How many parts the weight is partitioned. Default: 1. gather_out (bool, Optional): Whether to gather the output after computation. By default, the output on each partitions will be gathered after computation. Default: True. weight_attr (ParamAttr, Optional): The parameter attribute for the learnable weights(Parameter) of the specified operation. Default: None. bias_attr (ParamAttr, Optional): The parameter attribute for the bias of the specified operation. Default: None. name (str, Optional): The default value is None. Normally there is no need for user to set this property. Default: None. For more information, please refer to :ref:`api_guide_Name`. Returns: Tensor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:DISTRIBUTED) >>> import paddle >>> import paddle.distributed.fleet as fleet >>> paddle.enable_static() >>> paddle.set_device('gpu:%d'%paddle.distributed.ParallelEnv().dev_id) >>> fleet.init(is_collective=True) >>> data = paddle.randint(0, 8, shape=[10,4]) >>> emb_out = paddle.distributed.split( ... data, ... (8, 8), ... operation="embedding", ... num_partitions=2) zDThe type of size for paddle.distributed.split must be list or tuple.zCNumber of elements in size of paddle.distributed.split must be two.z?The type of operation for paddle.distributed.split must be str.rrz:The operation for paddle.distributed.split must be one of rTzpaddle.distributed.split cannot be used in dynamic graph mode, plese use ParallelEmbedding, ParallelRowLinear, ParallelColumnLinear instead.r)fleetzDTo use paddle.distributed.split, you must call fleet.init() firstly.zJWe only support to split the weight of embedding along the first axis now.zYThe length of the vocabulary must be divisible by num_partitions but received vocabulary=z num_partitions=NrFz)Number of rows of the weight for linear (z') must be divisible by num_partitions (r{rurWTz+Number of column of the weight for linear (z9The value of axis must be 0 or 1, but the value given is )r_r) isinstancertuplerstrr rZpaddle.distributed.fleetrZ _role_makerZ worker_indexZ worker_numrr1r)rrrZ operationr}rrrnror_Zsupported_operationsrr,r-rrZemb_outZ should_splitZ linear_sizerrrrsplits           r)NF)N)rrWN)NFrz)NN)rruTNNN)%r2rrZpaddle.autogradrZpaddle.base.data_feederrrZpaddle.distributedrZpaddle.frameworkrr r Z paddle.nnr Zpaddle.nn.utilsr Zcommunication.reducerrrr+rrIrNrQr rRrarbrrsrrrrrrrrsP     + >6- 0/ D7 l <