o *j@sdZddlmZmZmZddlZddlmZddZdej d e fd d Z d eej j d eej j ej j ffd dZdeej j d eej j ej j ffddZdeed eefddZdS)z- Copyright 2020 The Microsoft DeepSpeed Team )DictListTupleN)MoEcCs@d}d}|D]\}}t|trd}|j}||fSq||fS)NFrT)Z named_modules isinstancer num_experts)mZhas_moer_moduler h/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/modelscope/models/nlp/gpt_moe/moe/utils.pyhas_moe_layers s rparamreturncCst|dr |js dSdS)N allreduceTF)hasattrr)rr r r is_moe_paramsrparamscCs:gg}}|D]}t|r||q||q||fS)N)rappend)rZ shared_paramsZ expert_paramspr r r *split_params_into_shared_and_expert_paramss   rgroupcCsVg}g}|D] }|jdur&t|r||j|jq||j|jq||fS)aSplit grad of parameters into grads of non-expert params and grads of expert params. This is useful while computing grad-norms for clipping and overflow detection group (List[torch.nn.Parameter]): Args: The group of parameters to split Returns: Tuple[List[torch.nn.Parameter], List[torch.nn.Parameter]]: list of gradients for non MoE params, list of gradients of MoE params N)ZgradrrtoZdtype)rZ expert_gradsZ shared_gradsrr r r 0split_params_grads_into_shared_and_expert_params)s r param_groupsc Cst|tr t|}nt|tr|g}nt|ts!tdt|t}|D]}|dD] }t|r8||j q,q&t|}i}|D]O}i||d<|D]D}i||d|<|||d|d<d||d|d<| D]!}|dkr|dkrg||d||<qn||||d||<qnqLqB|D]'}g}|dD]}t|r||d|j d |q| |q||d<q| D]\}} | D] \} } | | qqt|S)zSplit parameters into different MoE groups for optimizer Args: param_groups (Tuple[Dict]): The list of parameter groups to split Returns: Tuple[Dict]: list of MoE/non-MoE groups for optimizer zUnknown param group type of rnameTZmoe) rtuplelistdict ValueErrortypesetradd group_namekeysritems) rZdata_parallel_group_namesZ param_grouprZ group_moekeyZori_key new_paramskvZk1Zv1r r r 4split_params_into_different_moe_groups_for_optimizerCsb              r+)__doc__typingrrrZtorchlayerrrZTensorboolrnn Parameterrrr+r r r r s*