o #jr@slddlmZddlmZddlZddlmZddlmZgZ ddd Z dddZ GdddeZ dS)) defaultdict)reduceN) framework) Optimizerc Cs|dur |\}}n||kr||fn||f\}}||d||||} | d||} | dkrj| } ||krN||||| | ||d| } n||||| | ||d| } tt| ||S||dS)a]Cubic interpolation between (x1, f1, g1) and (x2, f2, g2). Use two points and their gradient to determine a cubic function and get the minimum point between them in the cubic curve. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp59: formula 3.59 Args: x1, f1, g1: point1's position, value and gradient. x2, f2, g2: point2's position, value and gradient. bounds: bounds of interpolation area Returns: min_pos: the minimum point between the specified points in the cubic curve. Nrrg@)sqrtminmax) x1f1g1Zx2f2g2boundsZ xmin_boundZ xmax_boundZd1Z d2_squareZd2Zmin_posrW/var/www/html/Deteccion_Ine/venv/lib/python3.10/site-packages/paddle/optimizer/lbfgs.py_cubic_interpolates *( r-C6??& .>c ! Cs|} |}||||\} } d}t| |}tjd|jd|||f\}}}}d}d}|| kr| ||||ksD|dkrW| |krW||g}|| g}|| g}||g}nkt|| |krm|g}| g}| g}d}nU|dkr||g}|| g}|| g}||g}n>|d||}|d}|}t||||| |||fd}|}| }| }|}||||\} } |d7}| |}|d7}|| ks2|| krd|g}|| g}|| g}d}|d|d krd nd \}}|s|| krt|d|d| | krnt|d|d|d|d|d|d}d t|t|} tt|||t|| kr`|s:|t|ks:|t|kr]t|t|t|t|krTt|| }nt|| }d}nd}nd}||||\} } |d7}| |}|d7}| ||||ks| ||kr|||<| ||<| ||<|||<|d|dkrd nd \}}nEt|| |krd}n%|||||dkr||||<||||<||||<||||<|||<| ||<| ||<|||<|s|| ks||}||} ||} | | ||fS) ag Implements of line search algorithm that satisfies the strong Wolfe conditions using double zoom. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp60: Algorithm 3.5 (Line Search Algorithm). Args: obj_func: the objective function to minimize. ```` accepts a multivariate input and returns a scalar. xk (Tensor): the starting point of the iterates. alpha (Scalar): the initial step size. d (Tensor): search direction. loss (scalar): the initial loss grad (Tensor): the initial grad c1 (Scalar): parameter for sufficient decrease condition. c2 (Scalar): parameter for curvature condition. tolerance_change (Scalar): terminates if the change of function value/position/parameter between two iterations is smaller than this value. max_ls(int): max iteration of line search. alpha_max (float): max step length. Returns: loss_new (Scaler): loss of obj_func at final alpha. grad_new, (Tensor): derivative of obj_func at final alpha. alpha(Tensor): optimal step length, or 0. if the line search algorithm did not converge. ls_func_evals (Scaler): number of objective function called in line search process. Following summarizes the essentials of the strong Wolfe line search algorithm. Some notations used in the description: - `func` denotes the objective function. - `obi_func` is a function of step size alpha, restricting `obj_func` on a line. obi_func = func(xk + alpha * d), where xk is the position of k'th iterate, d is the line search direction(decent direction), and a is the step size. - alpha : substitute of alpha - a1 is alpha of last iteration, which is alpha_(i-1). - a2 is alpha of current iteration, which is alpha_i. - a_lo is alpha in left position when calls zoom, which is alpha_low. - a_hi is alpha in right position when calls zoom, which is alpha_high. Line Search Algorithm: repeat Compute obi_func(a2) and derphi(a2). 1. If obi_func(a2) > obi_func(0) + c_1 * a2 * obi_func'(0) or [obi_func(a2) >= obi_func(a1) and i > 1], alpha= zoom(a1, a2) and stop; 2. If |obi_func'(a2)| <= -c_2 * obi_func'(0), alpha= a2 and stop; 3. If obi_func'(a2) >= 0, alpha= zoom(a2, a1) and stop; a1 = a2 a2 = min(2 * a2, a2) i = i + 1 end(repeat) zoom(a_lo, a_hi) Algorithm: repeat aj = cubic_interpolation(a_lo, a_hi) Compute obi_func(aj) and derphi(aj). 1. If obi_func(aj) > obi_func(0) + c_1 * aj * obi_func'(0) or obi_func(aj) >= obi_func(a_lo), then a_hi <- aj; 2. 2.1. If |obi_func'(aj)| <= -c_2 * obi_func'(0), then alpha= a2 and stop; 2.2. If obi_func'(aj) * (a2 - a1) >= 0, then a_hi = a_lo a_lo = aj; end(repeat) reference: https://github.com/pytorch/pytorch rrdtypeFTg{Gz? )r)rr)rrg?) absr clonepaddledot to_tensorrrr )!obj_funcZxkalphadlossgradgtdc1c2tolerance_changeZmax_lsZd_normZloss_newZgrad_new ls_func_evalsZgtd_newZt_prevZf_prevZg_prevZgtd_prevdoneZls_iterbracketZ bracket_fZ bracket_gZ bracket_gtdZmin_stepZmax_steptmpZinsuf_progressZlow_posZhigh_posepsrrr _strong_wolfe>s X      3 "         K r0cseZdZdZ           dfdd Zd d Zd d ZddZddZddZ ddZ ddZ e j ddZ dddZZS)LBFGSa The L-BFGS is a quasi-Newton method for solving an unconstrained optimization problem over a differentiable function. Closely related is the Newton method for minimization. Consider the iterate update formula: .. math:: x_{k+1} = x_{k} + H_k \nabla{f_k} If :math:`H_k` is the inverse Hessian of :math:`f` at :math:`x_k`, then it's the Newton method. If :math:`H_k` is symmetric and positive definite, used as an approximation of the inverse Hessian, then it's a quasi-Newton. In practice, the approximated Hessians are obtained by only using the gradients, over either whole or part of the search history, the former is BFGS, the latter is L-BFGS. Reference: Jorge Nocedal, Stephen J. Wright, Numerical Optimization, Second Edition, 2006. pp179: Algorithm 7.5 (L-BFGS). Args: learning_rate (float, optional): learning rate .The default value is 1. max_iter (int, optional): maximal number of iterations per optimization step. The default value is 20. max_eval (int, optional): maximal number of function evaluations per optimization step. The default value is max_iter * 1.25. tolerance_grad (float, optional): termination tolerance on first order optimality The default value is 1e-5. tolerance_change (float, optional): termination tolerance on function value/parameter changes. The default value is 1e-9. history_size (int, optional): update history size. The default value is 100. line_search_fn (string, optional): either 'strong_wolfe' or None. The default value is strong_wolfe. parameters (list|tuple, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. The default value is None. weight_decay (float|WeightDecayRegularizer, optional): The strategy of regularization. \ It canbe a float value as coeff of L2 regularization or \ :ref:`api_paddle_regularizer_L1Decay`, :ref:`api_paddle_regularizer_L2Decay`. If a parameter has set regularizer using :ref:`api_paddle_ParamAttr` already, \ the regularization setting here in optimizer will be ignored for this parameter. \ Otherwise, the regularization setting here in optimizer will take effect. \ Default None, meaning there is no regularization. grad_clip (GradientClipBase, optional): Gradient cliping strategy, it's an instance of \ some derived class of ``GradientClipBase`` . There are three cliping strategies \ ( :ref:`api_paddle_nn_ClipGradByGlobalNorm` , :ref:`api_paddle_nn_ClipGradByNorm` , \ :ref:`api_paddle_nn_ClipGradByValue` ). Default None, meaning there is no gradient clipping. name (str, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. Return: loss (Tensor): the final loss of closure. Examples: .. code-block:: python >>> import paddle >>> import numpy as np >>> paddle.disable_static() >>> np.random.seed(0) >>> np_w = np.random.rand(1).astype(np.float32) >>> np_x = np.random.rand(1).astype(np.float32) >>> inputs = [np.random.rand(1).astype(np.float32) for i in range(10)] >>> # y = 2x >>> targets = [2 * x for x in inputs] >>> class Net(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... w = paddle.to_tensor(np_w) ... self.w = paddle.create_parameter(shape=w.shape, dtype=w.dtype, default_initializer=paddle.nn.initializer.Assign(w)) ... ... def forward(self, x): ... return self.w * x ... >>> net = Net() >>> opt = paddle.optimizer.LBFGS(learning_rate=1, max_iter=1, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn='strong_wolfe', parameters=net.parameters()) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print('loss: ', loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... opt.step(closure) ... >>> for input, target in zip(inputs, targets): ... input = paddle.to_tensor(input) ... target = paddle.to_tensor(target) ... train_step(input, target) ?NHz>rdc s|dur |dd}||_||_||_||_||_||_||_t|tj r-t dt |t t |_tjd|| | | dt|jdt sJ|j|_nt|jD] \} } | d|_qOd|_dS)Nz^parameters argument given to the optimizer should be an iterable of Tensors or dicts, but got r2) learning_rate parameters weight_decay grad_clipnamerparams)r8max_itermax_evaltolerance_gradr* history_sizeline_search_fn isinstancerZTensor TypeErrortyperdictstatesuper__init__Z_parameter_list_params enumerateZ _param_groups _numel_cache)selfr8r>r?r@r*rArBr9r:r;r<idxZ param_group __class__rrrIs8      zLBFGS.__init__cCs.i}|jD] \}}|||iqd|iS)aReturns the state of the optimizer as a :class:`dict`. Return: state, a dict holding current optimization state. Its content differs between optimizer classes. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def train_step(inputs, targets): ... def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... opt.clear_grad() ... loss.backward() ... return loss ... ... opt.step(closure) ... >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> n_iter = 0 >>> while n_iter < 20: ... loss = train_step(inputs, targets) ... n_iter = opt.state_dict()["state"]["func_evals"] ... print("n_iter:", n_iter) ... rG)rGitemsupdate)rMZ packed_statekvrrr state_dicts/zLBFGS.state_dictcCs$|jdurtdd|jd|_|jS)NcSs ||SN)numel)totalprrrs zLBFGS._numel..r)rLrrJrMrrr_numels  z LBFGS._numelcCsTg}|jD]}|jdurt|dg}n|jdg}||qtj|ddS)Nrr)Zaxis)rJr&rZ zeros_likereshapeappendconcat)rMZviewsrYviewrrr_gather_flat_grads   zLBFGS._gather_flat_gradc Cstd}|jD]*}|jgkrtdd|jnd}t||||||j||}||7}q||ks8JdS)NrcSs||SrVr)xyrrrrZ sz!LBFGS._add_grad..r)rJshaperrassignaddr]r\)rMr# directionoffsetrYrWrrr _add_grad s  zLBFGS._add_gradcCsdd|jDS)NcSsg|]}|qSr)r).0rYrrr sz&LBFGS._clone_param..)rJr[rrr _clone_paramszLBFGS._clone_paramcCs&t|j|D] \}}t||qdSrV)ziprJrre)rMZ params_datarYZpdatarrr _set_paramszLBFGS._set_paramcCs0|||t|}|}||||fSrV)rifloatrarn)rMclosurerbr#r$r% flat_gradrrr_directional_evaluates   zLBFGS._directional_evaluatec% stXtj}j}j}j}j}j}j }j } | dd| dd} t | } d} | dd7< } | |k}|rX| WdS| d}| d}| d}| d }| d }| d }| d }| d }d}||kr5|d7}| dd7<| ddkr| }g}g}g}tjd| jd}n| |}|tj||jd}||}|dkrt||kr|d|d|d|||||d||||}t|}d| vrdg|| d<| d}| }t|dddD] }|||||||<t||||| |qt||}}t|D]}|||||} t|||||| |q?|duri| }nt| || }| ddkrtdd| |}n|}| |}!|!| krnd}"|dur|dkrt d!}#fdd}$t"|$|#||| | |!\} } }}"#||| |k}n3#||||krt t } Wdn 1swY } | |k}d}"| |"7} | d|"7<|rn&|||krnt| ||kr%n| |kr+n ||kr1n||ks|| d<|| d<|| d<|| d <|| d <|| d <|| d <|| d <Wd| S1sawY| S)aPerforms a single optimization step. Args: closure (callable): A closure that reevaluates the model and returns the loss. Examples: .. code-block:: python >>> import paddle >>> paddle.disable_static() >>> inputs = paddle.rand([10, 10], dtype="float32") >>> targets = paddle.to_tensor([2 * x for x in inputs]) >>> net = paddle.nn.Linear(10, 10) >>> opt = paddle.optimizer.LBFGS( ... learning_rate=1, ... max_iter=1, ... max_eval=None, ... tolerance_grad=1e-07, ... tolerance_change=1e-09, ... history_size=100, ... line_search_fn='strong_wolfe', ... parameters=net.parameters(), >>> ) >>> def closure(): ... outputs = net(inputs) ... loss = paddle.nn.functional.mse_loss(outputs, targets) ... print("loss:", loss.item()) ... opt.clear_grad() ... loss.backward() ... return loss ... >>> opt.step(closure) Z func_evalsrn_iterrNr$r#old_ykold_skroH_diagprev_flat_grad prev_lossr2rg|=alrZ strong_wolfez only 'strong_wolfe' is supportedcs|||SrV)rr)rbr#r$rprMrrr"szLBFGS.step..obj_func)$rZno_gradZ enable_gradr8r>r?r@r*rBrArG setdefaultrorarr getnegr!rsubtractmultiplyr lenpopr^rangererfrr sum RuntimeErrorrlr0ri)%rMrpr8r>r?r@r*rBrArGZ orig_lossr%Z current_evalsrqZopt_condr$r#rtrurvrwrxryrsrcsZysZnum_oldrzqirZbe_ir'r+Zx_initr"rr{rstep%s )                      $ &                66z LBFGS.stepcCstd)z}Empty method. LBFGS optimizer does not use this way to minimize ``loss``. Please refer 'Examples' of LBFGS() above for usage.zeLBFGS optimizer does not use this way to minimize loss. Please refer 'Examples' of LBFGS() for usage.)NotImplementedError)rMr%Zstartup_programr9Z no_grad_setrrrminimizeszLBFGS.minimize) r2r3Nr4rr5NNNNN)NNN)__name__ __module__ __qualname____doc__rIrUr\rarirlrnrrrZnon_static_onlyrr __classcell__rrrOrr15s4\15    ar1rV)rrrr) collectionsr functoolsrrbaserZ optimizerr__all__rr0r1rrrrs     , x