0
Research Papers

Neural Dynamics and Newton–Raphson Iteration for Nonlinear Optimization

[+] Author and Article Information
Yunong Zhang

e-mail: zhynong@mail.sysu.edu.cn
School of Information Science
and Technology,
Sun Yat-sen University,
Guangzhou 510006, China

1Corresponding author.

Contributed by Design Engineering Division of ASME for publication in the JOURNAL OF COMPUTATIONAL AND NONLINEAR DYNAMICS. Manuscript received September 7, 2012; final manuscript received October 16, 2013; published online January 9, 2014. Assoc. Editor: Dan Negrut.

J. Comput. Nonlinear Dynam 9(2), 021016 (Jan 09, 2014) (10 pages) Paper No: CND-12-1141; doi: 10.1115/1.4025748 History: Received September 07, 2012; Revised October 16, 2013

In this paper, a special type of neural dynamics (ND) is generalized and investigated for time-varying and static scalar-valued nonlinear optimization. In addition, for comparative purpose, the gradient-based neural dynamics (or termed gradient dynamics (GD)) is studied for nonlinear optimization. Moreover, for possible digital hardware realization, discrete-time ND (DTND) models are developed. With the linear activation function used and with the step size being 1, the DTND model reduces to Newton–Raphson iteration (NRI) for solving the static nonlinear optimization problems. That is, the well-known NRI method can be viewed as a special case of the DTND model. Besides, the geometric representation of the ND models is given for time-varying nonlinear optimization. Numerical results demonstrate the efficacy and advantages of the proposed ND models for time-varying and static nonlinear optimization.

FIGURES IN THIS ARTICLE
<>
Copyright © 2014 by ASME
Topics: Optimization
Your Session has timed out. Please sign back in to continue.

References

Bestle, D., 2008, “Optimization of a Platform With Respect to Force Contact Conditions,” ASME J. Comput. Nonlinear Dyn., 3(4), p. 041011. [CrossRef]
Deshmukh, V., 2008, “Spectral Collocation-Based Optimization in Parameter Estimation for Nonlinear Time-Varying Dynamical Systems,” ASME J. Comput. Nonlinear Dyn., 3(1), p. 011010. [CrossRef]
Sedlaczek, K., and Eberhard, P., 2009, “Topology Optimization of Large Motion Rigid Body Mechanisms With Nonlinear Kinematics,” ASME J. Comput. Nonlinear Dyn., 4(2), p. 021011. [CrossRef]
Shin, H., Lee, S., In, W., Jeong, J. I., and Kim, J., 2011, “Kinematic Optimization of a Redundantly Actuated Parallel Mechanism for Maximizing Stiffness and Workspace Using Taguchi Method,” ASME J. Comput. Nonlinear Dyn., 6(1), p. 011017. [CrossRef]
Nocedal, J., and Wright, S. J., 1999, Numerical Optimization, Springer-Verlag, New York.
Boyd, S., and Vandenberghe, L., 2004, Convex Optimization, Cambridge University, New York.
Yang, Y., 2007, “Globally Convergent Optimization Algorithms on Riemannian Manifolds: Uniform Framework for Unconstrained and Constrained Optimization,” J. Optim. Theory Appl., 132(2), pp. 245–265. [CrossRef]
Kim, S. D., and Shin, B. C., 2009, “Adjoint Pseudospectral Least-Squares Methods for an Elliptic Boundary Value Problem,” Appl. Numer. Math., 59(2), pp. 334–348. [CrossRef]
Lee, H. C., and Chen, T. F., 2010, “A Nonlinear Weighted Least-Squares Finite Element Method for Stokes Equations,” Comput. Math. Appl., 59(1), pp. 215–224. [CrossRef]
Goglio, L., and Rossetto, M., 2004, “Comparison of Fatigue Data Using the Maximum Likelihood Method,” Eng. Fract. Mech., 71(4–6), pp. 725–736. [CrossRef]
Fares, B., Noll, D., and Apkarian, P., 2002, “Robust Control via Sequential Semidefinite Programming,” SIAM J. Control Optim., 40(6), pp. 1791–1820. [CrossRef]
Johansen, T. A., Fsosen, T. I., and Berge, S. P., 2004, “Constrained Nonlinear Control Allocation With Singularity Avoidance Using Sequential Quadratic Programming,” IEEE Trans. Control Syst. Technol., 12(1), pp. 211–216. [CrossRef]
Grudinin, N., 1998, “Reactive Power Optimization Using Successive Quadratic Programming Method,” IEEE Trans. Power Syst., 13(4), pp. 1219–1225. [CrossRef]
Ljung, L., and Söderström, T., 1983, Theory and Practice of Recursive Identification, MIT, London.
Feng, C. B., and Zhao, Y., 1992, “Time-Varying Nonlinear Programming and Its Realization via Neural Networks,” Proc. American Control Conf., 2, pp. 978–982.
Myung, H., and Kim, J.-H., 1997, “Time-Varying Two-Phase Optimization and Its Application to Neural-Network Learning,” IEEE Trans. Neural Netw., 8(6), pp. 1293–1300. [CrossRef] [PubMed]
Liu, G. H., Jing, L. L., Han, L. X., and Han, D., 1999, “A Class of Nonmonotone Conjugate Gradient Methods for Unconstrained Optimization,” J. Optim. Theory Appl., 101(1), pp. 127–140. [CrossRef]
Chang, H. C., and Prabhu, N., 2003, “Canonical Coordinates Method for Equality-Constrained Nonlinear Optimization,” Appl. Math. Comput., 140(1), pp. 135–158. [CrossRef]
Zhang, J., 2006, “A Robust Trust Region Method for Nonlinear Optimization With Inequality Constraint,” Appl. Math. Comput., 176(2), pp. 688–699. [CrossRef]
Ji, Y., Zhang, K.-C., Qu, S.-J., and Zhou, Y., 2007, “A Trust-Region Method by Active-Set Strategy for General Nonlinear Optimization,” Comput. Math. Appl., 54(2), pp. 229–241. [CrossRef]
Xue, X., and Bian, W., 2007, “A Project Neural Network for Solving Degenerate Convex Quadratic Program,” Neurocomputing, 70(13–15), pp. 2449–2459. [CrossRef]
Costantini, G., Perfetti, R., and Todisco, M., 2008, “Quasi-Lagrangian Neural Network for Convex Quadratic Optimization,” IEEE Trans. Neural Netw., 19(10), pp. 1804–1809. [CrossRef] [PubMed]
Zhang, Y., and Yi, C., 2011, Zhang Neural Networks and Neural-Dynamic Method, Nova Science, New York.
Mead, C., 1989, Analog VLSI and Neural Systems, Addison-Wesley, Reading, MA.
Mathews, J. H., and Fink, K. D., 2005, Numerical Methods Using MATLAB, Publishing House of Electronics Industry, Beijing.
Mitra, S. K., 2006, Digital Signal Processing - A Computer-Based Approach, 3rd ed., Tsinghua University, Beijing.
David, F. G., and Desmond, J. H., 2010, Numerical Methods for Ordinary Differential Equations: Initial Value Problems, Springer, UK.

Figures

Grahic Jump Location
Fig. 1

Trajectories of x*(t) and x(t) of CTND in Eq. (4) with different activation functions and γ-value to solve Eq. (19). (a) CTND with linear activation function. (b) CTND with power-sigmoid activation function.

Grahic Jump Location
Fig. 2

Residual errors |e(t)| of CTND in Eq. (4) with different activation functions and γ-value to solve Eq. (19). (a) CTND with linear activation function. (b) CTND with power-sigmoid activation function.

Grahic Jump Location
Fig. 3

Simulative results of GD in Eq. (6) by using different values of γ for online solution of Eq. (19). (a) Trajectories of x*(t) and x(t) of GD in Eq. (6). (b) Residual errors |e(t)| of GD in Eq. (6).

Grahic Jump Location
Fig. 4

Trajectories of x*(t) and x(t) and residual error |e(t)| of Eq. (20) for online solution of Eq. (19)

Grahic Jump Location
Fig. 5

Convergence performance of DTNDK in Eq. (8) with γ = 20 and τ = 0.01 for online solution of Eq. (19)

Grahic Jump Location
Fig. 6

Convergence performance of DTNDU in Eq. (11) with γ = 20 and τ = 0.01 for online solution of Eq. (19)

Grahic Jump Location
Fig. 7

MSSRE of DTNDK in Eq. (8) and DTNDU in Eq. (11) by using different values of γ and τ for online solution of Eq. (19). (a) With τ = 0.01 fixed. (b) With γτ = 0.2 fixed.

Grahic Jump Location
Fig. 8

Convergence performance of S-CTND in Eq. (15) with power-sigmoid activation function and different values of γ to solve Eq. (21)

Grahic Jump Location
Fig. 9

Convergence performances of S-DTND in Eq. (16) with power-sigmoid activation function and γτ=0.2 and NRI in Eq. (18) for static optimization

Grahic Jump Location
Fig. 10

Geometric representation of ND models for time-varying nonlinear optimization (and time-varying nonlinear/linear equation solving). (a) The proposed method approaches optimal solution x*(t). (b) Solutions with and without rotating trend considered. (c) The change of the gradient gx(x(t∧),t∧). (d) The gradient along the modified search direction.

Grahic Jump Location
Fig. 11

Analysis about the effect of different activation functions on the convergence performances of the presented ND models. (a) Activation functions ϕ(·). (b) Gradient along search direction.

Grahic Jump Location
Fig. 12

Convergence performance of the presented ND model in Eq. (13) by using γ=1 and different activation functions

Tables

Errata

Discussions

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related Journal Articles
Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In