期刊名称：Journal of Theoretical and Applied Information Technology
印刷版ISSN：1992-8645
电子版ISSN：1817-3195
出版年度：2021
卷号：99
期号：22
语种：English
出版社：Journal of Theoretical and Applied
摘要：Optimization is the process of finding parameters that return the maximum or minimum value of a function, where the function symbolizes the effort needed or the desired benefit. First-order stochastic gradient (SG) methods are often used to solve deep learning models that involve a hard non-convex optimization problem. Although second-order methods can ensure faster convergence, they have been less explored because processing time and costs are high. Optimizing deep learning models is a challenging problem; many deep learning companies spend a lot of their resources on training deep models. This paper proposes an implementation and evaluation of Newton's second-order optimization method, Hessian Free Optimization (HFO), on fully connected feed-forward networks, and enhances the method by the integration with some acceleration techniques such as Momentum and Root Mean Square Propagation (RMSProp). The paper also proposed a hybrid algorithm capable of combining two-degree orders, first-order, and second-order optimization methods. The hybrid algorithm can achieve better convergence (5% better in testing loss) compared to first-order methods with approximately the same time consumption.
