WIAS Preprint No. 2655, (2019)
Adaptive gradient descent for convex and non-convex stochastic optimization
- Ogaltsov, Aleksandr
- Dvinskikh, Darina
- Dvurechensky, Pavel
- Gasnikov, Alexander
- Spokoiny, Vladimir
2010 Mathematics Subject Classification
- 90C25 90C30 90C06
- Convex and non-convex optimization, stochastic optimization, first-order method, adaptive method, gradient descent, complexity bounds, mini-batch
In this paper we propose several adaptive gradient methods for stochastic optimization. Our methods are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.
- Proceedings of the 21th IFAC World Congress, R. Findeisen, S. Hirche, K. Janschek, M. Mönnigmann, eds., vol. 53 of IFAC PapersOnLine, Elsevier, 2020, pp. 1715--1720, DOI 10.1016/j.ifacol.2020.12.2284 .