WIAS Preprint No. 2655, (2019)

Adaptive gradient descent for convex and non-convex stochastic optimization


Authors

  • Ogaltsov, Aleksandr
  • Dvinskikh, Darina
  • Dvurechensky, Pavel
    ORCID: 0000-0003-1201-2343
  • Gasnikov, Alexander
  • Spokoiny, Vladimir
    ORCID: 0000-0002-2040-3427

2010 Mathematics Subject Classification

  • 90C25 90C30 90C06

Keywords

  • Convex and non-convex optimization, stochastic optimization, first-order method, adaptive method, gradient descent, complexity bounds, mini-batch

DOI

10.20347/WIAS.PREPRINT.2655

Abstract

In this paper we propose several adaptive gradient methods for stochastic optimization. Our methods are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.

Appeared in

  • Proceedings of the 21th IFAC World Congress, R. Findeisen, S. Hirche, K. Janschek, M. Mönnigmann, eds., vol. 53 of IFAC PapersOnLine, Elsevier, 2020, pp. 1715--1720, DOI 10.1016/j.ifacol.2020.12.2284 .

Download Documents