Cheng Mao

Scaling Limits of Constant-Stepsize SGD at Flat Minima

Jingyi Zhang, Cheng Mao, Debankur Mukherjee

For stochastic gradient descent (SGD) with a constant stepsize $α$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrtα$ scaling and a…Read more