Leena Chennuru Vankadara

Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara, Andrew Saxe

In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends…Read more