Haoming Meng

Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization

Haoming Meng, Anton Sugolov, Vardan Papyan

Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this observation, we introduce \emph{Depth-wise Gradient Augmentation}, a general optimization paradigm in…Read more