Ben Keigwin

Aurora: A Leverage-Aware Spectral Optimizer

Alec Dewulf, Dhruv Pai, Li Yang, Ashley Zhang, Ben Keigwin

We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform. This can lead to a self-reinforcing feedback loop whereby neurons receive persistently small updates and eventually do not con…Read more