Yousef Saad

Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization

Ziyuan Tang, Tianshi Xu, Yousef Saad, Yuanzhe Xi

Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ steps of the full-matrix Newton-Schulz update require…Read more