MAS_ Multi-stage Adaptive Fusion Optimizer Combining Muon, AdamW and SGD

Bojian Huang

Adaptive optimizers constitute core components for training deep learning models.However, mainstream optimizers suffer from irreconcilable inherent flaws. AdamWtends to induce over-preconditioning due to long-term accumulation of second-ordermoments, trapping models in sharp local minima and resulting in stagnant updates inthe late training stage. Muon delivers powerful capability to escape loss surfaces viaorthogonal spectral normalization, yet it lacks dimension-wise curvature adaptationand suffers from persistent oscillations in the late convergence phase. MomentumSGD achieves strong generalization and refined convergence, but it fails to cope withill-conditioned Hessian matrices and exhibits extremely slow convergence at the earlystage.