Sebastian Loeschcke

M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup, Anima Anandkumar

Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and…Read more