arxivcs.LGcs.AI2026-07-10
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but updat…