arxivcs.LGmath.OCstat.ML2026-07-07
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such…