arxivcs.LG2026-07-15
Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat
Grokking -- the delayed generalization of neural networks long after they have memorized their training data -- wastes thousands of training epochs and is notoriously unpredictable. Building on the recent result that Transformer attention is formally isomorphic to a thermodynamic…