CORTEXA
← Browse

Tejas Pradeep Shirodkar

2 papers indexed

arxivcs.LGmath.DGmath.OCstat.ML2026-06-28

Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

Tejas Pradeep Shirodkar

A deep network's loss is invariant to continuous symmetries of its parameters: the logit shift, the ReLU rescaling, the LayerNorm scale, the per-head attention rotation. Adam's per-coordinate preconditioner drifts along each symmetry orbit, which pulls the trajectory off the symm…

View free PDFSource page