arxivcs.LGcs.CLeess.SP2026-07-08
FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragrap…