arxivcs.LGmath.DS2026-07-20
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Sixu Li, Thomas Jacob Maranzatto, Jan Peszek, Trevor Teolis, Semih Akkoc, Konstantin Riedl, et al.
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embeddin…