Parviz Haggi-Mani

Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

Parviz Haggi-Mani, Irina Rish

Using the language of Wilsonian renormalization group theory (RG), we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point s…Read more