Tommaso Cerruti

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag

Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architecture…Read more