Winfried van den dool

AVQ-Attention: Adaptive Vector-Quantized Attention

Winfried van den dool, Patrick Forré, Amir Habibian, Yuki M. Asano, Max Welling

The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but applies uniform codebook capacity regardless of…Read more