Shreyas Kadekodi

LeRoPE: Learnable RoPE Frequencies Improve Language Modeling

Petros Karypis, Sean O'Brien, Shreyas Kadekodi, Rui Zhu, Julian McAuley

Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in…Read more