Jacob Zavatone-Veth

Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

Mary Letey, Yue M. Lu, Cengiz Pehlevan, Jacob Zavatone-Veth

Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has s…Read more