arxivcs.LGcs.CL2026-07-23
Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets
Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input. These selectors are often trained by distilling the attention patterns of a dense teacher, assuming that attention reveals which context the teacher actually uses.…