Swaroop Ghosh

Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

Debarshi Kundu, Swaroop Ghosh, Vasant Honavar

Data-adaptive sparse attention masks substantially outperform fixed patterns (e.g., BigBird and Longformer) and can even exceed dense attention on long sequences. Existing adaptive approaches---including SBM-Transformer, Dynamic Mask Attention, and NSA---typically require additio…Read more