Aritra Dutta

DnA: Denoising Attention for Visual Tasks

Ron Campos, Subhajit Maity, Xin Li, Srijan Das, Aritra Dutta

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose…Read more