Huiyin Xue

MultiHashFormer: Hash-based Generative Language Models

Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras

Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, man…Read more