Megi Dervishi

Separating Representation from Reconstruction Enables Scalable Text Encoders

Megi Dervishi, Mathurin Videau, Yann LeCun

While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via probing. Under this lens, the representations of BERT encoders become increasingly $\textit{unexploitable}$ by frozen probes, despi…Read more