Haoyan Luo

Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models

Haoyan Luo, Mateo Espinosa Zarlenga, Mateja Jamnik

Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence. We introduce Persistent Sparse Autoencoders (Persistent SAEs), which extend stand…Read more