Jacob Dunefsky

A Shared Subcircuit Lets LLMs Count Down Across Tasks

Jacob Dunefsky, Wes Gurnee, Emmanuel Ameisen

Writing a sentence of exactly twelve words; ending a DNA sequence at the right codon; formatting an ASCII table. These are all tasks that language models can do that requires tracking how many tokens remain before a target. In this work, we identify in Llama-3.1-70B-Instruct a ge…Read more