Romit Chatterjee

The Information Shadow: Measuring Structural Limits on What Language Models Can Learn

Priyansh Srivastava, Romit Chatterjee

Some limits on what language models know are not gaps in data coverage but structural properties of learning from text. We introduce the information shadow: the region of phenomena that a text-trained learner cannot acquire regardless of scale, comprising (I) structures language…Read more