Tavish Mankash

OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

Tavish Mankash, Vardhaman Kalloli, Keshava Prasad, Deepan Muthirayan

OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describ…Read more