A suite designed for research
On 3 April 2023 EleutherAI publishes Pythia, described in the paper “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling”. Unlike most language models released up to that moment, Pythia is not intended as a single model for production use, but as a systematic study tool for LLM behaviour during training and across scales.
The suite comprises eight models with sizes between 70 million and 12 billion parameters (70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B). All models are released under the Apache 2.0 licence.
Reproducibility and experimental control
Pythia’s distinctive feature is that all models in the suite were trained on the same dataset — The Pile — and in the same batch order. This experimental control allows researchers to isolate the effect of scale and number of training steps from other variables typically present when comparing different models.
For each model, EleutherAI published 154 intermediate checkpoints, logarithmically distributed along the training process. Researchers can therefore observe how specific capabilities — memorisation, reasoning, emergent behaviours — manifest and stabilise over time, with granularity much higher than what is available for other public models.
Research impact
Pythia has become the reference tool for studies on mechanistic interpretability, scaling laws, memorisation and training dynamics. Many academic works after 2023 have used Pythia checkpoints as an experimental base, precisely because they represent one of the few cases in which training code, data and model state at each intermediate phase are all fully accessible.
Licence and access
The combination of a permissive licence, a public dataset and complete checkpoints positions Pythia as an infrastructural contribution to the research community, rather than a product aimed at direct deployment. It is an example of open source release designed to enable further research, not only to provide usable weights.