Researchers pretrain a model using zero human data
A generator and a learner train each other on self-generated programs alone, and the resulting models show predictable scaling and early signs of in-context learning, the authors said.
Researchers from Stanford, Tel Aviv University and an independent lab pretrained a language model with no human-written text at all, according to a paper posted to arXiv on September 24. They trained it entirely through self-play between two networks.
https://x.com/askalphaxiv/status/2103778101033590958
In the setup, a generator network writes programs for a universal Turing machine, and a learner network predicts the resulting output bytes. The generator is trained with reinforcement learning to keep proposing sequences at the edge of what the learner can already handle. That produces what the authors call an adaptive curriculum.
An independent reproduction posted this week, built from scratch on two low-end GPUs, said it matched the paper's own checkpoints "identical to three decimals" across the paper's evaluation corpora. A second independent analysis recomputed the same metric and went further. It found that a small model developed basic arithmetic within its first 500 to 1,000 training rounds, well before broader in-context learning emerged.
The authors say the method could offer an "effectively unbounded" source of pretraining data, limited by compute rather than by how much human-written text exists. The models tested so far are tiny, from 99,000 to 24.4 million parameters, and the paper does not include the training code. It offers only evaluation scripts and checkpoints published on Hugging Face.