Key Points
- Student Grigorian trains tiny language models using nanoGPT and Microsoft Phi 1.5.
- Version 0 produced Victorian‑style gibberish; Version 5 generated correct grammar but many hallucinations.
- Current 700‑million‑parameter model reduces confabulations and includes real historical references.
- Model shows potential as a stylistic aid for historians and digital‑humanities research.
- All code, weights, and documentation are publicly released on GitHub.
- Future plans include models for other cities and cultures such as Chinese, Russian, and Indian.
An 1857 photographic portrait of Henry John Temple, also known as Lord Palmerston.
Background
Graduate student Grigorian set out to explore whether very small language models could convincingly reproduce period language from the 19th century. Using the open‑source nanoGPT architecture and Microsoft’s Phi 1.5 framework, the researcher built a series of experimental models, each larger and better trained than the last.
Model Development
The first iteration, dubbed Version 0, was trained on a modest 187 MB of data. It managed to echo Victorian‑style phrasing but produced nonsensical output. A subsequent Version 5 model achieved grammatically correct prose but filled the text with fabricated events, people, and dates. The current model, containing roughly 700 million parameters and trained on a rented A100 GPU, shows a marked reduction in such confabulations. According to Grigorian’s GitHub notes, scaling up both the model size and the quality of training data appears to help the system “remember” factual details from its corpus.
Results and Historical Accuracy
The newest model has begun to generate genuine historical references, including an authentic 1857 photographic portrait of Henry John Temple, also known as Lord Palmerston. While the model still occasionally hallucinates, the emergence of accurate facts—what the researcher calls a “factcident”—demonstrates that even tiny models can occasionally produce truthful historical content.
Implications for Research
Grigorian suggests that training language models on period texts could provide historians and digital‑humanities scholars with interactive tools for exploring extinct vernaculars or stylistic nuances. Such models could simulate conversations with a speaker from a bygone era, offering stylistic illumination even if factual rigor is not guaranteed.
Future Plans
The researcher plans to extend the approach to other locales, mentioning potential models focused on Chinese, Russian, or Indian city histories. All code, model weights, and documentation are released publicly on GitHub, inviting collaboration from the broader community.
Source: arstechnica.com