1930년대 텍스트로 학습된 빈티지 언어모델

Introducing talkie: a 13B vintage language model from 1930 Nick Levine, David Duvenaud, Alec Radford April 2026 GitHub Hugging Face 💬 Chat Claude chats with talkie, a 13B language model trained on pre-1931 text Connecting... Jump to new messages This is a 24/7 live feed of Claude Sonnet 4.6 prompting talkie-1930-13b-it in order to explore its knowledge, capabilities, and inclinations. talkie’s outputs reflect the culture and values of the texts it was trained on, not the views of its authors. Why vintage language models? Have you ever daydreamed about talking to someone from the past? What would you ask someone with no knowledge of the modern world? What would they ask you ? While we don’t have time machines yet, we can simulate this experience by training, in Owain Evans’s phrase, ‘vintage’ language models : LMs trained only on historical text. These models are fascinating conversation partners (watch Claude prompt talkie, our 13B 1930 LM, in the widget above). But we are also excited by the possibility that the careful study of the behaviors and capabilities of vintage LMs will advance our understanding of AI in general. Figure 1. In an early attempt to understand a vintage model’s anticipation of the future, we took nearly 5,000 historical event descriptions from the New York Times’s “On This Day” feature , calculated their surprisingness (measured as bits per byte of text) to our 13B model trained exclusively on pre-1931 text, and binned by decade. Figure 1b (draft). Surprisingness of NYT headlines across model sizes. Green = vintage (pre-1931), blue = modern (FineWeb). Higher opacity = larger model. Figure 1c (draft). Average surprisingness vs. model size, split by whether events occurred before or after the knowledge cutoff. Solid = pre-cutoff (up to 1930), dashed = post-cutoff (1931+). Figure 1d (draft). Same as Figure 1c, but indexed to the 4-layer model (y = 1.0 at 4L). Shows the relative reduction in surprisingness as model size increases. --> For example, we can evaluate LMs’ ability to predict the future. Inspired by Calcifer Computing’s work on Temporal Language Models , we calculated the surprisingness of short descriptions of historical events to a 13B model trained on pre-1931 text (Figure 1). We can see an increase after the knowledge cutoff, particularly pronounced in the 1950s and 1960s, followed by a plateau. We will continue to develop evals to measure with greater confidence how forecasting performance improves with model size and decays at longer horizons. Training larger vintage language models will allow us to uncover these scaling trends. Figure 2. Patents and a paper published after talkie’s knowledge cutoff. Left to right: helicopter patent (Sikorsky, 1935), Turing machines paper (Turing, 1936), xerography patent (Carlson, 1942). Similarly, we can test LMs’ abilities to come up with new ideas by seeing if they can arrive at inventions or scientific discoveries we know would arise after their knowledge cutoffs, such as those pictured in Figure 2. As Demis Hassabis has asked, could a model trained up to 1911 independently discover General Relativity, as Einstein did in 1915? Figure 3. We gave a Python programming test ( HumanEval ) to a series of pairs of vintage models (trained on pre-1931 text) and modern models (trained on the web), which have the same architecture. Left: This chart shows what percentage of problems each model would get right at least once, given 100 chances and randomly chosen Python functions as examples to learn from in-context. Right: An example of a successful solution to a Python coding problem produced by a vintage language model. The model had access to several other in-context examples to learn from. Contamination is a persistent problem for language models and causes us to overestimate the capabilities of LMs. Vintage LMs are contamination-free by construction, enabling unique generalization experiments, like examining whether a model with no knowledge of digital computers can learn to code in a modern programming language. Figure 3 (left-hand side) shows an early example of such a test, measuring how well models trained on pre-1931 text can, when given a few demonstration examples of Python programs , write new correct programs. While vintage models dramatically underperform models trained on web data (which includes code), we’ve found that they are slowly but steadily improving at this task with scale. There is still a long way to go before this capability is notable, however. All correct solutions generated by the vintage models are simple one-line programs (such as adding two inputs), or small modifications to in-context example programs. For instance, our model implemented the decoding function of a rotation cipher when given the encoding function . Although the solution (Figure 3, right-hand side) is only a single character edit (swapping an addition for a subtraction), this success suggests an understanding of inverse functions. We hope LMs with early knowledge cutoffs help the research community understand how well LMs can generalize beyond their pre-training data. Vintage language models could also teach us about the impact of data diversity in AI development. While modern models vary in disposition, capability, and behavior, they are all closely related to one another by having been trained, whether directly or indirectly (via distillation and synthetic data), on the web. How does this shape and constrain what they are? How much of what we think we know about LMs is about human language and culture in general, or about this one dataset—the web—in particular? Training on different sources may lead to very different kinds of models being created. Studying the ways in which they are similar and different could improve our understanding of language model personas, behaviors, and dispositions. Introducing talkie We have been excited to see a proliferation of vintage LM projects, including Ranke-4B , Mr. Chatterbox , and Machina Mirabilis . Alongside these efforts, we introduce talkie-1930-13b-base , a 13B language model trained on 260B tokens of historical pre-1931 English text. Additionally, we present a post-trained checkpoint turning our base model into a conversation partner without relying on modern chat transcripts or instruction-tuning data. talkie is the largest vintage language model we are aware of, and we plan to continue scaling significantly. As a next step, we are training a GPT-3-level model, which we hope to release this summer. A preliminary estimate also suggests we can grow our corpus to well over a trillion tokens of historical text, which should be sufficient to create a GPT-3.5 level model—similar in capability to the original ChatGPT. Benchmarking an LM from 1930 Figure 4. Evaluation accuracy vs. training compute for talkie-1930 (Vintage LM) and its modern twin trained on FineWeb . The vintage model underperforms the modern model on knowledge evals. Filtering out questions anachronistic from the perspective of 1930 roughly halves the performance gap between the vintage and modern models. To contextualize talkie’s capabilities, we built a “ modern twin ” that is identical architecturally but trained on modern web data (FineWeb) instead of pre-1931 text. On average, talkie underperforms its modern counterpart in standard LM evaluations, even after correcting for question anachronism, despite being trained with the same number of FLOPs (see Figure 4). But we have been encouraged by its similar performance on core language understanding and numeracy tasks. We suspect a combination of differences in data quality (poor optical character recognition) and corpus subject matter distribution explains why talkie-1930 underperforms on some benchmarks. To maximize the compute efficiency of future vintage language model training, we are developing a vint