What Is Model Collapse? I Trained a Model on Its Own Output, Three Times

The claim is everywhere: if AI systems keep training on text that other AI systems wrote, they get worse. It usually arrives as a warning about the future, with a photocopy-of-a-photocopy analogy attached and no numbers.

It is testable at small scale, so I tested it. I trained a language model on a million characters of Shakespeare, had it write a million characters of its own, trained a fresh model on that, and did it again. Three generations, plus a second run with one rule changed as a control. Every model was scored on the same real Shakespeare that none of them ever trained on, so “is it getting worse” has an answer instead of an analogy.

It is getting worse. What degrades, and what does not, is more specific than the popular version suggests.

The setup, in one paragraph

Generation 1 is a small transformer, 824,897 parameters, trained for 5,000 steps on the first 90 percent of the standard tinyshakespeare corpus. It is the same model I built for the piece on how language models work, where every part of it is laid out. It then writes 1,003,854 characters of its own, exactly the size of the text it learned from. Generation 2 is a brand new model, same architecture, same initialization seed, same number of steps, trained on that synthetic text and nothing else. Generation 3 is trained on generation 2’s output. The real text is never mixed back in, which is the condition that matters and which the control arm changes.

The score is cross-entropy loss on the last 10 percent of the real corpus, which no model in the chain ever saw. Lower is better. Two reference points, both measured under the same evaluation: a model that guesses uniformly among the 65 characters scores 4.174, and a model that knows nothing but how often each character occurs in the training text scores 3.351 on the same held-out block.

The result

Each round of training on synthetic text costs about 0.13 nats. Nothing dramatic happens, no cliff, no garbage: generation 3 is still well clear of the character-frequency baseline at 3.351. It is just measurably worse at real English than its parent, and then worse again.

What the degradation looks like

The first 230 characters each corpus produced, leading blank lines removed, so these are prefixes rather than excerpts I chose.

Generation 1’s output:

Look or:
Against
The same all empty?

LEONTES:
It is not thine, Clarence.

LEONTES:
Though the hangment am:
'Tis not to she?

HERMIONE:
Wo-more than follow it: and poors; I.
Clipnose wild be breathed in my daughter. O,
which, with

Generation 3’s output:

Look deareding to what his neighbour
be enter frience of me by stread;
To at wherewhere no bound boot! O, look
the toward; some on some! to be his more be:
I bold hope, to counter thee rights
Hme from thee loves sten nothing love

Both are nonsense as drama. Generation 1 still produces “hangment” and “Clipnose”, so it is not as though the first round was clean. What separates them is countable, and counting is the point.

What actually shrinks

I counted the word-like strings in each corpus two ways: how many distinct ones there are, how many of those distinct strings appear anywhere in real Shakespeare, and what share of all word occurrences are real words.

Those two percentage columns are different questions and it matters which one is being quoted. Three quarters of the words a generation 3 reader would encounter are still real English, because the common words stay common. But of the distinct strings in its vocabulary, only 7.7 percent are words at all. The list of things the model can write keeps growing, and almost all of the growth is invention.

The clearest version of this is in the speaker names, because a play script is full of lines like HERMIONE: and those are countable.

Two things move at once. Invented names multiply: the 154 real ones become 1,584 distinct headings, of which 1,509 are not characters in any play. And real names die out. Half of Shakespeare’s cast is gone by generation 3.

That middle column is the one worth remembering, and it survives the obvious objection. The three corpora do not contain the same number of headings, so I resampled each of them down to the smallest count, 4,823, and counted again: 111.3 real names for generation 1, 90.3 for generation 2, 75.0 for generation 3, averaged over 20 draws. The decline is not an artifact of counting different amounts of text. The rare ones go first, which is what you would expect: a name that appears a handful of times in the original has a small chance of being written at all by generation 1, a smaller chance of surviving generation 2, and effectively none after that. The survivors are strictly nested, generation 3’s set inside generation 2’s inside generation 1’s.

Which half of the effect this is

Shumailov and colleagues, who named the effect, split it in two. Early model collapse is when “the model begins losing information about the tails of the distribution”. Late model collapse is when the model converges on something with, in their words, “substantially reduced variance” and little resemblance to the original.

Three generations of a toy can only show the first. The tails are visibly going: that is exactly what the middle column of the speaker table is. The second half, the collapse into blandness that most popular retellings lead with, is not what I measured, and I should be careful not to claim it. The one metric that would speak to it, the character entropy of each corpus, does not move much and does not move cleanly: 3.3091 for the real text, then 3.3395, 3.3242 and 3.2846 across the three generations. It rises before it falls. On three generations of an 824,897 parameter model, that is not evidence of anything.

So what I can say is narrow. The rare things disappear, the invented things accumulate, and the next model treats both as the language. Whether it eventually flattens into repetition is a question this experiment does not reach.

The control: what happens if you keep the real data

Everything above uses the strictest rule I ran: each generation trains on synthetic text and nothing else. That rule is a choice, and it turns out to be the choice that decides the outcome.

So I ran a second arm. Generation 1 is the same model, reused rather than retrained, since nothing about it differs. Generation 2 trains on the real text plus generation 1’s output. Generation 3 trains on the real text plus both synthetic corpora. Nothing is thrown away. Same architecture, same initialization seed, same number of training steps, same scoring against the same held-out real text.

Replacing the data costs 0.1447 then 0.1178 nats. Accumulating it costs 0.0069 then 0.0179. After two rounds the two arms are 0.2377 apart.

The tails hold up better too. Resampled to the same 4,823 headings, the accumulating generation 3 retains 96.2 of the 154 real speaker names against 75.0 for the replacing one. On the surface of what it writes the gap is much smaller: 79.2 percent real word occurrences against 75.6, and 11.2 percent real distinct words against 7.7. Keeping the original data protected the model more than it protected the text the model produces at temperature 1.0.

I want to be honest about what this arm does and does not establish, because there is a large confound in it: the accumulating models train directly on real Shakespeare, and they are then scored on held-out real Shakespeare. Some of their advantage is that, not something deep about recursion. What the arm does show is that the degradation in the first table is not an unavoidable consequence of synthetic data being present. It is a consequence of synthetic data being the only thing present.

That is consistent with two results in the literature, and neither of them is mine. The Nature paper itself ran a variant that preserved 10 percent of the original data in each generation and reported only minor degradation. And Gerstgrasser and colleagues prove, in a linear-model analysis rather than for language models in general, that if data accumulate instead of being replaced then the test error has a finite upper bound that does not grow with the number of iterations, and report experiments in which accumulation avoids collapse. Two increments of my own are not a bound. They are a direction that points the same way.

What this does not show

A three-generation chain on an 824,897-parameter character model is not the internet. The things that could change the result, in the order I would attack them:

The control arm trains on the real text. Stated again because it is the largest confound here. A cleaner design would score both arms on a task neither of them was trained for.

The arms are not independent. They share generation 1, and each arm is a single chain with one set of seeds. I have no estimate of how much these numbers move between runs, because I did not run them twice.

The accumulating models see less real text per step. Both arms train for exactly 5,000 steps, but the accumulating pool is twice and then three times the size, so only about half and then a third of the batches come from real Shakespeare. Equal steps is not the same as equal exposure.

I trained from scratch each time. The Nature paper fine-tunes an existing model on each generation’s data instead, which is closer to how anyone would actually do this and preserves more of the original. Starting from nothing is the harsher version.

Sampling was plain temperature 1.0. No top-p, no filtering, and nobody chose which generated text was good enough to keep. Selection is exactly the step that might stop the drift, and there was none.

Character level, tiny model, three generations. Mine predicts one of 65 characters. A production model predicts one of tens or hundreds of thousands of word fragments and has roughly five orders of magnitude more parameters. Whether the cost per generation scales up, down or not at all is not something this can answer. A fourth generation of the replacing arm was started and stopped for time; it is not in any table here.

FAQ

Is model collapse inevitable?

Not on the evidence. It happens when each generation replaces the training data. When the data accumulates, Gerstgrasser and colleagues prove a bounded error in a linear-model analysis and report experiments where collapse does not occur, and the Nature paper’s own 10 percent preservation variant degraded only mildly. My control arm points the same way at small scale.

Does this mean AI-generated text on the web will ruin future models?

It means a closed loop degrades. The open questions, which no toy experiment answers, are what fraction of a real training crawl is synthetic, whether that fraction can be detected, and whether the human text that keeps arriving is enough. Those are measurements nobody has published for any frontier training set.

Why does the vocabulary grow instead of shrinking?

Because a character-level model can write strings that are not words, and does. Counted as distinct strings the vocabulary grows. Counted as real words it collapses, from 11,912 in the training text to 2,612 in generation 3. Growth in one number and loss of the original distribution in the other are the same process seen from two sides.

Can I run this myself?

Yes. The script is here. It runs both arms end to end and prints the two baselines, the losses and every corpus statistic in this article. The corpus is one download and the only dependency is PyTorch. Seeds are fixed, so on PyTorch 2.14 on CPU generation 1 scores 1.5791; on a different build expect about 1.58 rather than that exact figure. Under three hours per arm on two CPU threads, most of it spent writing the synthetic corpora rather than training.

The short version

Train a model on a model’s output and it gets worse at real text, measurably, by about 0.13 nats per round in this setup. What goes first is the rare material: half of Shakespeare’s named characters are gone after two rounds of recursion, while invented names multiply into the thousands. That is the early half of what the researchers call model collapse, and it is the half a small experiment can show. Keeping the original data in the mix changes the outcome, which makes this a question about how training sets get assembled rather than a prophecy about the internet.