AI Used to Be Creative — Then We Trained It Not to Be

The Strange Case of AI's Creative Decline

Here is an uncomfortable truth that AI companies would rather you not think about: GPT-2, released seven years ago, was a better creative writer than any model available today.

That is the central argument of a fascinating new piece in The Atlantic by Jasmine Sun, which explores why language models that have memorized centuries of great literature still cannot produce a single essay worth reading. The answer, it turns out, is that we deliberately trained the creativity out of them.

What Happened to the Weirdness?

Katy Gero, a poet and computer scientist who has been experimenting with language models since 2017, describes it perfectly: "You could be like, 'Continue this story: The man decided to take a shower,' and GPT-2 would be like, 'And in the shower, he was eating his lemon and thinking about his wife.' The models won't do that anymore."

That weirdness — that willingness to take unexpected creative leaps — was not a bug. It was arguably AI's most interesting capability. But it was systematically eliminated through a process called post-training, particularly Reinforcement Learning from Human Feedback (RLHF).

How Post-Training Kills Creativity

Here is how it works: after a model is pre-trained on vast amounts of text, human reviewers rate its outputs. The model learns to produce responses that score highly with these reviewers — which means responses that are safe, helpful, clear, and non-controversial.

The problem? Creativity is inherently risky. A surprising metaphor might be brilliant or it might be nonsensical. A whimsical tangent might delight or confuse. When you optimize for "consistently helpful and clear," you optimize against the very qualities that make writing interesting.

The result is what we all recognize as "AI prose": meaningless metaphors, endless "it's not this, but that" constructions, a cloying sycophantic tone, and of course, the beloved em dash used in every other sentence.

Even Sam Altman Admits the Problem

OpenAI CEO Sam Altman has predicted that large language models will soon be capable of "fixing the climate, establishing a space colony, and the discovery of all of physics." But when asked about creative writing specifically, he guessed that even future models — a hypothetical GPT-6 or GPT-7 — might produce only "a real poet's okay poem."

Think about that. The CEO of the world's most valuable AI company is essentially admitting that his technology may never write well. Not because it lacks capability, but because the optimization process that makes it useful for everything else actively suppresses the qualities that make writing good.

The Bigger Implication

This is not just about writing. It is about a fundamental tension in AI development: the qualities that make AI safe and predictable are the opposite of the qualities that make it creative and interesting.

Every time a model is fine-tuned to be more "helpful," it loses a little more of its ability to surprise. Every time RLHF pushes it toward "appropriate" responses, it moves further from the kind of lateral thinking that produces genuine insight.

The irony is profound. We built machines that can process more literature than any human could read in a thousand lifetimes, then trained them to write like corporate communications departments.

The Bottom Line

The next time someone tells you that AI will replace writers, ask them this: if these models have access to Shakespeare, Toni Morrison, and Jorge Luis Borges, why does every AI-generated paragraph sound like it was written by the same slightly nervous intern who really wants you to like them?

The answer is not that AI cannot be creative. GPT-2 proved that it can. The answer is that we decided creativity was less important than safety, helpfulness, and predictability. We got exactly what we optimized for — and what we optimized for turns out to be profoundly boring.