Machines Now Writing Better Poetry Than Humans, Universe Laughs Its Ass Off

modified
length
9 min read

TL;DR: In late 2024, a peer-reviewed study found that people can't tell AI poetry from human poetry, and prefer the machine's version. I tweeted about it at the time. Fifteen months later, the follow-up research tells a more complicated story. The surface won the first round. The depths are fighting back.

The Study Nobody Wanted to Read Carefully

In November 2024, Brian Porter and Edouard Machery published "AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably" in Scientific Reports. The title alone should have started bar fights in MFA programs worldwide. It didn't, mostly because poets don't read Nature and scientists don't read poetry.

The setup was clean. They took ten poets spanning six centuries of English literature, Chaucer, Shakespeare, Samuel Butler, Lord Byron, Walt Whitman, Emily Dickinson, T.S. Eliot, Allen Ginsberg, Sylvia Plath, Dorothea Lasky. Asked ChatGPT 3.5 to write short poems in each poet's style. Then showed both versions to 1,634 people and asked: which one is human?

Participants got it right 46.6% of the time. Worse than a coin flip. Literally worse than random guessing.

The AI poems were rated higher on 13 out of 14 qualitative dimensions. The biggest gap was rhythm. The machine had better rhythm than poets who spent their lives obsessing over rhythm.

The headlines wrote themselves. The nuance didn't.

The Part That Actually Mattered

Here's what tripped people up, and it's the detail that changes the conversation entirely: participants used difficulty of language as their primary heuristic for identifying human poetry. Complex, dense, ambiguous phrasing got tagged as "definitely human." Accessible, flowing language got tagged as "probably AI."

They had it backwards.

The human poems were the difficult ones. The AI poems were the clear ones. And clarity won, every single time, across nearly every metric the researchers measured.

There's something in that inversion worth sitting with. The machine's output was more immediately beautiful to a general audience than the human's, not because the machine understands beauty, but because it optimized for the patterns that humans associate with beauty. The surface was flawless. The question was whether the surface is the poem.

What Happened Next

Science doesn't end at one paper, even when the headline is good enough to stop reading. Over the following year, the study got the scrutiny it deserved.

A blogger named Soren Bjornstad ran his own replication, paying actual attention to the poems. He scored 96.6%, correctly identifying 84 out of 87. His argument was simple: the original participants didn't know what to look for. They held outdated mental models of what AI poetry sounds like, applied the wrong heuristics, and got predictably confused. Someone who reads poetry and has spent five minutes with ChatGPT can spot the tells immediately, the clichéd vocabulary, the relentless quatrains, the metaphors that gesture at depth without arriving anywhere.

He called it the Pepsi Challenge effect. In a blind sip test, people prefer the sweeter option. In a blind poem test, people prefer the clearer one. That doesn't mean Pepsi is better than Coke. It means lab conditions reward accessibility over complexity, and anyone who's spent time with either drink, or either kind of poem, knows there's more to preference than a first impression.

Then in November 2025, Dr. James O'Sullivan at University College Cork published a stylometric analysis comparing hundreds of short stories by humans and AI. The finding: GPT-3.5, GPT-4, and Llama 70B all produce writing that clusters tightly in stylistic space. Uniform word frequency, predictable pacing, a detectable fingerprint that persists even when the model is explicitly trying to sound human. Human authors, by contrast, scatter across the map, individual voices shaped by individual lives, messy and inconsistent in ways that turn out to be the whole point.

The machines write in a dialect. A very polished, very narrow dialect.

And in early 2026, a study out of Romania tested adolescents on the same question. Without attribution, the kids rated AI poems higher, simpler language, closer to how they actually talk. But once told which poems were AI-generated, they penalized them significantly. The label "AI" functioned as a quality discount regardless of actual quality. Female respondents and humanities students penalized hardest.

The pattern across all these studies is consistent: the surface impresses until you know where it came from. Then something else kicks in, call it provenance bias, call it authenticity seeking, call it whatever you want. The point is that humans don't evaluate poems purely on their merits. They never did.

Process and Output, or: Does the Suffering Matter

Photography didn't kill painting. Everyone knows this because everyone trots it out whenever AI generates something that makes artists nervous. But the parallel is worth examining more carefully than most people bother to.

When photography arrived in the 1830s, the debate wasn't really about whether photographs were beautiful. It was about whether beauty produced without the painter's hand counted as art. Baudelaire called photography "art's most mortal enemy." The Académie des Beaux-Arts spent decades arguing about whether photographers could claim authorship of their images.

The argument sounds familiar because it is familiar. Swap "photographer" for "prompt engineer" and "daguerreotype" for "language model" and you've got the same conversation, preserved in amber, waiting for each generation to crack it open again.

What actually happened with photography: painting didn't die. It mutated. Impressionism emerged partly because photography had cornered the market on faithful reproduction. Painters stopped competing on accuracy and started competing on perception. The technology didn't eliminate the art form, it forced the art form to figure out what it was actually for, beyond the mechanics of rendering.

Poetry might be in a similar moment. If a language model can produce verse that reads better than Dickinson to a room full of strangers, then the thing that makes Dickinson matter obviously isn't the surface quality of the text. It's something else. The biography, the context, the specific weight of a specific human choosing specific words while living a specific life.

Or maybe it's nothing else. Maybe the surface was always the point and we just told ourselves stories about depth to justify the gatekeeping. The original study didn't answer this. Fifteen months of follow-ups haven't either. They've just made the question harder to avoid.

The Aesthetic Turing Test

Alan Turing's original test was about conversation: can a machine fool a human into thinking it's human? We've been running informal aesthetic Turing tests for years now, AI art competitions, generated music, synthetic voices. But Porter & Machery were among the first to do it rigorously, with a proper sample size, against canonical works, with statistical controls.

The machine passed. Convincingly. Against a general audience.

It failed against anyone who knew what they were reading. Bjornstad's 96.6% is a reminder that "indistinguishable" is doing heavy lifting in that headline, indistinguishable to whom? A sommelier and a college freshman taste the same wine differently. Neither is wrong, exactly. They're just running different tests.

Study 2 of the original paper tried to address this. They told 696 participants upfront that some poems were AI-generated. People still couldn't identify them reliably, but they did rate poems lower when told they were AI-generated, even when they weren't. The Romanian study found the same thing with teenagers a year later. The label "AI" functioned as a quality discount regardless of actual quality.

That bias cuts both ways. It means we're not evaluating poems on their merits. We're evaluating them on their provenance. Which means the aesthetic Turing test might be the wrong test entirely, not because the machine fails it, but because humans don't actually use aesthetic criteria when they know the source.

Handmade

There's a bakery in Apt that charges twice what the supermarket charges for bread. Same flour, roughly. The bread is better, but not twice-as-good better. People pay because someone's hands were in it. Because you can watch the process through the window. Because the imperfections are evidence of a human choosing to be there at 4 AM instead of doing something easier.

That premium exists for a reason that has nothing to do with the bread itself. It's about participation in a story. The bread is a receipt for someone's morning.

Handmade poetry might work the same way. The poem on the page is a receipt for the life that produced it. Plath's poems hit different when you know about the oven. Ginsberg's "Howl" reads different when you know about Columbia, about Cassady, about the asylum visits. The words are the surface. The life is the weight underneath.

AI output comes without a lived context. That distinction is outside what Porter and Machery set out to measure. O'Sullivan's stylometric work gets closer: the machine's uniformity is itself a tell, a kind of uncanny valley of the written word. Not wrong, just too consistent. Too clean. Like bread that's always exactly the same shape.

Nobody rereads a bread machine's output with reverence. But nobody builds a museum around it, either.

Where It Stands

Fifteen months on, the picture is richer than the headline suggested.

The original study proved that general audiences can't distinguish AI poetry from human poetry in lab conditions, and that they prefer accessibility over complexity when reading blind. That finding stands. Nobody's seriously contested it.

What the follow-up work added: expertise matters enormously, AI writing has a detectable stylistic fingerprint that's getting better-documented by the year, and humans apply a provenance discount to AI work once they know its origin, regardless of quality. The machine writes well. The machine writes uniformly well. And that uniformity, paradoxically, is both its strength with casual readers and its weakness with anyone paying close attention.

The original study used GPT-3.5. We're two model generations past that now. The poetry has gotten better, the stylistic range wider, the tells subtler. But O'Sullivan's clustering analysis suggests the core limitation persists: language models converge on a narrow band of expression that, however polished, lacks the variance of genuine individual voice.

Maybe that changes. Maybe GPT-6 writes poems that scatter across stylistic space like actual humans. Or maybe the uniformity is structural, a consequence of how these systems learn, not a bug to be patched.

The unresolved question is whether newer models can produce the stylistic variation associated with individual human voices.

Sources:

  • Porter, B., Machery, E. AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Sci Rep 14, 26133 (2024). doi.org/10.1038/s41598-024-76900-1
  • O'Sullivan, J. et al. Stylometric comparisons of human versus AI-generated creative writing. Humanit Soc Sci Commun (2025). nature.com/articles/s41599-025-05986-3
  • Frontiers in Education: Human touch versus algorithm: reception of AI poetry among Romanian adolescents (2026). frontiersin.org
  • Bjornstad, S. In What Sense Is AI Poetry Indistinguishable from Human Poetry? controlaltbackspace.org