Shakespeare or ChatGPT? Study finds people prefer AI over real classic poetry

Readers are unable to reliably differentiate AI-generated from human-written poetry and are more likely to prefer AI poems, according to new research published in Scientific Reports. This tendency to rate AI poetry positively may be due to readers mistaking the complexity of human-written verse for incoherence created by AI and an underestimation of how human-like generative AI can appear.

Researchers Brian Porter and Edouard Machery, from the Department of History and Philosophy of Science, University of Pittsburgh, Cathedral of Learning, tested the ability of 1,634 participants to distinguish between AI-generated poetry and that written by a human poet.

Participants were presented with ten poems in random order: five written by ten well-known poets—including William Shakespeare, Lord Byron, Emily Dickinson, and T.S. Eliot—and five poems generated by ChatGPT3.5 in the style of these poets. Participants were more likely to guess that the AI poems had been written by a human, and the five poems considered least likely to be human-produced were all written by genuine poets.

In a second experiment, a different group of 696 participants assessed the poems for 14 characteristics such as quality, beauty, emotion, rhythm, and originality. Participants were randomly assigned to three groups where they were told the poems were written by a human, produced by AI, or given no information about the poem‘s origins.

Participants who were told that the poems were AI-generated gave lower ratings across 13 characteristics compared to participants who were told the poems were human-written, regardless of whether the poems were actually AI-generated or human-written. Participants who were told nothing about authorship rated AI-generated poems more favorably than human-written ones.

The authors suggest that participants preferred AI poems due to them being more straightforward and accessible than the work of the prominent poets. Additionally, participants expect to prefer human-written poetry and as they find the AI-generated poetry easier to interpret and understand, they misinterpret this preference as an indication that the poem was written by a human.