Political campaigns spend a lot of money hiring consultants to craft persuasive messages, but a new study co-authored by the Yale political scientist Joshua L. Kalla demonstrates that political professionals are no better than laypeople in predicting which messages will influence voters.

In the study, Kalla and his co-authors assessed how well sample groups of political practitioners – professionals working for political campaigns, polling firms and advocacy organizations – and members of the public could predict the effectiveness of 172 campaign messages on 21 policy issues, including legalizing marijuana, canceling student debt, and increasing border security.

They found that both groups performed only slightly better than chance and that practitioners were no more perceptive than laypeople in identifying messages that resonated with people.

“We found that neither political practitioners nor the general public are particularly accurate at predicting which persuasive messages are more effective than others,” said Kalla, an associate professor of political science at Yale’s School of Arts and Sciences. “This suggests that policy practitioners who craft language intended to persuade have fairly poor intuitions about what messages people will find persuasive. »

The study, published in the journal Proceedings of the National Academy of Scienceswas co-authored by David E. Broockman, Christian Caballero, and Matthew Easton, all of the University of California, Berkeley.

For the study, researchers collected 172 textual political messages that political practitioners used to support or oppose 21 separate issues. They pulled the messages from sources such as voter guides published by various advocacy organizations and the social media accounts of prominent politicians.

One example is a message used by the Marijuana Policy Project to support cannabis legalization: “Polls show that a strong and growing majority of Americans agree that it is time to end cannabis prohibition. Nationally, a recent Gallup poll found that 66% of respondents support legalizing marijuana use for adults.

To measure the effectiveness of these messages, the researchers conducted a large-scale survey experiment, in which they randomly assigned 23,167 participants to a treatment or control group. The treatment groups received messages about three specific issues: The control group saw no messages. The researchers then surveyed participants in both the treatment and control groups about their opinions on the questions, for a total of 67,215 participant observations. The researchers used this data to estimate the effectiveness of each message.

Next, they asked 1,524 policy practitioners with varying experience and expertise and 21,247 laypeople to predict the effectiveness of the messages. (Ninety-one percent of practitioners reported being directly involved in crafting the messages.) Both groups predicted the persuasiveness of the messages only slightly better than if they had guessed at random.

The study showed that members of the public thought others were more persuasive than the initial survey showed or what practitioners expected. But after accounting for these inflated expectations, practitioners did not make significantly better predictions than laypeople.

Among policy practitioners, the study found that experience or subject matter expertise did not translate into greater ability to identify effective messages.

The findings suggest that, rather than relying on intuition, policy practitioners should consider incorporating data science techniques into their evaluations of potential messages, said Kalla, a professor at the Institution for Social and Policy Studies in Yale.

“The main takeaway here is that policy practitioners have tools to help them identify effective messages without having to rely on intuition,” he said. “They could use survey experiments similar to what we did in this study. We’re already seeing political campaigns doing this, and I suspect more will adopt such techniques in the future. »