AI Writes the Emails, but Human Judgment Still Picks the Winner, Study Finds

Artificial intelligence has quietly transformed nearly every corner of the marketing profession, drafting emails, social media posts, advertisements and product descriptions at a speed no human team could match. Yet a fundamental problem has emerged alongside this explosion of machine-generated content: when a company can produce fifty versions of an email in the time it once took to write one, how does it decide which version will actually persuade its customers? A new study from Penn State’s Smeal College of Business and collaborators offers a technically grounded answer, proposing a framework that trains AI models to screen newly generated marketing content using the performance history of a business’s own past campaigns. The research, published in the Journal of Marketing Research, demonstrates on a large-scale email marketing campaign how data analysis and human evaluation can be combined to predict campaign success while dramatically reducing the need for costly, time-consuming traditional testing.

Wreetabrata Kar, assistant professor of marketing at Penn State and the study’s lead author, explains that marketers now deploy AI across almost every stage of their work, though content creation remains the most visible application. Teams rely on generative models to draft emails, social media copy, advertisements and product descriptions, and increasingly to produce images and short videos as well. The technology has moved well beyond writing, however. Marketers use AI to brainstorm campaign concepts, research markets and analyze customer data to divide audiences into groups for personalized messaging. Some firms employ predictive models to anticipate what customers will do next and to automate portions of the customer journey. The result is an unprecedented expansion of creative capacity, but also an unprecedented expansion of choices that must somehow be evaluated before a single message reaches a customer.

The core challenge, Kar argues, is that AI does not automatically know what works for a particular company. A message can sound creative and persuasive to a language model, yet that quality guarantees nothing about how a specific firm’s customers will respond. A general-purpose AI tool has never seen a company’s past campaigns and has no knowledge of how its customers reacted to them. That gap between generic fluency and firm-specific effectiveness is precisely what the new framework is designed to close. Rather than treating every piece of generated content as an unknown quantity requiring live experimentation, the approach leverages the rich, often underused record of historical campaign performance that most businesses already possess.

The technical heart of the method lies in how it connects new, untested messages to old, measured ones. Every company has a history of past campaigns, and each of those campaigns can be tied to measurable changes in customer behavior, such as purchases, clicks or conversions. A brand-new message, by contrast, has no history of its own, which is what makes evaluating it so difficult. The researchers solve this by having an AI model read each email message and convert its meaning into a set of numbers, effectively placing every message as a point on a high-dimensional map. Messages that mean similar things land close together on this map. For example, the phrases “Free Shipping on Orders $50+” and “Orders Over $50 Ship Free” end up side by side, while “Break Free from the Rules” lands far away despite sharing the word “free.” Older methods that simply count words cannot distinguish these cases, but the semantic representation generated by AI can.

Once every historical message occupies a position on this map, a newly generated message can draw insights from its nearest neighbors. If a candidate email sits close to past messages that performed well, the model predicts it will probably perform well too. If it sits far from anything the company has ever tried, the model flags that it cannot make a reliable prediction, and in that case the researchers recommend falling back on a traditional test. This is where the approach diverges sharply from conventional A/B testing, in which different versions of a message are sent to real customers and analysts wait to see which variant wins. Live testing takes time, consumes budget and, crucially, exposes some customers to the weaker message along the way. The new framework lets marketers skip the test entirely when their historical data already contains the answer, reserving genuine experimentation for truly novel ideas that lie outside the firm’s accumulated experience.

The framework’s second component addresses the generation side, and it is here that the study delivers one of its most striking findings about the limits of AI self-evaluation. The researchers gave an AI model the building blocks of a retailer’s past emails, elements such as discounts, free shipping offers and clearance sale announcements, and asked it to compose new messages. The model produced a flood of options within seconds, a substantial gain in productivity. But when the researchers asked the same model to select its own five best emails, those picks were predicted to perform poorly. Writing an email and judging it, the study concludes, are fundamentally different jobs. The solution was to split the labor: the AI model writes the candidate messages, while the company’s own historical data, interpreted through the AI-generated semantic map, judges them. Anything too distant from the company’s past experience is set aside, and the surviving candidates are ranked by their predicted performance.

Importantly, the framework does not sideline human marketers; it repositions them at the points of the process where judgment matters most. Marketers choose the building blocks from which new messages are composed, deciding which offers, themes and tones are appropriate for the brand. They also calibrate the acceptable level of risk, determining how far a candidate message may stray from past experience before it must be validated with a real test rather than a model prediction. Finally, the marketer makes the final selection from the shortlist, drawing on deep knowledge of the business and the nuanced context that no model can fully capture. AI performs the heavy lifting on both writing and reading, Kar notes, but people make the decisions, and with far more options on the table than ever before, that human judgment has become more valuable, not less.

The study, titled “EXPRESS: Evaluating Novel Unstructured Treatments with Generative AI: A Causal Prediction Framework,” was conducted using data and statistical analysis and published on 3 August 2026. Kar’s co-authors are Paul Ellickson and Guang Zeng of the University of Rochester and James Reeder of the University of Tennessee, Knoxville. Their combined expertise spans marketing, econometrics and machine learning, reflecting the interdisciplinary nature of a problem that sits at the intersection of causal inference and generative modeling. By framing content evaluation as a causal prediction task grounded in historical treatment effects, the researchers move beyond the superficial quality signals that language models excel at producing and toward measures tied to actual business outcomes.

The implications extend well beyond email marketing. Any organization that generates content at scale, from retail promotions to public health messaging, faces the same evaluation bottleneck: generative models can produce nearly unlimited variations, but live testing each one is expensive, slow and sometimes harmful to the audience receiving inferior variants. A framework that uses a firm’s own campaign history as a training signal for automated screening offers a scalable middle path between untested generation and exhaustive experimentation. It also carries a broader lesson for the adoption of AI across knowledge work. The technology’s greatest value emerges not when it replaces human judgment wholesale, but when it is paired with domain-specific data and human oversight, allowing machines to generate and rank options while people retain control over strategy, risk and the final call.

For marketers contemplating the next wave of AI tools, the message from University Park is clear and pragmatic. Speed and volume are no longer the binding constraints; discernment is. Companies that invest in connecting their generative systems to their own historical performance data, and that preserve meaningful human decision points in the workflow, stand to capture the productivity gains of AI without surrendering the contextual knowledge that makes marketing effective. Those that rely on the models’ own assessments of their output, the study suggests, may find that the most confident machine recommendations are also the least reliable.

Subject of Research: A causal prediction framework using generative AI to evaluate and rank AI-generated marketing content against historical campaign performance data.

Article Title: Q&A: How can marketers use AI effectively — without losing human judgment?

Article References: Q&A: How can marketers use AI effectively — without losing human judgment?. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: artificial intelligence, marketing, generative AI, A/B testing, Journal of Marketing Research, Penn State, email marketing, causal prediction, consumer behavior, human judgment, content evaluation, machine learning