A commissioning desk decides this week. The research that would tell you whether the decision is right comes back in 3 to 6 weeks. By then the piece has run, and the research arrives as a post-mortem rather than as something that shapes what happens next.
That is the first problem. Here is the second.
Readers are not reliable witnesses to their own taste. On Facebook, The Atlantic collects about 27 mentions for every one the National Enquirer gets. In the circulation figures, and in the number of people who actually search for each title, the two are level. Seth Stephens-Davidowitz uses that gap in Everybody Lies to make a plain point: what people say they read is the version they want to be seen reading, and it can sit a long way from what they actually open.
Put the two together and you have most editorial research. It asks readers to predict their own behaviour, they give the flattering answer and the result lands too late to use.
Content testing with a synthetic audience takes aim at both. You put the actual headline, format or cut of a trailer you are about to commission in front of a model built from your real subscribers, and read how each segment reacts in about a minute, before you commit any budget.
What can a publisher test with a synthetic audience?
Mostly the calls that get waved through in a commissioning meeting because there was never time to research them.
Decision | How it gets made now | What you can ask instead |
|---|---|---|
Whether to commission a series | Editorial judgement plus last quarter's traffic | Which segments would follow it to ep3, and what would make them stop |
What to call a podcast or newsletter | A shortlist argued in a room | What each name signals, and to which segment |
Which format a story should take | Whatever the desk usually does | Whether this audience wants the explainer, the video or the long read |
A subscription price or tier change | Benchmarking against competitors | What each segment expects to pay, and what makes it worth that |
A paywall or registration change | An A/B test after it ships | Which segment walks away, before it ships |
Whether a commercial partner reaches your readers | Two audience decks side by side | Where the two audiences genuinely overlap |
An AI-assisted product feature | An assumption about reader appetite | Which readers want it, and which quietly consider it a downgrade |
Why is this different for publishers specifically?
2 things are true of a publisher and of very few other businesses.
The first is timing. Your decision window is shorter than your research window, and no amount of hustle closes the gap. The desk moves faster than the panel can answer.
The 2nd is the opposite of what you would expect. You are not short of audience data. You have subscription records, engagement data and reader panels, years of it. What you are short of is permission to keep asking. Every survey spends a little of your readers' goodwill, and that well is not bottomless. The Times had 642,000 subscribers and a firm limit on how often it could go back to them.
A synthetic audience takes the routine questions so the real panel does not have to, which keeps the goodwill for the moments that genuinely need a person on the record. There is more on why that well is running dry in why customers stopped answering surveys.
How did The Times use this?
It ran as a shared research layer. Three teams pointed it at their own problems, rather than one workflow being imposed from the top.
Product stopped spending weeks on panel work and started getting reactions in seconds. One test is the reason people paid attention. Loyal readers were wary of AI features, while the readers The Times most wanted to win over were asking for exactly those features: summaries, AI explainers, video. Two opposite answers inside one audience. Average them and you get a lukewarm number that describes nobody, and that number would have set the product roadmap. Seeing the split, instead of the average, is what set it.
Marketing ran CRM work, copy testing and campaign evaluation without rationing questions. The example people quote back: four candidate names for a new business podcast, where the winner, The Business, tested fine with the general reader panel and much better with the exact group the show was chasing. The same shortlisting logic sits in message testing with synthetic audiences.
Commercial used it to check partnerships before signing them, asking whether a partner's audience actually overlapped with theirs rather than taking two decks at face value. That evidence had not existed before, and it let them back the deals that reached real readers.
Same team, same budget, roughly ten times the research volume across the three functions.
The full story is in The Times case study, and the tool they built on top of it is covered in The Times' synthetic audience tool built on Electric Twin.
But is it accurate on a real subscriber base?
Fair question. The honest answer is a number, a measure and a method, and you should refuse to buy without all three.
The method is holdout validation, and it runs on your own data. Part of what you already hold is kept back during setup, the audience is built from the rest, and it is then scored on questions it has never seen. Because the evaluation questions never went into building the audience, the personas answer from their characteristics rather than from something they were already told. Across studies that lands at up to 92% on NDAM.
Two measures sit behind any accuracy claim, and they are not interchangeable. 1-MAE asks how close the headline numbers are: if real readers say 40% agree and the model predicts 43%, that is a three-point miss. NDAM asks how well the whole distribution matches, the splits, the skews and the tails, which is the harder and more honest test.
Measure | Electric Twin | Traditional research | GPT-5, off the shelf |
|---|---|---|---|
1-MAE, the headline numbers | Up to 96% | 96% | 79% |
NDAM, the full distribution | Up to 92% | 94% | 65% |
Read those rows across, never down. An NDAM score and a 1-MAE score measure different things, so putting one next to the other overstates every gap it appears to show. The traditional research column is the human test-retest ceiling, which is the definition of as good as it gets, and the GPT-5 column is an off-the-shelf model answering directly, with no seed data and no persona methodology.
That is also the shape of the question to ask a vendor: which measure is this on, and can I see it on my own subscribers with the holdout held back by me? The measures and the protocol are in what a synthetic audience is and how it works, and the methodology sits on accuracy.
Can you test the actual asset, or only the idea of it?
Yes, and for a commissioning call that is the whole point. This is the stated-versus-revealed gap from the top of the page, closed from the other side. Rather than asking someone what they think about AI explainers, you show them the actual explainer and read the reaction. Reacting to a cut of a trailer is not the same as holding an opinion about the trailer's subject, and it is the reaction you want.
We validated our audiences against real human reactions to real assets: 18 images and 6 videos of up to three minutes, measured against a benchmark of 4,000 people. None of the opinions those people held about the assets went into building the audiences, so the audiences reacted cold rather than repeating something they had been told. The predictions tracked the real reactions closely, including on which of two assets won a straight head-to-head, which is the shape almost every format decision takes. Image and video work runs through creative testing, and the method is set out in how accurate synthetic audiences are.
The honest line for a newsroom: this is strong enough to shortlist a thumbnail, a trailer cut or a format treatment down to the ones worth backing. It is not the final word. Narrow with it, then commit with editorial judgement.
What did adoption actually take?
More than a year in, the thing The Times kept flagging was cultural, not technical.
Bring this into a research team and it splits people in two. Some see their capability multiply overnight. Others see a threat to the thing they are trusted for. That divide does not close because you waved a good accuracy score at it. It closes with visible human oversight, repeated validation against real responses, and a message that never wavers: this extends what researchers do.
There was a second lesson, about proof. Showing that any single editorial decision turned out better is close to unprovable, so instead they counted what they could count, research volume and speed to delivery, and put those in front of leadership. That was easier to prove than decision quality, and it was the argument leadership actually moved on.
What should you test first?
Start where being wrong is cheap to discover and expensive to ship. Three phases, in order, because you have to earn the right to test the decisions that carry weight.
Test something you already know the answer to. Re-run a question your last reader survey answered, and compare. This feels like a waste and is not. It is how the insight team walks into the newsroom with a number it can defend when someone asks why anyone should believe a synthetic audience.
Test the decisions nobody bothers to research. Names, subject lines, format calls, newsletter positioning. Low stakes, high frequency, currently decided by whoever talks loudest in the room. Any evidence is an upgrade, and nobody feels their job is under threat. This is where most of that tenfold jump in volume comes from.
Bring it to the decisions with money on them. Subscription tiers, paywall changes, a series commission, a partnership. By now it has a track record inside the building, which is what makes the answer usable in a room where budget is being signed off.
Get the order wrong and you start the credibility fight before you have any evidence to win it, which is how good pilots quietly die.
One more thing, on ownership. The Times ran it as a shared layer across product, marketing and commercial. The moment a single team owns it, it becomes a service-request queue, and you have rebuilt the bottleneck you were trying to escape.
Where does this fall over?
A thin or brand-new audience. Launch into readers you hold no data on and the model has nothing to stand on. Do the primary research first.
Anything that needs a name on the record. Regulated claims and legally mandated testing need real, identifiable people. No exceptions.
A replacement for the reader panel. The panel is now your calibration layer. Skip the periodic real-world checks and the audience drifts away from your actual readers without ever announcing it. Google Flu Trends predicted flu from search data, impressively at first, then drifted so far it was retired. A synthetic audience that is never re-checked goes the same way, quietly.
A stand-in for editorial judgement. It tells you what a modelled audience would say. Whether that should change what you commission is your call, and it should stay your call. A title that commissions purely to a predicted preference has stopped having a point of view, and readers notice that quickly.
What does a publisher need to start?
Real data about the readers you want to model: subscription records, engagement data, old survey waves, reader panel responses. We build the audience from that and validate it against a held-out slice of the same data before you rely on a word of it.
The sector walkthrough, from subscription pricing to the greenlight meeting, is on Electric Twin for media and entertainment. Pre-launch format work lives on concept and idea testing.
You ask, we answer
How is this different from analysing the audience data we already have?
Analytics tell you what readers have already done. A synthetic audience answers the question you have not asked yet, including about formats and products that do not exist.
Does this work for commercial and advertising teams?
Yes. The Times used it to model audience overlap before committing to partnerships, and put spend behind the deals that actually reached their readers.
Can we test commercially sensitive ideas?
That is one of the best reasons to use it. A pre-launch format or a competitor comparison stays inside the building instead of going out to a panel.
How often does it need re-validating?
Periodically, against fresh real responses. Readers move, and re-validation keeps the model moving with them.
