You have nine ideas for a new product and enough budget to test one properly. Which research should you commission?
Start with the question behind the question. You do not yet need a final verdict on one idea. You need to find the two ideas worth taking seriously.
That is where synthetic research changes the job. A synthetic audience lets you put all nine ideas in front of a model of your customers, see which ones hold up by segment, ask why, rewrite them and test again. Then you can spend traditional research budget on the ideas that survived.
The useful choice is rarely synthetic or traditional. It is which method should answer this question, at this point in the decision.
Watch: Electric Twin co-founder Alex Cooper on why traditional research falls short.
Synthetic vs traditional research at a glance
Traditional research | Synthetic research | |
|---|---|---|
Time to an answer | Weeks, with recruitment and fieldwork | About a minute |
Adding a question | Commission a new study or wait for the next wave | Ask the same audience straight away |
Iteration | The questionnaire is locked once fieldwork begins | Ask another question and run it again |
Panel fatigue | Real, and it limits how often you can ask | None. The real panel is left alone |
Small or niche segments | The sample can become too thin to interpret confidently | Explore defined segments in the same audience run |
Accuracy on NDAM | Around 94% for human test-retest | Up to 92% |
Regulated or clinical claims | Required for substantiation, with identifiable respondents on record | Useful for testing how a claim lands, but not for substantiation |
Physical or sensory testing | Required for taste, touch, packaging and in-person use | Not suitable |
Confidential or pre-launch questions | Sharing with a panel exposes the idea | The question stays inside the business |
Traditional research observes what real people say or do in a specific study. Synthetic research models an audience from real survey data and lets you question that audience repeatedly.
What is synthetic research good at?
Synthetic research is strongest while the decision can still change.
Say a team is choosing a campaign proposition. With traditional research, the practical limit might be two or three polished routes. Recruitment, fieldwork and budget encourage the team to narrow the list before anyone outside the room sees it.
A synthetic audience lets the team test the rougher, wider list first. It can compare the routes, show which audience segments react differently and let the team ask follow-up questions while the work is still open. The fourth or fifth idea costs no extra fieldwork and sometimes turns out to be the interesting one.
This makes synthetic research useful for:
Initial concept testing, when you need to narrow a wide field
Message and creative testing, while there is still time to rewrite the work
Pricing exploration, when you want to compare expectations across segments
Niche audience questions that would leave a traditional sample too thin
Confidential propositions or competitor comparisons that should stay inside the business
Small decisions, such as a subject line or product name, that would never justify a commissioned study
For a practical example, see message testing with a synthetic audience.
When do you still need real people?
Some questions need a person, not a model.
Anything on the record. Regulated claims, clinical studies and legally mandated consumer testing need identifiable real respondents. A synthetic audience can help you test how the wording lands, but it cannot provide legal substantiation.
Anything physical or sensory. If the question depends on taste, touch, smell, packaging in the hand, using a device or watching someone perform a task, traditional research is the method.
An audience with too little real data behind it. A synthetic audience is only as useful as the evidence used to build it. If the audience is genuinely new and there is little reliable survey data about the people in it, primary research comes first.
A final gate where the stakes demand direct evidence. Synthetic research can narrow the options and expose objections. A major launch may still deserve a final study with real people before the budget is committed.
This is why traditional research becomes more valuable when the two methods work together. You stop spending it on questions that can be answered earlier and use it where direct human evidence changes the decision.
What does the evidence say?
A comparison should not rest on speed alone. The answers have to stand up against real research.
Electric Twin uses holdout validation. Part of a real survey dataset is kept separate while the synthetic audience is built. The audience then answers questions from that hidden portion, and its answers are scored against what the real people said. The evaluation questions could not have been memorised during the build.
Across structured survey benchmarks, Electric Twin reports up to 96% on 1-MAE and up to 92% on NDAM.
The labels matter. 1-MAE measures how close predicted headline percentages are to the real percentages. NDAM is stricter because it measures the shape of the full response distribution and keeps the same scale regardless of how many answer options a question has.
On NDAM, human test-retest sits at around 94%. That is what happens when real people answer the same survey question twice. Electric Twin reaches up to 92%, two percentage points below that human benchmark. An off-the-shelf GPT-5 model answering directly, without Electric Twin's survey data and audience methodology, reaches 65% on the same measure.
Those figures are all on NDAM. They should not be compared with a 1-MAE score. The full definitions and validation method are set out in Electric Twin's accuracy methodology.
The full method, and what each measure does and does not cover, is on our accuracy methodology.
What happens when the human panels disagree?
There is another complication. Traditional research is not one fixed ground truth.
In the pre-registered Where You Ask Matters study, the same survey was given to 7,755 people across seven online sampling services, alongside an Electric Twin synthetic audience. Every source received the same questions and the results were weighted against the same recruitment targets.
The human panels did not return one clean answer. Voting intention moved by as much as 20 percentage points depending on the platform. After demographic weighting, the gaps shrank by more than 80%, which showed how much of the variation came from who each platform had recruited. No single human platform performed best across every measure.
The synthetic audience landed within the range of the human results across most questions.
That does not make human research unnecessary. It makes validation more honest. Any method, synthetic or traditional, should be judged by how it was sampled, what it was compared against and whether the result can be reproduced. The seven-panel study gives the full account.
How do you validate a synthetic audience on your own data?
Use the same holdout test with a survey you already trust.
Give the provider one part of the dataset and choose which questions to keep back. The audience is built without those answers. Then ask the hidden questions and compare the synthetic distribution with the real one.
This is more useful than accepting a platform-wide accuracy number on faith. It tells you how the method performs on your audience, your questions and your data. And it gives stakeholders something concrete to inspect when you need to defend the method choice.
Periodic comparisons with fresh real responses can also show when the audience needs updating. Traditional research supplies the new evidence. Synthetic research makes that evidence useful across many more decisions between waves.
Does synthetic research cost less?
Traditional research is usually commissioned study by study. A synthetic audience is built once and questioned repeatedly. The important change is therefore not a cheaper version of the same study. It is the number of questions a team can afford to investigate.
At The Times, product, marketing and commercial teams used one synthetic audience for concept testing, naming work and audience discovery. Research volume increased roughly tenfold across the three functions, with the same team and no extra budget.
“We went from rationing research to running it on demand. The same team, ten times the output, and we could prove it matched our real audience.”
Chris Courtney-Smith, Director of Data Operations, The Times
The full Times case study shows what changed in practice. Product teams could test feature ideas, marketing could compare names for a new podcast, and commercial teams could examine audience overlap before backing a partnership.
Which method should you choose?
Here is the rule we use: choose the method that matches the decision, rather than the method your team has always commissioned.
The rule most teams land on: run a synthetic audience first to narrow a wide field and to read a segment, then commission traditional research on what survived and has to stand up on the record. The real waves also keep the model honest, so the two get better the longer you run them together. Pricing questions run through pricing testing, and proposition work through message testing.
The question | Best fit | Why |
|---|---|---|
Which of nine propositions is worth developing? | Synthetic | The job is to narrow the field before expensive work begins |
Does the surviving proposition justify the campaign budget? | Both | Explore and refine synthetically, then use real respondents where final evidence matters |
What would this niche B2B segment pay? | Synthetic | A conventional sample may be too small or difficult to recruit |
Can we make this regulated claim in market? | Both | Test the wording synthetically, but use identifiable respondents for substantiation |
Does the new pack feel premium in the hand? | Traditional | The answer depends on a physical experience |
Which segment objects to this change, and why? | Synthetic | The useful answer is in the differences between segments |
Is our confidential competitor comparison landing? | Synthetic | The proposition can be tested without sending it to an external panel |
Which final route should we put live? | Both | Narrow synthetically, then use judgement, human research or an A/B test for the final gate |
The most effective sequence is often straightforward:
Use synthetic research to explore the full field
Ask follow-up questions and improve the strongest options
Take the survivors into traditional research when you need direct human evidence
Use fresh real responses to validate and update the synthetic audience
That sequence gives researchers a larger role in the decisions that matter. Instead of spending the whole budget proving out the first shortlist, they can help teams ask better questions, challenge the result and decide what deserves a real-world test.
If you want to try that sequence with your own audience and questions, book a demo.
