Simulated buyers are cheap and fast. Some forty studies and industry reports from 2023 to October 2026 say they are right often enough to use for three things: ranking ideas when the differences are big, telling whether a message works at all, and a first read on which well-known brand leads. They are wrong more often than they are right for everything underneath: how many people buy, what they would pay, what a smaller group thinks, and what changed since last quarter. Nobody who sells simulated research publishes the numbers that would let a buyer tell the two apart. The studies below do.
How to read the grades
- Usable: right often enough to act on a cheap, reversible decision.
- Usable with a human anchor: works once calibrated on some real answers; never alone.
- Not yet: wrong more often than useful in every fair test we found.
- No public test: nobody has published a fair comparison. Claims exist, data do not.
The map
Which of these ideas is stronger?
Usable as a first screen, when differences are big
Ranking 57 personal-care concepts reached 90% of the human panel's own test-retest reliability (9,300 people; authors from PyMC Labs and Colgate, one replication). On 17,681 real headline A/B tests, plain prompting was close to chance; on 2,599 such tests, a small model fine-tuned on human results picked decisive winners 52.8% of the time against 25.6% by chance; the raw predictions of a general model recovered 39% of the human effect.
Does this message work at all?
Usable for direction, not for size
Across 70 pre-registered US experiments, GPT-4 predictions correlated 0.85 with human results, but effect sizes were overstated and megastudies came out lower. 76% of main effects from 45 published marketing studies replicated, 68% once interactions counted; the counterintuitive findings were the ones missed.
Which brand leads?
Usable with a human anchor
Twins matched real soft-drink choices 79% of the time but overstated the well-known brands and showed far less variation (461 US consumers). The article gives no baseline for a plain guess from brand shares, so 79% cannot be read as "8 in 10 right". Simulated awareness of Starbucks 100% against 91% real, Dunkin' 100% against 87%. A US insurer nobody in France knows: 5% real awareness, 85% simulated.
How many buy, and how often?
Not yet
What will they pay?
Not yet, except hybrid designs
In a UK study of synthetic boosts to a real sample, one statistical provider broke the price order 68% of the time (33% when blended with humans) and overstated willingness to pay for sustainable products by more than 20 points in one segment. Against real willingness to pay from an incentive-compatible auction in four product studies, simulated buyers overstated it for the two more hedonic products, ranged from close to well below for the two more utilitarian ones, and their distributions differed in all four. Simulating from demographics alone flipped the sign of every product-origin coefficient in 18,144 real choices. A hybrid design halved conjoint error with 75% fewer human labels.
What does this group think?
Not yet
Women 35 to 44: 31% real, 14% simulated. Segment gaps inflated two- to fourfold; the wrong segment picked in 50 to 72% of cases. Spread about half of the human spread (SD 16 against 31); 48% of regression coefficients differed, signs flipped in a third of those. Values skew to English-speaking, Protestant Europe; the worst-matched group in Germany was AfD supporters.
What changed since last quarter?
Not yet
In the same UK study, the one synthetic data provider tested in both 2025 and 2026 called the direction of change correctly 47% of the time across all measures, and only 19% of the time on brand awareness, missing real movement in 13 of 16 brands. Presidential approval: humans 37% in January, 34% in April; the simulated panel said 46% both times. Error averaged 12 points over more than 120 trend questions whatever the real movement. No published case of a synthetic panel catching a move a human panel caught.
What share gives each answer?
Usable on high-salience topics, not elsewhere
Average error 2.0 and 2.8 points on UK snack toplines; 3.0 points on US political items. But 12 points on average across nearly 300 Pew questions; in a human-panel vendor's own tests, 23 points on health care and 20 on coffee. Humans were four times as likely to say "not sure"; on nearly half the questions, an option real people chose was chosen by no simulated respondent.
What will this particular person say?
Only with that person's own answers
After two-hour interviews, agents reached 82 to 86% of participants' own two-week test-retest consistency (1,052 people). Built from 500 earlier answers per person, twins correlated 0.20 with the same people on 164 new outcomes and beat an empty persona by a hair (0.748 against 0.734). From demographics alone, the best model equalled a lookup table (0.589 against 0.589).
Which ad will people love?
No public test
No independent comparison against System1, Kantar Link, USA Today Ad Meter or IPA cases exists, and no sealed forecast. Vendors claim agreement with human tests, for example "comparable ratings for more than 7 in 10 ads", without samples or definitions. Off-the-shelf multimodal labels can reverse the sign of an ad's measured effect; simulated viewers agree weakly and flatten subgroup differences.
What will the market share be?
One backtest, no forecast
600 simulated consumers tracked German car registrations over 41 months to within 0.44 points per brand-month, but only after calibration on the real data; the authors call it a backtest.
Five patterns behind the failures
- It knows the famous and forgets the rest. Big brands come out bigger, unknown brands come out known (5% real, 85% simulated), and global names attract positive attributes by default.
- Everyone sounds the same. Spread is about half the human spread, almost nobody says "don't know" (0% against 3%), and on half the questions an answer real people gave was never given at all.
- It never says no. Purchase, intent and willingness to pay run high; simulated customers do not walk away.
- It knows the past, not the present. Trend questions come out flat whatever happened; the error does not shrink when the world moves.
- It thinks in stereotypes. Groups are painted by their demographics, gaps between them inflated two- to fourfold, and the wrong group picked half the time.
One fact changes how to buy it: more simulated respondents do not make it more accurate (500 to 5,000 agents moved the error from 6.8 to 6.7 points: bias, not noise). Real human answers do: hybrid designs cut bias from 24 to 86% down to under 5%, and halved conjoint error.
What this means for a company
Use simulated buyers for: cutting twenty ideas to three before a human test; a first read on whether a message works and in which direction; a first read on which well-known brand leads a shortlist; rehearsing a questionnaire before fielding it.
Do not use them yet for: volume or revenue forecasts; price setting; sizing a segment or a minority audience; tracking; ad scores; any number that will be shown to a board without a human anchor next to it.
The one-sentence rule: treat a simulated panel as a clever colleague who has read everything and met no one. Good for a first opinion, not for a decision with money on it.
The hole in the record
- Vendors publish claims, not data. Only Qualtrics (one replication, eight attitude items) and PyMC Labs with Colgate-Palmolive publish both a method and a number; neither has been independently replicated. Kantar, System1, Toluna, Evidenza and Aaru publish results without samples, definitions or raw data.
- The professional bodies set rules for disclosure, not for accuracy. AAPOR (May 2026) calls synthetic responses the riskiest use of AI in survey research and finds no consensus that they can stand in for people; the ICC/ESOMAR Code (2025) requires disclosure of synthetic personas; the Insights Association says do not present them as equivalent without validation. None sets a threshold.
- Nobody has tested ads, and nobody has sealed a forecast. Every published number is a backtest on data the models may have seen. No pre-registered, time-stamped prediction of a human result exists in public.
- The record is lopsided. The harshest numbers come from human-panel vendors (Verasight, Strat7), the kindest from tool vendors or from tasks the models may have met in training (published studies, political polls). An independent referee is missing.
What Kesterley does with this
This is a description of design, not yet of proof. Kesterley runs simulated buyers as a blind panel (the model does not know which idea is the client's) against the client's real competitors, reads the result against the laws of buying, labels every simulated number as a simulation, and publishes its accuracy sealed in advance, whatever it shows. The first rows of our own in the table above: sealed predictions of human ad scores (UK Christmas advertising 2026 against System1, if started in time; the Super Bowl in February 2027 against USA Today Ad Meter), a test of whether a simulated panel reproduces the known patterns of buying (double jeopardy, duplication of purchase) in categories with public human panel data, and simulated brand choice against public market shares in the ten categories we already measure. Until those rows exist, our row is empty, and we say so.
Sources
[V] primary source read; [V abstract] abstract read; vendor marks a company's claim about its own product.
Pew Research Center, Data Labs, "Can AI Stand In for Human Survey-Takers? Not Really", 30 Sep 2026 [V]. Claude Opus 4.6 twins of American Trends Panel members, 291 questions (119 + 105 + 67). https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/
Pew Research Center, Data Labs, synthetic surveys and timely and topical questions, 30 Sep 2026 [V]. https://www.pewresearch.org/data-labs/2026/09/30/synthetic-surveys-and-timely-and-topical-questions/
STRAT7, "Synthetic data: Is this as good as it gets?", white paper 7 Jul 2026 (gated) and press release 14 Jul 2026 [V]; STRAT7 blog "Synthetic data: What your CMO needs to know", 19 Aug 2026 [V] (47%, 19%, 13 of 16 brands, willingness to pay in one segment), https://strat7.com/blogs/synthetic-data-what-your-cmo-needs-to-know/ . The study tests synthetic boosts of a real sample, not persona tools; the 47%, 19%, 68% and 33% figures are one statistical, non-LLM provider's. The +16% willingness to pay reported by Research Live is not in a primary source and is not used. About 3,000 UK adults, two anonymised providers. https://strat7.com/press-releases/synthetic-data-still-falls-short/ ; https://www.research-live.com/article/news/synthetic-data-survey-responses-significantly-different-to-humans-finds-study/id/5151157
STRAT7 blog, "What insight teams need to know about LLMs", 17 Mar 2026 [V]. UK, France, Malaysia; Claude, Mistral, Qwen. https://strat7.com/blogs/what-insight-teams-need-to-know-about-llms/
Kaiser, Kaiser, Schallner, Manewitsch (NIM), Rau (LMU), "Leaving Insight to Digital Twins?", NIM Marketing Intelligence Review 18(1), Apr 2026 [V]. 461 US soft-drink consumers, GPT-4o twins from demographics. https://www.nim.org/fileadmin/PUBLIC/12_NIM_MIR_Issues/MIR_KI_in_der_Mafo/MIR_AI_Market_Research/Kaiser_ea_AI_MR_Vol_18_No_1_ENG.pdf
Peng, Toubia et al. (Columbia), "Digital Twins as Funhouse Mirrors: Five Key Distortions", arXiv 2509.19088 v5, 19 Apr 2026 [V]. 19 pre-registered studies, 164 outcomes, GPT-4.1. https://arxiv.org/abs/2509.19088
Park et al. (Stanford), "LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals" (earlier "Generative Agent Simulations of 1,000 People"), arXiv 2411.10109 v3, 28 Jun 2026 [V]. 1,052 US adults, GPT-4o. https://arxiv.org/abs/2411.10109
Maier et al. (PyMC Labs, Colgate-Palmolive), arXiv 2510.08338 v3, 27 Oct 2025 [V]. 57 surveys, 9,300 US participants, GPT-4o and Gemini 2.0 Flash; stated intent only. https://arxiv.org/abs/2510.08338
Oetzel and Maiberger, "How price sensitive is AI? Benchmarking LLM-generated willingness to pay in pricing analytics", Journal of Marketing Analytics, 28 Sep 2026 [V abstract]. Four product studies, Becker-DeGroot-Marschak auction. Product-level percentages are not used: they are not in the abstract. https://link.springer.com/article/10.1057/s41270-026-00542-7
Chen, Zhu, Zheng (Stevens, UMass Boston), "When Synthetic Users Fail", arXiv 2607.26348 v2, 4 Oct 2026 [V]. General Social Survey and World Values Survey; Claude Haiku 4.5, Sonnet 4.6, Llama 3.1 and 3.3. https://arxiv.org/abs/2607.26348
Chen, "Simulated Customers Never Walk Away", arXiv 2606.20708, 16 Jun 2026 [V]. 2,790 real sales conversations, 793 with verified payment. https://arxiv.org/abs/2606.20708
Verasight Synthetic Sampling Reports I, III, IV, V (Morris, Leff, Enns; a human-panel vendor), Aug 2025 to Jun 2026 [V]. US panels of 1,500 to 2,000; GPT-4o-mini to GPT-5.4. https://verasight.io/reports/coffee-llm ; https://verasight.io/reports/synthetic-omnibus-survey ; https://verasight.io/reports/can-ai-digital-twins-replace-human-respondents ; https://verasight.io/reports/synthetic-sampling
Yeykelis, Pichai, Cummings, Reeves, arXiv 2408.16073 v2, Apr 2025 [V]. 133 findings from 45 Journal of Marketing studies, 19,447 personas. https://arxiv.org/abs/2408.16073
Miklian, Hoelscher, Katsos, arXiv 2603.00059 v4, Oct 2026 [V]. 420 developers, five LLMs. https://arxiv.org/abs/2603.00059
Ye, Yoganarasimhan, Zheng, "LOLA", Marketing Science 44(5), 2025 (arXiv 2406.02611) [V]. 17,681 Upworthy A/B tests. https://arxiv.org/abs/2406.02611
Ankargren, Persson, Schultzberg (Spotify), "When can LLMs replace humans in A/B tests?", 13 Aug 2026 [V]. https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests/
Clark, NeurIPS 2026 workshop, 2,599 A/B tests, fine-tuned Qwen2.5-3B [V]. https://neurips.cc/virtual/2026/169211
Ashokkumar, Hewitt, Ghezae, Willer, "Large language models can predict results of social science experiments", Nature 656, 2026 [V abstract; the 0.85 is from the working paper, the journal full text was not read]. https://gsb.stanford.edu/faculty-research/publications/large-language-models-can-predict-results-social-science-experiments ; https://ai4pb.stanford.edu/projects/predicting-results-of-social-science-experiments-using-large-language-models
Overgoor, Knight, Bart, SSRN 5494548, Sep 2025 [V abstract]. More than 10,000 human-labelled video ads. https://scholar.smu.edu/business_marketing_research/59/
Shrestha et al., arXiv 2606.07541, SocialLLM at ICWSM 2026 [V abstract]. 673 simulated viewers, 17-item scale. https://arxiv.org/abs/2606.07541
Brand, Israeli, Ngwe, "Using LLMs for Market Research", HBS WP 23-062 (2023) and MSI WP 25-136 (Nov 2025) [V for the MSI abstract; no figures from it are used]. https://library.hbs.edu/working-knowledge/can-ai-predict-whether-shoppers-would-pick-crest-over-colgate ; https://msi.org/working-paper/using-llms-for-market-research-2/
Hu, Gao, NeurIPS 2026 EconML workshop [V]. 18,144 real choices. https://neurips.cc/virtual/2026/169654
Zhang et al., "SocioVerse2", arXiv 2609.24911, 21 Sep 2026 [V]. 600 simulated consumers against German car registrations. https://arxiv.org/pdf/2609.24911
Lu et al., arXiv 2604.14575 [V abstract]. Hybrid conjoint inference. https://arxiv.org/abs/2604.14575 ; Gui and Toubia, arXiv 2312.15524 [V abstract]. https://arxiv.org/abs/2312.15524 ; Goli and Singh, arXiv 2305.02531, Marketing Science [V abstract]. https://arxiv.org/abs/2305.02531
Bisbee, Clinton, Dorff, Kenkel, Larson, "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models", Political Analysis 32, 2024 [V]. ANES 2016 to 2020, 7,530 personas. https://www.cambridge.org/core/product/B92267DC26195C7F36E63EA04A47D2FE
Santurkar et al., "Whose Opinions Do Language Models Reflect?", ICML 2023 [V abstract]. https://arxiv.org/abs/2303.17548 ; Wang, Morgenstern, Dickerson, arXiv 2402.01908, Nature Machine Intelligence [V abstract]. https://arxiv.org/abs/2402.01908
Aher, Arriaga, Kalai, "Using Large Language Models to Simulate Multiple Humans", ICML 2023 [V abstract]. Three of four studies replicate; "hyper-accuracy distortion". https://arxiv.org/abs/2208.10264
Tao et al., PNAS Nexus 2024 (arXiv 2311.14096) [V abstract]. Five OpenAI models against the World Values Survey. https://arxiv.org/abs/2311.14096
von der Heyde, Haensch, Wenz, Social Science Computer Review 2025 (arXiv 2407.08563) [V abstract]; Ma et al., ACL 2025 (arXiv 2412.13169) [V abstract]. German elections. https://arxiv.org/abs/2407.08563 ; https://arxiv.org/abs/2412.13169
Taday Morocho et al., arXiv 2602.18462 [V abstract]. Persona prompting harms under-represented groups. https://arxiv.org/abs/2602.18462
Krsteski et al., ACL 2026 (arXiv 2510.11408) [V abstract]. Synthesis alone biased 24 to 86%, under 5% with human rectification. https://arxiv.org/abs/2510.11408 ; Li and Ji, arXiv 2604.02458 [V abstract]. 59,508 people, 62 countries. https://arxiv.org/abs/2604.02458
Kinzinger, Hartmann, arXiv 2606.04592 [V abstract]. German SOEP, 500 people, 183 held-out questions, 78.8%. https://arxiv.org/abs/2606.04592 ; Toubia et al., Twin-2K-500, arXiv 2505.17479 [V abstract]. https://arxiv.org/abs/2505.17479
Kantar, Link AI, 19 Aug 2026 [V, vendor]. "Comparable ratings for more than 7 in 10 ads"; predictive model, not LLM respondents; no sample or definition. System1, Test Your Ad Screen, Aug 2026: trained on 18 million human emotional responses, results in low, medium and high bands, no published accuracy figure (trade press). The "9 in 10" match reported second-hand is not in a primary source and is not used.
Qualtrics, synthetic panels: Greenbook interview, 2 Feb 2026 [V, vendor]. One replication, eight attitude items, 518 matched respondents; the "12 times" is Cohen's d 0.07 against 0.87 and 0.88 for general models. Toluna reports an 81% average match rate between its synthetic personas and human respondents on ad tests (more than 4,000 validations, up to 92% in some countries) but publishes no sample sizes or raw data [V, vendor]. https://tolunacorporate.com/our-solutions/creative-and-campaigns/creative-pretest-instant/
AAPOR Task Force, "Responsible AI Integration in Survey Research", May 2026 [V]. https://aapor.org/wp-content/uploads/2026/05/Responsible-AI-Integration-In-Survey-Research.pdf ; ICC/ESOMAR International Code, Jul 2025 revision [V]; Insights Association, Courtright, 15 Jun 2026 [V]; MRS Delphi report, 16 May 2024 (cited via Research Live; no figures from it are used).
Li, Castelo, Katona, Sarvary, "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis", Marketing Science 43(2), 2024 [V abstract]. Agreement above 75% on perceptual maps. Separately: Arora, Chakraborty, Nishimura, "AI-Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators", Journal of Marketing 89(2), 43-70, 2025 [V abstract]: synthetic respondents got the direction and valence of quantitative answers right, and human-LLM hybrids beat either alone. https://doi.org/10.1177/00222429241276529 Kamruzzaman et al., EMNLP 2024 (arXiv 2406.13997) [V abstract]. https://arxiv.org/abs/2406.13997
Left out as unverified: Dillion et al. 2023 (r = 0.95 on moral judgements, secondary only); Evidenza "over 75% correlation" (vendor, no method); Nature Human Behaviour 74-country study of 13 Oct 2026 (headline statistics inconsistent across aggregators).