Synthetic user research means asking a language model to role-play your target customers and answer your questions. It is useful as rehearsal: sharpening interview questions, surfacing likely objections and testing how a pitch reads. It cannot tell you whether real people have the problem or will pay, so treat every answer as a hypothesis.
That is the whole position. The rest of this post is the evidence for it, a list of the jobs simulated respondents do well and badly, and a workflow that uses them for what they are good at without letting them stand in for customers.
What the research actually says
The optimistic case comes from Argyle and colleagues, published in Political Analysis in 2023. They conditioned GPT-3 on thousands of demographic backstories taken from real US survey participants and found the resulting "silicon samples" reproduced the response patterns of those subgroups in a nuanced way. Their conclusion was that a model with enough of this fidelity could be a useful proxy for some research questions.
The follow-up work is more cautious. Bisbee and colleagues, in the same journal in 2024, found ChatGPT's persona answers matched the averages in a major US election survey but varied much less than real people did, produced relationships between variables that often differed from the human data, and changed noticeably when the same prompt was rerun three months later. Their verdict was that this kind of synthetic data is not reliable enough for statistical inference.
Wang, Morgenstern and Dickerson, accepted at Nature Machine Intelligence, ran human studies with 3,200 participants and argued that the way current models are trained makes them likely to misportray and flatten demographic groups. Prompting a model with an identity tends to produce the stereotype of that identity rather than the range of people who hold it.
On the commercial side, a Harvard Business School study by Brand, Israeli and Ngwe found GPT's simulated shoppers behaved plausibly on everyday products: its estimated willingness to pay for fluoride toothpaste was $3.40 against $3.27 in an earlier human study. The authors still advised against replacing conversations with consumers, and noted the method is unproven for new or complex products. A startup idea is, almost by definition, a new product.
The closest test to what a founder would do comes from Nielsen Norman Group. They reran three studies they had already done with real participants, using synthetic users instead. In a study of online learning, a real participant said they had finished three of seven courses; the synthetic user said it had finished all of them. Real learners mostly avoided course discussion forums; synthetic learners said forums mattered to them. Feedback on a new concept was mostly positive and generic. NN/g's advice is to treat what synthetic users say as hypotheses that need testing.
Where simulated respondents earn their keep
Every job below has the same shape: the output is something you will check against reality, never something you will act on directly.
- Sharpening interview questions. Run your question list past a simulated panel and read the answers for vagueness. If every persona answers "it depends", the question is too abstract. If every persona answers yes, the question is probably leading. Rewriting before a real interview saves the conversations you can least afford to waste.
- Rehearsing objections. A persona told to be skeptical will raise the obvious objections: price, switching cost, trust, "we already use a spreadsheet". None of them are proven, but you should have an answer ready before a real buyer raises them.
- Running a pre-mortem. Ask the panel why they stopped using the product after a month. The answers are a list of failure modes to look for, not a prediction of which one will happen.
- Testing positioning copy. Show three versions of a one-line pitch and ask what each persona thinks the product does. Misreadings are useful even from a model, because a sentence that confuses a language model often confuses a busy person too.
- Getting oriented in an unfamiliar domain. If you are building for nurses or freight brokers and have never done either job, a simulated workday gives you vocabulary and a starting list of topics. NN/g found this desk-research use was where synthetic users were closest to real ones.
Where they mislead
These are the failure modes that matter for a founder deciding whether to build. Each one points the same way: toward an answer that is more positive, more uniform and more conventional than what you would hear from real customers.
| Failure mode | What it looks like | Why it happens |
|---|---|---|
| No real willingness to pay | Personas name a fair monthly price with confidence. | Nothing is at stake. A model cannot decide to spend money, so a stated price is a guess about what a person like this might say. |
| Sycophancy | Answers lean toward liking the idea, especially when the prompt makes it obvious you want them to. | Anthropic's research on sycophancy found that assistants trained on human preference tend to tell people what they want to hear. |
| Too little variety | Twelve respondents read like one opinion said twelve ways. | Bisbee et al. measured much lower variance than in real samples, and Wang et al. describe identity prompts flattening groups into a stereotype. |
| Borrowed opinions | Personas repeat what is commonly written online about a category. | A model's views come from its training text, which over-represents people who write publicly about a topic. |
| No discovery | Nothing in the transcript surprises you. | A model recombines problems that have already been described. The pain nobody has written about yet is the one a real interview can find and a simulation cannot. |
The last row is the one founders underestimate. The most valuable thing in a real interview is usually the problem you did not ask about.
A workflow that uses both
1. Write your questions about past behaviour
Ask what people did the last time the problem came up and what it cost them, not whether they would use your product. The customer interviews guide covers the question set in detail.
2. Run them past a simulated panel that includes a skeptic
Include at least one persona who is a poor fit or hard to convince. A panel of enthusiasts tells you nothing you did not already believe.
3. Rewrite the questions that got vague or uniform answers
This is the most reliable value you will get from the exercise. A question every persona answered the same way needs to be more specific.
4. Turn objections and themes into a hypothesis list
Write each as something a real interview can confirm or kill: "Buyers will not switch while their current tool is bundled with something they already pay for."
5. Talk to real people
Start with people already complaining about the problem in public. Our post on finding customer pain points shows where to look.
6. Compare the two
Where real customers disagreed with the simulated panel, write it down. That gap is information you could not have got any other way, and it is usually where the idea needs to change.
How the Truewick panel is built
Since we sell a simulated panel, here is exactly what it does, so you can judge its output against everything above.
You pick two to four seed personas, or start from a draft of three that the model is asked to write with at least one likely skeptic or poor fit. Each seed becomes three respondents that vary on age range, income, comfort with technology, skepticism and attitude to new products, for a panel of up to twelve simulated respondents. You ask up to five open-ended questions. When your pitch has a score, the drafted questions are aimed at its weakest dimensions, because those are the parts of the score real users can actually move.
Each respondent is interviewed in a separate model call and told to answer in character, not to be a cheerleader, and to say so when something would not work for them. A second step then reads every transcript and writes themes, objections, feature requests, a willingness-to-pay pattern, a verdict on each question and a recommended next step. Every interview is shown unedited alongside the summary, and the whole thing is labelled simulated throughout. A run costs two credits and takes about three minutes.
Everything in the failure-mode table applies to it. The willingness-to-pay figures in particular are what the personas said, which is exactly the number this post tells you not to trust. We show them because a price nobody on the panel would accept is still a useful warning, not because a price they would accept proves anything.