Synthetic vs Human Research Evaluation

Over the past year, I've been circling a question that feels important for where design and research are heading: can synthetic interviews give us something useful in the early stages of understanding a problem space? Not as a gimmick, and not as a replacement for real conversations, but as a practical part of shaping hypotheses.

Recently, I had the right conditions to try this properly. I was already running research with real participants, and I had enough background material to build credible personas. So I created a parallel synthetic track, running the same discussion guide through my own setup, transcript by transcript. Two studies side by side—one human, one simulated.

I built my own rig rather than using one of the platforms that have emerged in this space. I wanted to get my fingers into the soil—understand the pitfalls and benefits directly—especially since design is such a fluid, problem-fitting exercise. The value of a flexible setup that can adapt to different test types felt higher than convenience, so long as I could prove it generates valuable input.

This past weekend I cleaned the data and compared the two sets, building a Python-generated report looking at thematic coverage, emotional tone, specificity, and linguistic markers worth paying attention to. The point wasn't to prove anything definitive, but to see whether there's enough correlation to argue there's value here for designers who want to move faster without cutting corners.

I went in with a strong view: synthetic research isn't a replacement for talking to real people, but it might widen the aperture of what we already think we know. There's always that early stage where the team operates on half-formed assumptions about motivations, barriers, and emotional patterns. We chip away at those a few interviews at a time, refining questions as we go. What I wanted to test was whether synthetic interviews, grounded in good persona research, could accelerate that exploration—surfacing themes in advance, pressure-testing hypotheses so the real qualitative work becomes more focused.

What stands out is that synthetic interviews track the core thematic structure surprisingly closely, but with a different texture. They're confident, fluent, tidy. Real people are contradictory, uncertain, occasionally messy. So the role I see emerging isn't synthetic versus human, but layered. Start with a hypothesis, build personas from what you know, run a synthetic round to see where the cracks form, then sharpen your questions for real conversations. That feels closer to quantitative-before-qualitative thinking, except through narrative instead of numbers.

There's also the possibility of using synthetic interviews to simulate groups that are harder to recruit, creating a proxy view before spending the time and money finding them. If that works, we might treat insight as existing in different proportions—fully human, blended, or fully synthetic—each with its own strengths and blind spots.

I don't think synthetic interviews add new human insight on their own. What they could do is increase the quality of the early stage by making real interviews smarter. And perhaps the interesting question isn't "can synthetic replace human?" But "what mix gets us to better design decisions sooner?"