Making interviews countable
Filip Schouwenaars – 9 min read
Conveo runs AI-moderated qualitative research interviews. The moderator can run many conversations in parallel, so a single study regularly collects several hundred interviews within days, and sometimes more. Analyzing such vast amounts of data is not straightforward.
This post describes the part of our pipeline that makes that volume usable: topic encoding. It turns unstructured transcripts into topics and values that can be counted, split and queried, and every number keeps a link to the words behind it. We’ll go through how we decide what to count, how each interview gets coded, how the codes become charts, how the system tidies itself up during fieldwork, and how it all feeds an analysis agent.
This post also doubles as an experiment. I wanted to see how far Claude Opus 5.5 gets with explanatory animations, as I heard good things about it.
Throughout this post, we'll work through a hypothetical example: a meal-kit company that wants to know why subscribers cancel, and what would win them back.
Why not ask a model to read everything?
The obvious but naive approach is to put all transcripts into one prompt and ask a model for the main themes and how often each one comes up, but this breaks down quickly. Four hundred transcripts add up to millions of tokens, more than fits in a context window, and models get noticeably worse at recalling details long before they reach that limit. Every question would re-read the whole corpus, which is slow and expensive, and every new interview can invalidate past results. The numbers aren’t really counts either. A model that has skimmed a few million tokens estimates how often something came up, and the estimate moves between runs. And when it says 40% of participants mentioned price, there’s no way to see which participants, or what they said.
We therefore split the problem in two. First, decide what to count: a set of topics, each with its own list of possible values. Then code every interview against those topics, and store the results as rows. Counting becomes a database query. The judgment moves into many small decisions that can each be checked, because each one is tied to a specific participant answer. Market researchers will recognize this as coding open-ended answers against a code frame. We have automated both the code frame and the coding, with the ability to control every step of the pipeline. This makes all counts exact, repeatable and traceable.
Deciding what to count
When a study launches, a model reads the study design: the research briefing, the research objectives and the discussion guide. For each question in the guide, it proposes the topics worth coding. A topic has a name, a short definition that tells the coder what belongs in it, and a set of starting values. In our example, the question “What made you decide to cancel?” becomes a topic called Reasons for cancelling, with values like Price, Recipe fatigue and Delivery windows.
The model also makes two decisions per topic that shape how it gets coded later. The first is whether the list of values is fixed or open. Likelihood to resubscribe has a known scale, so its list is fixed. Reasons for cancelling can’t be listed in advance, because participants will name reasons nobody anticipated, so it starts with likely values and lets the coder add new ones. The second decision is whether a participant can land on one value or several. A participant has one likelihood to resubscribe, but can give several reasons for cancelling.
When an earlier study in the same organization asked the same question, the model can reuse that study’s topic, definition and values included, so results stay comparable from one study to the next.
Coding every interview
Coding starts as soon as an interview ends. Each topic is coded separately for each interview. For a topic that belongs to a question, the coder only sees the part of the conversation where that question was discussed, which keeps it focused. It returns the values that apply and, for each value, the participant messages that support it. Only the participant’s own messages count as evidence. The moderator’s turns are there for context.
If a participant says something that matches none of the existing values and the topic’s list is open, the coder adds a new value. If the list is fixed, it has to pick from the list or return nothing. Values are always written in the study’s reporting language, whatever language the interview was in, so a study that runs in eight countries ends up with one set of values. Because topics are coded independently, a failed call affects one topic in one interview, and a single topic can be re-coded without touching the others.
Counting and splitting
Once interviews are coded, counting becomes a query. The unit is the interview, so a participant who brings up price three times counts once for Price. Every code sits next to everything else we coded about that interview and know about the participant, so any chart can be split by anything else: an attribute such as age or country, or another topic. Splitting Reasons for cancelling by Likelihood to resubscribe shows whether people who left over price are more open to coming back than people who got bored of the recipes. Every bar also opens the interviews and messages behind it, so we maintain traceability to the quotes that produced it.
Keeping the topics clean
A set of topics generated before fieldwork is an informed hypothesis, but when actual interviews have happened, some topics may turn out to be rare. Two topics could turn out to measure the same thing. Open lists can collect near-duplicate values like Price and Too expensive. And participants also bring up themes nobody planned for, which is especially valuable to detect emerging trends. That's why the pipeline revisits the topics, once halfway through fieldwork and again when the study wraps up. Each quality pass does four things.
- It merges topics that measure the same thing. One model call proposes merges and a second call checks each pair. Existing codes move to the surviving topic, so nothing needs to be re-coded.
- It adds topics for themes that emerged in the interviews, found by reading the per-interview summaries, and then codes every completed interview for them.
- It hides topics that almost nobody mentions. Hidden topics are never deleted.
- It merges near-duplicate values within a topic.
Researchers can still change anything. They can add topics, rewrite a topic’s definition, which is the instruction the coder works from, and re-run the coding over all interviews, keeping their manual corrections if they want to. In practice they rarely need to. Customers left the generated topics as they were in about 9 out of 10 studies.
Answering complex queries
Bar charts of coded topics and breakdowns by attributes and other topics are useful, but our clients often require more powerful ways to slice and dice the data. For that, Conveo features Talk to your data, an agent that answers any questions about a study through a chat interface. Every chat gets its own sandboxed environment in which everything we know about the study is queryable: the topics, values and codes, the participant attributes, and the transcripts themselves.
The agent answers by querying that structured dataset, and returns charts with quotes pulled from the transcripts. Because the codes are structured, a question like “among people who left over price, what would win them back, by country?” gets turned into a precise query, with the result coming straight from the coded data, again linking back to the exact quotes where it got coded from.
The same structure works across studies. After topics are generated for a new study, a model matches each one against the topics already shared across the organization’s studies. When a chat covers several studies, the agent can compare and pool linked topics, so a study in Q1 can easily be related to a similar study in Q3.
The tip of the iceberg
Topic encoding converts what participants said into something countable, but is only one stage of a longer pipeline that makes sense of these topics and builds a bigger picture: summaries per question and per research objective, insights that pull evidence together across topics, personas, and the reports and quote reels that put the findings in front of a team. There's enough there for several follow-on posts!
Also for topic encoding itself there's a lot we haven't touched on here: eval suites for generation, coding and the quality passes, guardrails for the edge cases that real studies produce, and the plumbing that keeps thousands of small coding jobs cheap, fast and recoverable. We're constantly working to measure and enhance this pipeline, as it's the foundation of all insights surfaced across Conveo.
How I wrote this post
I started by drafting the outline of this post: what each section should cover, how the paragraphs should be organized, and specifics on how the animations should look and behave. I handed that brief to Claude Opus 5.5 in Claude Code, rooted in our codebase. Before writing anything, I asked it to check every claim in my outline against the code and our docs, which sharpened some details. Claude then wrote the text and all the animations. Aside from one small miss, the animations were correct on the first pass, quite impressive! I finally did a thorough manual pass through all paragraphs, so that the post speaks my language.
