← Blog

Why We're Having the Wrong Debate About GEO Tools

What polling research can teach us about AI visibility, and why good research design matters more than prompt volume.

In this article
A ballot box with voting slips beside a questionnaire and charts: counting votes and polling research as complementary perspectives.

Imagine someone dismissing polling research as useless: people give different answers to the same question. So, the argument goes, we can't measure their attitudes reliably.

Even the same person doesn't always give the same answer. A local election can bring different issues and priorities into focus than a national election. What someone prefers today may look different two weeks from now. Even the precise wording of a question can influence the answer we get.

Survey researchers have understood all of this for decades. That's why they have established methods for selecting samples, testing questions, examining differences in responses, and estimating statistical uncertainty.

Variation doesn't make answers unmeasurable. It means we need appropriate methods to study them.

We hear a very similar argument about Generative Engine Optimization (GEO): AI systems give different answers to the same prompt. Their responses are probabilistic. Therefore, the argument goes, those responses can't be measured rigorously. Then come the other objections: synthetic prompts are artificial, personas are made up, 100 prompts aren't enough, and different GEO tools contradict each other anyway.

Some of these objections do expose methodological weaknesses. Others ask a method to deliver something it was never designed to deliver. The real problem is often that we judge every GEO analysis by the standards of traditional SEO rank tracking.

1. Vote Counting or Polling Research?

SEO tracking works much like vote counting: it records observable rankings and search result features for a defined set of keywords. The measurement follows a largely deterministic logic: under the same conditions, it should produce the same result.

I'm not going to get into whether those keyword sets are representative, or how much of the measured search volume comes from people rather than bots. SEO tools have their own methodological problems, after all.

GEO analysis has more in common with the polling research that accompanies an election. It adds a probabilistic perspective: not every survey produces the same result. What matters is whether stable patterns emerge across enough observations. GEO analyses can measure, for example, how often a brand appears in AI responses. They can also examine different user intents, explore associations, track developments, or use experiments to test hypotheses about factors that influence responses.

SEO TRACKING
Closer to vote counting

GEO ANALYSIS
Closer to market research

Observe rankings
Count positions
Compare trends over time

Where do I rank?

Define questions
Build samples
Study response patterns

How often, and why?

Both approaches can produce very good results. The difference lies less in their potential accuracy than in the range and complexity of the research questions they address.

SEO is primarily a measurement problem. GEO is also a research problem.

2. Different Answers? Measure the Distribution.

An AI recommends a hotel in one response and leaves it out of the next. If you expect a stable ranking position, that looks like an error. If you study response behavior empirically, you first see it as a characteristic of the system you're investigating.

Suppose a model mentions a brand 63 times across 100 repetitions of the same question. The observed mention rate is 63 percent. That doesn't establish a brand's share of recommendations across real-world AI use, but it is a finding we can interpret for this prompt, this system, and the documented conditions. Repeated measurements reveal response variability. Combined with careful sampling design, they also help us assess how robust comparisons across prompts and models really are.

We need to separate two sources of uncertainty: repeating the same prompt reveals variability in model responses. It doesn't tell us whether that prompt accurately represents real-world demand. One hundred identical queries are not one hundred different user questions.

Different results from two GEO tools aren't automatically a contradiction, either. If they use different models, questions, regions, or definitions of a “recommendation,” they may be measuring different things. A meaningful comparison therefore starts by disclosing the measurement conditions.

3. The Real Challenge Is Research Design

Synthetic prompts are often criticized because nobody can prove that real users ask those exact questions. Yet we often have little choice but to develop questions ourselves. In traditional search, many people use identical search phrases. AI prompts are often individual, one-off formulations. A real user prompt is therefore not automatically more representative than a synthetic one.

Traditional market researchers write their own questionnaires, too. What matters isn't whether a question was constructed artificially, but whether it can produce the insight we're looking for.

For an exploratory question like “In which decision-making situations is a food delivery service recommended?”, well-chosen synthetic prompts can be extremely useful. What matters is less the exact wording than the intent the AI system recognizes in the question. This is the foundation of Intent First Tracking: rather than monitoring individual prompt formulations in isolation, we systematically study the user intents behind them and the responses they elicit.

That isn't enough, however, to support a claim like “30 percent of all real AI queries about food delivery services mention a particular provider.” We would also need to know how frequently the different user intents occur in actual AI use.

The same applies to personas. Target audiences, social groups, and segments are established tools in market and social research. In GEO, we can examine how recommendations differ for budget-conscious users, families, or people with specific dietary preferences. These personas help us systematically represent different decision-making situations. But they aren't automatically representative of real user groups. And when a language model plays a persona, that isn't a survey of a real person.

When Prompt Selection Creates GEO Success

Brand research offers a particularly clear example. It distinguishes between unaided brand awareness and aided brand awareness.

“What food delivery services come to mind?” measures unaided brand awareness. “Are you familiar with DoorDash or Uber Eats?” measures aided brand awareness. Both approaches provide valuable information, but they measure different things. No serious researcher would combine the two and report a single unaided brand awareness score.

This distinction matters just as much in GEO. Yet open-ended market questions and prompts that already name specific brands can end up in the same visibility score.

The risk is especially high when prompt lists are generated automatically from traditional SEO keyword lists or sources such as Google's “People Also Ask.” Open-ended market questions, brand comparisons, and questions about individual providers can get mixed together. If all the responses then go into a single visibility score, the result may depend more on the composition of the prompt list than on a brand's actual visibility.

If you put the brand in the question, don't be surprised when it shows up in the answer.

Three Prompt Types Compared

Market

“Which food delivery services would you recommend?”

Quantitative analysis: Brand mentions, omissions, and relative frequency of mentions among competitors.

Compare

“How do DoorDash and Uber Eats compare?”

Qualitative analysis: Arguments, advantages, disadvantages, and attributed characteristics.

Brand

“What are the pros and cons of DoorDash?”

Qualitative analysis: Brand associations, factual accuracy, and sentiment.

Market, Compare, and Brand describe different research objectives. Aided and unaided brand awareness describe a second dimension: is the brand being studied already named in the prompt? Compare prompts are often aided, while Market prompts aren't necessarily unaided.

A good GEO tool should let us capture, filter, and analyze both dimensions separately.

This becomes particularly problematic when comparing results over time. Gradually increase the proportion of Brand prompts, and measured brand visibility can rise significantly even though the AI system's response behavior hasn't changed at all.

This can happen unintentionally as a prompt set keeps expanding. But it can also be exploited deliberately. An unscrupulous agency could increase the proportion of brand-specific prompts and then sell the rising visibility score as proof that its GEO work is succeeding.

The metric goes up. That doesn't establish a real improvement in GEO performance.

That's why anyone commissioning GEO services or evaluating their results should understand the underlying methodology. What types of prompts are being used? What proportion does each type account for? How are the results weighted? And has the composition changed since the last measurement?

For meaningful comparisons over time, each group must remain separately analyzable. An aggregate metric needs a consistent, justified composition or transparent weighting. A change in the measurement isn't automatically a change in what we're measuring.

The sheer number of prompts is often overrated, too. A study with 10,000 poorly selected questions can be systematically biased. One hundred carefully developed questions, on the other hand, can yield valuable insights into a clearly defined research question.

Under the right conditions, more data reduces random uncertainty. It doesn't fix a bad research design.

4. GEO Can Do More Than Count Visibility

The most exciting possibilities of GEO analysis extend beyond a single visibility metric. Open-ended questioning of models can uncover associations and recurring patterns in their responses. Quantitative studies can then test how often those patterns occur under defined conditions. Experiments can investigate whether targeted changes to content have an observable effect.

Empirical researchers know this combination as a mixed-methods approach. First, we explore and formulate hypotheses. Then we test them using measurements designed differently. A qualitative study doesn't have to be representative. Its purpose may be precisely to uncover questions we haven't considered before.

An exploratory analysis of Haribo that I created using our internal Entity Decoder at GPT Insights shows what this can look like. The language model we examined strongly associates the brand with gummy bears, childhood, and nostalgia. It also reveals tensions: indulgence meets health concerns, and global brand recognition meets regional differences in recipes. Analyses like this don't give us definitive answers about why a model recommends a brand. But they show us which associations and connections deserve a closer look.

A single response reveals only part of what a language model associates with a brand. Repeated queries can show which associations are particularly stable and which appear only occasionally. Those less frequent responses can be especially interesting. They may reveal connections that a single query would leave hidden.

GEO analyses can do more than observe what a model says. They can also develop hypotheses and investigate the conditions under which AI search systems respond that way.

5. The Empirical Research Toolkit

The comparison with market research is more than a metaphor. Over decades, empirical researchers have developed methods for dealing with differing responses, sampling error, and uncertain results. Much of that can be applied to GEO.

Three principles matter most.

First: Know what you want to measure. A research question needs a clear definition. And terms such as visibility, mention, or recommendation must be defined in ways that allow us to measure them reliably. Researchers call this operationalization.

Second: Ask the right questions. Samples must fit the research question. Prompts should be tested for unintended steering, different user intents deliberately included, and assumptions about their weighting disclosed. Not every study has to be representative. But every study must explain which claims its findings can support.

Third: Scrutinize the results. Repeated measurements help capture variability. Independent studies and different methods can strengthen confidence in findings, while controlled experiments help test possible causes.

Total Survey Error is a useful concept here. It reminds us that random variability isn't the only source of error. A biased sample, unclear measurement definitions, or errors in analysis can also distort the results.

Ten thousand additional AI responses won't fix a systematically misselected sample.

A good GEO analysis therefore needs more than reliable measurement technology. It needs a research design that fits its research question.

A good GEO tool should make that possible. Configurability is a clear advantage over rigid automation here. The tool shouldn't dictate the research question. The research question should determine how the tool is used.

6. What Really Makes a Good GEO Tool

The toolkit makes one thing clear: a method isn't inherently good or bad. What matters is whether it fits the research question and supports the claim being made. That gives us six practical questions to ask.

Exploratory questioning of a model doesn't have to represent a real user market. A tool that reports a brand's overall share of AI recommendations, however, must explain how its prompt sample represents real-world AI use. This is the distinction we should use to assess quality.

Six questions for practice:

Research question: What specific claim should the study enable us to make?

Sample: Why were these particular prompts and contexts selected?

Operationalization: What counts as a mention, recommendation, or citation?

Reliability: How consistent are results across repeated measurements and assessments?

Validity: Do the method and metric support the claim being made?

Transparency: Are the conditions, limitations, and changes documented?

These requirements are standard practice in empirical research. Professional associations such as AAPOR and research institutions such as Pew Research Center have spent decades studying question effects, sampling problems, weighting, sources of error, and transparency. GEO can draw on that experience without treating surveys of people and experiments with AI as equivalent.

7. The Wrong Question for the GEO Community

Perhaps the debate would be less heated if we saw GEO analyses as a family of empirical research methods, rather than simply a new form of rank tracking. Not every study has to be representative. Not every study has to provide qualitative explanations. And not every investigation has to establish a causal effect.

But a study must deliver what it promises. And that's still a frequent problem in the GEO market. Some tools claim to know the search volume of AI prompts or infer brands' overall shares of AI recommendations from limited samples. Others present scraped user questions as a representative picture of AI use, even though those data capture only a limited slice of it. Often, it's unclear where the data come from, what they actually represent, and what their limitations are.

Skepticism about these promises is justified. But concluding that GEO analyses are inherently unreliable or even pointless would be throwing the baby out with the bathwater. Rather than dismissing every method alike, we should distinguish more carefully between the questions each can actually answer.

The polling analogy takes us a long way, but it has limits. Polling researchers survey real people from a definable population. In GEO, we study systems whose response behavior depends on the model version, context, and many other factors. We also often lack a reliable sampling frame for real user questions. Clever statistics don't make those limitations disappear.

That's precisely why we need more methodological clarity and fewer sweeping judgments. Asking which GEO method is best often means we're already asking the wrong question. First, we need to establish what we want to know. Then we can decide which method is suitable and how far its findings can take us.

Uncertainty isn't a reason to give up on research. It's a reason to do better research.

Hanns Kronenberg

Thoughts on search, AI and the web.

← Back to the blog