THINK BIG
GEO Measurement · AI Visibility
Asked AI Once and Called It? — AI Answers Are Probabilities, Not Rankings: How to Measure GEO Properly
AI answers change every time. Re-ask the same question minutes apart and roughly two-thirds of the cited sources will be different. To know whether AI mentions you, ask repeatedly, track for weeks, and look at ranges — not single results.
1. The problem: one screenshot proves nothing
“ChatGPT just recommended us!” “I asked three AIs and none of them mentioned us.” We hear both versions constantly. Behind each one is the same habit: ask once, treat it as the answer. That works for traditional search, where the same keyword returns the same rankings minutes apart. AI doesn’t work that way.
A team at the University of St. Gallen published a paper in April 2026 titled “Don’t Measure Once,” studying this problem directly. From January to March 2026, they asked the same sets of questions to four AI search services (ChatGPT, Perplexity, Google AI Mode, Gemini) every day across four industries — 8 questions per industry — tracking continuously for 45–46 days and re-asking the same question up to 10 times within the same day. (arXiv 2604.07585)
Key findings: day-to-day turnover is high. Cited sources on consecutive days overlap only 34–42% on average. Same-day re-asks show the same instability: re-asking minutes apart overlaps only 32–43% of cited sources — nearly the same as asking on different days. The researchers concluded the instability comes from how models generate text, not from external algorithm updates.
Brands are more stable than sources. Brand mentions overlap 45–59% day-over-day, more than citation sources. Some questions are very stable, some aren’t at all. Seeing just one or two questions reveals the personality of those questions, not your overall visibility.
AI doesn’t always search the web. ChatGPT answered 57.8% of questions without any citations — because it only triggers web search for certain queries, and relies on model memory for the rest.
One line from the paper’s abstract fits as this article’s headline: AI search visibility should be described as a distribution, not a single data point.
2. What the research says: how many times is enough?
The paper gives concrete thresholds. To estimate “the rate at which a brand is mentioned” with enough stability (standard error below 0.10), you need at least 7 repetitions per question per day; for citation sources, at least 8. For time, a single brand requires at least 10 consecutive days to bring the standard error below 0.10, and 24 days to drop below 0.05. The authors recommend using a 2–4 week rolling window. They also advise: set a separate baseline for each AI, don’t average across AI tools; use brand mention rate as the primary signal and citation URL tracking as a diagnostic.
Repeated questioning also detects AI fabrications. A 2024 Nature paper proposed: ask a model the same question multiple times; if the answers contradict each other, that’s a signal the model may be hallucinating. (Farquhar et al., Nature 2024) One answer is a sample; repetition reveals the real picture.
Technique effectiveness can only be demonstrated with rigorous measurement. A 2026 review of 45 GEO studies found that low source overlap and high run-to-run variability are pervasive in commercial audits; and no technique has been shown to improve discoverability stably, long-term, or across platforms. (arXiv 2607.14035) If your measurement is noisy, you can’t tell whether a technique worked or you got a lucky sample.
But don’t break the rules to measure. Google’s spam policies prohibit unauthorized automated queries to Google Search, including scraping for ranking data. The compliant path: use each AI’s official API, supplement with manual UI checks, and state which channel you used in every report.
Official data is now available. Bing Webmaster Tools launched an AI Performance report in public preview in February 2026, showing how often your pages are cited in Copilot and Bing AI summaries, which pages are cited, and how that changes over time. (Bing Webmaster Blog)
3. Three mistakes we made measuring our own AI visibility
Before launching our AI visibility service, we measured our own website first. We made three mistakes. All three are now written into our methodology.
Mistake 1: running one round and thinking we saw the truth. Our first baseline was one question, one ask, one round. We barely showed up anywhere and felt discouraged. Then we checked Google Search Console and saw several of our guide pages ranked on the first page for their target queries. After reading the St. Gallen study, we confirmed: one round is statistically insufficient for any conclusion, positive or negative.
Mistake 2: a dirty test environment pollutes results. On October 4, 2026, we ran a proxy test using an internal AI tool and found it had read notes stored on the test computer and incorporated them into answers. The results looked uncannily accurate — but that’s not what regular users see. OpenAI’s documentation notes that when memory is enabled, ChatGPT may use stored memories to rewrite search queries. Rule established: all measurements run in a clean environment — logged out, no conversation history, no memory, fresh session.
Mistake 3: treating “mentioned,” “cited,” and “described correctly” as the same thing. AI can mention your name without linking to your page. It can link to your page while describing you incorrectly. We experienced this firsthand — an AI stated with complete confidence that our company had nothing to do with AI. Now every run records four things separately: did AI search the web, did it mention us, did it cite our URL, did it describe us accurately.
Our revised methodology in brief:
| Item | Our current practice |
|---|---|
| Questions | Layered by intent: broad, contextual, conditional, brand comparison, factual; 3–5 phrasings per intent, including English versions |
| Lock questions | Freeze the question set once built; add but never edit, so comparisons are valid over time |
| Repetitions | Each question, each AI, multiple times per day with at least 10 minutes between runs |
| Duration | Consecutive days; report on a 2–4 week rolling window |
| Separation | Separate baseline for each AI; no cross-AI averages |
| Recording | Did it search; was brand mentioned; position in answer; was URL cited; was description accurate |
| Reporting | Every figure includes a 95% confidence interval; overlapping intervals between periods = “stable,” not “growth” |
| Channel | State API vs. UI for each measurement; no automated scraping of Google pages |
4. Six principles for checking AI visibility yourself
1. Ask the same question several different ways. Think like a customer: “Which [service] companies do you recommend in [city]?” “What’s good for a five-person company?” “How does A compare to B?”
2. Ask each question at least 5–7 times. Record how many of those times you were mentioned. Look at the rate, not just “yes or no.”
3. Use a clean environment. Log out or use a private window, disable memory, start a fresh session. Otherwise you’re seeing a personalized answer, not a representative one.
4. Track four things separately. Did it search the web; did it mention you; did it cite your URL; did it describe you accurately.
5. Track for at least two weeks before talking about trends. One-day fluctuations are mostly noise. Look at each AI separately — don’t average across platforms.
6. Don’t scrape Google. It violates Google’s policies. For large-scale measurement, use official APIs or work with a team that does it compliantly.
If you’re evaluating a GEO vendor, ask five questions: How many questions, how many phrasings? How many repetitions per question? How many consecutive days? API or UI? Do the numbers include confidence intervals? If they can’t answer, discount the report.
Bottom line: AI answers are probabilities, not rankings. One ask is luck. Doing it properly gives you data.
Honest numbers. Readable reports. Verifiable methods.
(The following reflects Think BIG’s own position and is separate from the neutral reporting above.)
Our AI visibility reports follow this methodology and lay the method open in every report:
One “denominator and method” page: how many questions, how many phrasings, how many repetitions per day, how many days, which AIs, which channel — all stated. You can re-run any query and verify.
Trigger rate listed separately: did AI search the web for this question? Questions AI doesn’t search require different interventions than ones it does search.
Description accuracy listed separately: not just whether AI mentioned you, but whether what it said was true. Inaccuracies are logged and tracked for correction.
Separate recommendations per AI: different AIs favor different sources, so recommendations are split by platform.
Tiered conclusions: each finding is labeled [count], [comparison], or [inference]; overlapping confidence intervals between periods = “stable,” not “growth.”
We don’t promise rankings or sales. We promise honest numbers, readable reports, and methods you can verify.
FAQ
- Q1: Why does the same AI question get different answers each time?
- AI models introduce randomness when generating answers, and the sources they cite change along with the answer. A 2026 University of St. Gallen study found that re-asking the same question minutes apart overlaps only about 32–43% of cited sources on average — roughly the same as asking on consecutive days. The instability comes from how models generate text, not from external algorithm changes.
- Q2: How many times do I need to ask AI the same question to get a reliable result?
- The same study estimates you need at least 7 repetitions per question per day for brand mention rates, or 8 for citation rates, to bring the standard error below 0.10. For tracking trends, you need at least 10 consecutive days. The authors recommend using a 2–4 week rolling window.
- Q3: Can I write a script to automatically query Google AI Mode for measurement?
- No. Google’s spam policies prohibit automated queries to Google Search without permission, including scraping for ranking data. The compliant approach is to use each AI’s official API alongside manual spot-checks, and to document which channel you used in your report.
- Q4: What’s the difference between AI “mentioning” me and “citing” me?
- A mention means your name appears in the answer. A citation means the answer links to your page as a source. They don’t always happen together — and even when both occur, the description of your company may still be inaccurate. Track them separately.
- Q5: Is there an official tool to see how often AI cites my website?
- Bing Webmaster Tools launched an AI Performance report in public preview in February 2026, showing how often your pages are cited in Copilot and Bing AI summaries. On the Google side, AI-driven traffic is rolled into the “Web” search type in Search Console and can’t be broken out separately.
This article combines industry reporting with Think BIG’s first-hand measurement experience. All cited research and official documents are linked; figures follow the original papers’ stated conditions, platforms, and time windows. Data about our own site is shown only as relative observations. AI tools are not named, and nothing here is directed at any company or product.