Your AI visibility score moved three points. Here are four honest answers to whether it means anything.

Your dashboard says AI visibility went from 30% to 33% this month. You’ve got 50 prompts running daily on one engine, so 1,500 answers a month. Somebody is about to put that three-point gain in a slide. So is it real? We can give you four answers, and all four are correct.

Those are our calculations, with a between-prompt correlation of 0.5 and the drift figure as assumptions we’ll explain below. The basic math isn’t ours. Gumshoe published the margin formula two weeks after the SparkToro study raised the question [3], Cloro has the sample sizes [12], and Switas has the design effect. [13] What’s missing from most of that is the first column.

Two questions wearing one number

SparkToro and Gumshoe had volunteers run prompts about 2,960 times and found that lists almost never repeated. [1] That sets up the wrong fear: that the score is random. Graphite then tested whether runs of one prompt behave as independent draws, and they do, with a variance ratio of 1.02. [8] So a single prompt’s rate over a short window follows the textbook.

The trouble starts when you ask about anything wider, because every prompt has its own rate. You may be near 100% on one and near zero on another, and 30 runs of a prompt that never names you are 30 copies of one fact. So there are two different claims:

• This list. Only run noise counts. Margins are smaller than the textbook says, because prompts where you always or never appear contribute none.
• Our category. The list is itself a sample of what people might ask, and the prompts become your real sample size. Volunteers writing their own prompts for one intent reached a semantic similarity of just 0.081, so that population is enormous. [1]

Vendors usually quote the first margin and let you hear the second. Evertune’s example gives ±1 point on 100 prompts run 100 times. [6] That’s right for those 100 prompts. As a statement about the brand in its category, by our arithmetic it’s about ±5 at a correlation of 0.3 and ±6 at 0.5.

That also explains why vendors seem to contradict each other. Profound tested 753 prompts and found once a day lands within about 2 points of ten times a day. [4] Meikai ran 250,000 responses over a frozen list of 500 and got the same answer. [5] Evertune says tight margins arrive at 100 repetitions. [19] They’re measuring different things: a pooled portfolio against a single prompt. All of them sell the cadence they recommend, and all can be correct. You have to know which you’re buying.

These assume 30% visibility, one run a day, a correlation of 0.5, and no drift. That correlation is ours. Nobody publishes it, so we derived two anchors from vendor tables: about 0.33 from Profound’s standard errors, and 0.86 to 0.91 from Momentic’s finding that same-day repeats disagree only 4.5% of the time. [4][11] Your vendor holds the raw data and could compute yours. Ask them to.

Watch the words “per engine,” too. A plan selling 4,500 answers a month is often 50 prompts across three models, each run once a day. [14][7] That’s the 50 row, not a 4,500 row. Each slice you take divides your prompts again: a dashboard that looks fine in total will show ±5 on a 10-prompt topic, and ±23 on one prompt.

Two ways to fool yourself with a clean test

Change the list. In our simulation, with fresh prompts each month and nothing else moving, the textbook test flagged 62% of months as significant. Raising volume from 1,500 to 6,000 answers changed nothing. Swap prompts in and you’re comparing two different samples, so ask for a like-for-like cut.

Ignore the platform. Profound’s daily wobble in portfolio visibility was 2.19 points at one run a day and 1.95 at ten, an 11% cut, against a floor of 1.92. [4] That’s the platform really changing, not error, but you didn’t cause it. Whatever your budget, the smallest change you can detect is about four times the platform’s month-to-month drift. Nobody publishes that figure, so we assumed one point: at that level, 5 points is detectable and 3 points never is. We tested this on real data too. Split the study’s public answers into the earlier and later half of a roughly week-long window, with no intervention at all, and 13.1% of brands look significantly different, against 4.4% for a random split. [2] Early and late volunteers may differ, so that mixes drift with who answered, but it shows the direction.

So a significant move still doesn’t mean your work did it. You need a control: competitors on the same list, or a group of prompts you held out and didn’t act on.

What survives

Citation share deserves extra suspicion. Citations inside one answer are a joint draw, with published design effects from 2.1 on SearchGPT to 4.3 on Perplexity, so counting citations overstates your sample several times. [10] Sielinski’s example has Tom’s Guide at 9.5% and Runner’s World at 6.0% on 200 queries, a gap well inside the interval. [9] He co-founded a firm selling this, so note the interest, but the arithmetic checks.

Position needs more care than the slogan. SparkToro’s post calls ranking “baloney.” [1] We rebuilt the study’s public data, 2,958 answers with every brand total matching the site, and split each prompt and model into two independent halves. At 30 runs a half, top-3 share correlated 0.91 between halves, against 0.87 for inclusion and 0.78 for mean rank. [2] So what’s dead is narrower: single-run rank and whole-list order. A brand with more than 50% inclusion sat at its own usual rank in only about half of its appearances. Report inclusion as the default, add top-3 share for brands named often, and never report a rank from one run.

The headline claim also has a soft edge. Fewer than one in 100 pairs of lists match “no matter the question,” says the post. On the public data, that’s true at the median, but 14 of 36 prompt and model combinations reach 1% or more, and one hits 23%. In tight categories the lottery framing overshoots. Top brands are stable: Smartsites appeared in 85 of 95 Google answers in its category, an interval of about 82% to 94%. [1]

Then there’s what none of this reaches. Graphite says tools run logged out or through APIs [8], and it found API and logged-out answers differ beyond noise, with a similarity of 0.48 across channels against 0.70 to 0.76 within them. In Profound’s test, synthetic personal context roughly halved list overlap. [15] So a clean number can still describe nobody you sell to. Our piece on ChatGPT’s search behavior covers why logged-out reads differ.

One tool’s methodology page says, “A CMO does not want a confidence interval.” [18] Maybe not. But a CMO does want to avoid telling the board that a three-point gain came from the content team, and then watching it vanish next month. Print the denominator, the band, and the control next to the number. If the move still clears all three, you’ve got something worth a slide.

Sources

1. SparkToro · New research: AIs are highly inconsistent when recommending brands or productsvolunteer survey, late 2025, co-run with Gumshoe, a tracking vendor
2. Gumshoe and SparkToro · Public study results2,958 answers; the splits and repeat rates here are our own reanalysis
3. Gumshoe · How much data do you need to measure AI visibility with confidence?the published margin formula, which treats observations as independent
4. Profound · Is once a day enough?753 prompts on 7 platforms, June 2026, from a vendor defending its own cadence
5. Meikai · How often should you run AI visibility prompts?250,000 responses over a frozen list of 500 prompts, July 2026, vendor
6. Evertune · How Evertune measures AI visibilitypublished margins of error that assume independent answers, vendor
7. Peec AI · Getting started, part 1: Understanding promptsprompts run once every 24 hours, described as building toward statistically meaningful insights
8. Graphite · Demystifying randomness in AIindependence tests, API against logged-out answers, and position error, from an agency with public data
9. Sielinski · Quantifying uncertainty in AI visibilitypreprint, and the author co-founded IQRush, which sells this
10. Sielinski · From stochastic to stable: rank stability in AI searchpreprint on citation clustering and design effects by platform
11. Momentic · How to build a prompt set worth tracking29,344 repeat groups, July to August 2026, from an agency tool
12. Cloro · AI visibility sample size: how many prompts and runsthe two-proportion arithmetic, from a vendor
13. Switas · Run an AI visibility panel that can tell a real change from model noisethe cluster design effect already published by a consultancy
14. Peec AI · Pricingprompts and monthly answers per plan
15. Profound · How does personal context change LLM responses?synthetic profiles, October 2026, vendor
16. Ahrefs · Brand Radar methodologymonthly refresh and a 90-day reporting window
17. Semrush · Where does the data in Semrush’s AI Visibility Toolkit come from?how prompts are captured and how the composite score is built
18. LLM Pulse · How to measure AI visibility without fooling yourselfa vendor stating that its comparison is descriptive and not a significance test
19. Evertune · How many repetitions of a single prompt does it take to understand your AI visibility?10,700 ChatGPT prompts, July 2026, from a vendor selling deep sampling

MS

Editor-in-chief

Mikołaj Salecki

Writes about media, tech, and AI business for people who actually run digital. Former agency lead. Skeptic of frameworks that read better than they perform.

LinkedIn ↗Facebook ↗salecki.digital ↗

More articles →