How do you sample AI answers often enough to separate signal from noise?
Treat every AI answer as one yes-or-no draw and size the sample from the binomial formula: a brand mention rate near 50% needs 97 answers for a 95% interval of ±10 points and 385 for ±5 points, and a rate near 10% needs
Treat every AI answer as one yes-or-no draw and size the sample from the binomial formula: a brand mention rate near 50% needs 97 answers for a 95% interval of ±10 points and 385 for ±5 points, and a rate near 10% needs 35 and 139. In our monitoring of four AI engines over four weeks, a week of daily runs on one prompt missed the four-week rate by more than 10 points in 44% of cases, so one prompt sampled daily is not enough on its own.
- Daily runs behave like independent draws. The lag-1 autocorrelation of "brand named" between consecutive days was 0.07. Yesterday's answer tells you almost nothing about today's.
- Weeks drift more than draws predict. Weekly mention rates varied about 1.6 times as much as the binomial model allows, so add about 60% to the formula's run count if your runs fall in one week.
- A weekly number needs a prompt set, not one prompt. Seven daily runs of one prompt put 95% of weekly estimates within 30 points of the four-week rate. A set of 50 prompts run daily gives 350 answers a week per engine, enough for about ±5 points at a 25% rate.
To sample AI answers for a mention rate you can trust:
- Fix the prompt text, engine, country and brand list. Change one of them and you start a new series.
- Choose the precision you need: ±10 points is enough to rank brands, ±5 points to report a change of 10 points.
- Compute the runs with n = 1.96² à p à (1 - p) / E², with p your expected mention rate (use 0.5 if you do not know it) and E the margin.
- Multiply n by about 1.6 when the runs come from a single week, for the drift between weeks.
- Reach n by pooling: more prompts in the set, more engines, more days. One prompt run once a day gives 7 answers a week.
- Report each rate with its interval, and call a change only when it passes a two-proportion test.
How do you sample AI answers often enough to separate signal from noise?
You count answers, not days. A mention rate is the share of answers that name the brand, and each answer either names it or does not. That is a binomial proportion, and its 95% interval has a half-width of about 1.96 Ć ā(p(1 - p) / n). Solve for n and you get the number of answers you need for a margin E:
n = 1.96² à p à (1 - p) / E²
| Mention rate p | Runs for ±10 points | Runs for ±5 points |
|---|---|---|
| 0.1 | 35 | 139 |
| 0.3 | 81 | 323 |
| 0.5 | 97 | 385 |
Halving the margin multiplies the runs by four. The worst case is p = 0.5, which is why 0.5 is the safe guess for a brand you have not measured yet.
The formula in Python:
import math
def runs_needed(p, margin, z=1.96):
"""Answers needed for a 95% interval of +/- margin around a mention rate p."""
return math.ceil(z * z * p * (1 - p) / margin ** 2)
for p in (0.1, 0.3, 0.5):
print(p, runs_needed(p, 0.10), runs_needed(p, 0.05))
# 0.1 35 139
# 0.3 81 323
# 0.5 97 385
The formula assumes each answer is an independent draw from the same rate. The next two sections test both halves of that assumption on real answers.
Are AI answers on consecutive days independent draws?
Almost. We ran a fixed set of prompts once a day on Google AI Overview, Google AI Mode, ChatGPT and Gemini from October 17 to November 13, 2026, and recorded which tracked brands each answer named. For every prompt, engine and brand that was named at least once, we took the 28 daily outcomes as one series.
The correlation between "named today" and "named tomorrow", measured inside each series, was 0.07 overall and 0.05 to 0.11 per engine. An engine that names a brand today is barely more likely to name it tomorrow than its average rate says. So for sample-size purposes, a run on a new day is close to a fresh draw.
Most of these series sit far from 0 and 1. The median four-week mention rate was 11%, with a middle half from 4% to 33%. Only 3.2% of the brand, prompt and engine combinations were named in every run. At the median rate of 11%, the formula asks for 38 runs for ±10 points and 152 for ±5.
How far off is a weekly rate from 1, 3 or 7 samples?
We simulated a tracker that samples each prompt k days a week. For each series and each of the four weeks, we drew k days at random from that week, took the share that named the brand, and compared it with the series' four-week rate. The table covers series that were named on some days and not on others, the case where sampling matters:
| Samples per week | Typical error (RMSE) | Binomial prediction | Within ±10 points | Within ±5 points | 95% of errors below |
|---|---|---|---|---|---|
| 1 | 34 points | 34 points | 44% | 31% | 82 points |
| 3 | 20 points | 20 points | 48% | 33% | 43 points |
| 7 | 14 points | 13 points | 56% | 37% | 30 points |
At one and three samples a week, the error matches the binomial prediction. At seven, it is about 10% larger, because the weeks themselves differ. Over all series, the spread of weekly rates around the four-week rate was 1.58 times the binomial variance (1.5 to 1.7 per engine). That is drift, slow changes in what the engine names, on top of the draw-to-draw noise.
Two practical results follow. First, a weekly number for one prompt on one engine is noise at any daily cadence: with seven runs, 95% of weekly estimates fall within 30 points of the four-week rate, and only 56% within 10. Second, the formula's run counts are a floor. If the runs come from one week, multiply by about 1.6: a rate near 50% then needs about 155 runs for ±10 points and about 615 for ±5.
Does pooling prompts fix it?
For a rate over a prompt set, yes. The formula counts answers, and a set of prompts multiplies the answers you get per day. A set of 50 prompts run once a day on one engine gives 350 answers a week. At a 25% mention rate that is a 95% interval of about ±4.5 points, or about ±5.7 points after the 1.6 drift factor. Add a second engine and the weekly count doubles.
Pooling changes the question. A rate over 50 prompts is the brand's visibility across those buyer questions, not on any one of them. That is usually the number a team wants to track. For a single prompt that matters on its own, such as the one your product page targets, the only route to a tight interval is more runs of that prompt: with runsPerDay at 4, one prompt gives 28 answers a week and reaches the 97 runs of the ±10 case at p = 0.5 in about three and a half weeks.
We did not run several runs a day in this data set, so whether runs in the same day are as independent as runs on different days is not measured here.
How do you know a change is real?
Compare two periods with a two-proportion test, not two point estimates. The runs you need in each period to detect a move from p1 to p2 with 95% confidence and 80% power are:
n = (1.96 + 0.84)² à (p1(1 - p1) + p2(1 - p2)) / (p1 - p2)²
A move from 30% to 40% needs about 353 answers in each period. A move from 30% to 50% needs about 91. Small changes in a mention rate need large samples, and a dashboard line that moves 5 points week over week on a few dozen answers per point is showing noise.
import math
def runs_per_period(p1, p2, z_a=1.96, z_b=0.84):
"""Answers per period to detect p1 -> p2 (95% confidence, 80% power)."""
return math.ceil((z_a + z_b) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2) ** 2)
print(runs_per_period(0.3, 0.4), runs_per_period(0.3, 0.5))
# 353 91
How was this checked?
- Data. Our own monitoring of a fixed set of prompts about search, SERP and AI-tracking APIs, run once a day per engine through logged-out sessions, October 17 to November 13, 2026. Bing AI Mode joined the set on November 7 and is left out, because it has fewer than 21 days.
- Brand matching. A deterministic, word-bounded match against a fixed list of vendor names, on the answer text with link anchors removed. A citation link is not counted as a mention.
- Series. One series per prompt, engine and brand named at least once in the window, with at least 21 days that had an answer. The four-week rate includes the week being estimated, which makes the errors above slightly smaller than they would be against an independent target.
- Simulation. 200 random draws per series and week for 1 and 3 samples, seed 42. Seven samples use the whole week.
- Limits. Each prompt ran once a day, so this data cannot separate run-to-run noise inside one day from day-to-day change. Two runs in the same hour can be more alike than two runs on different days. The prompts are in one market, API buyers, and other markets can drift faster or slower.
This post is published by cloro, which sells an API that returns AI engine answers as structured data. The answers above came through it. A longer treatment of prompt counts and periods is in the AI visibility sample size guide. To run the same sampling on your own prompts, the open-source geo-tracker has a runsPerDay setting from 1 to 24 and stores every answer in your own Postgres.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes ā full credit and traffic to the original publisher.