Measuring AI visibility means tracking how often your pages are cited in AI answers, across a stable set of prompts, run repeatedly, and compared against your own prior baseline. There is no Search Console equivalent covering all four engines. The programme below can be run by hand, needs no paid tooling, and produces numbers a leadership team can act on.
Set a baseline before you change anything
The most common failure in a measurement programme is starting to measure after the work has begun. A baseline recorded after the first round of changes is not a baseline, it is a midpoint with nothing to compare against.
Before any content is rewritten, any schema is added, or anyone posts anything anywhere, run the full prompt set once and record the results. That record is the only thing that lets you say, three months later, that citations went up. Without it, every improvement is an anecdote.
This matters most for teams inheriting a programme already underway. If no pre-work baseline exists, say so plainly in every report. A measurement programme that names its own gaps is more credible than one that quietly hopes nobody asks.
What the baseline record must contain
For each prompt, record the following at baseline and again at every run after it:
- The exact prompt text, copied verbatim.
- The engine tested.
- The date and time of the run.
- Whether your domain was cited, yes or no.
- If cited, which URL, and where in the response it appeared.
- If not cited, which domains were cited instead, up to five.
- The run number, since you will be running each prompt more than once.
One row per prompt per engine per run is enough. Resist adding a sentiment column, a position column or a quality score at this stage. Most of them come out again later, and the section on misleading metrics explains why.
What the baseline does not tell you: it does not tell you whether the answer that cited you recommended you, ignored you, or used you as the cautionary example. It does not tell you whether your prompt set reflects what buyers actually ask. Record it anyway. A limited baseline beats none.
Building a prompt set that holds up over time
The prompt set is the instrument. If the instrument changes, the measurements stop being comparable, and this is where most reports quietly go wrong.
How to choose prompts
Start with what buyers type when they are evaluating a purchase in your category. Not awareness questions, not brand questions. Evaluation questions: what someone asks once they have identified the problem and are deciding what to use. In order of reliability:
- Sales call recordings. The questions prospects ask before committing are the prompts, almost word for word.
- Support tickets. What customers ask after buying usually mirrors what prospects ask before.
- Search Console queries containing evaluation language: "vs", "alternative", "best for", "how does".
- Reddit threads in the subreddits Part 4 helped you find, specifically the posts asking for recommendations.
Avoid prompts broad enough to produce an encyclopaedia entry, and avoid prompts so narrow they can only resolve to you. The useful band is category-level evaluation questions: specific enough to have a defensible answer, open enough that several vendors could plausibly be named.
How many prompts you need
This is a practitioner convention rather than a published standard. A set of 30 to 50 prompts across three to five topic clusters gives enough coverage to detect real movement while staying small enough to run by hand. Under 20 and the results are too exposed to the variance described in the next section. Over 100 and consistency suffers, because sets that large tend to drift as people quietly add and drop items.
When you may change the set
Retire a prompt when the product or category it names no longer exists. Add prompts only at the start of a measurement period, labelled as new so they do not contaminate the trend. Never remove a prompt because the result is unflattering. A prompt that consistently shows a competitor cited and you absent is the most valuable row in the sheet. Deleting it hides the problem instead of fixing it.
Non-determinism: why a single run means nothing
This is the load-bearing idea in this part, and every other measurement decision follows from it.
AI engines do not return the same output twice for the same prompt. This is not an edge case, it is how the systems work. The model samples from a probability distribution over next tokens, and even with temperature pinned at zero the output can still move, because variance also comes from the serving layer: floating-point arithmetic, batch scheduling and hardware parallelism are not things a temperature setting controls.
What the research shows
The clearest measurement of this is "Non-Determinism of 'Deterministic' LLM Settings" (arXiv, August 2024), which ran five LLMs ten times each under settings intended to be deterministic. The authors report "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%", and conclude that "none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings".
A more recent preprint, "Quantifying non-deterministic drift in large language models" (arXiv, January 2026), ran repeated-prompt experiments on gpt-4o-mini and llama3.1-8b at temperatures of 0.0 and 0.7 and reports that nondeterminism persists at temperature 0.0, with different patterns by model size, deployment and prompt type. It is a single-author preprint rather than peer-reviewed work, so treat it as corroboration rather than proof.
The consequence for citation measurement is direct. Run a prompt once, record whether you appeared, and you have one sample from a distribution. That is not a citation rate. It is one draw.
How repeated sampling fixes it
Run each prompt at least five times per measurement period, spread across different times of day, recording every run separately. Then compute a presence rate for that prompt: runs in which your domain was cited, divided by total runs.
A prompt run five times that cites you three times has a presence rate of 60 percent. Next month the same prompt returns 40 percent. That is a signal. A single run that says yes one month and no the next is noise wearing a suit.
| Runs per prompt per period | What it produces | Use it |
|---|---|---|
| One | A single draw from a non-deterministic system, indistinguishable from noise | No |
| Three | Minimum viable signal, acceptable for internal weekly tracking | With caution |
| Five | A presence rate with enough samples to show real movement | Yes |
| Ten | High-confidence presence rate | Yes, for your key prompts |
Spread the runs across the day. Engines serve different model versions and different retrieval results at different times, and clustering every run into one session understates the variance you are trying to measure.
The metrics that hold up
Three metrics survive contact with a sceptical leadership team. Everything else is a proxy for one of them or a number that misleads.
Citation count against your own baseline
This is the primary metric. Count how many times your domain was cited across all prompts and all runs in the period, and compare it to the same count at baseline. Report it as a percentage change against your own prior performance, not as a raw number and not against anyone else's benchmark.
An illustrative example, with invented figures for a generic company: forty prompts across four engines, each run five times, is 800 runs per period. If the domain appeared in 120 runs at baseline and 180 now, citations are up 50 percent against baseline. That number depends on no third-party tool and no vendor-defined metric.
What it does not tell you: nothing about whether the citations were favourable. "Company X has poor support" is a citation. Read a sample of the responses you appear in each period and note the framing. That reading is qualitative rather than a metric, and it is the only way to know whether citation growth is good news.
Presence rate across the prompt set
For each prompt, take the fraction of runs citing you, then average across the set. Presence rate beats raw count for tracking, because it normalises for the number of runs: if you ran 800 this period and 600 last period because the set grew, the raw counts are not comparable and the rates are.
What it does not tell you: whether the prompt set reflects real buying questions. A presence rate of 80 percent on a set made entirely of your own brand name means nothing at all. The metric is only ever as good as the instrument.
Share of voice, as context
Record which other domains are cited alongside or instead of you, aggregate across the set, and calculate each domain's share of total citations.
Share of voice belongs in the report as competitive context, not as the headline. It tells you who gets cited when you do not, which is how you choose what to study next. It cannot tell you whether your own citation count is good, because the denominator moves for reasons that have nothing to do with you.
The metrics that mislead
These turn up in AI visibility reports constantly. Each has a plausible surface logic and a fatal flaw underneath.
Average position
Some tools report the average position your domain occupies among an answer's citations. The flaw is that AI responses have no stable position structure. A response citing three sources in a numbered list is structurally different from one weaving citations through prose, and "first in the list" and "first inline mention" are not the same thing. Averaging across both produces a number with no referent.
Single-run screenshots
A screenshot showing your domain in one response proves your domain appeared once, on one engine, for one prompt, on one day. Given everything above, the same prompt tomorrow may look nothing like it. Screenshots are good for illustrating a point in a deck. They are not measurement, and presenting them as evidence of performance is how a programme loses credibility the first time someone checks.
Sentiment scores
Applying automated sentiment analysis to AI-generated text stacks one probabilistic system on top of another and compounds the error. A sentiment score drawn from five responses to one prompt on one day has too many sources of variance to act on. If you want to know how you are being characterised, read the responses.
Anything from one query on one day
The general rule: a number derived from a single run of a single prompt on a single day is a data point, not a metric. Data points are inputs to metrics.
First-party signals you actually own
Prompt tracking tells you what the engines are doing. Two sources you own tell you what happens around it: whether crawlers arrive at all, and whether anyone clicks.
AI crawler hits in server logs
Every major engine identifies itself by user-agent, and Part 3 lists the tokens. Filter your access logs for them and count hits per page per week:
grep -E "OAI-SearchBot|Claude-SearchBot|PerplexityBot|Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn
That returns your URLs ranked by crawler hits. Pages with zero hits from retrieval agents are not in the candidate pool at all, whatever your citation tracking says, and that is usually a Part 3 problem rather than a content one.
What it does not tell you: a crawler hit proves the engine fetched the page. It does not prove the engine extracted anything from it, used it in an answer, or showed it to anyone. Crawl activity is necessary for citation, not sufficient.
Referral traffic from assistant domains
When someone clicks a citation, the visit arrives with a referrer from the engine's domain. Monitor chatgpt.com, perplexity.ai, claude.ai, copilot.microsoft.com and gemini.google.com. ChatGPT also appends a campaign parameter to links it generates, which survives in your logs even where the referrer is stripped, so filter for it separately and confirm against your own data rather than taking anyone's word for the current format:
grep "utm_source=chatgpt" access.log | awk '{print $7}' | sort | uniq -c | sort -rn
In GA4 there is a default channel for this. Google's documentation defines AI Assistant as "the channel by which users arrive at your site from sources like ChatGPT, Gemini, Deepseek, Copilot, or Grok". Perplexity is not among the sources Google lists, so if Perplexity matters to you, build a custom channel group in Admin with a source condition matching perplexity\.ai rather than assuming the default covers it.
What it does not tell you: how often you were cited without a click, which is most of the time. Referral traffic measures the overlap of citation and curiosity. A page cited constantly in answers that fully resolve the question may send almost no traffic, and that is not a failure, it is the mechanism working.
What the platforms report natively
This section describes the state in September 2026. It has changed more than once in the past year and it will change again, so re-check before relying on it.
Google Search Console
In June 2026 Google published "Introducing Search Generative AI performance reports in Search Console", adding a view of how content performs in Google's generative AI surfaces. What it gives you is impression-level data for Google's own surfaces. What it does not give you is which prompts triggered an appearance, what the response said, or who was cited alongside you, and it covers none of ChatGPT, Claude or Perplexity. Useful, and not a substitute for the programme in this part.
Bing Webmaster Tools
Microsoft went further. AI Performance launched in public preview in February 2026 and reports total citations, cited pages, grounding queries and page-level citation activity for Copilot and Bing's AI answers. Note what grounding queries actually are: the queries the system generates internally to retrieve content, not the prompts users typed. Microsoft describes the data as a sample rather than a complete record. In June 2026 it added intents, topics, citation share and competitor comparison.
If Copilot matters to your audience, this is the most granular first-party citation data available anywhere today, and it is free. It is also Microsoft-only.
Cadence, and what a believable monthly report looks like
Monthly is the right reporting cadence for most teams. Track weekly internally to catch problems early, but do not present weekly numbers to leadership: the samples are too small to survive the variance, and a chart that jumps around every week trains people to ignore it.
The tracking sheet
Keep a raw tab with one row per run, and a summary tab with one row per prompt and one column per period holding the presence rate. Always derive the summary from the raw data rather than typing rates in by hand.
| Column | Contents | Notes |
|---|---|---|
| Prompt ID | Short stable identifier, P01 onwards | Never reused, even after a prompt is retired |
| Prompt text | The exact prompt, verbatim | Copy from a locked reference sheet, never retype |
| Topic cluster | Evaluation, comparison, use case | Lets you roll up by cluster |
| Baseline presence rate | The fraction from the pre-work run | Written once, never updated |
| Period presence rates | One column per period, labelled by month | =COUNTIF(run_range,"Y")/COUNTA(run_range) |
What the report contains
- The headline. Total citations this period against baseline, as a percentage change. One number, one comparison.
- Presence rate trend. Average presence rate per engine per month since baseline, which shows which engines are moving.
- Competitive context. The five domains most often cited instead of you, with counts.
- A qualitative sample. Three to five verbatim responses citing you, chosen to show the range of framing. This is the part that tells leadership whether the growth is good news, and no number can do it.
What it does not contain: automated sentiment scores, average position, single-run screenshots dressed up as evidence, or comparisons against industry benchmarks that were not produced from the same prompt set on the same engines.
State the limits in the report itself
Put this in every report rather than in a footnote. The citation count measures how often your pages appear in answers to the prompts you track. It does not measure how often buyers encounter those answers. It does not measure the far larger universe of prompts you are not tracking. And it does not measure revenue. Connecting visibility to money needs first-party attribution: the referral sessions, the form submissions, and CRM records that tie a session source back to a closed deal. The programme here produces the first number honestly. The second one is a different project, and saying so is what keeps the first one trusted.
Common questions
How do I measure AI visibility without a dedicated tool?
Run your prompt set by hand in each engine, record one row per run in a spreadsheet, and calculate presence rates from the raw data. No paid tooling is required. A browser, a spreadsheet and about half an hour per engine per week covers a set of 40 to 50 prompts run several times each.
How often should I run the full prompt set?
Weekly for internal tracking, monthly for reporting. Weekly runs catch problems early, such as a presence rate dropping after a site change. Monthly aggregation smooths out the run-to-run variance enough to produce numbers that hold up in front of leadership. Do not report weekly figures upward; the samples are too small.
Why does the same prompt give a different answer each time?
Because these systems are not deterministic. The model samples from a probability distribution, and even at temperature zero the serving infrastructure introduces variance through floating-point arithmetic, batching and hardware parallelism. Published research running identical settings repeatedly found accuracy swings of up to 15 percent between runs. Repeated sampling is the only defence.
What is the difference between a citation and a Search Console impression?
A citation is a confirmed appearance of your domain in an AI answer, observed by running the prompt yourself. A Search Console impression counts how often your URL appeared in one of Google's generative AI surfaces. They measure different things on different engines with different methods, so the two numbers are not comparable.
Can I prove a specific change caused a citation increase?
Not from citation tracking alone. Engine behaviour, competitor content and query patterns all move at once, so causation is out of reach. What you can show is correlation: presence rate rising in the periods after a change and holding flat in periods without one. Date every change so it can be aligned to the timeline.
This is the method we run for clients. If you would rather hand it over than do it yourself, the free AI Visibility Audit is where that starts.
Get a free AI Visibility Audit