Track AI Brand Mentions Over Time: Schedules and Drift

Posted on Oct 28 • Originally published at datatooly.xyz

To track AI brand mentions over time, run the same buyer questions in the same assistants with the same settings on a schedule (monthly by hand, weekly with a script). Log every brand and cited domain per answer and compare runs prompt by prompt. Record the model and whether web search ran, to separate real change from updates and noise.

The first check tells you where you stand. This guide is about the second and later checks: what to hold constant, what causes drift, and how to tell a real change from noise. The comparison script at the end runs on the same CSV log as the free ChatGPT brand check.

If any of these change between runs, a change in your numbers may come from your setup, not the assistants.

Save the raw answers, not just your tallies, so you can re-score old runs if you refine the rules.

Some changes are real, some come from the assistant, and some are noise.

Running more often doesn't make a single run more accurate. More answers per run does.

This script compares two logs prompt by prompt (won, lost, up, down, same) and tests whether the overall mention rate changed by more than chance would explain:

import csv import math import sys from collections import defaultdict BRAND = " Northwind " def load ( path ): """ {(assistant, prompt_id): [True/False per run]} - did the answer name BRAND? """ out = defaultdict ( list ) for r in csv. DictReader ( open ( path, encoding = " utf-8 " )): brands = [ b. strip () for b in r [ " brands_in_order " ]. split ( "; " )] out [( r [ " assistant " ], r [ " prompt_id " ])]. append ( BRAND in brands ) return out def two_proportion_p ( k1, n1, k2, n2 ): """ Two-sided p-value for ' the mention rate changed ' (normal approximation).

""" p = ( k1 + k2 ) / ( n1 + n2 ) se = math. sqrt ( p * ( 1 - p ) * ( 1 / n1 + 1 / n2 )) if se == 0: return 1.0 z = ( k2 / n2 - k1 / n1 ) / se return math. erfc ( abs ( z ) / math. sqrt ( 2 )) before, after = load ( sys. argv [ 1 ]), load ( sys. argv [ 2 ]) print ( f " { ' prompt ': < 16 }{ ' before ': > 8 }{ ' after ': > 8 } change " ) for key in sorted ( set ( before ) | set ( after )): b, a = before. get ( key, []), after. get ( key, []) if not b or not a: print ( f " { ' / '. join ( key ): < 16 } only in one run: not comparable " ) continue rb, ra = sum ( b ) / len ( b ), sum ( a ) / len ( a ) change = " won " if rb == 0 and ra > 0 else " lost " if rb > 0 and ra == 0 else ( " up " if ra > rb else " down " if ra < rb else " same " ) print ( f " { ' / '. join ( key ): < 16 }{ sum ( b ) } / { len ( b ): < 6 }{ sum ( a ) } / { len ( a ): < 6 } { change } " ) k1 = sum ( sum ( v ) for v in before. values ()); n1 = sum ( len ( v ) for v in before. values ()) k2 = sum ( sum ( v ) for v in after. values ()); n2 = sum ( len ( v ) for v in after. values ()) print ( f " \n mention rate { k1 } / { n1 } = { k1 / n1:. 0 % } -> { k2 } / { n2 } = { k2 / n2:. 0 % }, " f " p = { two_proportion_p ( k1, n1, k2, n2 ):.

2 f } " ) Enter fullscreen mode Exit fullscreen mode python compare_runs.py log_2026-09.csv log_2026-10.csv on two small example logs (the brands are placeholders, not real measurements):

prompt before after change chatgpt/P1 1/3 3/3 up chatgpt/P2 0/3 1/3 won chatgpt/P3 3/3 2/3 down mention rate 4/9 = 44% -> 6/9 = 67%, p = 0.34 Enter fullscreen mode Exit fullscreen mode The headline looks like a 23-point gain. With 9 answers per month, p = 0.34 says a change this size would turn up often by chance alone. Same rates with more answers, using the same test:

A rule of thumb from this: on 100 answers a run, a 20-point move is solid evidence; a 10-point move needs about 200 answers a run before it stands out. The prompt-by-prompt view is still useful at small sizes. It tells you where to look, such as which cited sources changed on a prompt you lost, even when the total isn't conclusive.

The free LLM visibility query builder gives you a ready-to-run audit input from your brand, category and competitors, with a fixed example of the output. It doesn't call any AI model from your browser.

LLM Visibility Monitor on Apify is built for scheduled runs. It saves the generated question set so every run asks the same questions, compares each run with your previous run for the same brand, country and language (questions won, lost and unchanged), and records the model it requested and the model that actually answered on every row. You can schedule it weekly through Apify. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly query builder and the Apify actor. The comparison script works on any hand-collected log.

Sources: OpenAI: overview of OpenAI crawlers, OpenAI Help Center: Searching the web with ChatGPT, and the two-proportion z-test (computed in the script above).