Add an AI-visibility regression test to your CI: fail the build when ChatGPT stops recommending you

Disclosure: I build the AI Search Visibility Tracker Actor used here on Apify. The output below is a real run (October 22, 2026, US, English, HubSpot as the tracked brand, Salesforce and Pipedrive as competitors). The test was run for real against those saved rows with pytest; I did not make new calls for this post.

You test your code on every commit. You probably do not test the thing that now sends a growing share of buyers your way: whether ChatGPT and Perplexity still name your product when someone asks for a recommendation.

This post adds a small pytest file that fails when your brand's mention rate or your own-domain citation rate falls under a threshold. It is meant to run on a schedule (weekly is plenty), not on every push.

Each run gives you one check row per prompt per engine, plus summary rows with the rates already computed. The summary row for the tracked brand across both engines looked like this (real values):

{ "type": "summary", "engine": "all", "brand": "HubSpot", "is_your_brand": true, "checks": 2, "answers": 2, "mentions": 2, "mention_rate": 1, "share_of_voice": 0.3333, "avg_position": 1, "top_position_rate": 1, "citations": 2, "citation_rate": 1, "sentiment_positive": 2, "sentiment_negative": 0 } Enter fullscreen mode Exit fullscreen mode and the competitor rows have the same shape ( is_your_brand: false ), so you can also assert that a competitor is not suddenly first. The per-answer check rows carry brand_mentioned, brand_position, competitors_mentioned, your_cited_urls and cited_domains.

""" AI-visibility regression test: fails when AI assistants stop recommending your brand. Live mode: APIFY_TOKEN=... pytest -q test_ai_visibility.py (costs about $0.04 per prompt per engine) Offline mode: AI_VIS_FIXTURE=tracker_rows.json pytest -q test_ai_visibility.py (replays saved rows, free) """ import json, os import pytest import requests BRAND, DOMAIN = " HubSpot ", " hubspot.com " COMPETITORS = [ " Salesforce | salesforce.com ", " Pipedrive | pipedrive.com " ] PROMPTS = [ " What is the best CRM for a small business? " ] ENGINES = [ " chatgpt ", " perplexity " ] MIN_MENTION_RATE = 0.5 # share of answers that must name the brand MIN_CITATION_RATE = 0.5 # share of answers that must cite your domain @pytest.fixture ( scope = " session " ) def rows (): if os. environ. get ( " AI_VIS_FIXTURE " ): return json. load ( open ( os. environ [ " AI_VIS_FIXTURE " ])) r = requests. post ( " https://api.apify.com/v2/acts/wrenfield-data~ai-search-visibility-tracker/run-sync-get-dataset-items ", params = { " maxTotalChargeUsd ": 2.00 }, # hard cap: the test can never spend more than $2 headers = { " Authorization ": f " Bearer { os. environ [ ' APIFY_TOKEN ' ] } " }, json = { " brands ": [ BRAND ], " domains ": [ DOMAIN ], " competitors ": COMPETITORS, " prompts ": PROMPTS, " engines ": ENGINES }, timeout = 600 ) r. raise_for_status () return r. json () @pytest.fixture ( scope = " session " ) def mine ( rows ): s = [ x for x in rows if x. get ( " type " ) == " summary " and x. get ( " engine " ) == " all " and x. get ( " is_your_brand " )] assert s, " no summary row for your brand: the run failed or returned no answers " return s [ 0 ] def test_answers_came_back ( rows ): ok = [ x for x in rows if x. get ( " type " ) == " check " and x. get ( " status " ) == " ok " ] assert len ( ok ) >= len ( PROMPTS ) * len ( ENGINES ) / 2, " more than half of the checks failed; do not trust the result " def test_brand_is_named ( mine ): assert mine [ " mention_rate " ] >= MIN_MENTION_RATE, f " { BRAND } named in only { mine [ ' mention_rate ' ]:.

0 % } of answers " def test_own_domain_is_cited ( mine ): assert mine [ " citation_rate " ] >= MIN_CITATION_RATE, f " { DOMAIN } cited in only { mine [ ' citation_rate ' ]:. 0 % } of answers " Enter fullscreen mode Exit fullscreen mode Two modes: live (calls the Actor, capped at $2 per run by maxTotalChargeUsd ) and offline (replays saved rows for free, which is how you develop the test and how I ran it for this post).

test_ai_visibility.py::test_answers_came_back PASSED [ 33%] test_ai_visibility.py::test_brand_is_named PASSED [ 66%] test_ai_visibility.py::test_own_domain_is_cited PASSED [100%] ============================== 3 passed in 0.07s =============================== Enter fullscreen mode Exit fullscreen mode Making it a scheduled job A GitHub Actions workflow on a weekly cron is enough:

on: schedule: [{ cron: " 0 14 * * 1" }] # Mondays jobs: ai-visibility: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pip install pytest requests - run: pytest -q test_ai_visibility.py env: APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }} Enter fullscreen mode Exit fullscreen mode (I have not run this workflow file; the pytest file above is what I executed.)

If you want a field added to the output, say so in the comments.