ai newsSa., 31. Okt.

Giving an AI assistant a memory of your Google search traffic that does not go stale

Picture this. You ask your AI assistant which pages bring in the most search traffic. It answers with confidence. The numbers are from three months ago, for a site you stopped managing in August. Nothing crashed. The assistant just kept a memory of data that is no longer true. I wanted a memory of

dev.toSamstag 06:32 · 31.10.4 Min. Lesezeit
Bild: dev.to

You ask your AI assistant which pages bring in the most search traffic. It answers with confidence. The numbers are from three months ago, for a site you stopped managing in August.

Nothing crashed. The assistant just kept a memory of data that is no longer true.

I wanted a memory of Google search traffic that stays current, and that forgets what it should. So for Mergetober I built a Google Search Console connector for Cognee. The work is for issue #4793.

Cognee turns documents into a knowledge graph that an agent can search. The connector feeds it from Search Console. You sign in with Google, pick your properties, and it syncs how they perform in search.

Search Console gives you numbers, not prose. One row per date, query and page would be millions of documents, and an LLM would read every one. So the connector groups the data by calendar month. Each month of a property becomes:

All of a property's documents share one node set. The same query in two months becomes one entity in the graph, so you can ask how it changed.

source = google_search_console_source ( sites = [ " sc-domain:example.com " ], # leave it out for every verified property months = 3, ) await cognee. remember ( source, dataset_name = " search_console ", primary_key = " id ", write_disposition = " merge " ) Enter fullscreen mode Exit fullscreen mode Sign-in is OAuth 2.0 with the read-only scope. Run login.py once, in the browser, and the token is cached in a private file.

Two things make Search Console different from a normal source.

First, the newest data is late and preliminary. Google publishes final numbers about two days behind, and it can still revise the last days. So the connector reads final data only. A month stays open until it is a few days older than that lag. An open month is read again on every run. Then it is read one last time and closed, and a closed month is never read again. Only documents whose text changed are sent on, so an unchanged run makes no LLM call.

Second, the data is sampled. Google returns the top rows by clicks and leaves out rare queries for privacy. That means a query that disappears from a response is not a deleted query, and the rows never add up to the totals. Every document says so.

A property you remove, deselect, or lose verification on is forgotten whole, with its documents. I added a few safety rules, because a wrong delete is worse than a missed one:

First I tested against a fake: an in-memory copy of the Search Console API. It aggregates real fact rows, sorts by clicks, hides the newest days, and withholds rare queries. 110 tests pass on Python 3.10 and 3.12, and on Cognee 1.6.4 and 1.6.6. They include a run of the whole Cognee pipeline with a mocked LLM. The 106 that skip that pipeline run also pass on Python 3.14. But a fake only proves what you thought to fake.

Here is the honest part. My only property is a personal portfolio, and it had no search traffic in the last 80 days. So the real sync found no data. It created no documents and raised no errors. The second run skipped the two closed months:

run 1 {'sites': 1, 'periods': 3, 'closed': 0, 'documents': 0,...} run 2 {'sites': 1, 'periods': 1, 'closed': 2, 'documents': 0,...} Enter fullscreen mode Exit fullscreen mode The bug only a real Google could find That empty property showed me something my fake got wrong.

When there is no data, I expected Google to answer with an empty response. It does not. For a query with no dimensions, it returns one row of zeros: {'clicks': 0, 'impressions': 0,...}.

My code read "one row" as "there is data". It would have written an overview that says "0 clicks" for every empty month. Worse, it would have walked straight past my own safety rule. A month that had real documents and then got a zero row back would have been overwritten with zeros, instead of being kept.

Now a row with zero impressions counts as no data. I changed the fake to behave the way Google does, so a test fails if this comes back.

The same run settled a smaller question. Google accepts the lowercase type and dataState values that I send.

I did not test any of this against real traffic: documents built from real queries and pages, a recall on real data, a changed month, or forgetting a real property. Those paths are covered by the fake-API tests and the mocked-LLM pipeline test, not by a live account. I also never saw a real 429 from Google.

Rare queries are left out of the lists by design. A long tail of low-click pages shows up in the totals only. Sitemaps and URL inspection are not ingested. The README lists these.

The example is in packages/connector/google-search-console/examples/. You need a Google Cloud project with the Search Console API enabled, an OAuth client of type Desktop app, and a verified property. The README walks through it.

If you point it at a property with real traffic, I would love to hear what breaks.

Quelle: dev.to (dev.to). Original öffnen