Your AI backfill passed the quality check. More than a third of it was wrong.

Say you inherit an orders table. The load breaks now and then, and someone reruns it by hand. A spreadsheet of Monday corrections gets pasted in. The city column has "SP", "São Paulo", "Sao Paulo" and "sao paulo", and about one row in ten is blank because nobody could work out the city.

Nothing in that table fails. Nothing in it is traceable either. And the moment somebody says "let's just have the AI fill the blanks", you've got one more way for a row to be wrong that looks exactly like a row that's right.

The fix is a path: four steps, in order. Snowflake and BigQuery each have a name for every one of them, and both are below. Under each step there's also a piece of a single Python file that runs on DuckDB with synthetic data, so you can watch each step do its job without a cloud account. Every output block from that file is a real run, seeded, and reproduces exactly. The Snowflake and BigQuery snippets follow the vendor docs linked in each section. I haven't run them for this article.

First, the as-is, and the checks it passes:

""" From hand fixes to an audited AI backfill, on a laptop. Synthetic data, seeded, so every number below is reproducible. Runs on DuckDB so the SQL reads the same as it would on Snowflake or BigQuery. pip install duckdb jsonschema """ import hashlib import random import duckdb import jsonschema random. seed ( 7 ) con = duckdb. connect () # --------------------------------------------------------------------------- # AS-IS: 1,000 orders, the city column the way a hand-maintained table ends up. # --------------------------------------------------------------------------- CITIES = [ " São Paulo ", " Campinas ", " Santos ", " Rio de Janeiro ", " Belo Horizonte " ] SP_SPELLINGS = [ " São Paulo ", " Sao Paulo ", " SP ", " sao paulo " ] # same city, four ways def make_order ( i ): true_city = random. choices ( CITIES, weights = [ 50, 15, 10, 15, 10 ])[ 0 ] kind = " clean " note = f " Entregar na portaria, { true_city } " city = random. choice ( SP_SPELLINGS ) if true_city == " São Paulo " else true_city r = random. random () if r < 0.06: # blank, but the note names the city city, kind = None, " blank_clear " elif r < 0.09: # blank, and the note names two cities city, kind = None, " blank_ambiguous " other = random. choice ([ c for c in CITIES if c!= true_city ]) note = f " Cliente de { other }, entregar no escritório em { true_city } " elif r < 0.10: # blank, and the note names no city at all city, kind = None, " blank_empty " note = " Entregar na portaria " return ( i, city, note, true_city, kind ) rows = [ make_order ( i ) for i in range ( 1, 1001 )] con. execute ( """ CREATE TABLE raw_orders (order_id INT, city VARCHAR, delivery_note VARCHAR, true_city VARCHAR, kind VARCHAR) """ ) con. executemany ( " INSERT INTO raw_orders VALUES (?,?,?,?,?) ", rows ) print ( " == AS-IS: the usual checks == " ) print ( con. execute ( """ SELECT COUNT(*) AS row_count, COUNT(DISTINCT order_id) AS distinct_ids, COUNT(*) - COUNT(order_id) AS null_ids FROM raw_orders """ ). fetchall ()) print ( con. execute ( """ SELECT city, COUNT(*) AS orders FROM raw_orders GROUP BY city ORDER BY orders DESC """ ). fetchall ()) Enter fullscreen mode Exit fullscreen mode == AS-IS: the usual checks == [(1000, 1000, 0)] [('Campinas', 131), ('sao paulo', 129), ('São Paulo', 123), ('Rio de Janeiro', 123), ('Sao Paulo', 117), ('SP', 115), (None, 111), ('Belo Horizonte', 85), ('Santos', 66)] Enter fullscreen mode Exit fullscreen mode A thousand rows, a thousand distinct IDs, no null keys.

Every count a dashboard would show is green. And by city, Campinas is your biggest market, which it isn't: São Paulo is split four ways and adds up to 484. The true_city and kind columns are the answer key a synthetic dataset gets to have.

Your real table doesn't, and that's the whole problem.

Every later step reads from this one. You can't check, fill or judge a table someone still edits by hand, because you can't tell the edit from the data.

Two things make a load automated rather than scheduled. It runs without a person. And corrections arrive as data the load consumes, not as edits to the table it produces.

On Snowflake that's a task. The detail that catches people: newly created tasks are created suspended. Nothing runs until you resume it.

-- the rebuild-from-raw logic lives in a view (same SQL as the Python below); -- the task only materialises it CREATE OR REPLACE TASK load_orders WAREHOUSE = transform_wh SCHEDULE = '60 MINUTE' AS CREATE OR REPLACE TABLE orders AS SELECT * FROM orders_clean_v; ALTER TASK load_orders RESUME; -- without this line, the task exists and never runs Enter fullscreen mode Exit fullscreen mode On BigQuery it's a scheduled query. Its detail: unless you give it a service account, it runs with the credentials of the person who created it. Give it one.

bq mk \ --transfer_config \ --target_dataset = mydataset \ --display_name = 'load_orders' \ --schedule = 'every 24 hours' \ --params = '{"query":"SELECT * FROM mydataset.orders_clean_v","destination_table_name_template":"orders","write_disposition":"WRITE_TRUNCATE"}' \ --data_source = scheduled_query \ --service_account_name = loader@myproject.iam.gserviceaccount.com Enter fullscreen mode Exit fullscreen mode WRITE_TRUNCATE isn't decoration. A load that rebuilds the target from raw every time is safe to run twice, and that is what lets you stop being afraid of it. Here's the same idea locally, with an alias table for the spellings and a corrections feed instead of a spreadsheet:

# --------------------------------------------------------------------------- # STEP 1: Automate the load. One code path from raw to clean, no hand edits. # Corrections arrive as data (a feed table), not as edits to the target. # --------------------------------------------------------------------------- con. execute ( """ CREATE TABLE city_aliases AS SELECT * FROM (VALUES ( ' são paulo ', ' São Paulo ' ), ( ' sao paulo ', ' São Paulo ' ), ( ' sp ', ' São Paulo ' ), ( ' campinas ', ' Campinas ' ), ( ' santos ', ' Santos ' ), ( ' rio de janeiro ', ' Rio de Janeiro ' ), ( ' belo horizonte ', ' Belo Horizonte ' ) ) t(alias, city) """ ) con. execute ( " CREATE TABLE corrections (order_id INT, city VARCHAR, reason VARCHAR) " ) con. execute ( " INSERT INTO corrections VALUES (42, ' Santos ', ' customer called support ' ) " ) def load_orders (): """ The whole load. Safe to run twice: it rebuilds from raw every time.

""" con. execute ( """ CREATE OR REPLACE TABLE orders AS SELECT r.order_id, COALESCE(c.city, a.city) AS city, CASE WHEN c.city IS NOT NULL THEN ' correction ' WHEN a.city IS NOT NULL THEN ' source ' ELSE NULL END AS city_source, r.delivery_note FROM raw_orders r LEFT JOIN city_aliases a ON lower(trim(r.city)) = a.alias LEFT JOIN corrections c ON c.order_id = r.order_id """ ) # a fingerprint of the whole table, so "same result" is checked, not assumed return con. execute ( """ SELECT COUNT(*), md5(string_agg(order_id || ': ' || coalesce(city, ' ∅ ' ), ', ' ORDER BY order_id)) FROM orders """ ). fetchone () print ( " \n == STEP 1: run the load twice == " ) print ( " run 1: ", load_orders ()) print ( " run 2: ", load_orders ()) print ( con. execute ( """ SELECT city, COUNT(*) FROM orders GROUP BY city ORDER BY 2 DESC """ ). fetchall ()) Enter fullscreen mode Exit fullscreen mode == STEP 1: run the load twice == run 1: (1000, '9fb6899493deff927b480901368c59d5') run 2: (1000, '9fb6899493deff927b480901368c59d5') [('São Paulo', 484), ('Campinas', 130), ('Rio de Janeiro', 123), (None, 111), ('Belo Horizonte', 85), ('Santos', 67)] Enter fullscreen mode Exit fullscreen mode Same fingerprint twice. São Paulo is one city again. The correction for order 42 moved one row from Campinas to Santos, and the row says so in city_source.

That column is the start of provenance, and step 3 depends on it.

Now the 111 blanks are a measured property of the table instead of a feeling.

On Snowflake, quality checks are data metric functions. They run on the table's schedule, one hour by default, and every DMF on that table shares that one schedule. You can't run the null check every five minutes and the accepted-values check daily on the same table.

ALTER TABLE orders SET DATA_METRIC_SCHEDULE = '60 MINUTE'; -- built-in: nulls in the column ALTER TABLE orders ADD DATA METRIC FUNCTION SNOWFLAKE. CORE. NULL_COUNT ON ( city ); -- custom: values outside the list you accept CREATE OR REPLACE DATA METRIC FUNCTION governance. dmfs. unknown_city_count ( arg_t TABLE ( arg_city VARCHAR ) ) RETURNS NUMBER AS $$ SELECT COUNT ( * ) FROM arg_t WHERE arg_city NOT IN ( 'São Paulo', 'Campinas', 'Santos', 'Rio de Janeiro', 'Belo Horizonte' ) $$; ALTER TABLE orders ADD DATA METRIC FUNCTION governance. dmfs. unknown_city_count ON ( city ); -- results land here; measurement_time is when the check actually ran SELECT measurement_time, metric_name, argument_names, value FROM SNOWFLAKE.

LOCAL. DATA_QUALITY_MONITORING_RESULTS WHERE table_name = 'ORDERS' ORDER BY measurement_time DESC; Enter fullscreen mode Exit fullscreen mode On BigQuery, the same check is a rule in a data quality scan. Google's default here is the right one, and worth knowing before someone changes it: rows with null values automatically fail a rule unless ignoreNull is true.

Set it to true and nulls are, in Google's words, "trivially considered passing."

# spec.yaml rules: - column: city dimension: VALIDITY setExpectation: values: [ ' São Paulo', ' Campinas', ' Santos', ' Rio de Janeiro', ' Belo Horizonte' ] ignoreNull: false threshold: 1.0 Enter fullscreen mode Exit fullscreen mode gcloud dataplex datascans create data-quality orders-city-check \ --location = us-central1 \ --data-quality-spec-file = spec.yaml \ --data-source-resource = //bigquery.googleapis.com/projects/myproject/datasets/mydataset/tables/orders Enter fullscreen mode Exit fullscreen mode The same rule locally, so you can watch what one flag does:

# --------------------------------------------------------------------------- # STEP 2: Check quality where it lands. # An accepted-values rule, the same shape as a BigQuery setExpectation rule. # --------------------------------------------------------------------------- def accepted_values_rule ( column, values, ignore_null, threshold = 1.0 ): """ Pass ratio the way the rule defines it: passing rows / total rows. ignore_null=False mirrors the documented default: a null row fails. """ in_list = ", ". join ( f "' { v } '" for v in values ) null_outcome = " 1 " if ignore_null else " 0 " passing, total = con. execute ( f """ SELECT SUM(CASE WHEN { column } IS NULL THEN { null_outcome } WHEN { column } IN ( { in_list } ) THEN 1 ELSE 0 END), COUNT(*) FROM orders """ ). fetchone () ratio = passing / total return round ( ratio, 3 ), " PASS " if ratio >= threshold else " FAIL " print ( " \n == STEP 2: accepted-values rule on city == " ) print ( " ignoreNull = true: ", accepted_values_rule ( " city ", CITIES, ignore_null = True )) print ( " ignoreNull = false: ", accepted_values_rule ( " city ", CITIES, ignore_null = False )) Enter fullscreen mode Exit fullscreen mode == STEP 2: accepted-values rule on city == ignoreNull = true: (1.0, 'PASS') ignoreNull = false: (0.889, 'FAIL') Enter fullscreen mode Exit fullscreen mode Same table, same 111 blanks, opposite verdicts. The failing one is telling the truth.

Hold on to that, because in the next step a table with 42 wrong values is going to pass this exact rule.

This is the step everyone wants to start with, and it's the one with the trap.

Both platforms let you hand the model a schema. On Snowflake, AI_COMPLETE takes a response_format, and Snowflake's docs describe what it does with it: it "verifies each generated token against your structured output definition."

SELECT order_id, AI_COMPLETE ( model => 'mistral-large2', prompt => 'Return the delivery city named in this note: ' || delivery_note, response_format => { 'type': 'json', 'schema': { 'type': 'object', 'properties': { 'city': { 'type': 'string' }}, 'required': [ 'city' ]} } ) AS fill FROM orders WHERE city IS NULL; Enter fullscreen mode Exit fullscreen mode On BigQuery, AI.GENERATE_TABLE takes an output_schema, written like a CREATE TABLE column list, against a remote model you've created over Gemini:

SELECT * FROM AI. GENERATE_TABLE ( MODEL `mydataset.gemini_model`, ( SELECT order_id, CONCAT ( 'Return the delivery city named in this note: ', delivery_note ) AS prompt FROM mydataset. orders WHERE city IS NULL ), STRUCT ( 'city STRING' AS output_schema ) ); Enter fullscreen mode Exit fullscreen mode Here's the trap. A schema check proves the shape of the answer. It can never prove the value. "Campinas" is a perfectly valid string. So is the wrong city.

Locally, stand_in_model plays the model. It isn't an LLM and doesn't pretend to be one: it takes the first city it finds in the note, and guesses São Paulo when there's none. That's a fair impression of a cheap model on an ambiguous note, and it lets the run reproduce without an API key.

# --------------------------------------------------------------------------- # STEP 3: Backfill with AI, and label it as AI. # stand_in_model is NOT an LLM. It behaves like a cheap one on this task: # it returns the first city it can find in the note, or its best guess. # Swap in AI_COMPLETE or AI.GENERATE_TABLE and the shape of the problem is the same. # --------------------------------------------------------------------------- FILL_SCHEMA = { " type ": " object ", " properties ": { " city ": { " type ": " string " }}, " required ": [ " city " ]} PROMPT = " Return the delivery city named in this note: {note} " MODEL = " stand-in-v1 " def stand_in_model ( note ): found = [ c for c in CITIES if c in note ] found. sort ( key = note. index ) # first mention wins return { " city ": found [ 0 ] if found else " São Paulo " } # no city? guess the big one blanks = con. execute ( " SELECT order_id, delivery_note FROM orders WHERE city IS NULL " ). fetchall () fills, schema_failures = [], 0 for order_id, note in blanks: answer = stand_in_model ( note ) try: jsonschema. validate ( answer, FILL_SCHEMA ) # the check the platforms give you except jsonschema. ValidationError: schema_failures += 1 continue prompt_hash = hashlib. sha256 ( PROMPT. format ( note = note ). encode ()). hexdigest ()[: 12 ] fills. append (( order_id, answer [ " city " ], MODEL, prompt_hash )) con. execute ( " CREATE TABLE fills (order_id INT, city VARCHAR, fill_model VARCHAR, prompt_hash VARCHAR) " ) con. executemany ( " INSERT INTO fills VALUES (?,?,?,?) ", fills ) con. execute ( """ CREATE OR REPLACE TABLE orders AS SELECT o.order_id, COALESCE(o.city, f.city) AS city, CASE WHEN f.city IS NOT NULL THEN ' ai_fill ' ELSE o.city_source END AS city_source, f.fill_model, f.prompt_hash, o.delivery_note, CAST(NULL AS DOUBLE) AS judge_score FROM orders o LEFT JOIN fills f USING (order_id) """ ) print ( " \n == STEP 3: the backfill == " ) print ( " blanks: ", len ( blanks ), " | filled: ", len ( fills ), " | schema failures: ", schema_failures ) print ( " accepted-values rule now: ", accepted_values_rule ( " city ", CITIES, ignore_null = False )) # the answer key only a synthetic dataset has print ( con. execute ( """ SELECT r.kind, COUNT(*) AS fills, SUM(CASE WHEN o.city <> r.true_city THEN 1 ELSE 0 END) AS wrong FROM orders o JOIN raw_orders r USING (order_id) WHERE o.city_source = ' ai_fill ' GROUP BY r.kind ORDER BY r.kind """ ). fetchall ()) Enter fullscreen mode Exit fullscreen mode == STEP 3: the backfill == blanks: 111 | filled: 111 | schema failures: 0 accepted-values rule now: (1.0, 'PASS') [('blank_ambiguous', 36, 36), ('blank_clear', 67, 0), ('blank_empty', 8, 6)] Enter fullscreen mode Exit fullscreen mode Read that slowly.

Every blank filled, no schema failures, and the quality rule that failed at 0.889 now passes at 1.0. Meanwhile 42 of the 111 fills are wrong: all 36 where the note named two cities, and 6 of the 8 where it named none. The check didn't get weaker.

The table got more confident.

That's why the step title says label it as AI. The city_source, fill_model and prompt_hash columns are what keep a fill from passing as a source value. Without them, the 42 wrong rows look exactly like the 484 São Paulo rows that came from the source. With them, you can at least ask the question. And a trap you can't query for is a warning, not a tool, so here's the query:

# The trap query. Every AI-filled value that nothing has checked beyond its shape. TRAP_QUERY = """ SELECT order_id, city, fill_model, delivery_note FROM orders WHERE city_source = ' ai_fill ' AND (judge_score IS NULL -- never judged OR strpos(lower(delivery_note), lower(city)) = 0) -- value not in its own source """ print ( " unchecked fills: ", len ( con. execute ( TRAP_QUERY ). fetchall ())) print ( " of those, value not in the note: ", con. execute ( f " SELECT COUNT(*) FROM ( { TRAP_QUERY } ) WHERE strpos(lower(delivery_note), lower(city)) = 0 " ). fetchone ()[ 0 ]) Enter fullscreen mode Exit fullscreen mode unchecked fills: 111 of those, value not in the note: 8 Enter fullscreen mode Exit fullscreen mode The SQL is plain enough to port to either warehouse; check your platform's substring function when you do.

Notice the second line. The cheap check, "is the value even in its own source?", catches the 8 guesses. It can't catch the 36 ambiguous ones, because the wrong city really is in the note.

That's the job for step 4.

One Snowflake detail for the comments: when a field marked required can't be extracted, AI_COMPLETE raises an error rather than inventing one. That protects you from the empty answer. It doesn't protect you from the confident wrong one.

A judge is a second model that scores the first one's output. For a backfill, the metric that fits is groundedness: is the filled value supported by the source text it came from?

On Snowflake, TruLens metrics run server-side as part of AI Observability, and the judge model is a parameter. Its default is llama3.1-70b. Your app has to be instrumented so the note is recorded as the context the answer came from; Snowflake's TruLens guide covers that part.

from trulens.core.run import RunConfig run_config = RunConfig ( run_name = " city_backfill_v1 ", source_type = " TABLE ", dataset_name = " ORDERS_FILLED ", dataset_spec = { " RECORD_ROOT.INPUT ": " DELIVERY_NOTE " }, llm_judge_name = " llama3.1-70b ", # the default; set it anyway, so the run says what judged it ) run = tru_app. add_run ( run_config = run_config ) # tru_app: your app, registered with TruLens run. start () run. compute_metrics ( metrics = [ " groundedness " ]) Enter fullscreen mode Exit fullscreen mode On Google, TruLens runs from Python with its Google provider, which reaches Gemini through the Gemini API or Vertex AI and defaults to gemini-2.5-flash. The groundedness call returns a score between 0.0 (not grounded) and 1.0 (grounded) plus the judge's reasons:

from trulens.providers.google import Google judge = Google () # model_engine defaults to "gemini-2.5-flash"; vertexai=True for Vertex AI score, reasons = judge. groundedness_measure_with_cot_reasons ( source = " Cliente de Campinas, entregar no escritório em São Paulo ", statement = " The delivery city is Campinas. ", ) Enter fullscreen mode Exit fullscreen mode I'm not printing an output for that one, because it needs a key and the number would be one run of one model. Think about what it's being asked, though. "Campinas" is in the source. A groundedness judge is checking support, not correctness, and the note does support Campinas. It just doesn't support it as the delivery city.

So the rule is the last clause of the step: grade the judge on a hand-checked sample before you trust a threshold. TruLens's own judge-benchmarking notebook says the same thing plainly: "Use a representative human-labeled dataset for real decisions."

Locally, the judge's scores are synthetic stand-ins, drawn from one Beta distribution per case. The assumptions are in the code where you can see and change them. The arithmetic of grading the judge is real.

# --------------------------------------------------------------------------- # STEP 4: Judge the fill with a second model, then grade the judge. # Real call, per row (TruLens, Google provider; Snowflake runs the same metric server-side): # from trulens.providers.google import Google # score, reasons = Google().groundedness_measure_with_cot_reasons( # source=delivery_note, statement=f"The delivery city is {city}.") # Here the scores are SYNTHETIC stand-ins so the calibration runs without an API key. # The assumptions are below. Change them and re-run. # --------------------------------------------------------------------------- JUDGE_ASSUMPTIONS = { # (alpha, beta) of a Beta distribution per case " value in note, one city ": ( 9, 1.5 ), # clearly supported, scores high " value in note, two cities ": ( 6, 2.5 ), # the city IS in the source, so it looks supported " value not in note ": ( 1.5, 6 ), # nothing to ground on, scores low } def stand_in_judge ( city, note ): in_note = city in note n_cities = sum ( c in note for c in CITIES ) case = ( " value not in note " if not in_note else " value in note, two cities " if n_cities > 1 else " value in note, one city " ) return round ( random. betavariate ( * JUDGE_ASSUMPTIONS [ case ]), 3 ) for order_id, city, note in con. execute ( " SELECT order_id, city, delivery_note FROM orders WHERE city_source = ' ai_fill '" ). fetchall (): con. execute ( " UPDATE orders SET judge_score =? WHERE order_id =?

", [ stand_in_judge ( city, note ), order_id ]) print ( " \n == STEP 4: judge every fill == " ) print ( " unchecked fills after judging: ", len ( con. execute ( TRAP_QUERY ). fetchall ())) # Grade the judge: a hand-checked sample. Here "hand-checked" = the synthetic answer key. sample = con. execute ( """ SELECT o.order_id, o.judge_score, (o.city = r.true_city) AS human_says_correct FROM orders o JOIN raw_orders r USING (order_id) WHERE o.city_source = ' ai_fill ' ORDER BY hash(o.order_id) LIMIT 40 """ ). fetchall () print ( f " \n hand-checked sample: { len ( sample ) } fills, " f " { sum ( 1 for s in sample if not s [ 2 ]) } of them wrong " ) print ( f " { ' threshold ': > 9 } | { ' agree ': > 5 } | { ' wrong but passed ': > 16 } | { ' right but rejected ': > 18 } " ) for t in ( 0.5, 0.7, 0.9 ): agree = sum (( score >= t ) == ok for _, score, ok in sample ) wrong_passed = sum ( score >= t and not ok for _, score, ok in sample ) right_rejected = sum ( score < t and ok for _, score, ok in sample ) print ( f " { t: > 9 } | { agree: > 2 } / { len ( sample ) } | { wrong_passed: > 16 } | { right_rejected: > 18 } " ) Enter fullscreen mode Exit fullscreen mode == STEP 4: judge every fill == unchecked fills after judging: 8 hand-checked sample: 40 fills, 16 of them wrong threshold | agree | wrong but passed | right but rejected 0.5 | 28/40 | 11 | 1 0.7 | 28/40 | 7 | 5 0.9 | 27/40 | 2 | 11 Enter fullscreen mode Exit fullscreen mode No threshold works. Lower it and wrong fills sail through; raise it and you throw away good ones without getting rid of the bad.

Without the hand-checked sample you'd never know, because the judge's scores look perfectly reasonable on their own.

The sample tells you something more useful than a threshold, though. The wrong fills cluster in one place: notes that name two cities. The judge is being asked a question the data can't answer. That isn't a model problem, it's a data problem, and data problems get routed:

# No threshold fixes it, because the judge is being asked a question the data can't answer: # a note that names two cities supports both. That's a data problem, so route it as one. city_count = " + ". join ( f " (strpos(delivery_note, ' { c } ' ) > 0)::INT " for c in CITIES ) con. execute ( f """ CREATE TABLE fill_quarantine AS SELECT order_id, city, judge_score, delivery_note, CASE WHEN strpos(lower(delivery_note), lower(city)) = 0 THEN ' value not in source ' ELSE ' source names more than one city ' END AS reason FROM orders WHERE city_source = ' ai_fill ' AND (strpos(lower(delivery_note), lower(city)) = 0 OR ( { city_count } ) > 1) """ ) con. execute ( """ UPDATE orders SET city = NULL, city_source = ' quarantined ' WHERE order_id IN (SELECT order_id FROM fill_quarantine) """ ) print ( " \n == STEP 4b: route what the source can ' t support == " ) print ( con. execute ( " SELECT reason, COUNT(*) FROM fill_quarantine GROUP BY reason ORDER BY 1 " ). fetchall ()) sample = con. execute ( """ SELECT o.order_id, o.judge_score, (o.city = r.true_city) AS human_says_correct FROM orders o JOIN raw_orders r USING (order_id) WHERE o.city_source = ' ai_fill ' ORDER BY hash(o.order_id) LIMIT 40 """ ). fetchall () print ( f " hand-checked sample of what ' s left: { len ( sample ) } fills, " f " { sum ( 1 for s in sample if not s [ 2 ]) } wrong " ) for t in ( 0.5, 0.7, 0.9 ): agree = sum (( score >= t ) == ok for _, score, ok in sample ) print ( f " threshold { t }: agree { agree } / { len ( sample ) } " ) print ( " accepted-values rule, nulls failing: ", accepted_values_rule ( " city ", CITIES, ignore_null = False )) Enter fullscreen mode Exit fullscreen mode == STEP 4b: route what the source can't support == [('source names more than one city', 36), ('value not in source', 8)] hand-checked sample of what's left: 40 fills, 0 wrong threshold 0.5: agree 40/40 threshold 0.7: agree 35/40 threshold 0.9: agree 23/40 accepted-values rule, nulls failing: (0.956, 'FAIL') Enter fullscreen mode Exit fullscreen mode Forty-four fills go to quarantine with a reason a person can act on. What's left grades clean against the sample, and now a threshold means something: at 0.5 the judge agrees with the human on all 40.

At 0.9 it's throwing good fills away. That's a decision you can only make once you've graded it.

And the last line is the one I'd frame. The quality rule fails again, at 0.956, because 44 cities are honestly unknown.

The temptation is to start at step 3, because step 3 is the demo. You point a model at the blanks, the blanks disappear, the dashboard goes green. In this run, that green table had 42 wrong cities in it and passed the accepted-values rule at 1.0.

The table I'd actually ship fails that same rule at 0.956. Forty-four rows are null with a reason attached, every fill says which model wrote it, and the judge's threshold was chosen against rows a person checked. It looks worse on the dashboard and it's right.

None of that is possible out of order. You can't label fills as AI if the load doesn't keep a city_source column. You can't measure blanks honestly if someone set ignoreNull to quiet the alert. You can't grade a judge if you can't tell its rows from the source rows. Each step reads from the one before it, so the order isn't a recommendation. It's the product.

def which_step ( symptom ): return { " someone reruns the load by hand ": " 1. Automate the load ", " corrections live in a spreadsheet ": " 1. Automate the load ", " one value, several spellings ": " 1.

Automate the load (alias table) ", " blanks nobody can fill ": " 2. Check quality, then 3. Backfill ", " a quality check that always passes ": " 2.

Check quality (how are nulls counted?) ", " AI-filled values you can ' t tell apart ": " 3. Backfill, label it as AI ", " a judge score nobody has checked ": " 4. Judge, then grade the judge ", " all of the above ": " 1.

Start at the load ", }[ symptom ] print ( " \n == Which step, when you inherit X == " ) for s in [ " someone reruns the load by hand ", " corrections live in a spreadsheet ", " one value, several spellings ", " blanks nobody can fill ", " a quality check that always passes ", " AI-filled values you can ' t tell apart ", " a judge score nobody has checked ", " all of the above " ]: print ( f " { s: < 40 } -> { which_step ( s ) } " ) Enter fullscreen mode Exit fullscreen mode == Which step, when you inherit X == someone reruns the load by hand -> 1. Automate the load corrections live in a spreadsheet -> 1. Automate the load one value, several spellings -> 1.

Automate the load (alias table) blanks nobody can fill -> 2. Check quality, then 3. Backfill a quality check that always passes -> 2.

Check quality (how are nulls counted?) AI-filled values you can't tell apart -> 3. Backfill, label it as AI a judge score nobody has checked -> 4. Judge, then grade the judge all of the above -> 1.

Start at the load Enter fullscreen mode Exit fullscreen mode The principle An AI fill you can't tell from a source value is a hand edit at scale.

What does your team still fix by hand every week?

The data in this article is synthetic and the judge scores are stand-ins with their assumptions in the code. Treat it as a teaching starting point, not a turnkey way to decide which values to trust.

I'm Vinicius Fagundes, an independent principal data engineer and MBA lecturer in São Paulo. I write about the data layer underneath the decisions at vf-insights.com.