Postmortem: The Tuesday Our AI Bill Had No Owner

I've read a lot of postmortems. The best ones aren't about blame. They're about the gap between what happened and what the system let you see.

This one is about that gap. Specifically, the gap between "our AI bill went up" and "here's exactly why."

This is a composite of patterns I've seen across platform teams, written in the standard blameless format. The names are made up. The lesson isn't.

At 07:40 on a Tuesday, our daily cost alert fired. Vertex AI spend in the shared platform project had roughly tripled overnight.

Nine services call Gemini from that project. All of them use the same service account. All of them call the same model.

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

The root cause turned out to be a retry loop in the review step of a nightly ticket summarization workflow. Finding it took over two hours of Slack threads, log digging, and educated guessing.

The deeper root cause: none of our model calls carried request labels. We had cost data, but no way to slice it.

Monday, 16:30 A developer ships prompt_version=v4 for the ticket summarization workflow. The new prompt asks the review step to be stricter about tone.

Monday, 23:00 Nightly ticket summarization job starts. The stricter reviewer starts rejecting drafts. Each rejection triggers a rewrite with the full ticket history plus the previous draft plus review notes. Context grows on every loop.

07:52 On-call SRE opens Cloud Billing. Filters by project. Sees one project, one service (Vertex AI), one SKU group. No breakdown below that.

08:15 Slack thread in #platform: "Did anyone ship anything AI-related yesterday?"

08:40 Three teams reply "no." One replies "maybe? we updated a prompt."

09:05 SRE starts grepping application logs across services for model call volume. Logs are inconsistent. Some services log token counts, most don't.

09:30 A developer notices the summarization job ran for six hours instead of the usual forty minutes.

09:54 Root cause confirmed: review step looping on rejections, with context growing each time.

10:10 Prompt rolled back to v3. Retry cap lowered. Job rerun successfully.

Why did spend triple? The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

Why didn't we see it during the night? Our only cost signal was the daily billing alert. Billing data isn't real-time, so it can't catch a loop while it's happening.

Why did root cause take two hours? Billing showed one number for the whole project. Nothing told us which service, workflow, or step was responsible.

Why couldn't billing break it down? None of our Vertex AI calls carried request labels.

Why weren't we labeling? We label infrastructure (VMs, buckets, clusters) through Terraform. Model API calls have no resource to label, so they fell through the cracks. Nobody owned the gap.

Before the action items, here's the feature we were missing. If you already know it, skip to section 8.

Naming note: Vertex AI docs are moving under the Gemini Enterprise Agent Platform name. If a docs link redirects there, it's the same feature.

In Google Cloud, labels live at three layers:

Request labels are key-value pairs you attach to each model call. Vertex AI forwards them to Cloud Billing, where you can filter and group costs in billing reports and query them in the BigQuery billing export.

Here's the simplest possible Gemini call with labels:

from google import genai from google.genai import types client = genai. Client ( vertexai = True, project = " your-project-id ", location = " global " ) response = client. models. generate_content ( model = " YOUR_GEMINI_MODEL ", contents = " Summarize this support ticket in two lines. ", config = types. GenerateContentConfig ( labels = { " team ": " support ", " env ": " prod ", " use_case ": " ticket_summary ", } ), ) Enter fullscreen mode Exit fullscreen mode That's it. Three labels would have turned our two-hour investigation into one query.

Each one is below, with the actual implementation.

The first temptation is to let every team pick their own labels. Don't. Six months later you'll have team=Support, team=support, team=support-team, and team=cx all meaning the same thing.

Notice that step and prompt_version alone would have pointed straight at our root cause: step=review, prompt_version=v4.

Two reasons. Cardinality: billing data is aggregated, and a label with a million unique values makes every report useless and your queries expensive. Visibility: label values land in billing data, and anyone with billing viewer access can read them.

Our rule: if a label can have more than about 100 values, it belongs in logs, not billing.

And we treat the taxonomy like an API contract. It's written down, versioned, and changes go through review. Renaming a key later breaks every historical query.

Application code no longer calls the model directly. Every call goes through one wrapper that merges default labels, validates required keys, sanitizes values, and logs token usage.

# llm_client.py import json import re import contextvars from contextlib import contextmanager from google import genai from google.genai import types REQUIRED_KEYS = ( " team ", " env ", " use_case " ) _bad_chars = re. compile ( r " [^a-z0-9_-] " ) _ctx_labels = contextvars. ContextVar ( " llm_labels ", default = {}) def _clean ( value: str ) -> str: return _bad_chars. sub ( " _ ", str ( value ). strip (). lower ())[: 63 ] def _clean_key ( key: str ) -> str: key = _clean ( key ) return key if key [: 1 ]. isalpha () else f " k_ { key } " [: 63 ] @contextmanager def llm_labels ( ** labels ): """ Attach labels to every model call made inside this block. """ token = _ctx_labels. set ({ ** _ctx_labels. get (), ** labels }) try: yield finally: _ctx_labels. reset ( token ) class LabeledClient: def __init__ ( self, project: str, location: str, defaults: dict ): self. _client = genai.

Client ( vertexai = True, project = project, location = location ) self. _defaults = defaults def generate ( self, model: str, contents, labels: dict | None = None, config: types. GenerateContentConfig | None = None ): merged = { ** self. _defaults, ** _ctx_labels. get (), ** ( labels or {})} missing = [ k for k in REQUIRED_KEYS if k not in merged ] if missing: raise ValueError ( f " Missing required LLM labels: { missing } " ) merged = { _clean_key ( k ): _clean ( v ) for k, v in merged. items ()} config = config or types. GenerateContentConfig () config. labels = { ** ( config. labels or {}), ** merged } response = self. _client. models. generate_content ( model = model, contents = contents, config = config ) _log_usage ( model, merged, response ) return response def _log_usage ( model: str, labels: dict, response ): usage = response. usage_metadata print ( json. dumps ({ " message ": " llm_usage ", " model ": model, " labels ": labels, " prompt_tokens ": getattr ( usage, " prompt_token_count ", None ) or 0, " output_tokens ": getattr ( usage, " candidates_token_count ", None ) or 0, " cached_tokens ": getattr ( usage, " cached_content_token_count ", None ) or 0, " total_tokens ": getattr ( usage, " total_token_count ", None ) or 0, })) Enter fullscreen mode Exit fullscreen mode Three design decisions worth explaining:

llm = LabeledClient ( project = " your-project-id ", location = " global ", defaults = { " team ": " support ", " env ": " prod ", " app ": " helpdesk " }, ) response = llm. generate ( model = " YOUR_GEMINI_MODEL ", contents = " Summarize this support ticket in two lines. ", labels = { " use_case ": " ticket_summary ", " prompt_version ": " v3 " }, ) Enter fullscreen mode Exit fullscreen mode AI-3: Propagate labels through multi-agent workflows Our summarization workflow has four steps across three agents. Passing a labels dict through every function signature would never survive code review.

The llm_labels context manager in the wrapper solves this. Set labels once at the top, and every call inside inherits them:

with llm_labels ( use_case = " ticket_summary ", prompt_version = " v4 ", agent = " orchestrator " ): plan = llm. generate ( model = " YOUR_GEMINI_MODEL ", contents = plan_prompt, labels = { " step ": " plan " }) with llm_labels ( agent = " writer_agent " ): draft = llm. generate ( model = " YOUR_GEMINI_MODEL ", contents = draft_prompt, labels = { " step ": " draft " }) with llm_labels ( agent = " reviewer_agent " ): review = llm. generate ( model = " YOUR_GEMINI_MODEL ", contents = review_prompt, labels = { " step ": " review " }) Enter fullscreen mode Exit fullscreen mode contextvars works correctly with asyncio, so this holds up when agents run concurrently.

If this had been in place on Monday night, the billing breakdown would have shown one line towering over the rest: agent=reviewer_agent, step=review, prompt_version=v4. Investigation over.

Some of our teams also call partner models through Vertex AI. For those, labels travel as a base64-encoded JSON header. Here's a Claude call as the example:

import base64 import json import google.auth import google.auth.transport.requests import requests PROJECT = " your-project-id " MODEL = " YOUR_CLAUDE_MODEL " creds, _ = google. auth. default ( scopes = [ " https://www.googleapis.com/auth/cloud-platform " ]) creds. refresh ( google. auth. transport. requests. Request ()) labels = { " team ": " support ", " env ": " prod ", " use_case ": " ticket_summary " } labels_header = base64. b64encode ( json. dumps ( labels ). encode ( " utf-8 " )). decode ( " utf-8 " ) url = ( f " https://aiplatform.googleapis.com/v1/projects/ { PROJECT } " f " /locations/global/publishers/anthropic/models/ { MODEL }:rawPredict " ) resp = requests. post ( url, headers = { " Authorization ": f " Bearer { creds. token } ", " Content-Type ": " application/json ", " X-Vertex-AI-Labels ": labels_header, }, json = { " anthropic_version ": " vertex-2023-10-16 ", " max_tokens ": 512, " messages ": [{ " role ": " user ", " content ": " Summarize this support ticket in two lines. " }], }, ) Enter fullscreen mode Exit fullscreen mode Two things to watch:

We folded this into the same wrapper so app code never needs to know which path it's on.

A wrapper only helps if people use it. Five lines of bash close the gap:

#!/usr/bin/env bash # Fail the build if anyone calls the model directly instead of through LabeledClient. if grep -rn "models.generate_content(" --include = "*.py" src/ | grep -v "llm_client.py"; then echo "Direct model calls found. Use LabeledClient from llm_client.py so labels are attached." exit 1 fi Enter fullscreen mode Exit fullscreen mode It's crude. It works. If you want something smarter later, swap it for an AST-based lint rule. Ship the crude one first.

The billing console is fine for a quick look. For real analysis, we use the BigQuery billing export. The standard export table is named gcp_billing_export_v1_<BILLING_ACCOUNT_ID>, and labels live in an array of key-value structs.

Query 1: Label coverage. This is the first number we put on a dashboard. What percentage of AI spend actually carries labels?

SELECT DATE ( usage_start_time ) AS day, ROUND ( SUM ( cost ), 2 ) AS total_cost, ROUND ( SUM ( IF ( EXISTS ( SELECT 1 FROM UNNEST ( labels ) WHERE key = 'team' ), cost, 0 )), 2 ) AS labeled_cost, ROUND ( 100 * SAFE_DIVIDE ( SUM ( IF ( EXISTS ( SELECT 1 FROM UNNEST ( labels ) WHERE key = 'team' ), cost, 0 )), SUM ( cost ) ), 1 ) AS coverage_pct FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` WHERE usage_start_time >= TIMESTAMP_SUB ( CURRENT_TIMESTAMP (), INTERVAL 30 DAY ) AND service. description LIKE '%Vertex AI%' GROUP BY day ORDER BY day DESC; Enter fullscreen mode Exit fullscreen mode On the day of the incident, ours was effectively 0%. Our target is 95% in prod.

Query 2: Net cost by team, use case, and step. The query we wish we'd had at 07:52.

WITH ai AS ( SELECT DATE ( usage_start_time ) AS day, ( SELECT value FROM UNNEST ( labels ) WHERE key = 'team' ) AS team, ( SELECT value FROM UNNEST ( labels ) WHERE key = 'use_case' ) AS use_case, ( SELECT value FROM UNNEST ( labels ) WHERE key = 'step' ) AS step, ( SELECT value FROM UNNEST ( labels ) WHERE key = 'env' ) AS env, cost + IFNULL (( SELECT SUM ( c. amount ) FROM UNNEST ( credits ) c ), 0 ) AS net_cost FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` WHERE usage_start_time >= TIMESTAMP_SUB ( CURRENT_TIMESTAMP (), INTERVAL 30 DAY ) -- Check how AI usage appears in your export first: -- SELECT DISTINCT service.description FROM... and adjust this filter. AND service. description LIKE '%Vertex AI%' ) SELECT day, team, use_case, step, env, ROUND ( SUM ( net_cost ), 2 ) AS net_cost FROM ai GROUP BY day, team, use_case, step, env ORDER BY day DESC, net_cost DESC; Enter fullscreen mode Exit fullscreen mode Note the credits line. The cost column doesn't include credits, so skip that and your numbers won't match the invoice.

Query 3: Unit economics. Join labeled cost with your own business metrics:

SELECT a. day, ROUND ( SUM ( a. net_cost ), 2 ) AS ai_cost, t. tickets_closed, ROUND ( SAFE_DIVIDE ( SUM ( a. net_cost ), t. tickets_closed ), 4 ) AS cost_per_ticket FROM ai AS a -- the CTE from Query 2 JOIN `your_project.app_metrics.daily_tickets` AS t USING ( day ) WHERE a. use_case = 'ticket_summary' AND a. env = 'prod' GROUP BY a. day, t. tickets_closed ORDER BY a. day DESC; Enter fullscreen mode Exit fullscreen mode daily_tickets is your own table. This is the query that turns an AI bill into a business conversation. "AI is expensive" gets projects cancelled. "This feature costs us a fraction of a cent per ticket" gets budgets approved.

This is the action item that would have caught the loop at 23:30 instead of 07:40.

Billing data is not real-time. Usage reaches Cloud Billing at varying intervals, and the BigQuery export updates throughout the day. That's fine for chargeback. It's useless for catching a runaway loop in the middle of the night.

The fix: emit your own usage metrics with the same labels, in real time, and reconcile against billing daily.

The wrapper already logs usage_metadata as structured JSON. On Cloud Run or GKE, JSON on stdout becomes a structured log entry automatically. From there, we created a log-based metric:

# llm_tokens_metric.yaml filter: ' jsonPayload.message="llm_usage"' metricDescriptor: metricKind: DELTA valueType: DISTRIBUTION unit: " 1" labels: - key: use_case - key: team - key: env - key: step valueExtractor: ' EXTRACT(jsonPayload.total_tokens)' labelExtractors: use_case: ' EXTRACT(jsonPayload.labels.use_case)' team: ' EXTRACT(jsonPayload.labels.team)' env: ' EXTRACT(jsonPayload.labels.env)' step: ' EXTRACT(jsonPayload.labels.step)' bucketOptions: exponentialBuckets: numFiniteBuckets: 32 growthFactor: 2 scale: 1 Enter fullscreen mode Exit fullscreen mode gcloud logging metrics create llm_total_tokens --config-from-file = llm_tokens_metric.yaml Enter fullscreen mode Exit fullscreen mode Now we have a token metric we can chart and alert on, sliced by the exact same dimensions as the bill.

If you already run OpenTelemetry, do the same with a counter and put the labels on as attributes. Same idea, different pipe.

Daily reconciliation. Compare logged token totals against the billing export for the same labels. If they drift more than a few percent, something is calling the model outside the wrapper. That drift check is our second line of defense after CI.

Not a labels fix, but it would have limited the blast radius. Every review loop now has a hard ceiling on rewrites, and the reviewer gets a trimmed context on retries instead of the full growing history. When the cap is hit, the job flags the ticket for a human instead of trying forever.

Labels tell you where the fire is. Retry caps keep the fire small.

We didn't flip a switch. Here's how we rolled it out without breaking prod:

Labels aren't magic. Here's what's still true after all of the above:

We were labeling the wrong layer. Our Terraform tagged every VM and bucket perfectly. But our fastest-growing cost line had no resource to tag. When the shape of your spend changes, your attribution has to change with it.

Cost is a reliability signal. A loop that burns tokens is a bug, even when the output is eventually correct. It deserves real-time alerting, not a next-morning email.

Attribution is a developer job. Only code can attach labels. FinOps can define the taxonomy and SRE can build the alerts, but if developers don't own the labels, nothing else works.

The best postmortem action items are boring. One wrapper. Three required labels. One coverage query. None of it is exciting. All of it would have turned a two-hour investigation into a five-minute one.

Your turn: if your AI bill spiked tomorrow morning, how long would it take your team to explain it? Five minutes? Two hours? Never? Tell me in the comments. 👇