Version Your AI Feature: Prompts and Evals Before Model Config

You shipped the AI feature. The model name is pinned in a Slack thread. The prompt that actually drives behavior lives in a random PR — or worse, only in the running container.

Support tickets get worse summaries. The agent starts inventing policies. Latency looks fine. Cost looks fine. Nobody can answer the only useful question: what changed?

If you cannot diff the prompt, you cannot debug the drop.

Model swaps are noisy. Someone posts "we moved to X," configs update, dashboards spike, people notice.

Prompt edits are silent. A one-line tweak to the system message can rewrite product behavior with no deploy note, no tagged release, and no rollback button. Eval datasets are the same story: if the golden examples live in a spreadsheet on someone's laptop, you have no regression signal when the prompt drifts.

Treat the prompt as code. Treat the eval set as the test suite. Treat model and provider config as infrastructure-as-code that comes after those two.

AWS Prescriptive Guidance puts it plainly: prompts are critical software artifacts that need versioning, a store or registry, PR-style review, and environment promotion — not ad-hoc text files. See Managing the generative AI prompt lifecycle.

You do not need a platform rewrite. You need three artifacts next to the feature.

Keep the prompt text in the repo so every change is a diff and every release is a tag. If you outgrow flat files, use a prompt registry that still versions commits — Langfuse documents organizing, versioning, and testing prompts at scale in Prompt management at scale.

A small JSONL (or YAML) set of inputs plus expected properties is enough to start. Run it on every prompt PR. Grow it from production failures. Without this, "looks better" is your only test.

Pin model IDs, temperature, timeouts, and fallbacks in a config file the app loads. Easy to log. Easy to roll back. Still secondary to prompts and evals — because swapping the model without a fixed prompt and eval suite just moves the noise around.

Same bar as application code: reviewer, CI check, merge. Langfuse's Prompt CI/CD write-up frames the loop as version → gate → rollout. That is the right mental model even if your "registry" is just a folder.

Something this simple already beats Slack archaeology:

ai/ prompts/ support_summarizer.v3.md evals/ support_summarizer.jsonl config/ support_summarizer.yaml Enter fullscreen mode Exit fullscreen mode support_summarizer.v3.md — the system/user template, with a short changelog header.

support_summarizer.jsonl — one object per line: input ticket, expected fields, maybe a rubric label.

support_summarizer.yaml — model id, provider, temperature, max tokens, fallbacks.

On a prompt PR, CI loads v3 + the JSONL, scores the run, and fails the build if recall/precision/rubric drops past a threshold. Reviewers read the prompt diff the way they read a service change — because it is a service change.

{ "id": "t-1042", "input": "Customer wants refund after 45 days; policy is 30.", "expect":{ "tone": "firm_empathetic", "must_cite_policy": true, "must_not_invent_exception": true }} Enter fullscreen mode Exit fullscreen mode Example config:

feature: support_summarizer prompt: prompts/support_summarizer.v3.md eval: evals/support_summarizer.jsonl model: primary: openai/gpt-4.1-mini fallback: anthropic/claude-haiku-4-5 temperature: 0.2 Enter fullscreen mode Exit fullscreen mode Bump the prompt version when behavior changes. Keep the eval set growing from real misses. Log the model string every call so ops can still correlate provider incidents — without pretending the model string is the product.

Prompt only in application code strings. Hard to review. Hard to A/B. Easy to "fix" in a hotfix and forget.

Evals that never ship. A notebook on one laptop is not a gate. Put the file in the repo and run it in CI.

Versioning the model first. Useful, but incomplete. You can pin gpt-4.1-mini forever and still ship a silent prompt regression on Tuesday.

No ownership. If anyone can edit production prompts without a PR, you will eventually ship a Monday morning mystery.

Pick one AI feature. Move its prompt into prompts/. Add 20–50 golden examples from the last month of complaints. Wire a cheap CI job that fails on a clear regression. Require a human on the prompt PR.

That is not bureaucracy. That is how you get a diff when quality drops.

What's the first artifact you put under version control for an AI feature?

My lean: A and B together. Model IDs are easy to log later. If you cannot diff the prompt, you cannot debug the drop.

Which letter matches your team today — and what broke the last time quality slipped?