Do you log your LLM prompts and responses? Four answers, and the one I'd pick

Friday AI Question. This week's question is one every team shipping an LLM feature has to answer, usually after the first weird bug report:

Do you log your LLM prompts and responses?

Drop your letter in the comments. Here's how I think about each one.

An LLM feature fails in ways a status code won't show you. The request succeeds, the answer is wrong, and the only evidence is the text that went in and the text that came out. If you never kept that text, you can't reproduce the bug, you can't build an eval set from real traffic, and you can't explain to a user why the bot said what it said.

But prompts are also where users paste the things they shouldn't: names, emails, account numbers, contract text, health details. The OWASP Top 10 for LLM Applications lists Sensitive Information Disclosure as one of its top risks, and your logs are one more place that information can leak from. Privacy laws such as the GDPR also include a storage limitation principle: personal data shouldn't be kept longer than you need it for its purpose.

So the real question isn't "log or don't log". It's "what do you keep, for how long, and who can read it".

Good: nothing to leak, nothing to delete, an easy privacy story.

Bad: you're debugging blind. Every "the bot gave a weird answer" ticket becomes guesswork, and you have no real-world examples to turn into evals. Fine for a prototype; hard to defend for a feature people depend on.

Good: you can still watch cost, latency, error rates and stop reasons, and spot that something changed. This is a solid baseline for every LLM app.

Bad: you'll know that quality dropped but not why. Metadata tells you a request was long and expensive; it doesn't tell you the retrieved context was the wrong document.

Good: you can reproduce real failures, read traces end to end, and sample production traffic into an eval set, while limiting the blast radius.

Bad: it takes real work: a redaction step, a retention job, separate access controls. Redaction is never perfect, so treat the store as sensitive anyway.

Good: maximum debuggability, zero effort.

Bad: every customer secret anyone ever pasted now lives in your log platform, readable by anyone with log access, kept long after it's useful. That's the option that turns a small incident into a big one.

Start with B on day one, because cost and latency metrics are cheap and always useful. Move to C as soon as the feature has real users, because you can't improve what you can't read. Avoid D.

LLM observability tools (Langfuse, for example, which I covered in an earlier post) make the "read traces end to end" part easy. The policy part (what to mask, how long to keep it, who can see it) is still your decision.

So, A, B, C or D? And if you picked C, how many days do you keep prompt logs?