Langfuse — Stop Debugging LLM Apps Blind

Your support RAG returned a confident answer from the wrong policy doc.

If you have no traces, the postmortem is a Slack thread: "model issue?", "prompt drift?", "embeddings?", "try GPT-whatever." That is not debugging. That is guessing with an API key.

Langfuse is open-source LLM observability. You get traces (one user request), nested spans (each pipeline step), generations (LLM calls with tokens/cost/latency), and scores you attach when a human or eval marks the answer good or bad. The goal is simple: open the bad request and see which step failed.

Without instrumentation, production only shows you step 4. Langfuse makes 1–3 visible in the same tree, with timing and cost on every LLM call.

pip install langfuse openai Enter fullscreen mode Exit fullscreen mode Set env vars (Cloud or self-hosted — same SDK):

export LANGFUSE_PUBLIC_KEY = "pk-lf-..." export LANGFUSE_SECRET_KEY = "sk-lf-..." export LANGFUSE_BASE_URL = "https://cloud.langfuse.com" # EU # US: https://us.cloud.langfuse.com # JP: https://jp.cloud.langfuse.com Enter fullscreen mode Exit fullscreen mode Optional sanity check (not for hot paths):

from langfuse import get_client get_client (). auth_check () Enter fullscreen mode Exit fullscreen mode Docs: OpenAI integration (Python), SDK overview.

If you already use the OpenAI Python SDK, Langfuse's drop-in records prompts, completions, usage, cost, latency, and API errors by changing the import:

# before: import openai from langfuse.openai import openai # or: from langfuse.openai import OpenAI, AsyncOpenAI completion = openai. chat. completions. create ( model = " gpt-4o-mini ", messages = [ { " role ": " system ", " content ": " Answer only from the context. " }, { " role ": " user ", " content ": " What is the refund window? " }, ], name = " support-answer ", # shows up as the generation name in Langfuse ) print ( completion. choices [ 0 ]. message. content ) Enter fullscreen mode Exit fullscreen mode You can also pass Langfuse attributes via metadata (e.g. langfuse_session_id, langfuse_user_id, langfuse_tags ) on the same call. See the custom trace properties section in the docs.

A single OpenAI call becomes one generation. A RAG pipeline needs a trace tree. Decorate each step:

from langfuse import observe, get_client, propagate_attributes from langfuse.openai import openai @observe ( name = " retrieve " ) def retrieve ( query: str, k: int = 4 ) -> list [ dict ]: # your vector store / BM25 / hybrid search return search_chunks ( query, k = k ) @observe ( name = " build_prompt " ) def build_prompt ( query: str, chunks: list [ dict ]) -> list [ dict ]: context = " \n\n ". join ( c [ " text " ] for c in chunks ) return [ { " role ": " system ", " content ": " Answer only from the context. If missing, say you don ' t know. " }, { " role ": " user ", " content ": f " Context: \n { context } \n\n Question: { query } " }, ] @observe ( name = " rag_answer " ) def rag_answer ( query: str, user_id: str, session_id: str ) -> str: with propagate_attributes ( user_id = user_id, session_id = session_id, tags = [ " support-rag " ], metadata = { " pipeline ": " support-rag-v2 " }, ): chunks = retrieve ( query ) messages = build_prompt ( query, chunks ) completion = openai. chat. completions. create ( model = " gpt-4o-mini ", messages = messages, name = " generate ", ) return completion. choices [ 0 ]. message. content answer = rag_answer ( " What is the refund window for annual plans?

", user_id = " u_42 ", session_id = " s_99 ", ) # Short-lived scripts / serverless: flush before exit get_client (). flush () Enter fullscreen mode Exit fullscreen mode Nested @observe calls become parent/child spans automatically. The OpenAI drop-in generations nest under the active observation. Instrumentation docs: Observe wrapper.

1. Reproduce or open the failing request in Langfuse

Find the trace by user_id / session_id / time window. You should see something like:

If generate looks fluent but wrong, do not start by swapping models. Open retrieve. Check:

In the wrong-policy case, the generate span is innocent. The retrieve span shows high-confidence chunks from an older refund FAQ. Root cause: stale index or missing metadata filter — not "the LLM hallucinated."

3. Attach a score so the failure is searchable

Inside the active observation (or afterward via create_score with trace_id ):

from langfuse import get_client langfuse = get_client () # While still inside an @observe / start_as_current_observation context: langfuse. score_current_span ( name = " retrieval_relevance ", value = 0.2, data_type = " NUMERIC ", comment = " Top chunks from outdated refund FAQ; annual-plan policy missing ", ) langfuse. score_current_trace ( name = " answer_correctness ", value = 0, data_type = " BOOLEAN ", comment = " Wrong refund window cited ", ) Enter fullscreen mode Exit fullscreen mode Score types include NUMERIC, CATEGORICAL, BOOLEAN, and TEXT. Full API: Scores via SDK.

4. Fix the span that failed, then re-run the same query

New trace should show better retrieve output and a passing score. That pair — before/after traces with scores — is your regression evidence for the next prompt or index change.

When a production AI answer fails, do you open a trace — or start a Slack thread of guesses?