AI Agent Observability: How to Trace Multi-Turn Conversations, Tool Calls and Agent Workflows" published: false

AI agent observability is the practice of recording what an AI agent received, did and returned across a whole conversation, including model calls, tool calls, retrieved context, token usage, latency and errors, so that a team can debug the agent, control its cost and verify its behavior. This guide explains what to record, why the session is the right unit of analysis, and how to keep traces trustworthy. Why isn’t traditional monitoring enough for AI agents? A traditional application follows a fixed path: a request reaches a service, the service reads a database, and a response comes back. An AI agent does not. One user request can trigger several model calls, external tools, knowledge retrieval and even other agents, and the path changes from one run to the next.

User request → Agent planning → LLM call → Tool execution → Knowledge retrieval → Another agent → Final response

Traditional monitoring answers “Is my application working?” LLM observability for agents has to answer a harder question: “Why did the agent give this answer?” The final answer alone does not explain what happened. When an agent fails, a team needs to see what information it received, which tools it used, what those tools returned and which decisions the model made. Why trace at the session level?

A single conversation can contain several turns, and each turn can contain model calls, tool calls, retrieval steps and agent decisions. If you only look at individual requests, you see isolated model calls. You cannot see that the third turn was slow because the second turn’s tool returned a very large payload, or that cost climbed because the agent kept retrying.

Both the performance problem and the cost problem live in the sequence. Session-level tracing connects these steps into one workflow. The attributes that make it work are a session ID, a turn index, parent/child links between steps, and hand-off markers when one agent calls another.

What should an AI agent trace record? Model calls. Record the model name and version, generation parameters such as temperature, input and output tokens, latency, finish reason, errors and retries.

Without these, a team cannot explain why behavior changed between two runs. Tool calls. Record the tool name and version, the arguments (redacted where needed), the result status and size, and the duration.

Tools such as database queries, search systems and APIs are a critical part of most agents. Cost attribution. Record tags for the model, application, team or business line, and environment.

With them you can answer which model uses the most tokens, which workflow is expensive and which agent creates unexpected cost. Versions. Record the prompt version, model version, framework version and tool version.

A small update to any of them can change the final answer. Content policy. Traces can contain sensitive data, so decide for every field whether to store it, sample it, redact it or never store it.

Operational metadata such as timings and token counts is usually safe to store. Secrets and personal data should not be stored. How do you keep traces trustworthy?

Trust in an AI system is built on evidence. A read-only tool call, such as a database query, should return the same data for the same input, as long as the underlying data has not changed. Model output varies from run to run.

If a trace records tool results and model output separately, a team can tell whether a wrong answer came from bad data, a failing tool or the model’s reasoning. That separation is what makes an agent’s behavior verifiable. Do frameworks change what you record?

The checklist is the same whichever framework you use, but the effort of collecting it is not. Teams build agents in different ways: in code with LangChain, as stateful graphs with LangGraph, on low-code platforms such as Dify, or on frameworks such as OpenClaw. If every framework needs its own tracing setup, session traces become patchy.

When you choose tooling, check that the frameworks your teams actually use are covered, and use shared attribute names, such as the OpenTelemetry semantic conventions for generative AI, so traces stay portable. From AI agent observability to AI agent operations Observability tells a team what happened. The next step is helping teams operate systems with the help of AI agents that use observability data to diagnose incidents, inspect system health, analyze database performance and assess capacity.

Disclosure: I work at Bonree (博睿数据). Bonree ONE is our AI-driven observability platform. Its AI observability capability covers session-level tracing and token cost analysis for AI applications, and collects data from applications built on LangChain, LangGraph, Dify and OpenClaw.

Bonree ONE·Sage AI is the intelligent operations workspace in Bonree ONE, with ready-to-use agents for scenarios such as incident diagnosis and database performance analysis, and support for creating custom agents. The checklist in this article does not depend on any product. FAQ What is the difference between LLM observability and AI agent observability?

LLM observability focuses on individual model calls. AI agent observability also follows tool calls, retrieval, hand-offs and the whole multi-turn session.Should I store every prompt and output? No.

Decide per field. Sampling or redaction is a common default, and secrets and personal data should not be stored.Why record tool results separately from model output? Read-only tool results are repeatable and model output is not, so separating them shows whether a failure came from data, tools or reasoning.Is there a ready-made checklist?

Yes. The GitHub repository ai-agent-observability-checklist contains the full checklist, a content capture policy and an example trace.