Mastering Datadog AI Agent Observability: A Beginners Guide

When something goes wrong, identifying the actual cause isn’t always straightforward due to non-deterministic nature of LLM applications.

Adding logs helps, but logs alone often aren’t enough for LLM applications.

You might have no visibility of the complete workflow if you are using logs alone.

This is where Datadog Agent Observability comes in.

Datadog gives you end-to-end visibility into your AI application’s execution.

Instead of guessing what happened, you can trace a user’s request through out the complete execution flow.

Implementing Datadog agent observability involves 3 core components:

⚪ Traces — shows the complete journey of a single request.

⚪🟡 Spans — represent individual operations within a trace.

⚪🟡⚪ Instrumentation is the process of capturing this information automatically or manually and sending it to Datadog for analysis.

Press enter or click to view image in full size

A trace represents the complete execution path of a single request handled by the LLM application.

A trace captures everything that happens while the application processes that request, including LLM calls, agent decisions, tool calls, and other operations.

A trace can contain multiple spans, where each span represents an individual operation within the overall request.

An agent trace groups together all workflows and contextual operations such as the tool calls and reasoning steps executed by an agent to complete a user request.

"What's the weather in Nairobi?" Enter fullscreen mode Exit fullscreen mode The AI agent might:

All of these operations can be represented within the same trace.

TRACE ──────────────────────────────── trace_id: trace_001 ──────────────────────────────── Agent / Workflow Span span_id: span_001 trace_id: trace_001 │ ├── LLM Span │ span_id: span_002 │ trace_id: trace_001 │ │ Input: │ "What's the weather in Nairobi?" │ │ Output: │ "I need to check the weather." │ ├── Tool Span │ span_id: span_003 │ trace_id: trace_001 │ │ Input: │ { │ "location": "Nairobi" │ } │ │ Output: │ { │ "temperature": 23, │ "condition": "Partly cloudy" │ } │ └── LLM Span span_id: span_004 trace_id: trace_001 Input: Weather data from tool Output: "It's 23°C and partly cloudy in Nairobi." ──────────────────────────────── Enter fullscreen mode Exit fullscreen mode TRACE SAMPLING Trace Sampling is a method datadog uses to determine the number of traces to retain for observability.

Agent Observability billing is based on the volume of spans sent.

Setting a sample rate is a direct way to control the cost for agent obervability.

Suppose your application receives: 10,000 agent requests And each request generates approximately: 10 spans You could therefore send to DataDog approximately: 10,000 × 10 = 100,000 spans Enter fullscreen mode Exit fullscreen mode Since Agent Observability billing is based on the volume of spans sent, this can become expensive at scale.

You can configure the sample rate (a float between 0.0 and 1.0)

sample rate = 0.5 Enter fullscreen mode Exit fullscreen mode The above sample rate means approximately 50% of traces (50,000) will be retained instead of 100,000.

While trace sampling allows you to control the ingestion volume and storage costs for your individual traces and spans, it does not impact these operational metrics

A span represents a single operation in a trace.

A span serves as the fundamental building block of a trace

A span attribute describes the operation represented by the span.

Span attributes represent the content of the work being done within the span

For example: Suppose the span represents this operation:

SELECT * FROM users WHERE id = 123; Enter fullscreen mode Exit fullscreen mode The span could contain the following span attributes:

@db.system:postgres @db.operation:SELECT @db.table:users Enter fullscreen mode Exit fullscreen mode WHERE DO SPAN ATTRIBUTES COME FROM? Automatic instrumentation Datadog’s APM libraries can automatically collect information.

This means there is no need to write code to record values for such fields.

For example, your application makes an HTTP request:

GET /products Enter fullscreen mode Exit fullscreen mode The instrumentation may automatically record things like:

@http.method: GET @http.route: /products Enter fullscreen mode Exit fullscreen mode 2. Manual/custom instrumentation This means we have to explicitly specify the fields to record.

This is useful because it allows you to put business-specific information into your traces.

For example: Suppose your application calculates the shopping cart value:

cart_total = 1299.99 Enter fullscreen mode Exit fullscreen mode You might instrument your application so the span contains:

@cart.total:1299.99 Enter fullscreen mode Exit fullscreen mode Example:

from ddtrace import tracer # Get the current active span span = tracer. current_span () if span: # Manually set a custom attribute span. set_tag ( ' cart.total ', 1299.99 ) Enter fullscreen mode Exit fullscreen mode RESERVED SPAN ATTRIBUTES Datadog has certain reserved span attributes

Reserved span attributes are automatically instrumented and are available on every span.

You do not need to perform any manual setup to ensure these attributes are collected for your traces:

env: environment name (e.g. production) where the service is running.

service: The name of the service that generated the span.

operation_name: The specific name of the operation being performed.

resource_name: A human-readable name for the resource associated with the operation (e.g., an endpoint path or a database query).

status: The operational status of the span (e.g., ok or error).

span_id: The unique identifier for that specific span.

trace_id: The unique identifier that groups all spans belonging to the same request or workflow.

timestamp: The exact time the span occurred.

type: The category of operation (e.g., web, db, or cache).

link: References connecting the span to other spans or traces.

ingestion_reason: The reason why the trace or span was retained by the Datadog ingestion pipeline.

When working with reserved attributes, don’t put @ in front.

service:checkout env:production status:error Enter fullscreen mode Exit fullscreen mode rather than:

@service:checkout @env:production Enter fullscreen mode Exit fullscreen mode 2. SPAN TAGS Span tags are contextual information about where the span came from for example infrastructure/environment context.

hostname:web-server-17 availability-zone:us-east-1a cluster-name:production-cluster container_name:api-frontend Enter fullscreen mode Exit fullscreen mode These tags provide context about the infrastructure or orchestration environment where the service is running

Annotations are additional pieces of context that you can attach to a span to make the trace more useful for analysis.

Annotations describe business-specific context about an operation.

This helps explain operations from a business perspective.

A checkout span could have annotations like customer_tier=premium or order_type=subscription.

Spans automatically capture common information such as: duration, status, errors, input, output

But sometimes you want to add your own context.

Example: Suppose you’re building a customer-support AI.

You may want to know additional information about each request, such as:

In Datadog, annotations are filtered using attribute search syntax, where attributes are prefixed with an @ symbol.

@customer_tier:premium @language:en @ticket_category:refund @department:electronics Enter fullscreen mode Exit fullscreen mode Using ddtrace, you can attach this business context to the current span with LLMObs.annotate():

from ddtrace.llmobs import LLMObs from ddtrace.llmobs.decorators import tool @tool def handle_support_request ( message, customer_id ): response = process_support_request ( message, customer_id ) LLMObs. annotate ( tags = { " customer_tier ": " premium ", " language ": " en ", " ticket_category ": " refund ", " department ": " electronics " } ) return response Enter fullscreen mode Exit fullscreen mode The resulting span now contains both its normal observability information and your additional business context:

Tool Span: handle_support_request ──────────────────────────────────────── Duration: 4.2 sec Status: OK Input: "I want a refund for my headphones." Output: "I'll help you process your refund..." Annotations: customer_tier: premium language: en ticket_category: refund department: electronics ──────────────────────────────────────── Enter fullscreen mode Exit fullscreen mode Using annotations, allow you to filter, group, and analyze traces based on your application’s specific business context.

For example, you could filter traces where:

@customer_tier:premium Enter fullscreen mode Exit fullscreen mode to investigate premium-customer requests.

Datadog allows spans to be categorized according to the type of operation they represent.

Instead of representing every operation as a generic span (Span), Datadog allows spans to be categorized according to the type of operation they represent:

An Agent is an AI system that can decide what actions it needs to take and in what order to accomplish a goal.

An Agent span represents the complete execution of an AI agent.

Example: Consider a travel-planning agent that helps a user plan a trip.

from ddtrace.llmobs.decorators import agent from openai import OpenAI llm = OpenAI () # Define tools def search_flights ( destination ): # Tool execution logic pass def search_hotels ( destination ): # Tool execution logic pass def check_weather ( destination ): # Tool execution logic pass @agent def plan_trip ( customer_query ): # The LLM analyzes the request and decides which tools to use llm_response = llm. chat. completions. create ( model = " gpt-4 ", messages = [{ " role ": " user ", " content ": customer_query }], tools = [ search_flights, search_hotels, check_weather ] ) return llm_response. choices [ 0 ]. message. content Enter fullscreen mode Exit fullscreen mode 2. Root Span A root span is defined by its position in the trace hierarchy.

A span is considered a root span, if it is the span at the very top, with no parent span.

An agent span may be the root span, particularly when the agent execution is the top-level operation you’re tracing.

But if the agent is itself invoked by another operation — such as a workflow — it can be a child span.

def travel_workflow ( customer_query ): validate_request ( customer_query ) plan_trip ( customer_query ) save_itinerary () return " Workflow complete " Enter fullscreen mode Exit fullscreen mode Conceptually, the trace looks like this:

ROOT SPAN: travel_workflow │ ├── Span: validate_request │ ├── Agent Span: plan_trip │ │ │ ├── LLM Span: understand_requirements │ │ │ ├── Tool Span: search_flights │ │ │ ├── Tool Span: search_hotels │ │ │ ├── LLM Span: compare_options │ │ │ ├── Tool Span: check_weather │ │ │ └── LLM Span: generate_itinerary │ └── Span: save_itinerary Enter fullscreen mode Exit fullscreen mode 3. Workflow Span A workflow span represents the execution of a complete, high-level process within an application.

A workflow in Agents Observability is a predefined sequence of operations that may include LLM calls, tool calls, data processing, and other steps required to complete a task.

A Workflow Span represents the complete execution of the workflow.

Unlike an Agent, which can autonomously decide which actions to take, a workflow generally follows a predefined sequence of operations defined by the application.

Example: Consider a customer support ticket workflow:

The workflow can be implemented using the @workflow decorator:

from ddtrace.llmobs.decorators import workflow, llm, tool, retrieval, task @tool def retrieve_customer_data ( customer_id ): # Logic to fetch customer information from an external database pass @llm def classify_ticket ( ticket ): # LLM call to categorize the support ticket pass @retrieval def search_knowledge_base ( category ): # Logic to search a vector database or external KB pass @llm def generate_response ( ticket, customer_data, knowledge ): # LLM call to generate the ticket response pass @task def send_response ( response ): # Logic to format/send the response (no external call) pass @workflow def process_support_ticket ( ticket ): customer_data = retrieve_customer_data ( ticket. customer_id ) ticket_category = classify_ticket ( ticket ) knowledge = search_knowledge_base ( ticket_category ) response = generate_response ( ticket, customer_data, knowledge ) send_response ( response ) return response Enter fullscreen mode Exit fullscreen mode 4. Task Span A task span represents an individual processing operation performed by an application that don’t call an external service.

It is typically used for application-level logic that happens within the application itself, rather than an operation that calls an external service.

Example: Consider an application that calculates a discount for a customer.

The task can be implemented using the @task decorator:

from ddtrace.llmobs.decorators import task @task def calculate_discount ( price, customer_type ): if customer_type == " premium ": return price * 0.80 return price Enter fullscreen mode Exit fullscreen mode 5. LLM Span An LLM span represents an LLM call.

It captures the model invocation, including the model used, input, output, token usage, latency, and status.

from ddtrace.llmobs.decorators import llm @llm ( model_name = " gpt-4 " ) # model_name is optional def generate_product_description ( product_data ): response = llm. complete ( prompt = f " Generate description for: { product_data [ ' name ' ] } " ) return response. text Enter fullscreen mode Exit fullscreen mode The above example could be represented as:

A tool represents calls to external services like databases, APIs made by an AI application.

A tool span represents the details of the external operation, such as what tool was called, the input provided, the result returned, and how long the operation took.

Example: Consider an AI customer-support assistant, that calls an external API to fetch order details:

from ddtrace.llmobs.decorators import tool @tool def fetch_inventory_status ( product_id ): inventory = database. query ( " SELECT quantity, location FROM inventory WHERE product_id =? ", product_id ) return inventory Enter fullscreen mode Exit fullscreen mode 7. Embedding span An Embedding Span represents the operation of converting text or other data into an embedding vector using an embedding model.

from ddtrace.llmobs.decorators import embedding @embedding ( model_name = " text-embedding-ada-002 " ) # model_name is required, model_provider is optional def generate_product_embedding ( product_text ): vector = embedding_model. encode ( product_text ) return vector. tolist () Enter fullscreen mode Exit fullscreen mode 8. Retrieval Span A retrieval span represents the operation of searching for and retrieving relevant information from a data source, such as a vector database.

The retrieval span captures the search operation and the information returned by it.

For example, in an AI shopping assistant, a user might ask:

“Find me a black running shoe under $100.”

The retrieval operation can be instrumented using:

from ddtrace.llmobs.decorators import retrieval @retrieval def search_product_catalog ( query ): query_vector = generate_embedding ( query ) results = vector_db. similarity_search ( query_vector, top_k = 5 ) return results Enter fullscreen mode Exit fullscreen mode Conclusion Agent observability helps you see what happens inside your AI application when it processes a request.

Datadog uses traces, spans, and instrumentation to help you follow a request from beginning to end, inspect individual steps, and find out why something failed or took too long.

Logs are useful, but observability gives you a clearer picture of how the different parts of your application work together.