How to Debug AI Agent Failures in .NET Using OpenTelemetry and Distributed Tracing

Debug AI Agent Failures in.NET Using OpenTelemetry

AI agent observability in.NET: learn how to debug AI agent failures with OpenTelemetry, distributed tracing, custom C# spans, metrics, and LLM monitoring.

ai-agent-observability-dotnet-opentelemetry

AI agent observability in.NET starts with a simple problem: an agent can return a wrong answer even when every API responds successfully. A database query runs without errors, the LLM request returns HTTP 200, and the vector database returns results. Yet the final response is incomplete, irrelevant, or slow.

Traditional logging can tell you a request failed. But an agent does several things before it answers:

When something goes wrong in that workflow, you need visibility into each operation, its duration, its dependencies, and its errors. That's what OpenTelemetry and distributed tracing provide.

In this tutorial we'll build an observability foundation for an AI-powered ASP.NET Core application using C#, OpenTelemetry, custom spans, metrics, and structured logging.

A user asks a support agent, "Find the latest order and summarize its delivery status." The agent searches the customer database, retrieves the order, calls a shipping API, sends the data to an LLM, and writes a response. The whole request takes 8 seconds, and a traditional log shows this:

2026-11-03 10:30:00 Information: Request received 2026-11-03 10:30:08 Information: Response generated Enter fullscreen mode Exit fullscreen mode It doesn't say where the eight seconds went. Was the database slow? The shipping API? Did the LLM retry? A distributed trace answers that:

AI Agent Request 8000 ms │ ├── Load Conversation 100 ms ├── Search Customer Database 250 ms ├── Shipping API 1200 ms ├── LLM Request 5800 ms └── Generate Final Response 650 ms Enter fullscreen mode Exit fullscreen mode These timings are illustrative, not benchmark results.

Together they make AI agent observability in.NET practical. Microsoft documents the approach here: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel

dotnet new webapi -n AIObservabilityDemo cd AIObservabilityDemo dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Exporter.Console dotnet add package OpenTelemetry.Instrumentation.AspNetCore dotnet add package OpenTelemetry.Instrumentation.Http Enter fullscreen mode Exit fullscreen mode These provide hosting integration, console export, ASP.NET Core instrumentation, and outgoing HttpClient instrumentation. For production, add an OTLP exporter and a telemetry backend, and check package versions first.

Configure tracing and metrics in Program.cs:

using OpenTelemetry.Metrics; using OpenTelemetry.Resources; using OpenTelemetry.Trace; var builder = WebApplication. CreateBuilder ( args ); builder. Services.

AddControllers (); builder. Services. AddOpenTelemetry ().

ConfigureResource ( resource => resource. AddService ( serviceName: "AIObservabilityDemo", serviceVersion: "1.0.0" )). WithTracing ( tracing => tracing.

AddAspNetCoreInstrumentation (). AddHttpClientInstrumentation (). AddSource ( "AIObservabilityDemo.Agent" ).

AddConsoleExporter ()). WithMetrics ( metrics => metrics. AddMeter ( "AIObservabilityDemo.Agent" ).

AddAspNetCoreInstrumentation (). AddHttpClientInstrumentation (). AddConsoleExporter ()); builder.

Services. AddHttpClient (); builder. Services.

AddScoped < AiAgentService >(); builder. Services. AddSingleton < AgentMetrics >(); var app = builder.

Build (); app. UseHttpsRedirection (); app. MapControllers (); app.

Run (); Enter fullscreen mode Exit fullscreen mode AddAspNetCoreInstrumentation() captures incoming requests, AddHttpClientInstrumentation() captures outgoing ones, AddSource() registers our custom ActivitySource, and ConfigureResource() names the service. The console exporter is for local development.

Automatic instrumentation sees HTTP requests, but it doesn't know about prompt construction, RAG retrieval, tool selection, or agent execution. Add those with an ActivitySource:

using System.Diagnostics; public static class AgentTelemetry { public static readonly ActivitySource ActivitySource = new ( "AIObservabilityDemo.Agent" ); } Enter fullscreen mode Exit fullscreen mode Then wrap each stage in a nested activity:

public class AiAgentService { public async Task < string > ProcessAsync ( string question ) { using var activity = AgentTelemetry. ActivitySource. StartActivity ( "AI Agent Execution" ); activity?.

SetTag ( "agent.type", "CustomerSupport" ); try { using var retrieval = AgentTelemetry. ActivitySource. StartActivity ( "RAG Retrieval" ); var documents = await SearchDocumentsAsync ( question ); retrieval?.

SetTag ( "rag.documents_count", documents. Count ); using var llm = AgentTelemetry. ActivitySource.

StartActivity ( "LLM Request" ); var response = await GenerateResponseAsync ( question, documents ); activity?. SetStatus ( ActivityStatusCode. Ok ); return response; } catch ( Exception ex ) { activity?.

SetStatus ( ActivityStatusCode. Error, ex. Message ); activity?.

AddException ( ex ); throw; } } // Simulated operations, not real vector database or LLM integrations. private async Task < List < string >> SearchDocumentsAsync ( string question ) { await Task. Delay ( 200 ); return [ "Document one", "Document two" ]; } private async Task < string > GenerateResponseAsync ( string question, List < string > documents ) { await Task. Delay ( 500 ); return "This is an example AI response."; } } Enter fullscreen mode Exit fullscreen mode The trace now shows the parent agent operation with its child operations, so you can see which stage to investigate.

Send a POST /api/agent/ask request from any simple controller that calls ProcessAsync, and read the spans in the console.

Agents use tools to reach databases, APIs and services, and one slow tool can slow the whole response. A reusable wrapper makes each tool call its own span:

public class AgentToolExecutor { public async Task < T > ExecuteAsync < T >( string toolName, Func < Task < T >> operation ) { using var activity = AgentTelemetry. ActivitySource. StartActivity ( "AI Agent Tool" ); activity?.

SetTag ( "tool.name", toolName ); try { var result = await operation (); activity?. SetStatus ( ActivityStatusCode. Ok ); return result; } catch ( Exception ex ) { activity?.

SetStatus ( ActivityStatusCode. Error, ex. Message ); activity?.

AddException ( ex ); throw; } } } Enter fullscreen mode Exit fullscreen mode Use the same pattern around LLM calls, and record the model name, duration, and any failure. Useful LLM signals include request duration, success and failure counts, retry count, provider status, timeouts, and token usage. For token usage, read the usage data in the provider's response.

Don't estimate it from response length. And don't retry a failed LLM call blindly: use bounded retries, timeouts, and provider-specific rate-limit handling.

Security: keep complete prompts, customer records, tokens, API keys and confidential tool arguments out of telemetry. Record only the metadata you need for debugging.

Traces help you investigate one request. Metrics show behavior across many:

using System.Diagnostics.Metrics; public class AgentMetrics { private readonly Meter _meter = new ( "AIObservabilityDemo.Agent" ); private readonly Counter < long > _requests; private readonly Counter < long > _failures; private readonly Histogram < double > _duration; public AgentMetrics () { _requests = _meter. CreateCounter < long >( "ai.agent.requests" ); _failures = _meter. CreateCounter < long >( "ai.agent.failures" ); _duration = _meter.

CreateHistogram < double >( "ai.agent.duration", unit: "ms" ); } public void RecordRequest () => _requests. Add ( 1 ); public void RecordFailure () => _failures. Add ( 1 ); public void RecordDuration ( double ms ) => _duration.

Record ( ms ); } Enter fullscreen mode Exit fullscreen mode Avoid high-cardinality labels such as user IDs, conversation IDs or raw prompts. They inflate telemetry storage and cost.

Use ILogger<T> with message templates, like _logger.LogError(ex, "Agent operation failed: {Operation}", operation), instead of writing to the console. Structured logs are easy to filter by operation name and other safe metadata, and you can correlate them with traces through trace identifiers. Don't log sensitive data just because it might help debugging.

Console output is fine for development, but production needs centralized storage and visualization. A typical setup:

ASP.NET Core AI Application | OpenTelemetry | OpenTelemetry Collector | Jaeger / Prometheus | Grafana Enter fullscreen mode Exit fullscreen mode Jaeger visualizes traces, Prometheus stores metrics, and Grafana builds dashboards. To export with OTLP:

dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol Enter fullscreen mode Exit fullscreen mode. WithTracing ( tracing => tracing. AddAspNetCoreInstrumentation (). AddHttpClientInstrumentation (). AddSource ( "AIObservabilityDemo.Agent" ). AddOtlpExporter ()) Enter fullscreen mode Exit fullscreen mode Set the endpoint through configuration, for example OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317. It must match your collector's receiver and protocol, and production should use secured transport. Docs: https://opentelemetry.io/docs/languages/dotnet/exporters/

Debugging agents in production needs visibility into the whole execution workflow. A success or failure log doesn't tell you whether a problem happened in retrieval, model inference, database access, or tool execution.

AI agent observability in.NET combines OpenTelemetry's automatic instrumentation with custom ActivitySource spans, structured logging, and application-specific metrics, so you can see how each operation behaves. Build it into the architecture from the start, not after the first production failure.

More practical AI engineering guides for.NET developers: AI Engineering Foundry

What is OpenTelemetry in.NET? An open-source observability framework that lets.NET applications generate, collect, and export traces, metrics, and logs.

Why does distributed tracing matter for AI agents? It shows how an agent processes a request across components, so you can find slow operations, failed tool calls, and dependency problems.

Can OpenTelemetry monitor LLM API calls and RAG pipelines? Yes. Instrument outgoing HTTP calls and add custom spans around embeddings, vector searches, retrieval, and LLM generation to capture duration, errors, and safe metadata.

Does OpenTelemetry evaluate answer quality? No. It monitors execution. Answer quality needs separate evaluation and testing.

Which tools visualize OpenTelemetry data? Jaeger for traces, Prometheus for metrics, Grafana for dashboards, and compatible APM platforms.