What 586 Real Traces Taught Us About LLM Agent Determinism

Posted on Nov 4 • Edited on Nov 4 • AI-assisted

How temperature selectively degrades agent reasoning — and what that means for observability.

On November 3, 2026, we ran 586 traces of a LangGraph folder-research agent against AgentPulse, our metadata-only observability platform. The agent used Ollama with qwen2.5:7b on an RTX 4060 (Legion), plus 6 traces from a Pixel 8 Pro running qwen3b via llama.cpp in Termux.

The experiment: same 10 questions, same corpus, four temperatures (0, 0.3, 0.7, 1.0). We measured trace shapes — span counts, durations, tool calls — to understand how sampling temperature affects agent behavior.

Why this matters: If you're building observability for AI agents, you need to know what "normal" looks like. Without a determinism baseline, every alert is a guess.

At temp=0, qwen2.5:7b produced identical trace shapes across 20 runs per question. Nine out of ten questions had standard deviation 0.0 on span counts. The tenth (Q1) had sd 0.4 — still remarkably tight.

This is the baseline. When temp=0, trace-shape changes are genuine regression signal, not noise. Your alert thresholds can be tight.

Here's what surprised us. Temperature doesn't add uniform noise. It selectively degrades complex reasoning while simple lookups survive.

The pattern: Multi-file synthesis questions degrade at 0.3. Simple lookups survive to 1.0. The breakdown is question-dependent, not just temperature-dependent.

Implication: Your eval thresholds can't just be "wider at higher temp." They need to be per-question-type. A simple lookup at temp=0.7 should still be deterministic. A complex reasoning task at temp=0.3 is already noisy.

Our first eval prototype used mean comparison. It got 7/10 "correct" — but the 3 false positives were instructive.

At temp=0.7, traces become bimodal: sometimes the agent does full work (8 spans), sometimes it produces a degenerate 2-span trace. The mean shifts, triggering a "regression" alert. But the median and IQR tell the real story: high variance with overlapping IQR = noise, not regression.

We validated this on 500 traces. Mean-based detection fails. Median/IQR works.

We validated AgentPulse across three very different machines:

The phone was the hard one. We fought through:

But it works. The phone agent runs, generates traces, and uploads them to the same workspace. Metadata-only observability works on mobile.

From these findings, we built the quality-warning evidence adapter — the statistical engine that connects real trace data to AgentPulse's alerting system.

The adapter is live on GitHub (codex/quality-warning-evidence, commit 3df33f0). The API integration follows.

Temperature doesn't just make agents "more random." It selectively kills complex reasoning first. Your monitoring needs to know the difference.

586 traces. 3 devices. 1 night. The data is real, the findings are publishable, and the code is on GitHub.