Your Agent Harness Is Paying LLM Prices for Classifier Decisions

Posted on Oct 14 • Originally published at Medium

Keep your LangGraph router. Let a decision model take the calls it's sure about, and prove the savings on your own traffic before you change anything.

The AI engineering community has settled on a fundamental architectural principle: a raw foundation model is not an agent; an agent is a model plus a harness.

The harness is everything surrounding the model: memory, state schemas, guardrails, tool bindings, and the control-flow graph that orchestrates what happens next.

Look closely at the control flow in a typical LangGraph workflow. Between an incoming user request and the final synthesized response, the graph evaluates multiple decision points:

Almost none of these choices produce free-form text. They evaluate input context and pick one discrete item from a known set of routes.

As analyzed in The Router Is the Harness, the decision layer of an agent is fundamentally a classification problem, yet production systems routinely pay generative LLM prices for it: roughly $0.000006 per decision for a typed classifier versus ~$0.02 for a frontier LLM call.

[ Incoming Request ] │ ▼ ┌───────────────────────────┐ │ TypeSafe Jev Fast-Path │ │ (~100ms System 1 reflex)│ └─────────────┬─────────────┘ │ ┌──────────────────┴──────────────────┐ │ │ Confidence ≥ Threshold Confidence < Threshold (Clear intent) (Ambiguous / Edge case) │ │ ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ Direct Route Dispatch│ │ Existing LLM Router │ │ ⚡ ~100ms / No LLM │ │ 🤖 Trusted Fallback │ └──────────────────────┘ └──────────────────────┘ Enter fullscreen mode Exit fullscreen mode Even when context caching shrinks token costs, the primary operational penalties remain: latency, jitter, and failure modes. A generative model requires multiple token-generation steps and is vulnerable to prompt drift, formatting hallucinations, rate limits ( 429 ), and upstream API outages. A typed classifier evaluates the input in a single pass, responds deterministically in ~100ms, and cannot be prompt-injected into an invalid route.

Because you would be trading a router you have painstakingly tuned, tested, and trust for a model you haven't validated on your data.

TypeSafe's Jev is a purpose-built decision model, but enterprise engineering teams should not blindly trust third-party confidence scores or unverified benchmarks with mission-critical routing.

The battle-tested architectural pattern is confidence-gated routing:

That pattern is exactly what langgraph-jev-router implements as a drop-in library for LangGraph.

Adopting it cannot make routing worse than it is today. If Jev has any doubt, or if the network blips, your trusted router executes exactly as it does right now.

A common misunderstanding is asking: "Which industry is this built for?"

langgraph-jev-router is not an industry-specific application. It is an infrastructure primitive for agentic harnesses, akin to an HTTP connection pool or an edge-caching layer.

It applies to any agentic system in any domain where an LLM is currently used to decide control flow:

The industry benchmarks we publish (manufacturing, supply chain, healthcare, telecom) are not boundaries of the tool—they are empirical stress tests on messy, real-world domain vocabularies to prove that the architecture holds up under enterprise conditions.

Integrating langgraph-jev-router requires adding a single decorator to your existing routing function.

from typing import Literal from langchain_core.messages import HumanMessage, AIMessage from langgraph.graph import StateGraph, START, END, MessagesState from langgraph_jev_router import jev_first, attach_routes # 1. Define your routing options with clear docstrings/descriptions ROUTES = { " billing ": " Invoices, credit card charges, payment failures, and refund requests. ", " technical ": " System errors, stack traces, API timeouts, and login failures.

", " account ": " Password resets, team invites, SSO, and user profile management. ", } # 2. Add one decorator to your existing router @jev_first ( routes = ROUTES, threshold = 0.75, # Only take over when 75%+ confident ) def route_ticket ( state: MessagesState ) -> Literal [ " billing ", " technical ", " account " ]: # ── YOUR EXISTING LLM ROUTER (Unchanged, acts as 100% safety fallback) ── print ( " 🤖 Ambiguous or low confidence: Falling back to LLM router...

" ) # Your standard OpenAI / Gemini / Claude call runs here return " technical " # 3. Build your LangGraph as usual builder = StateGraph ( MessagesState ) builder. add_node ( " billing ", lambda s: { " messages ": [ AIMessage ( " Handling Billing " )]}) builder. add_node ( " technical ", lambda s: { " messages ": [ AIMessage ( " Handling Tech Issue " )]}) builder. add_node ( " account ", lambda s: { " messages ": [ AIMessage ( " Handling Account " )]}) builder. add_conditional_edges ( START, route_ticket ) for node in [ " billing ", " technical ", " account " ]: builder. add_edge ( node, END ) # 4. Attach routes and compile # attach_routes reads node docstrings or route dicts as descriptions graph = attach_routes ( builder ). compile () Enter fullscreen mode Exit fullscreen mode When an incoming message arrives:

Many LangGraph developers use prebuilt abstractions like create_supervisor from langgraph-supervisor, where the routing function is compiled internally and cannot be decorated directly.

For these graphs, langgraph-jev-router provides install_shortcut:

from langgraph_jev_router import install_shortcut # Intercept before the prebuilt supervisor node handle = install_shortcut ( workflow, before = " supervisor " ) app = workflow. compile () Enter fullscreen mode Exit fullscreen mode When Jev is confident, the request skips the supervisor's LLM call entirely and routes directly to the specialist worker node. When uncertain, control passes to the supervisor node unaltered.

Most of the engineering in langgraph-jev-router went into handling non-ideal edge cases gracefully:

One of the most critical bugs uncovered during testing occurs in cyclic multi-agent graphs:

[Router] ──> [Worker Agent] ──> [Tool Execution] ──> [Router] ──>... Enter fullscreen mode Exit fullscreen mode In a loop-shaped graph, a worker agent finishes an action and hands control back to the router. A naive fast-path router inspecting only the original user request will see the initial prompt again and re-route right back to the same worker agent—creating an infinite execution loop.

At that stage in the conversation, the real question is no longer "Who should handle this request?" but rather "Is the task complete, or does it require another iteration?" Answering that requires analyzing the full conversation history—a task suited for the reasoning LLM.

To prevent this, langgraph-jev-router automatically inspects message sender history: Jev only activates on fresh user requests. Any internal loop or agent-to-agent handoff automatically routes to your full LLM.

Engineering leaders should never adopt routing changes based on vendor claims. You need to prove the behavior on your own production traffic.

langgraph-jev-router includes built-in Shadow Mode. In shadow mode, your existing LLM router makes 100% of the live routing decisions, while Jev executes asynchronously in parallel to observe, record, and evaluate agreement:

# Enable zero-risk shadow observation export JEV_ROUTER_SHADOW = 1 export JEV_ROUTER_LOG = jev_decisions.jsonl # Run your normal traffic for a few days... Enter fullscreen mode Exit fullscreen mode Then generate an automated agreement and savings report using the CLI:

jev-router report jev_decisions.jsonl --llm-cost-per-call 0.002 Enter fullscreen mode Exit fullscreen mode Output:

====================================================================== SHADOW MODE REPORT: 425 decisions compared (Your existing router decided all traffic; Jev observed in parallel) ====================================================================== Threshold | Jev Autonomous % | Agreement with LLM Router 0.60 | 82.4% (n=350) | 94.1% 0.70 | 74.2% (n=315) | 96.8% <-- Recommended Threshold 0.80 | 68.5% (n=291) | 98.2% 0.90 | 58.1% (n=247) | 99.1% ---------------------------------------------------------------------- Recommendation: Set threshold=0.70 to eliminate ~74% of routing LLM calls while maintaining 96.8% agreement with your current router. Enter fullscreen mode Exit fullscreen mode (Note: Agreement is measured against your current LLM router. When disagreement occurs, manual inspection often reveals the classifier chose the correct category while the LLM drifted.)

To measure langgraph-jev-router against real-world data containing specialized technical jargon, we built an open-source evaluation suite: jev-router-benchmark.

We evaluated 425 real industrial plant accident reports from the IHM Stefanini Industrial Safety and Health Analytics Database (CC0 1.0 Universal Public Domain). Incidents involve complex operational hazards: molten metal splashes, hydraulic press pinch points, high-voltage flashovers, and chemical inhalation.

We benchmarked a standard Google Gemini Flash router against langgraph-jev-router across 50 real manufacturing incidents.

How does threshold selection affect safety and LLM call reduction? We executed a multi-threshold sweep from 0.60 to 0.95 across the incident dataset:

Architectural Takeaway: Even if you configure an ultra-conservative threshold of 0.95, Jev still eliminates 56% of routing LLM calls at an average latency of 137ms. Engineering teams have a predictable knob to tune cost savings against accuracy preservation.

Different tools address model routing at different layers of the AI stack:

If you are a CTO, VP of Engineering, or Principal Architect evaluating this architecture, here is the operational reality:

Empirical claims require open benchmarks. In the upcoming articles of this series, we examine full benchmark evaluations across diverse operational domains:

All benchmark code, raw dataset references, and calibration scripts are publicly available and reproducible.

pip install langgraph-jev-router Enter fullscreen mode Exit fullscreen mode Verify your setup:

export TYPESAFE_API_KEY = "your_api_key" jev-router check Enter fullscreen mode Exit fullscreen mode GitHub Repository: https://github.com/JTankariya/langgraph-jev-router Benchmark Suite & Data: https://github.com/JTankariya/jev-router-benchmark Note: langgraph-jev-router is an independent open-source project released under the MIT license. Not affiliated with LangChain, Inc. or TypeSafe AI.