Your Java LLM agent works in the demo. Friday night in production is another story.

Picture an agent that handles customer refunds. In the demo it was perfect: it reads the request, checks the order, calls the refund tool and returns clean JSON. Everyone is happy.

Then comes the first Friday night in production.

Monday morning, two questions arrive: what did that weekend cost, and what exactly did the agent do? Nobody can answer with confidence.

The scene is made up. But if you've run an LLM agent in production, you've probably met at least one piece of it. And notice: none of these problems is about how smart the model is.

TL;DR — ARA is an open source (Apache 2.0) Java 21 runtime for LLM agents, built on LangChain4j. It handles output contracts, human approval, provider failover, spend caps and execution traces, and runs fully offline for tests. On Maven Central as io.github.xmor:ara-runtime:1.0.3.

Stripped down, an agent is a loop: send a request to the model, run the tool it picked, hand back the result, repeat until there's a final answer. You can write that in an afternoon, which is why so many tutorials stop there.

These share one trait: you shouldn't leave them to the model. An LLM can help decide what to do. It shouldn't be the only thing responsible for the rules that govern how that decision gets executed.

I work on the JVM, and I wanted these rules in one place: explicit, in Java code you can read, with tests you can repeat. That's how ARA started.

ARA ( Agent Runtime Architecture ) is an open source runtime for LLM agents. It's written in Java 21, licensed under Apache 2.0 and published on Maven Central.

It's not a new model, and it doesn't replace the LLM integration layer. It sits on top of LangChain4j, which handles the providers (OpenAI, Anthropic, Ollama, Mistral, any OpenAI-compatible endpoint), and takes care of how agents run. It's a library, not a container: no mandatory annotations, no reflection to discover components, no Spring requirement.

The principle behind it: the model's probabilistic decisions and the runtime's deterministic rules should have separate owners.

If you want to follow along with the code, the repository is on GitHub. Let's go back to Friday night.

The model wraps its JSON in a Markdown fence, or forgets a required field. The downstream service expects structured data and breaks.

A common fix is asking a second LLM to check the first one's answer. But a deterministic validation doesn't need another model call.

In ARA you declare an AgentContract with checks written in Java:

AgentContract contract = AgentContract. builder () // Input side: runs before the model sees anything. addInputProcessor ( InputSanitizer. instance ()) // blocks known prompt-injection patterns. addInputProcessor ( ContentTruncator. to ( 4000 )) // hard cap on input length // Declares the expected shape and appends it to the system prompt. outputSchema ( JsonSchemaValidator. forOutput ( SCHEMA )) // Output side: plain Java, zero tokens, applied in order. addOutputProcessor ( MarkdownFenceStripper. instance ()) // strips the Markdown code fence around the JSON (6:52 PM). addOutputProcessor ( JsonSchemaValidator. forOutput ( SCHEMA )) // rejects missing or wrong fields // On rejection, send the answer back with the reason, at most twice (these calls cost tokens). outputRepairAttempts ( 2 ). build (); // The contract is bound to the agent: every execute() goes through it AraAgent agent = runtime. createAgent ( config, contract ); Enter fullscreen mode Exit fullscreen mode Two separate mechanisms are at work.

The first is deterministic. MarkdownFenceStripper removes the 6:52 PM code fence before the downstream service ever sees it, and the schema validator catches the missing field. These checks run in plain Java and consume no tokens.

The second is recovery. With outputRepairAttempts(2), the model gets its rejected answer back along with the reason, and two chances to fix it. Those calls do cost tokens, so account for them separately.

If the answer still doesn't satisfy the contract, the task fails explicitly instead of passing invalid data along.

The point isn't to make the model infallible. It's to make sure its mistakes don't get accepted unchecked.

An agent can have access to the right tool, understand the request and even write a convincing justification. None of that means it should execute immediately.

// Holds pending requests in memory (they don't survive a restart, see the limits below) ApprovalGate gate = new InMemoryApprovalGate (); // Level 1: enable approvals on the runtime AraRuntime runtime = AraRuntime. builder (). llmClient ( llmClient ). approvalGate ( gate ). build (); AraAgent agent = runtime. createAgent ( AgentConfig. builder (). agentId ( "refund-agent" ). agentType ( "support" ) // Level 2: opt this agent in; every tool call now waits for a human. humanApprovalRequired ( true ) // Tools are opt-in per agent: this one can only call "refund". enabledTools ( List. of ( "refund" )). build () ); Enter fullscreen mode Exit fullscreen mode From then on, when the agent calls refund, the runtime parks the call and creates an approval request carrying the tool, its arguments and an expiry. Thanks to Java 21 virtual threads, waiting doesn't hold a platform thread, so many pending requests don't cost you resources.

The operator has three answers: approve (the refund goes through as is), reject with a reason (the agent receives it and can change course), or modify the arguments (correct the amount to €400 and the refund goes through with that).

// What the operator's UI (or your own endpoint) calls, one of these per request gate. submit ( requestId, new ApprovalDecision. Approved ()); // run as proposed gate. submit ( requestId, new ApprovalDecision. Rejected ( "Too expensive" )); // the agent sees the reason gate. submit ( requestId, new ApprovalDecision. Modified ( newArgumentJson )); // run with corrected arguments, e.g. €400 Enter fullscreen mode Exit fullscreen mode ApprovalDecision is a sealed interface, so a switch over it must handle all three cases or it won't compile:

// Sealed interface: remove a case and this no longer compiles switch ( decision ) { case ApprovalDecision. Approved a -> proceed (); case ApprovalDecision. Rejected r -> abort ( r. reason ()); case ApprovalDecision. Modified m -> proceed ( m. newPayload ()); } Enter fullscreen mode Exit fullscreen mode ( proceed() and abort() stand in for your own code.)

If nobody answers before the deadline, the call fails. The default is to do nothing.

One thing to know: the gate applies per agent, not per tool. With humanApprovalRequired(true), every tool call of that agent waits for a human. So keep sensitive actions in a dedicated agent, like the refund-agent above, and let read-only agents run without the gate.

The value isn't an "Approve" button. It's an explicit control point between the agent's decision and the action that changes the outside world. An agent can propose an action without having the authority to execute it.

A retry may help, but repeating the same request against an unavailable service isn't a resilience strategy.

In ARA you configure an ordered chain of models:

// "smart", "local" and "cheap" are names of clients registered on the runtime: // no credentials live in the agent config AgentConfig config = AgentConfig. defaults (). agentType ( "resilient" ). primaryLlm ( LlmProfile. of ( "smart" )). fallbackLlms ( List. of ( // tried in declaration order LlmProfile. of ( "local" ), LlmProfile. of ( "cheap" ) )) // Move to the next model only on retryable errors (network, 5xx, 429); // auth errors and bad requests stop the chain. llmSelectionPolicy ( LlmSelectionPolicy. FAILOVER ). build (); Enter fullscreen mode Exit fullscreen mode The interesting part is deciding when to switch. A network error, a 5xx or a rate limit moves on to the next model.

An invalid API key or a malformed request stops the chain instead: it would fail identically on every model, and switching would only clutter your logs.

Then there's the growing queue. Each model in the chain sits behind a circuit breaker: after three consecutive failures it's skipped outright, so an outage stops costing you a timeout on every request. After thirty seconds a single probe call decides whether to reopen it. Health always comes from real traffic, never from background probes, which would cost money and burn the very rate limit you're trying to protect.

What's left is spend. In the model profile you set unit prices and a spending cap per agent. Every response reports tokens and cost, and when the cap is exceeded, execution stops. You find out from your code, not from Monday's invoice.

One caveat: a fallback model isn't equivalent to the primary one. Quality, latency and cost can all change. The chain is an explicit architectural choice, not a promise of identical results.

A reliable system doesn't just react to errors. It lets you understand what happened.

In ARA every response carries its own history. AgentResponse.steps() holds the steps the runtime recorded, including tool calls with their arguments and results, even when the execution fails. In that case you get the partial trace, which is exactly what you need to see where it broke. If you use OpenTelemetry, ARA emits a span per execution, per LLM call and per tool dispatch.

Two caveats. An operational trace records steps, calls and events, not the model's internal reasoning. And tracing doesn't remove the need to protect data: tool arguments and request contents still need to follow your own security and retention rules.

You can, and both are good libraries. Resilience4j gives you circuit breakers and retries; LangChain4j gives you the provider layer.

The work is in the glue: deciding that a 429 should fail over but a 401 shouldn't, wiring a spend cap into the same call path, parking a tool call on a human decision without tying up a thread, and making sure validation runs before output leaves the agent. ARA is that glue in one place, built on top of LangChain4j rather than instead of it.

If you'd rather own the glue, that's a fair choice. The four sections above are a decent checklist either way.

One thing I care about a lot: being able to develop and test agents without depending on a remote provider.

ARA includes ScriptedLlmClient, a test client that replays predefined responses. It doesn't simulate a real model's intelligence. It lets you check the runtime's behavior repeatably, under controlled conditions.

<dependency> <groupId> io.github.xmor </groupId> <artifactId> ara-runtime </artifactId> <version> 1.0.3 </version> </dependency> Enter fullscreen mode Exit fullscreen mode And write:

// AraRuntime is AutoCloseable: try-with-resources shuts it down cleanly try ( AraRuntime runtime = AraRuntime. builder () // Stub client: replays a fixed script, one entry per LLM call. llmClient ( ScriptedLlmClient. script (). thenFinalAnswer ( "Virtual threads are lightweight JVM threads." ). build ()). build ()) { AraAgent agent = runtime. createAgent ( AgentConfig. defaults (). agentType ( "assistant" ). systemPrompt ( "You are a concise technical assistant." ). build ()); // Runs the real execution loop; only the model's answer is canned AgentResponse response = agent. execute ( AgentTask. of ( "Explain virtual threads" )); System. out. println ( response. content ()); } Enter fullscreen mode Exit fullscreen mode No API key, no network, no cost. The runtime is the real one; only the answer is prepared in advance. The same approach works for tests: success paths, errors, tool calls, recovery conditions, without every test depending on an external provider.

ARA tests itself this way.

When you want a real model, add ara-adapters and change one line:

// Pass this instead of ScriptedLlmClient: the rest of the code stays the same LlmClient llm = AraLlmClientFactory. openAi (). apiKey ( System. getenv ( "OPENAI_API_KEY" )). modelName ( "gpt-4o" ). build (); Enter fullscreen mode Exit fullscreen mode You need Java 21 and Maven.

People tend to oversell their own projects. So here's where you can check every claim in this post yourself.

And one number: the sources contain 2,215 tests, all run offline by mvn verify.

I'd rather tell you the limits now than have you discover them later.

If any of these is a blocker for you, open an issue. It's exactly the feedback that decides what comes next.

ARA started in the evenings, after work. By day I'm a consultant, and I've seen enough systems to know the problem rarely ends when the code works in the ideal case.

The project is built with heavy AI assistance, and I say so openly. Using AI to write code doesn't make design less important; it makes it more necessary. That's why the architectural rules are written down, dependencies are checked on every build, and every module has a test-coverage threshold that can only go up.

AI speeds up the writing. It doesn't take over responsibility for the decisions.

ARA is an independent project with no company behind it. It grows only if developers who find it useful say so. Three things you can do, from quickest to most involved:

Build autonomous AI agents and multi-agent systems in plain Java 21. LLM integration, tool calling, deterministic I/O contracts, RAG and human-in-the-loop — no annotations, no reflection, no Spring, no Kotlin.

Quick start · Documentation · Examples · Website · Discussions

ARA is a Java 21 agent runtime: a place where agents actually run, with sessions memory, execution strategies, cost budgets, approval gates and a full execution trace — not a thin wrapper around a chat completion endpoint.

It works with OpenAI, Anthropic, Ollama, Mistral, LM Studio, Groq and any OpenAI-compatible endpoint, and it is published on Maven Central as io.github.xmor:ara-runtime.

Your code ──▶ AraRuntime ──▶ Agent (strategy + contract + session) │ ┌──────────┼───────────┐ ▼ ▼ ▼ LlmClient Tools Retriever OpenAI/Claude/ (yours, (in-memory Ollama/Mistral parallel or Qdrant) on vthreads) Quick start — 60 seconds, no API key 1. Add the dependency …