How I Stopped an LLM From Skipping the Investigation, Looping Forever, and Inventing Pods — Without LangChain

An agent loop is just a for loop. The hard part is the invariants inside it.

LLMs misbehave in predictable ways: they skip steps, invent tools, loop forever, and fabricate evidence. Every failure can be caught by a small, deterministic guardrail that replies with feedback instead of crashing.

You know the routine. The pager fires, you open a terminal, and you start the ritual:

kubectl get pods -A kubectl describe pod payment-service-6f7b8d9c0-ghi56 -n production kubectl logs payment-service-6f7b8d9c0-ghi56 -n production --tail = 50 kubectl get events -n production Enter fullscreen mode Exit fullscreen mode Four commands, three context switches, and a lot of squinting before you spot the line that matters:

config validation failed: missing required field 'DB_PASSWORD' Enter fullscreen mode Exit fullscreen mode The investigation follows a predictable shape, but it still takes a senior engineer’s instincts to run it efficiently. That combination — a repeatable process, with judgment calls at every step — is exactly what LLM agents should be good at.

So I built one. You ask it a question in plain English:

python agent.py "Which pods in the production namespace are unhealthy, and why?" Enter fullscreen mode Exit fullscreen mode It decides which tools to call, digs through several rounds of evidence on its own, and returns a report like this:

payment-service-6f7b8d9c0-ghi56 — CrashLoopBackOff, 14 restarts Evidence: logs show missing required field 'DB_PASSWORD' Fix: restore the missing Secret entry, then roll out the deployment inventory-service-59d8f7c6a-mno90 — ImagePullBackOff Evidence: events show image v0.9.9-beta — manifest unknown Fix: push the missing image or correct the tag Enter fullscreen mode Exit fullscreen mode Every conclusion cites the tool result it came from.

One question in, a structured health report out — every step logged.

Getting a demo to produce that output once was easy. Getting it to produce that output reliably, even when the model misbehaves, was the real work. That is what this article is about.

An agent loop is, at its core, embarrassingly simple:

That fits in a single for loop. The hard part is not the loop itself.

The hard part is the invariants that have to hold inside the loop: every tool_call gets exactly one matching tool message, the loop always terminates, and the model never reports a fact it didn't observe.

Frameworks hide those invariants behind abstractions. That’s great — until production breaks and you need to know exactly what the model saw at step 4, and why it called list_pods for the third time.

So I wrote the loop by hand, using only the low-level openai and mcp SDKs. The entire agent fits in about 350 lines you can read in one sitting.

The key piece of glue is surprisingly small. The LLM only understands OpenAI’s tool format, and the server only speaks MCP:

{ " type ": " function ", " function ": { " name ": tool. name, " description ": tool. description or "", " parameters ": tool. inputSchema, # MCP JSON Schema -> OpenAI parameters }, } Enter fullscreen mode Exit fullscreen mode Because the tool list is fetched from the server at runtime, switching from the mock to a real cluster is a single line in.env:

MCP_SERVER_COMMAND=npx -y mcp-server-kubernetes Enter fullscreen mode Exit fullscreen mode agent.py doesn't change at all.

This is the heart of the project. Every guardrail below exists because a model actually did the thing it prevents.

Symptom: You ask for a health check, and the model confidently replies “All pods look healthy” — without calling a single tool. It has no idea what the cluster looks like. It is pattern-matching on what a health report usually says.

Fix: Force tool use on the first step only, then let the model decide freely after that:

tool_choice = " required " if step == 1 else " auto ", Enter fullscreen mode Exit fullscreen mode Why only step 1? If you force tools on every step, the model can never stop and write the report. You’ve replaced “answers too early” with “never answers.”

The second layer: not every OpenAI-compatible provider honors tool_choice. So if the model still answers with zero tool calls, the harness pushes back:

if tool_calls_made == 0 and step < cfg [ " max_steps " ]: messages. append ({ " role ": " user ", " content ": " You have not called any tool yet. Investigate the " " cluster with the provided tools before answering. ", }) continue Enter fullscreen mode Exit fullscreen mode Lesson: don’t trust a single API parameter to enforce behavior across providers. Enforce it in your own code.

Symptom: The model calls a tool that doesn’t exist, such as delete_pod.

That case is worth pausing on. A hallucinated tool name is not just a bug. In an ops context, it is a safety issue.

Fix: A whitelist built from what the server actually exposes. If the model asks for something else, the call is rejected — and the model is told what it can use:

if name not in known_tools: messages. append ({ " role ": " tool ", " tool_call_id ": tc. id, " content ": f " Unknown tool ' { name } '. " f " Available tools: { ', '. join ( sorted ( known_tools )) }. ", }) continue Enter fullscreen mode Exit fullscreen mode 3. Malformed arguments Symptom: get_pod_logs({"name": "x", "tail": "fifty"}). A required namespace is missing, and tail is a string instead of an integer.

You could forward that to the server and let it fail. But that wastes a round trip, and server error messages are often vague.

Fix: Pre-flight validation. Every MCP tool already ships with a JSON Schema, so the harness compiles one Draft 2020–12 validator per tool at startup, and checks arguments before the call:

errors = sorted ( validator. iter_errors ( args ), key = lambda e: [ str ( p ) for p in e. absolute_path ], ) if errors: detail = "; ". join ( f " { '. '. join ( map ( str, e. absolute_path )) or ' <root> ' }: { e. message } " for e in errors ) # -> "Invalid arguments for get_pod_logs: <root>: 'namespace' is a # required property; tail: 'fifty' is not of type 'integer'" Enter fullscreen mode Exit fullscreen mode The model gets a precise, field-level correction it can act on in the next step.

One more detail: if a tool’s schema is itself invalid, the harness logs a warning and skips validation for that tool instead of crashing. A guardrail should never become a new single point of failure.

Symptom: The model calls list_pods(namespace="all"), gets the result, and calls list_pods(namespace="all") again. And again. Each loop burns tokens and moves nothing forward.

Fix: Signature-based duplicate detection. Each call is normalized to tool:json(args, sort_keys=True), so {"a": 1, "b": 2} and {"b": 2, "a": 1} count as the same call.

A duplicate is not executed again. The model gets the cached result plus a nudge:

[Duplicate call] You already called this tool with identical arguments; the result has not changed. Base your next step on it, try a different tool, or write the final report. Enter fullscreen mode Exit fullscreen mode After MAX_REPEATS duplicates, the harness stops the investigation and forces a summary.

The protocol trap that bit me: my first instinct was to break as soon as the repeat limit was hit. That corrupts the conversation.

An assistant message can contain several tool_calls, and the API requires every one of them to have a matching role="tool" message. Break halfway through, and the next request is rejected outright.

So the harness sets a flag, finishes answering every call in the current round, and only then exits:

if repeat_count >= MAX_REPEATS: # Do not break mid-round: every tool_call in this assistant # message still needs a matching tool message, or the next # request would be rejected. force_summary = True Enter fullscreen mode Exit fullscreen mode This is the kind of invariant a framework hides from you — until it doesn’t.

Break mid-round and the next request fails with a 400. Finish the round, then exit.

Symptom: The report is well written and confident — and it mentions ghost-app-12345-abcde, a pod that does not exist.

This is the most dangerous failure, because it looks correct. An on-call engineer may spend 20 minutes chasing a ghost.

Fix: A grounding check. Before accepting a final answer, the harness extracts every pod-like name from the report and verifies that it appeared in at least one tool result:

POD_NAME_RE = re. compile ( r " \b[a-z0-9]+(?:-[a-z0-9]+){2,}\b " ) def _invented_pods ( answer: str, evidence: str ) -> list [ str ]: return sorted ( { n for n in POD_NAME_RE. findall ( answer ) if n not in evidence } ) Enter fullscreen mode Exit fullscreen mode If any are found, the model is told exactly which names to remove or correct.

That’s a narrow check. But it targets the highest-cost hallucination: sending a human after something that doesn’t exist.

Symptom: The model keeps finding “one more thing” to check.

Fix: A hard MAX_STEPS budget (10 by default). When it runs out, the harness appends a final instruction and makes one more call with no tools parameter at all:

resp = await llm. chat. completions. create ( model = cfg [ " model " ], messages = messages ) Enter fullscreen mode Exit fullscreen mode The model physically can’t call another tool. It has to summarize what it has already gathered. The loop is guaranteed to terminate.

Symptom: The MCP subprocess hiccups, the arguments aren’t valid JSON, or the server returns isError. Naive code raises an exception, and the whole investigation is lost.

Fix: None of these raise. Each one becomes a tool result string sent back to the model:

except Exception as exc: return f " Tool call failed ( { name } ): { exc } " Enter fullscreen mode Exit fullscreen mode The model can then decide to retry, pick another tool, or explain the gap in its report. A human engineer does the same thing when kubectl logs fails: you note it, and move on.

Step back from the individual fixes, and a few principles emerge. I think they apply to almost any production agent.

The model is a reasoning engine, not an executor. Every action it proposes passes through deterministic checks — whitelist, schema, deduplication, grounding — before it touches anything real.

None of the guards above crash the run. Each one rejects the action and tells the model how to fix it. Errors become observations, and observations are what the model reasons over. This turns a fragile pipeline into a self-correcting one.

Every pushback has a budget: one evidence pushback, MAX_REPEATS duplicates, MAX_STEPS steps, and a final tool-less summary. Self-correction without a bound is a liability. With a bound, termination is guaranteed by construction — not by hoping the model behaves.

This one is easy to overlook. The docstrings on the MCP tools steer the model’s strategy:

The response already includes this pod’s events, so do not call get_events for the same pod; use get_pod_logs next if it is CrashLoopBackOff.

Do not use this for ImagePullBackOff or Pending pods: their containers never started, so no logs exist.

Those two sentences teach the model to skip wasted calls. If you own the MCP server, the tool documentation is your highest-leverage prompt.

Every run writes the complete conversation — system prompt, every tool call, every result, the final answer — to a JSONL trace. When the agent reaches a strange conclusion, you replay exactly what it saw. Trace-writing itself never raises, because diagnostics must not break the thing they’re diagnosing.

How do you write reliable tests for something built on top of an unpredictable LLM?

You don’t test the model. You test the harness against a model that misbehaves on purpose.

The test suite spins up a real local HTTP server that impersonates an OpenAI-compatible API. Each test scripts a “bad model,” response by response. The MCP side stays fully real: calls go through the actual client to the actual mock server.

A scripted fake LLM over real HTTP, a real agent loop, and a real MCP server — no API key required.

def test_unknown_tool_is_rejected ( fake_llm ): def script ( body, n ): if n == 1: return _completion ( tool_calls = [ _tool_call ( " c1 ", " delete_pod ", { " name ": " x " }) ]) joined = " ". join ( _tool_contents ( body )) assert " Unknown tool ' delete_pod '" in joined assert " list_pods " in joined # the whitelist is offered back return _completion ( content = " That tool does not exist; sticking to the whitelist. " ) Enter fullscreen mode Exit fullscreen mode There are six tests, one per failure mode:

None of them needs an API key, and all of them run in CI in seconds.

This is the part I’d most encourage other agent builders to copy: each guardrail should have a test that proves it fires.

Faster incident response. A four-command investigation with manual correlation becomes one question. The agent follows the same path a senior SRE would: overview first, then the unhealthy pods, then logs or events depending on the failure type.

Reports you can trust and audit. Every conclusion cites its source tool, the grounding check filters invented resources, and the full trace is on disk. When the report says “missing DB_PASSWORD ”, you can see the exact log line it read.

Safe by default. The agent only has read-only tools, and the whitelist means it cannot call an action it wasn’t given — however confidently it hallucinates one.

No model lock-in. Any OpenAI-compatible endpoint works: OpenAI, DeepSeek, Qwen, or a local Ollama model for teams that can’t send cluster data to a third party. No data leaves your network unless you want it to.

Small enough to own. About 350 lines of agent code and no framework. Your team can read it in an afternoon, audit it before a security review, and change it without fighting abstractions.

A pattern, not just a tool. Nothing in the guardrail layer is specific to Kubernetes. Swap the MCP server, and the same loop can diagnose databases, CI pipelines, or cloud billing anomalies. Any read-only investigation over structured tools fits.

Being honest about limits is part of building something people can trust:

Next on the roadmap: schema-validated structured reports, an offline evaluation set built from saved traces, and approval-gated write actions.

The pager still fires. But now the first thing you run is a single question — and in the time it takes to open your laptop, you have a report that names the broken pods, explains why each is broken, cites the evidence, and suggests a fix.

The LLM does the reasoning. A few hundred lines of deterministic code keep it honest.

Diagnosing unhealthy Kubernetes Pods usually means manually chaining kubectl get pods, describe, logs, and get events until the root cause shows up. This project automates that investigation with a hand-written Agent Loop — no agent framework, just the low-level openai and mcp SDKs — so an LLM calls Kubernetes tools over MCP, digs in over multiple rounds on its own, and produces a structured cluster health report.