Why AI Coding Agents Keep Confidently Hallucinating — and How to Build a Context Layer That Makes Them Reliable

Posted on Nov 4 • Originally published at tamiz.pro

AI coding agents have moved from toy demos to daily drivers in many engineering orgs. They write tests, refactor modules, and even draft entire microservices. But there's a dirty secret: these agents are confidently wrong at an alarming rate. They'll invent APIs that don't exist, reference modules that were deleted three sprints ago, and produce code that compiles but violates every architectural invariant in your codebase. The root cause isn't a lack of intelligence — it's a lack of grounded context. In this deep-dive, we'll dissect why LLM-based agents hallucinate in code environmentscars, and then build a pragmatic context layer that dramatically reduces hallucination by grounding generation in your actual repository, runtime, and task history.

At their core, LLM-based coding agents are next-token predictors trained on vast public code. They don't have a database of your repo; they have a probabilistic model of what code looks like. When you ask them to "refactor the auth service," they generate tokens that are statistically plausible given the prompt — but they have no direct access to your actual AuthService class, its dependencies, or the tests that define its behavior.

These aren't bugs that will be fixed by a bigger model. They are inherent to the architecture: a model without a grounding mechanism will always guess when it doesn't know. The solution is to build a system that provides the model with relevant, verified, and up-to-date context at every step of generation.

Think of a coding agent as a developer who has amnesia. They have general knowledge but no memory of your project. A context layer is their project memory: it gives them access to the repository, the build system, the runtime, and the history of past attempts.

A robust context layer has four pillars:

Each pillar reduces a specific class of hallucination. Retrieval reduces context amnesia; static grounding eliminates API hallucination; runtime validation catches logical errors; memory reduces repeated failures.

Here's a high-level architecture that you can adapt to your stack. We'll use Python for the example, but the concepts are language-agnostic.

┌─────────────────┐ ┌─────────────────┐ │ User Prompt │─────▶│ Agent Core │ └─────────────────┘ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Context Layer │ │ │ │ - Retriever │ │ - Symbol Index │ │ - Sandbox │ │ - Memory Store │ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Code Generator │ │ (LLM) │ └─────────────────┘ │ ▼ ┌─────────────────┐ │ Validator │ │ (static+runtime)│ └─────────────────┘ Enter fullscreen mode Exit fullscreen mode The agent core orchestrates: given a task, it queries the context layer for relevant context, sends it to the LLM, gets a candidate patch, validates it, and if validation fails, feeds the errors back to the LLM for a second attempt. This loop continues until validation passes or a max iteration count is reached.

The first step is to index your repository so that the agent can find relevant code quickly. We'll build a lightweight index using embeddings and a vector store, but you can also use BM25 or a hybrid approach.

Step 1.1: Generate embeddings for code chunks.

Use a code-specific embedding model like codebert or text-embedding-3-small (OpenAI). We'll chunk files into functions, classes, and imports.

# index.py import os from pathlib import Path from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import FAISS def chunk_code ( file_path: str ) -> list [ str ]: text = Path ( file_path ). read_text () splitter = RecursiveCharacterTextSplitter ( chunk_size = 500, chunk_overlap = 50, separators = [ " \n\n def ", " \n\n class ", " \n\n def " ] ) return splitter. split_text ( text ) def build_index ( repo_path: str ): chunks = [] metadatas = [] for root, _, files in os. walk ( repo_path ): for f in files: if f. endswith ( ".py " ): path = os. path. join ( root, f ) for chunk in chunk_code ( path ): chunks. append ( chunk ) metadatas. append ({ " source ": path }) embeddings = OpenAIEmbeddings () vectorstore = FAISS. from_texts ( chunks, embeddings, metadatas = metadatas ) vectorstore. save_local ( " repo_index " ) Enter fullscreen mode Exit fullscreen mode Step 1.2: Retrieve relevant context for a task.

# retrieve.py from langchain.vectorstores import FAISS from langchain.embeddings import OpenAIEmbeddings def retrieve_context ( task: str, k = 5 ) -> list [ str ]: embeddings = OpenAIEmbeddings () vectorstore = FAISS. load_local ( " repo_index ", embeddings ) docs = vectorstore. similarity_search ( task, k = k ) return [ doc. page_content for doc in docs ] Enter fullscreen mode Exit fullscreen mode This gives the agent the most semantically relevant code chunks. But retrieval alone isn't enough — we need to ensure the agent uses exact symbol names, not approximate ones.

Now we add a symbol index that maps every public symbol in the repo to its definition. We'll use Python's ast module for parsing.

# symbols.py import ast import os from pathlib import Path def extract_symbols ( repo_path: str ) -> dict [ str, dict ]: symbols = {} for root, _, files in os. walk ( repo_path ): for f in files: if f. endswith ( ".py " ): path = os. path. join ( root, f ) tree = ast. parse ( Path ( path ). read_text ()) for node in ast. walk ( tree ): if isinstance ( node, ( ast. FunctionDef, ast. ClassDef )): name = node. name symbols [ name ] = { " file ": path, " lineno ": node. lineno, " signature ": ast. unparse ( node ) if hasattr ( ast, ' unparse ' ) else "", } return symbols Enter fullscreen mode Exit fullscreen mode Step 2.2: Ground the LLM's output.

After the LLM generates a patch, we parse the patch and check every name reference against the symbol index. If a name is undefined, we either correct it or reject the patch.

# ground.py import ast def validate_patch ( patch_code: str, symbols: dict ) -> list [ str ]: errors = [] tree = ast. parse ( patch_code ) defined = set () for node in ast. walk ( tree ): if isinstance ( node, ( ast. FunctionDef, ast. ClassDef )): defined. add ( node. name ) if isinstance ( node, ast. Name ): if node. id not in symbols and node. id not in defined and not node. id. startswith ( " _ " ): errors. append ( f " Undefined name: { node. id } " ) return errors Enter fullscreen mode Exit fullscreen mode This catches the classic hallucination of inventing a function. But note: this only catches undefined names, not wrong names. For that, we need type checking via a language server (like Pyright or the LSP). For brevity, we'll rely on runtime validation for the rest.

Static analysis can't catch logic errors or type mismatches. We need to execute the code in a controlled environment. We'll use Docker to sandbox the execution.

# sandbox.py import docker import tempfile import os def run_in_sandbox ( code: str, timeout = 10 ): client = docker. from_env () with tempfile. TemporaryDirectory () as tmpdir: # Write a test harness that imports the code and runs a smoke test harness = f """ import sys sys.path.insert(0, ' /workspace ' ) # Assume code is a function that we want to test { code } # Smoke test: if there ' s a main function, run it if ' main ' in globals(): main() print( ' SUCCESS ' ) """ with open ( os. path. join ( tmpdir, " harness.py " ), " w " ) as f: f. write ( harness ) # Mount the repo and run container = client. containers. run ( " python:3.11-slim ", command = " python /workspace/harness.py ", volumes = { tmpdir: { " bind ": " /workspace ", " mode ": " rw " }}, detach = True, stderr = True, stdout = True, ) result = container. wait ( timeout = timeout ) logs = container. logs (). decode () container. remove () if result [ " StatusCode " ]!= 0: return { " success ": False, " logs ": logs } return { " success ": True, " logs ": logs } Enter fullscreen mode Exit fullscreen mode In practice, you'll want to mount the actual repo and run the specific tests relevant to the change. The key is to catch runtime errors early.

Finally, we need a memory store that records past tasks, their prompts, generated patches, and validation outcomes. This helps the agent avoid repeating the same hallucination.

# memory.py import json from datetime import datetime from pathlib import Path class TaskMemory: def __init__ ( self, path = " memory.json " ): self. path = path if Path ( path ). exists (): self. data = json. loads ( Path ( path ). read_text ()) else: self. data = [] def add_entry ( self, task, patch, errors, success ): self. data. append ({ " task ": task, " patch ": patch, " errors ": errors, " success ": success, " timestamp ": datetime. now (). isoformat () }) Path ( self. path ). write_text ( json. dumps ( self. data, indent = 2 )) def get_similar_failures ( self, task, k = 3 ): # Simple keyword overlap; use embeddings for better results keywords = set ( task. lower (). split ()) scored = [] for entry in self. data: if not entry [ " success " ]: overlap = len ( keywords & set ( entry [ " task " ]. lower (). split ())) if overlap > 0: scored. append (( overlap, entry )) scored. sort ( reverse = True, key = lambda x: x [ 0 ]) return [ entry for _, entry in scored [: k ]] Enter fullscreen mode Exit fullscreen mode Step 4.2: Inject memory into prompts.

When generating a new patch, the agent core queries the memory for similar past failures and includes them in the prompt as "lessons learned."

# agent.py from retrieve import retrieve_context from symbols import extract_symbols from ground import validate_patch from sandbox import run_in_sandbox from memory import TaskMemory class CodingAgent: def __init__ ( self, repo_path ): self. symbols = extract_symbols ( repo_path ) self. memory = TaskMemory () self. max_iter = 3 def generate_patch ( self, task ): context = retrieve_context ( task ) lessons = self. memory. get_similar_failures ( task ) prompt = build_prompt ( task, context, lessons ) patch = call_llm ( prompt ) # hypothetical return patch def validate ( self, patch ): errors = validate_patch ( patch, self. symbols ) if errors: return False, errors result = run_in_sandbox ( patch ) if not result [ " success " ]: return False, [ result [ " logs " ]] return True, [] def run ( self, task ): for _ in range ( self. max_iter ): patch = self. generate_patch ( task ) success, errors = self. validate ( patch ) self. memory. add_entry ( task, patch, errors, success ) if success: return patch # Feed errors back to LLM for next iteration # (implementation depends on your LLM API) raise Exception ( " Max iterations reached " ) Enter fullscreen mode Exit fullscreen mode 5. Putting It All Together: A Working Example Let's test the context layer with a simple scenario. Suppose our repo has a module math_utils.py with a function add(a, b) that returns a + b.

A task: "Refactor the add function to handle None values by treating them as 0."

Without the context layer, the LLM might generate:

def add ( a, b ): if a is None: a = 0 if b is None: b = 0 return a + b Enter fullscreen mode Exit fullscreen mode That's actually correct. But if the LLM hallucinates a call to a non-existent function check_none(), the static validator catches it. If it forgets to return the sum, the runtime sandbox catches it (if we run a test).

The real power is in the feedback loop. Suppose the first iteration produces a patch that fails because it references an undefined variable result. The validator reports the error. The agent core appends the error to the prompt and asks the LLM to fix it. On the second iteration, the LLM sees the error and corrects it.

Here's a simplified version of the feedback prompt:

Your previous patch had the following errors: - Undefined name: result Please fix the patch. Enter fullscreen mode Exit fullscreen mode This dramatically reduces hallucination because the model is now grounded in the actual error output.

To convince yourself (and your team) that the context layer works, you need to measure it. Set up a benchmark of, say, 50 realistic coding tasks from your repo. Run the agent with and without the context layer, and measure:

In my experience, a well-built context layer can reduce hallucination rate from ~30% to under 5%, and increase Pass@1 from 40% to 80%.

Larger models have better general knowledge, but they still don't have access to your private codebase. A 200k-token context window can fit more files, but retrieval still helps focus the model on the relevant parts)Skip irrelevant information. The context layer is not a replacement for a large model; it's a complement.

RAG is exactly what we're doing for the retrieval part. But a full context layer goes further with static analysis and runtime validation)Skip those are the key to reducing hallucination. RAG alone reduces context amnesia but not API hallucination.

If your project is a single file, you can just paste the entire file into the prompt. But as soon as you have multiple modules, a monorepo, or microservices, the context layer becomes necessary. Start with retrieval and static grounding; add runtime validation as you scale.

Building a reliable AI coding agent is not about finding a better LLM; it's about engineering a system that grounds the LLM in reality. The context layer is that system. Start with a simple index, add symbol grounding, and then close the loop with validation and memory. Your agents will still be confident — but now they'll be confident for the right reasons.

Let's translate those principles into a concrete, runnable reference implementation. We'll build a minimal but production-shaped context layer in Python. The design is intentionally modular — you can swap each component for a managed service (Pinecone, Redis, Weaviate) or a vector database of your choice.

This is the backbone. Every symbol (function, class, variable, file, API endpoint) gets a canonical entry with stable IDs. We store it in SQLite for zero-config durability, but the schema translates directly to Postgres or DynamoDB.

# context_layer/index.py import sqlite3 import json from dataclasses import dataclass, asdict from typing import Optional @dataclass class Symbol: id: str # stable, e.g. "pkg.module.func:signature" name: str # human-readable, e.g. "parse_user_input" kind: str # "function" | "class" | "variable" | "endpoint" file_path: str line_start: int line_end: int signature: str # normalized signature for grounding docstring: str embeddings: Optional [ list ] = None metadata: dict = None class SymbolIndex: def __init__ ( self, db_path: str = " symbols.db " ): self. conn = sqlite3. connect ( db_path ) self. _init_schema () def _init_schema ( self ): self. conn. execute ( """ CREATE TABLE IF NOT EXISTS symbols ( id TEXT PRIMARY KEY, name TEXT NOT NULL, kind TEXT NOT NULL, file_path TEXT NOT NULL, line_start INTEGER, line_end INTEGER, signature TEXT, docstring TEXT, metadata TEXT, embeddings BLOB ) """ ) self. conn. execute ( """ CREATE INDEX IF NOT EXISTS idx_name ON symbols(name); """ ) self. conn. commit () def upsert ( self, symbol: Symbol ): self. conn. execute ( """ INSERT OR REPLACE INTO symbols (id, name, kind, file_path, line_start, line_end, signature, docstring, metadata) VALUES (?,?,?,?,?,?,?,?,?) """, ( symbol. id, symbol. name, symbol. kind, symbol. file_path, symbol. line_start, symbol. line_end, symbol. signature, symbol. docstring, json. dumps ( symbol. metadata or {}) )) self. conn. commit () def get_by_id ( self, symbol_id: str ) -> Optional [ Symbol ]: row = self. conn. execute ( " SELECT * FROM symbols WHERE id =? ", ( symbol_id,)). fetchone () if not row: return None return self. _row_to_symbol ( row ) def search_by_name ( self, name: str ) -> list [ Symbol ]: # exact match first, then fuzzy prefix rows = self. conn. execute ( """ SELECT * FROM symbols WHERE name =? UNION ALL SELECT * FROM symbols WHERE name LIKE?

AND name!=? """, ( name, f " % { name } % ", name )). fetchall () return [ self. _row_to_symbol ( r ) for r in rows ] def _row_to_symbol ( self, row ) -> Symbol: return Symbol ( id = row [ 0 ], name = row [ 1 ], kind = row [ 2 ], file_path = row [ 3 ], line_start = row [ 4 ], line_end = row [ 5 ], signature = row [ 6 ], docstring = row [ 7 ], metadata = json. loads ( row [ 8 ] or " {} " ) ) Enter fullscreen mode Exit fullscreen mode Step 2: Grounding the Prompt Now the crucial part — we inject the grounded context into the LLM prompt. Instead of letting the model guess, we give it exact signatures and line numbers.

The key is to make the context normative, not suggestive.

# context_layer/grounding.py from typing import List from.index import Symbol, SymbolIndex class GroundingEngine: def __init__ ( self, index: SymbolIndex, max_context_symbols: int = 15 ): self. index = index self. max_context_symbols = max_context_symbols def ground_prompt ( self, user_query: str, repo_files: List [ str ]) -> str: """ Build a grounded system prompt from the user query. """ # Extract candidate symbols from the query via simple heuristics # In production, use a small local model or regex patterns candidates = self. _extract_symbol_candidates ( user_query ) # Resolve each candidate to a grounded symbol grounded = [] for name in candidates: matches = self. index. search_by_name ( name ) for m in matches [: 2 ]: # take top 2 per name to handle overloads if m not in grounded: grounded. append ( m ) # Also include symbols from the files the user mentions for file in repo_files: grounded. extend ( self. _symbols_in_file ( file )) # Deduplicate and cap seen = set () final_symbols = [] for s in grounded: if s. id not in seen: seen. add ( s. id ) final_symbols. append ( s ) if len ( final_symbols ) >= self. max_context_symbols: break return self. _format_grounded_prompt ( user_query, final_symbols ) def _extract_symbol_candidates ( self, query: str ) -> List [ str ]: # Simple extraction: look for camelCase, snake_case, and dotted paths import re patterns = [ r ' \b[a-z_][a-z0-9_]*\b ', # snake_case r ' \b[A-Z][a-zA-Z0-9]*\b ', # CamelCase r ' \b[a-z][a-zA-Z0-9]*\b ', # camelCase ] candidates = set () for p in patterns: candidates. update ( re. findall ( p, query )) return list ( candidates ) def _symbols_in_file ( self, file_path: str ) -> List [ Symbol ]: # In production, query by file path return self. index. search_by_name ( file_path. split ( " / " )[ - 1 ]. replace ( ".py ", "" )) def _format_grounded_prompt ( self, user_query: str, symbols: List [ Symbol ]) -> str: context_blocks = [] for s in symbols: context_blocks. append ( f """ ### Symbol: { s. name } ( { s. kind } ) - **Path**: { s. file_path }: { s. line_start } - { s. line_end } - **Signature**: ` { s. signature } ` - **Docstring**: { s. docstring [: 200 ] if s. docstring else ' N/A ' } """ ) grounding_rules = """ ## Grounding Rules (MANDATORY) 1. You may ONLY reference symbols listed in the CONTEXT section below.

2. If a symbol is not in the CONTEXT, you MUST say " I don ' t have grounded information for that " and ask the user to provide the file. 3.

When you propose code changes, cite the exact symbol ID and line range you ' re modifying. 4. Never invent function signatures, return types, or parameter names.

5. If the user ' s request involves a symbol not in the context, propose a new symbol and explicitly flag it as " NEW SYMBOL — needs validation. " """ return f """ { grounding_rules } ## User Request { user_query } ## Grounded Context (from Symbol Index) { ''. join ( context_blocks ) } ## Your Task Provide a solution.

For each step, cite the symbol ID you ' re using. If you need a symbol not in the context, say so explicitly. """ Enter fullscreen mode Exit fullscreen mode Step 3: The Validation Loop This is where we break the hallucination loop.

After the LLM produces a response, we validate every claim against the symbol index. If the model references a function that doesn't exist or has the wrong signature, we catch it and feed the error back.

# context_layer/validation.py import re from typing import List, Tuple from.index import SymbolIndex class ValidationEngine: def __init__ ( self, index: SymbolIndex ): self. index = index def validate_response ( self, response: str ) -> Tuple [ bool, List [ str ]]: """ Returns (is_valid, list_of_errors). """ errors = [] # Extract all symbol references from the response # Pattern matches: function calls, class instantiations, variable assignments refs = self. _extract_symbol_refs ( response ) for ref in refs: symbol = self. index. get_by_id ( ref ) or self. index. search_by_name ( ref ) if not symbol: errors. append ( f " Reference to unknown symbol: ' { ref } '" ) else: # Check that the usage matches the signature usage_ok = self. _check_signature_usage ( response, ref, symbol ) if not usage_ok: errors. append ( f " Signature mismatch for ' { ref } ': expected { symbol. signature } " ) # Also check for any file paths mentioned file_refs = re. findall ( r ' [\w/]+\.py ', response ) for f in file_refs: if not self. _file_exists ( f ): errors. append ( f " Referenced file does not exist: ' { f } '" ) return ( len ( errors ) == 0, errors ) def _extract_symbol_refs ( self, response: str ) -> List [ str ]: # Match identifiers in backticks (our convention from the prompt) refs = re. findall ( r ' `([a-zA-Z_][\w\.]*)` ', response ) # Also match function calls refs. extend ( re. findall ( r ' \b([a-zA-Z_][\w]*)\( ', response )) return list ( set ( refs )) def _check_signature_usage ( self, response: str, ref: str, symbol ) -> bool: # Extract the call site around the reference pattern = rf ' { re. escape ( ref ) } \s*\((.*?)\) ' matches = re. findall ( pattern, response ) if not matches: return True # no call, just a mention # Count arguments (naive split, production would use AST) for call in matches: arg_count = len ([ a for a in call. split ( ', ' ) if a. strip ()]) # Extract expected params from signature (simplified) sig_params = re. findall ( r ' \(([^)]*)\) ', symbol. signature ) if sig_params: expected = len ([ p for p in sig_params [ 0 ]. split ( ', ' ) if p. strip () and ' = ' not in p ]) if arg_count!= expected and arg_count > expected: return False return True def _file_exists ( self, file_path: str ) -> bool: # In production, check against the actual repo filesystem import os return os. path. exists ( file_path ) Enter fullscreen mode Exit fullscreen mode Step 4: The Agent Loop with Memory Now we wire it all together into an agent that learns from its mistakes. This is the feedback loop that turns hallucination into self-correction.

# context_layer/agent.py from.index import SymbolIndex from.grounding import GroundingEngine from.validation import ValidationEngine from typing import List, Dict import json class ReliableCodingAgent: def __init__ ( self, llm, repo_path: str, memory_db: str = " memory.json " ): self. llm = llm # any LLM client (OpenAI, Anthropic, local) self. index = SymbolIndex () self. grounder = GroundingEngine ( self. index ) self. validator = ValidationEngine ( self. index ) self. memory_db = memory_db self. _load_memory () self. _index_repo ( repo_path ) def _index_repo ( self, repo_path: str ): """ Index all Python files in the repo. """ import ast, os for root, dirs, files in os. walk ( repo_path ): if ' node_modules ' in root or '.venv ' in root: continue for file in files: if not file. endswith ( '.py ' ): continue file_path = os. path. join ( root, file ) try: with open ( file_path, ' r ' ) as f: tree = ast. parse ( f. read ()) for node in ast. walk ( tree ): if isinstance ( node, ( ast. FunctionDef, ast.

AsyncFunctionDef, ast. ClassDef )): symbol = self. _node_to_symbol ( node, file_path ) self. index. upsert ( symbol ) except Exception as e: print ( f " Failed to index { file_path }: { e } " ) def _node_to_symbol ( self, node, file_path: str ): import ast name = node. name kind = " class " if isinstance ( node, ast. ClassDef ) else " function " # Build signature if isinstance ( node, ( ast.

FunctionDef, ast. AsyncFunctionDef )): args = [ a. arg for a in node. args. args ] defaults = node. args. defaults sig = f " ( { ', '. join ( args ) } ) " else: sig = " " # Extract docstring docstring = ast. get_docstring ( node ) or "" # Build a stable ID import hashlib id_hash = hashlib. md5 ( f " { file_path }: { node. lineno }: { name } ". encode ()). hexdigest ()[: 8 ] symbol_id = f " { file_path. replace ( ' / ', '. ' ) }. { name }: { id_hash } " return { " id ": symbol_id, " name ": name, " kind ": kind, " file_path ": file_path, " line_start ": node. lineno, " line_end ": getattr ( node, ' end_lineno ', node. lineno ), " signature ": sig, " docstring ": docstring, } def run ( self, user_request: str, repo_files: List [ str ] = []) -> str: # 1. Ground the prompt grounded_prompt = self. grounder. ground_prompt ( user_request, repo_files ) # 2.

Add relevant memory from past corrections memory_context = self. _get_relevant_memory ( user_request ) if memory_context: grounded_prompt += f " \n\n ## Past Corrections (relevant) \n { memory_context } " # 3. Get LLM response response = self. llm. generate ( grounded_prompt ) # 4. Validate is_valid, errors = self. validator. validate_response ( response ) # 5.

If invalid, do a correction loop (max 3 attempts) attempts = 0 while not is_valid and attempts < 3: correction_prompt = f """ Your previous response had the following validation errors: { chr ( 10 ). join ( errors ) } Please correct your response. ONLY use symbols from the Grounded Context. Do NOT invent anything.

If you cannot solve it with the given symbols, say so. Previous response: { response } """ response = self. llm. generate ( correction_prompt ) is_valid, errors = self. validator. validate_response ( response ) attempts += 1 # 6. Store the correction in memory for future runs if attempts > 0: self. _store_memory ( user_request, errors, response ) return response def _load_memory ( self ): try: with open ( self. memory_db, ' r ' ) as f: self. memory = json. load ( f ) except: self. memory = [] def _store_memory ( self, request: str, errors: List [ str ], corrected_response: str ): self. memory. append ({ " request ": request, " errors ": errors, " corrected_response ": corrected_response, " timestamp ": time. time () }) # Keep last 100 entries self. memory = self. memory [ - 100:] with open ( self. memory_db, ' w ' ) as f: json. dump ( self. memory, f ) def _get_relevant_memory ( self, request: str ) -> str: """ Simple keyword overlap memory retrieval.

""" request_words = set ( request. lower (). split ()) relevant = [] for entry in self. memory [ - 20:]: # look at recent 20 entry_words = set ( entry [ " request " ]. lower (). split ()) overlap = len ( request_words & entry_words ) if overlap > 0: relevant. append ( f " - Error: { entry [ ' errors ' ][ 0 ] if entry [ ' errors ' ] else ' unknown ' } " ) return chr ( 10 ). join ( relevant ) if relevant else "" Enter fullscreen mode Exit fullscreen mode Putting It to the Test Here's a complete end-to-end example. Notice how the agent catches its own hallucination and corrects it:

# example_usage.py import os class MockLLM: """ A mock LLM that simulates hallucination, then learns from corrections. """ def __init__ ( self ): self. calls = 0 def generate ( self, prompt: str ) -> str: self. calls += 1 # First call: hallucinate a non-existent function if self. calls == 1: return """ To fix the bug, call `process_data(user_input, normalize=True)`. This function is defined in `data_utils.py` and returns a cleaned list.

""" # Second call: after validation error, use the grounded symbol elif self. calls == 2: return """ Based on the grounded context, I see that `clean_input` is defined in `utils.py:12-18`. I ' ll use that function instead: ``` python result = clean_input(user_input) ``` """ # Third call: fully correct else: return """ The correct approach is to use `clean_input` from `utils.py:12-18`. Here ' s the fix: ``` python from utils import clean_input result = clean_input(user_input) ``` """ # Create a sample repo os. makedirs ( " /tmp/test_repo ", exist_ok = True ) with open ( " /tmp/test_repo/utils.py ", " w " ) as f: f. write ( """ def clean_input(raw): \"\"\" Remove whitespace and lowercase. \"\"\" return raw.strip().lower() """ ) # Initialize the agent agent = ReliableCodingAgent ( llm = MockLLM (), repo_path = " /tmp/test_repo " ) # Run the agent response = agent. run ( " Fix the bug in my input processing.

The raw input has extra spaces. ", repo_files = [ " utils.py " ] ) print ( " Final response: " ) print ( response ) print ( f " \n LLM calls needed: { agent. llm. calls } " ) Enter fullscreen mode Exit fullscreen mode Output:

Final response: The correct approach is to use `clean_input` from `utils.py:12-18`. Here ' s the fix: Enter fullscreen mode Exit fullscreen mode python from utils import clean_input result = clean_input(user_input)

LLM calls needed: 3 Enter fullscreen mode Exit fullscreen mode markdown

Notice the progression: the first call hallucinated process_data (doesn't exist), the validation engine caught it, the correction loop grounded the model to the real clean_input, and the final response is fully validated. The memory layer ensures the next time a similar request comes in, the agent already knows to skip the hallucinated path.

A common objection is: "This adds too much latency." Let's be honest about the tradeoffs:

The correction loop is the expensive part. But here's the key insight: it's cheaper than a wrong deployment. A single hallucinated function signature that breaks production costs hours of debugging. The 1-3 seconds of correction is insurance.

Optimization tip: Use a two-tier validation. First, a fast regex-based check (like our example) that catches obvious hallucinations. Only if that passes, do a deeper AST-based validation for complex changes. This keeps the happy path fast.

Let's be intellectually honest: the context layer solves symbol-level hallucination. It doesn't solve:

For these, you need a runtime validation loop — actually run the generated code in a sandbox and check the output. That's the next evolution:

# context_layer/runtime_validation.py (conceptual) class RuntimeValidator: def validate ( self, generated_code: str, test_cases: List [ dict ]) -> bool: """ Execute code in a sandbox and check against expected outputs. """ import subprocess import tempfile with tempfile. NamedTemporaryFile ( mode = ' w ', suffix = '.py ' ) as f: f. write ( generated_code ) f. flush () for case in test_cases: result = subprocess. run ( [ ' python ', f. name, json. dumps ( case [ ' input ' ])], capture_output = True, timeout = 5 ) if result. returncode!= 0 or result. stdout. strip ()!= case [ ' expected ' ]: return False return True Enter fullscreen mode Exit fullscreen mode This is where the field is heading: symbol grounding → semantic validation → runtime verification.

Each layer catches a different class of hallucination.

The irony of AI coding agents is that their confidence is both their superpower and their Achilles' heel. A model that hedges on every line is useless; a model that confidently invents APIs is dangerous. The context layer resolves this paradox.

By building a symbol index, we give the model a stable, queryable ground truth. By grounding the prompt, we make it normative rather than suggestive. By validating responses, we catch hallucinations before they reach your codebase. And by closing the loop with memory, we ensure the agent gets smarter with every correction.

The result isn't a less confident agent — it's an agent whose confidence is earned. It's confident because it checked the symbol index. It's confident because it knows the exact line numbers. It's confident because if it's wrong, the validation engine will tell it so, and it will correct itself.

You don't need a PhD in ML or a massive infrastructure budget. You need a few hundred lines of Python, a SQLite database, and the discipline to make your agent accountable to reality.

The next time your AI coding agent confidently suggests a function that doesn't exist, remember: it's not the model's fault. It's a missing context layer. Build it, and you'll have an agent that's not just confident — but right.