Article: Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures

InfoQ Homepage Articles Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures

Explore how Azul helps Java teams reduce JVM vulnerability risk, strengthen application security, and prepare their Java environments for agentic AI. Discover practical strategies for securing Java workloads while maintaining performance and reliability. Learn more.

Agents must reason across tools, sessions, users, and live systems, given that in an engineered system we can describe the context as a control plane. Context as a control plane is an architectural paradigm in which organizational, behavioral, and situational data (the who, what, and why) serve as the centralized governing layer for system operations, rather than being hardcoded directly into individual applications, AI models, or microservices. In our world, this will determine whether an agent acts like a reliable software system. Providing the wrong context highly increases the likelihood of the model hallucinating.

Knowledge of the system is our policies, governance, and hard truth. Knowledge acts as the system’s official textbook. It holds core business logic, strict policies, and databases. Back to our omelet example, part of the knowledge would be the need for eggs and a pan.

Memory is the critical layer that bridges historical data and real-time signals into a reasoning-ready state. What needs to be remembered for the current task? This is usually information given from knowledge, historical data, and current live events.

Our goal in agentic systems is, when given the task context, system knowledge, and memory, to turn data into the right memory state at the right moment.

Beyond that, there are limitations to this dynamic architecture paradigm; hence, we need to systematically address the economic costs of large language model (LLM) limitations and the technical, architectural roots of hallucinations. As developers and architects, learning to avoid the common "culprits" of agentic systems mismanagement is crucial. Dive in to learn how to design a sophisticated, multi-tiered architecture for the next generation of reliable agentic AI.

An LLM is only as good as how we use it. While the model generates, our system determines what it sees. To avoid hallucinations, we need a trustworthy system around it.

Leveraging context, knowledge, and memory. To architect the system, we will separate the model from the surrounding machinery. Going forward, for the machinery around it, we will call it a context engine.

The context engine is part of the context control plane. This is the orchestrator responsible for selecting, filtering, ranking, structuring, compressing, safeguarding, and delivering information before it ever hits the LLM inference endpoint. It pulls information from knowledge, categorizes it in memory, and determines what the agent's context should be at any given millisecond.

This is a meaningful shift from simple prompt engineering to systems engineering.

Prompt engineering assumes the task is to craft a better instruction. Context engineering assumes the real task is to build a dynamic substrate that continuously assembles the right state for reasoning. The prompt is no longer about "What should I ask the model?" It is "What is the minimal, highest-value context the model needs in order to reason well right now?"

In a production-grade agentic architecture, we require dynamic systems that dictate what an agent "knows" at any given millisecond. Our system goal is to take what’s relevant and compile a new prompt encompassing the required information to improve the model's success likelihood.

To understand how the Context Engine behaves, consider the analogy of a modern software compiler. A compiler translates high-level source code into machine-executable instructions; the context engine translates vast oceans of unstructured, disconnected raw, real-time, and historical data into a structured, reasoning-ready state. It builds a deterministic state machine that allows the probabilistic agent to execute safely. It ensures that inputs are authenticated, validated against business logic, stripped of personally identifiable information (PII), and securely sandboxed before the LLM generates a plan of action.

Effective software architecture for autonomous AI agents requires reconciling several fundamentally conflicting paradigms of data management and data flow.

A robust agentic architecture must gracefully balance long-term "episodic" memory with immediate "sensory" input. Meaning that to build a useful context engine, our architecture must reconcile two very different categories of state: historical memory and real-time input.

Historical memory is the long-lived layer. It includes prior interactions, past decisions, user preferences, domain rules, internal documentation, and accumulated observations. This is what gives an agent continuity. It allows the system to behave as though it has experience.

Real-time input is the short-lived layer. It includes the current task, the latest events, live telemetry, fresh tool outputs, and environmental changes. This is what keeps the agent grounded in the present.

While vector databases excel at retrieving historical, semantic facts, agentic workflows frequently pivot on sub-second, real-time context. For example, if an AI DevOps agent is triaging a server outage, retrieving an architectural runbook from a vector store (episodic memory) is only half the battle. The agent also needs the live error log stream from the last five seconds (sensory memory). The architecture must fuse these batch and streaming sources asynchronously, ensuring the agent acts on a unified, up-to-date reality.

This architecture must have balance. If it over-indexes on history/episodic memory, the agent becomes stale. It will keep replaying yesterday’s assumptions into today’s reality. If it over-indexes on real-time data, the agent becomes reactive and forgetful. It may respond to the latest signal but fail to incorporate the broader context that gives that signal meaning.

Luckily for us, this tension is not new. Distributed systems have always had to reconcile durable state with transient events. Agent systems inherit the same problem, but with an additional challenge: the reasoning layer is probabilistic, expensive, and bounded by a limited context window. This adds an important layer of constraint optimization to the context architecture design - deciding which information to keep, which can be compressed, which must be revalidated, and which should be excluded entirely. The goal is not just freshness or completeness. The goal is usefulness under constraint.

Figure 1: Historical Data vs Real-Time Memory Streams (Source: created by the author with Gemini)

One of the most common architectural mistakes in AI systems is treating retrieval as if it were enough, ie, operating under the semantic equivalent of a SQL SELECT *, assuming more information must be better. Architectures that attempt to consistently retrieve large volumes of data will inevitably fail at scale. We can think of retrieval as how we get data into the system, yet it does not guarantee that the data is useful, timely, coherent, or appropriately ranked for the decision at hand.

This is what standard RAG pipelines are all about. They are often built to retrieve raw, fragmented, and overlapping data chunks. In actuality, using this approach would most likely result in the system dumping 10 unedited wiki pages into a prompt, making the LLM waste compute cycles trying to sort the noise from the signal - yet failing due to relevant signals buried under less important and more noisy ones.

Hence, the right information and shape of the data, which we refer to as knowledge, matter more than raw data.

True Knowledge Curation focuses on cumulative insight. Instead of injecting raw data, the architecture employs background workers/ML pipelines that distill, aggregate, and rank retrieved data by relevance, using the up-to-date system state. To achieve that, we need to build dedicated pipelines that transform raw data into structured knowledge graphs or condensed summaries. For example, taking a large database and creating a pipeline that summarizes a million data entries into one entry every 12 hours. This way, we ensure the model receives a small, synthesized body of knowledge (1 entry point in the database) rather than a chaotic stream of consciousness (the whole database, sometimes).

Foundational models like GPT-4 and Claude, as well as vector databases like Pinecone and Milvus, have become an accessible commodity. Everyone can access them today through their services. Yet, this is not the same for context; the architectural principles for working with context are highly domain-specific.

Every company should design its own context based on the business domain and the product it serves. For example, a fintech firm might choose to curate context for an autonomous fraud-analysis agent by prioritizing transaction timestamps, geolocation anomalies, and strict compliance rules. This will look entirely different for a legal tech firm.

A legal tech firm might choose to curate context based on state law for use with an agent that generates contracts. We are learning that the context itself cannot be generic; it must reflect the unique domain logic, security boundaries, and vocabulary of the specific product.

Treating context as a continuous lifecycle allows us, as engineering teams, to manage knowledge systematically and directly address the root causes of model failure and prohibitive inference costs. The Lifecycle Stages

Generally speaking, there are 4 stages: Selection, Injection, Compression, Expiration & Refresh. Figure 2 describes how each stage could be a dedicated module in our architecture or a standalone service.

Figure 2: Agentic AI Systemic Context Lifecycle (Source: created by the author with Gemini)

The first and most important step. This is where the system identifies which data points are likely to matter for the current task. The system's goal is to identify the absolute highest-value data points across multiple data stores.

Advanced architectures have moved beyond simple cosine similarity in vector spaces, implementing hybrid search algorithms that combine dense semantic search with sparse keyword search (e.g., BM25) and knowledge graph traversal to ensure precision. In this process, we want to understand intent, task type, risk, user role, and temporal relevance. It is important to remember that the same underlying corpus may yield different contexts for a troubleshooting agent, a research agent, and a support summarization agent.

This is where architecture begins to diverge from product. As part of the design requirements, we need to identify and define what is risky and what is actionable.

Once the system knows what matters, it has to place that information into the model’s working context (aka context window) in a useful form. While the end result might look like simple string concatenation, the path to getting there requires more deliberate design. That design involves semantic templating, ensuring the data is formatted to match the user's explicit intent, including necessary system instructions, few-shot examples, and output schema constraints (such as JSON definitions). To understand the importance of this step, consider how it affects the entire system. In our world, this placement of information changes how models reason. Hence, order, grouping, and structured state matter.

Processing a large context is expensive. To respect token limits, historical data should be iteratively compressed. Utilizing smaller, faster LLMs (or even deterministic algorithms) to summarize conversation histories or condense long documents saves massive amounts of space and reduces latency for the primary reasoning model. Test summaries carefully to avoid distortions. There are statistical models available today to assess summary validity, such as the Perplexity (PPL) Ratio, Token-Level Kullback-Leibler (KL) Divergence, the Chi-Square Test for Categorical Outputs, and others.

Context that never expires eventually turns into a liability. Agents need mechanisms to retire stale assumptions, refresh live facts, and separate durable truths from temporary observations. A system preference might remain stable across months or change overnight.

A product configuration might change in minutes. A support incident may be relevant for a day and misleading a week later. Without expiration and refresh, systems fall into stale data loops: they keep reusing context that was once correct but is now simply persuasive.

Hence, constantly testing and validating the system through feedback loops is a must. This should be tailored to the system's end goal. For example, in a car-sharing system, the goal is to satisfy users; hence, if user satisfaction drops suddenly, it’s best to explore underlying factors that may need refreshing.

LLM hallucinations are rarely the result of a "broken" model; they are almost always the symptom of an unmanaged, impoverished context window. Because LLMs are probabilistic, when an agent lacks the explicit, factual data required to answer a prompt or execute a task, its underlying mechanism forces it to "guess" based on latent training weights.

Proactive context management acts as a strict guardrail. By saturating the model with highly relevant, high-confidence knowledge and explicitly instructing it to reason only based on the provided context, we can effectively suppress latent, inaccurate associations. In a productionized agentic system, a well-engineered context engine becomes our primary defense against hallucinated actions.

Processing a large context is notoriously expensive. Every token injected into the prompt window incurs both a computational latency cost and a strict financial cost. Token Economics is the engineering discipline of optimizing "attention real estate" within the LLM's limitations.

The recent expansion of context windows (e.g., models boasting 128k to 1M+ token limits) has tempted developers towards lazy architecture. They dump entire code repositories or massive troves of documents into the prompt. This drastically increases Time-to-First-Token (TTFT) latency, skyrockets API billing costs, and, ironically, degrades model performance.

The model suffers from the "Lost in the Middle" phenomenon, shown in Figure 3. This problem was identified by Stanford researchers Nelson F. Liu et al. in 2023, where the model fails to retrieve facts buried in the center of massive context windows and only focuses on edge case information.

The larger the bottom part of the U shape (context window), the less accurate the response will be. This research explains why we want the context sent to the model to be relatively small. This approach makes the U bottom part smaller as well - hence, improving outcomes.

Architects must weigh the marginal utility of every retrieved document against its token cost and relevancy. Just because we can send 100,000 tokens doesn't mean we should. Strict context budgeting, iterative pruning, and caching mechanisms are mandatory for scalable economics.

Figure 3: Lost in the middle problem - recall vs accuracy (Source: created by the author with Gemini - based on Lost in the Middle scientific paper by Stanford)

Designing systemic approaches to memory requires establishing firm boundaries for how agents retain, update, or discard knowledge across sessions. Drawing inspiration from human cognitive architecture, systems should be multi-tiered:

There are a few recurring failure patterns that show up again and again in agentic systems.

Implement Strict TTL (Time-To-Live) Policies

Not all memories are created equal. Treat agent memory like a standard caching layer by applying category-specific expiration dates.

The next generation of AI systems' success will be determined primarily by architecture design. While the industry continues to debate model parameters and raw capabilities, the core challenge for engineering teams looks remarkably familiar: state management, signal prioritization, fault isolation, data freshness, security, and cost control.

The winning architectures will treat themselves as essential infrastructure, curate context, view memory as a strict system design problem, and elevate knowledge curation to a first-class engineering discipline. This is the real shift from chatbots to autonomous agents.