Article: AI Changes the Data Interface, Not the Query Engine

InfoQ Homepage Articles AI Changes the Data Interface, Not the Query Engine

The question did not come from a whiteboard exercise. It appeared after a real workflow successfully produced its first report. We were analyzing a disaster-recovery (DR) islanding exercise. During the exercise, one region was intentionally isolated, and traffic was expected to fail over to surviving regions. For every service, we needed to answer three questions:

We exported service metrics at thirty-second resolution, wrote them as Parquet files to S3-compatible object storage, and let an AI agent generate a structured analysis report. Once the workflow ran end to end, an attractive idea emerged:

If the agent can read files from object storage and produce the answer, why do we need tables, a catalog, a lakehouse, or SQL at all?

The question sounds reasonable because the user-facing experience has changed. Instead of opening a dashboard or writing a query, an engineer asks the agent to analyze the exercise. The agent identifies the relevant data, generates DuckDB SQL or Python as needed, executes the analysis through deterministic tools, and explains what happened.

For example, to verify whether traffic shifted from Region A to Region B during failover, the agent may generate a read-only DuckDB query over a predefined set of tables and time ranges. Before execution, the query is validated against an allowlist of accessible datasets and operations, then executed in a sandboxed analysis runtime. The runtime returns structured query results, which deterministic application code normalizes into a fact packet containing fields such as traffic volume by region, timestamps, and percentage changes.

A simplified implementation looks like this query:

generated_sql = """ SELECT region, SUM(request_count) AS requests FROM traffic_metrics WHERE ts BETWEEN:start_time AND:end_time GROUP BY region """ validate_sql( generated_sql, allowed_tables={"traffic_metrics"}, allowed_operations={"SELECT"}, ) rows = sandbox.execute( generated_sql, params={ "start_time": exercise_start, "end_time": exercise_end, }, ) fact_packet = { "metric": "request_count", "window": [exercise_start, exercise_end], "by_region": { row["region"]: row["requests"] for row in rows }, } In this simplified example, the agent generates only the analytical query. A deterministic validation layer rejects access to non-allowlisted tables or non-read-only operations before execution. The query then runs in a sandboxed runtime and the returned rows are converted by application code into a structured fact packet.

The model receives this packet (for example, request counts for Regions A and B) rather than inventing or directly estimating those values.

The difficult part of DR analysis is not retrieving a time series. It is deciding what the time series means. Suppose Region A is isolated and Region B remains active. A healthy failover should produce a recognizable sequence:

Each statement requires context. A QPS increase is not automatically a successful failover. It may be too small to account for the traffic lost in the isolated region. A lower success rate is not automatically a DR failure if the metric includes expected business rejections. A latency spike at the cutover boundary may be harmless if the system quickly returns to a steady state. We therefore used two baselines and a steady-state window:

The analysis collected four core signal families:

For each service, deterministic code calculated facts such as traffic absorption ratio, SLO violations, latency change, and the duration of sustained degradation. The agent then interpreted those facts in the context of the service, the exercise configuration, and the available evidence. This distinction is fundamental. The hard part is often the judgment, but the judgment is only trustworthy when the facts underneath it are deterministic.

The workflow used three logical layers: data preparation, deterministic computation, and agent-driven interpretation, with reproducibility treated as a cross-cutting requirement. The data-preparation layer used reusable DuckDB build logic, which was generated once and then reused, to transform raw metrics into self-describing assets in object storage. For each analysis request, the agent selected the relevant inputs, generated constrained SQL or Python, and orchestrated deterministic execution in DuckDB/Python.

The resulting structured fact packets were then interpreted by the agent to produce the final report. Throughout the workflow, the input data, configuration, reusable build logic, deterministic facts, and agent execution metadata were retained so that the analysis could later be reconstructed, replayed, and evaluated.

Figure 1. The AI agent initiates and orchestrates the analysis, deterministic tools compute the facts, and the agent interprets the resulting fact packet. Source: Authors’ own illustration, created with assistance from ChatGPT.

The physical data lived in S3-compatible object storage. Each exercise had a self-describing directory:

s3://<bucket>/<root-prefix>/ index.json <exercise-id>/ manifest.json exercise_config.json metrics-*.parquet service_dimension.parquet The files served different purposes:

The manifest was important because a folder of files is not yet a trustworthy data asset. Consumers need to know what is present, which schema was used, whether the write completed, and whether the files can be validated later.

DuckDB was used on both sides of the storage boundary. During ingestion, it read the raw metric records, sorted them, wrote compressed Parquet files, and generated validation statistics for the manifest. During analysis, it read the Parquet files directly, filtered the relevant windows, joined service metadata, and computed the fact packet used by the agent. This architecture benefited from several properties of DuckDB:

The point was not that DuckDB is universally better than Spark, Trino, or a managed lakehouse. The point was that the workload did not yet need a shared distributed engine.

The agent did not receive hundreds of thousands of raw metric rows and attempted to count errors or average latency itself. Instead, it received a compact, deterministic evidence packet for each service. A simplified packet looked like this example:

{ "service_id": "service-a", "exercise_window": "post-cutover-steady-state", "expected_failover_qps": 1200, "observed_qps_increase": 910, "traffic_absorption_ratio": 0.758, "success_rate_baseline": 0.9992, "success_rate_during_exercise": 0.9941, "service_slo": 0.9990, "p99_latency_baseline_ms": 84, "p99_latency_during_exercise_ms": 191, "sustained_degradation_minutes": 18, "data_quality_flags": [] } The agent used the packet to classify the outcome, explain the supporting evidence, and propose the next investigation step. A corresponding output might be:

{ "classification": "partial_failover", "confidence": "high", "evidence": [ "The surviving region absorbed approximately 76% of the expected traffic.", "Availability remained below the service SLO during the steady-state window.", "The p99 latency increase was sustained for 18 minutes." ], "recommended_action": "Review routing capacity and downstream dependency limits." } The exact numbers came from the engine. The model's responsibility was to determine how those numbers should be interpreted and communicated.

A simple threshold-based rule was not always sufficient. For example, a service might temporarily exceed its p99 latency threshold immediately after cutover while traffic had already shifted correctly and availability remained within the service objective. A rule that treated any threshold violation as a failure would classify the service as degraded. The agent could instead compare the transition window with the steady-state window, recognize that latency returned to baseline within a few minutes, and classify the event as a successful failover with transient degradation rather than a sustained DR failure.

More importantly, some cases contained conflicting evidence that could not be resolved by a single threshold or temporal rule. A service might absorb the expected traffic and recover its own latency, yet still remain at risk because a downstream dependency was operating near saturation. Conversely, a Tier-2 asynchronous service could show a temporary backlog increase without violating its availability objective, while the same pattern in a Tier-0 synchronous dependency would warrant escalation.

In these cases, the deterministic layer still produced the underlying facts (i.e.,traffic shift, latency, success rate, backlog, dependency saturation, service tier, and SLO status), but the agent combined those facts with service context to determine which evidence was most important and how the event should be classified.

The agent therefore added value not by replacing deterministic computation, but by resolving context-dependent and sometimes conflicting evidence that would otherwise require an increasingly large set of service-specific rules.

The most reusable lesson from the project is what I call the determinism split: separate exact computation from contextual reasoning.

Table 1. A practical separation between exact computation and contextual reasoning.

This division avoids a common anti-pattern, using an LLM as a query engine. A language model should not read hundreds of thousands of rows and calculate an error count, percentile, or average. That approach is slow, expensive in tokens, and difficult to reproduce. More importantly, it assigns deterministic work to a probabilistic system. The safer pattern is:

The agent can decide which question to investigate, dynamically generate constrained DuckDB SQL or Python, and orchestrate its execution. The generated code may be proposed probabilistically, but the resulting numerical facts must come from deterministic execution rather than from the model itself.

The upstream collection run across all services took roughly an hour with concurrent workers. The resulting analytical dataset was compact enough that an embedded engine could process it close to the agent without introducing a shared data platform. That observation matters because architecture decisions should follow the actual workload rather than the label big data. More than a million rows can sound large in a design review, but row count alone is not a sufficient reason to deploy a distributed lakehouse stack. Column count, compression, query shape, concurrency, access patterns, and data movement all matter.

The following is a representative example based on the workflow used in the exercise; service identifiers and values have been anonymized.

For one service, the deterministic execution layer produced a fact packet showing that the surviving region absorbed approximately seventy-six percent of the expected failover traffic, availability fell below the service SLO during the steady-state window, and p99 latency remained elevated for eighteen minutes. At the same time, the service continued to serve traffic and did not experience a complete outage.

The agent interpreted these facts as a partial failover rather than a binary success or failure. Its explanation cited the traffic absorption ratio, the sustained SLO violation, and the latency degradation, and recommended reviewing routing capacity and downstream dependency limits.

The conclusion was then validated against the underlying deterministic facts and the exercise timeline. Reviewers could trace each statement in the agent’s explanation back to fields in the fact packet and rerun the same DuckDB/Python analysis against the retained input data and configuration. Once the conclusion was confirmed, the resulting action was to investigate whether routing capacity or a downstream bottleneck had prevented the surviving region from absorbing the full expected load.

This validation step was important: the agent proposed the interpretation and next action, but the numerical evidence remained reproducible and independently verifiable.

The prototype only became an engineering system after we treated reproducibility as a first-class requirement. For every exercise, we retained a replay bundle containing:

This replay bundle made it possible to rerun an analysis offline, investigate a disputed conclusion, or compare the behavior of two agent versions against the same evidence. We also encoded data semantics explicitly. One subtle example was the meaning of a missing value. Missing QPS or error-count data could represent zero activity in some contexts. Missing success-rate or latency data often meant no traffic was available to measure, not zero percent success or zero milliseconds of latency.

Filling every missing value with zero would have created false failures and false recoveries. The schema and data-quality flags needed to preserve that distinction so the engine and the agent interpreted the dataset consistently.

Agent plus DuckDB is not the final architecture for every analytical system. It works when the workload has the right shape. Four signals indicate where the lightweight architecture begins to encounter pressure. They do not require adopting a complete lakehouse stack at once. Each signal points to a different capability that can be introduced incrementally, including a catalog for shared discovery and governance, an open table format for safe concurrent writes and evolution, or a distributed query engine when computation must move closer to large shared datasets.

If dashboards, scheduled jobs, notebooks, or other teams need direct access, the data needs a stable contract. Those consumers should not have to call the original agent or reverse-engineer a bucket layout. A shared catalog allows them to discover a table by name, understand its schema, apply permissions, and query it through standard tools.

At terabyte scale, pulling files toward a local or development environment for analysis becomes inefficient. The query should execute where the data is stored (i.e., near-data processing ), with partition pruning, predicate pushdown, and distributed processing. Only the result should move back to the agent.

DuckDB can join local files efficiently when both sides are small enough. The boundary appears when the other side is an organization-wide authoritative dataset that is already registered in a shared catalog and is too large or too frequently updated to export safely. At that point, it is more reliable to register the new data in the same environment and execute the join through the shared engine.

A single-writer convention is simple and effective for an early workflow. It breaks down when multiple producers update the dataset, schemas evolve independently, or readers require consistent snapshots. An open table format such as Apache Iceberg adds table metadata, snapshots, schema evolution, and transactional commit semantics on top of the Parquet files. A managed lakehouse can then combine three capabilities:

These capabilities are valuable, but they introduce integration, governance, compute, and operational cost. They should be adopted in response to real sharing, scale, and transaction requirements, not because every AI project is assumed to need a lakehouse on day one.

Table 2. Choose the platform based on access patterns, scale, and write semantics.

The decision is not AI or SQL nor is it files or tables. The real decision is:

Is an embedded, agent-driven analytical engine sufficient, or does the workload require a managed, shared data platform?

AI agents make data analysis more accessible. They can translate an operational question into analytical steps, select relevant evidence, classify behavior, and write a report that an engineer can act on. They do not eliminate the need for precise execution.

In our DR analysis workflow, Parquet and DuckDB performed the mechanical work: storage, filtering, aggregation, validation, and joins. The agent performed the contextual work by deciding how to interpret traffic absorption, availability, and latency together, and communicating the result. The architecture remained intentionally lightweight because the real workload did not yet require a shared lakehouse.

What changed after operating the workflow was our view of where the analytical interface should live. Engineers no longer needed to interact directly with the query engine for every investigation, but the engine remained essential for producing trustworthy facts. The agent became the layer that planned the analysis, generated the required SQL or Python, and interpreted the evidence.

The main architectural trade-off is therefore not AI versus SQL, or files versus tables. It is how much analytical infrastructure needs to sit behind the agent for the workload at hand. A small embedded engine may be sufficient for a focused workflow; shared catalogs, open table formats, and distributed engines should be introduced only as sharing, scale, write concurrency, or governance create a concrete need. The engine remains; the interface changes.