When a data pipeline fails, can an AI agent explain why?

A data pipeline fails. You open the logs, find the failed run, check the input file, and try to work out what happened.

Did the source send bad data? Did processing fail? Is the evidence you need even available?

That investigation is the part I'm exploring with Hermes Data Reliability: a small AWS lab that uses Hermes Agent, built by Nous Research, to investigate failed data loads and propose a next step.

The goal is to give an engineer something useful to review: an explanation, the evidence behind it, and an honest account of what remains unclear.

The example pipeline is deliberately simple:

Producer Lambda -> S3 input -> SQS -> Processor Lambda -> S3 output Enter fullscreen mode Exit fullscreen mode The producer creates a JSON file containing five records. The processor checks that each record has a numeric amount, doubles it, and writes the result back to S3.

For the failure scenario, the producer puts this in the third record:

{ "record_id": 3, "amount": "not-a-number" } Enter fullscreen mode Exit fullscreen mode The processor rejects it during validation. Now there's a concrete failure to investigate, with a known answer we can compare the agent's explanation against.

This small example keeps the investigation easy to follow. We can inspect the input, the processing attempt, and the logs without needing to understand a whole company's data platform first.

First, collect the signal. A collector reads configured CloudWatch logs and applies matching rules defined in YAML. When a rule matches, it saves an incident in SQLite.

Then, investigate. A separate worker picks up the incident and starts Hermes. The agent chooses which available evidence to inspect within the permissions and limits set for that investigation. For a lab run, that can include nearby logs, the input file, and saved processing-attempt details.

Finally, write up the findings. The report includes the failed stage, an explanation, uncertainty, a proposed action, and references to the evidence used.

For the malformed record, a useful explanation would look something like this:

Validation failed because the third record contains a string in amount, where the processor requires a number. Correct that value at the source and review how to reprocess the affected file. The available evidence does not establish that a corrected run has succeeded.

That's an illustrative summary, rather than output from a measured live run. An actual report also needs references to evidence returned during its investigation.

The collector continues working while Hermes is busy or unavailable. Incidents and collection progress are saved in the database, so a restart can resume from saved state.

This also gives ordinary code a clear job: read the configured sources, apply the rules, and queue the matches. The agent handles the investigation after a match.

There are limits on investigation time, tool calls, and attempts. Interrupted and failed attempts still count. A difficult incident shouldn't create an endless loop of model calls.

The current investigation tools provide read-only access. Engineers decide which changes to make.

Access is enforced by code outside the model. Each tool request is checked against the approved resources, time window, and remaining budget. AWS credentials stay outside the model container.

The host also records which evidence each tool call returned. Before an automatic finding is accepted, its citations are checked against those records.

That check has a specific purpose: confirm that the cited evidence was actually returned during this investigation. It cannot prove that the agent interpreted the evidence correctly. Reviewing the explanation still matters.

Logs and file contents are treated as untrusted input, too. Text inside a log does not get to grant permissions or tell the agent to change its task.

An investigation can run into missing permissions, unavailable input, or incomplete results.

Those limitations belong in the report. A failed read doesn't establish that a file is missing. A partial search doesn't establish that no relevant event happened.

The project includes three evaluation scenarios: a malformed record, a log containing misleading instructions, and an unavailable input file. They let us examine both the diagnosis and how the agent handles evidence it should distrust or cannot inspect.

These are a starting point for evaluation. Broader reliability needs more cases and measured results.

Today, the lab focuses on explicit failures found through configured CloudWatch log matches. Detection of silent stoppages and overdue output, along with Snowflake integration, is still future work.

I'm building this to make the investigation after a pipeline failure easier: follow the evidence, explain what happened, and suggest a next step an engineer can review. I'd love to shape it around the problems you actually run into.

If you work with data pipelines, which recurring failure would you want an agent to investigate first?

Maybe it's a file that arrives late, a schema change that breaks a load, or a job that reports success while some data never reaches its destination. I'm looking for real examples I can recreate in the lab and run the agent against.

Leave a comment below or open a GitHub issue. A short description is enough to start:

A simplified setup or an anonymized example is perfect. I'd like to turn these into reproducible scenarios and test what the agent can explain, where it gets stuck, and what evidence would help it do better. Your recurring headache could help shape the next lab example.

If this is useful to you, give the project a star on GitHub. It helps others discover it, and I'd appreciate the support.