Semantic Chaos Monkey: Testing Ambiguous AI Instructions 🐒

Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle Submission 💸

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

This is my submission for the Hacktoberfest Open-Source AI Challenge — Week 2: Side Hustle.

What if an instruction is technically correct but still dangerous to execute?

Reset the database, migrate the schema, and restore the latest backup if necessary.

It sounds straightforward. But which database should be reset? What does “latest” mean? When exactly is a restore necessary?

Different engineers—or AI agents—could interpret the same instruction differently and produce entirely different action sequences. In an infrastructure environment, that ambiguity can have serious consequences.

That's the problem I'm tackling with Semantic Chaos Monkey (SCM), an AI-assisted adversarial testing platform designed to identify ambiguity in operational procedures and AI-agent instructions.

Instead of merely checking whether an instruction reads well, SCM is designed to:

The core idea is simple: test what an instruction could cause, not just what it appears to mean.

SCM is intended for DevOps engineers, AI-agent developers, platform engineering teams, and organizations that rely on operational runbooks.

For a startup, it could eventually help catch ambiguous deployment procedures before they reach production. For an AI engineering team, it could provide a reproducible way to test whether an agent's instructions leave room for conflicting interpretations.

The product is being designed with a local-first approach, keeping the initial testing environment isolated from real infrastructure.

Project status: Product definition and MVP development.

The complete end-to-end workflow is not yet verified as a working demo. The repository documents the proposed architecture, requirements, safety constraints, and implementation plan.

GitHub repository: https://github.com/HerambInamke/week-2-DEV-Challenge

I'll update this section with a live demo or walkthrough when the working MVP is ready.

🔗 GitHub: https://github.com/HerambInamke/week-2-DEV-Challenge

The repository is released under the Apache-2.0 license.

The intended architecture uses FastAPI and Pydantic for API contracts and validation, a replaceable model-provider interface for AI-generated interpretations, and a deterministic simulation engine for evaluating their consequences.

The focus is not just generating an AI response. It is making the reasoning outcomes testable, reproducible, and reviewable.

I approached SCM as an engineering problem rather than another general-purpose AI chatbot.

The proposed system follows a structured workflow.

Given an operational procedure, the AI layer proposes three to five plausible interpretations, including assumptions, source references, and ordered actions.

Each interpretation must conform to structured schemas, parameter requirements, preconditions, and a fixed catalog of permitted actions. Unsupported actions are rejected rather than executed through a fallback mechanism.

The accepted action sequences are evaluated against identical initial states in a synthetic environment. Deterministic application logic handles the simulated state transitions.

SCM is designed to compare action ordering, affected resources, resulting state, and configured safety invariants. This helps distinguish harmless wording differences from interpretations that could produce materially different outcomes.

The system proposes a clearer version of the original instruction and evaluates the revision against the same configured tests. The intention is to verify whether the identified ambiguity has actually been addressed rather than simply assuming that the rewritten instruction is safe.

One of the most important architectural decisions is separating AI-generated interpretations from their execution.

The model proposes possible meanings; deterministic application logic validates and simulates their consequences. The initial scope excludes production credentials, live infrastructure changes, unrestricted shell execution, and real deployment operations.

A language model suggesting an action sequence is not proof that the sequence is safe. SCM is being designed around that distinction.

Note: The architecture and workflow above describe the intended MVP. They should not be interpreted as a claim that every feature has already been implemented.

Reliable AI systems should not be something only large companies with proprietary infrastructure can investigate.

Open innovation allows developers to inspect the implementation, challenge assumptions, contribute adversarial test cases, extend the simulator, and adapt the approach to their own engineering workflows.

For SCM, openness matters in three specific ways.

Transparency: Engineers should be able to inspect how proposed actions are validated, how simulated outcomes are compared, and which rules determine whether a test passes or fails.

Reproducibility: Saved test cases and execution traces can make it possible to review the same ambiguity without depending on another AI generation producing identical wording.

Flexibility: A replaceable model-provider interface is intended to support experimentation with different models without tying the entire platform to one vendor.

A closed AI API can generate interpretations, but that alone does not provide a transparent, deterministic testing environment tailored to a team's operational rules.

By developing SCM in the open, I want to make it possible for other engineers to challenge the design, contribute test scenarios, and help establish better practices for testing natural-language instructions.

I'll add a shareable DevRelay agent-session link here once one is available.

I'll update this section after confirming the applicable categories and eligibility requirements in the official challenge rules.

As AI agents become more involved in software development, infrastructure management, and operational workflows, the instructions we give them become increasingly important.

A procedure can look perfectly reasonable while leaving critical decisions undefined. Reviewing the wording alone will not always reveal the consequences of those missing details.

Semantic Chaos Monkey is my attempt to explore a more rigorous approach: turn ambiguous instructions into testable scenarios, compare their consequences, and verify proposed corrections.

The project is still taking shape, and there is plenty of engineering work ahead. I'm sharing it early because open-source feedback can help challenge the assumptions and improve the design before the system grows further.

If you're working on AI agents, DevOps, or software reliability, I'd welcome your feedback and contributions.

🔗 Explore the project: https://github.com/HerambInamke/week-2-DEV-Challenge