Agentic AI Workflow auf AWS untersucht Produktionsvorfaelle ohne Vollzugriff

Can AI help investigate production incidents without giving it unrestricted access to your AWS environment?

Imagine your production API starts returning HTTP 503 errors. Latency increases, CPU utilization spikes, and your logs show database connection pool timeouts.

A developer usually needs to open CloudWatch, inspect logs, compare metrics, investigate dependencies, and determine what to do next.

What if an AI-assisted workflow could collect the evidence, summarize the incident, recommend troubleshooting steps, and prepare a remediation plan for human approval?

In this tutorial, we'll explore how to build that workflow using AWS generative AI services and a small Python project.

Each AWS service has a different role in the workflow.

Amazon Bedrock provides managed access to supported foundation models through APIs.

You can use it to summarize application logs, classify incidents, explain deployment errors, generate reports, and recommend troubleshooting steps.

Amazon Bedrock Agents can coordinate foundation models with configured tools and knowledge bases to support multi-step tasks.

For example, an operations agent could collect approved metrics, retrieve a runbook, analyze an incident, and recommend a next step.

An agent should never receive unrestricted AWS permissions just because it can reason about operational problems.

Knowledge Bases support retrieval-augmented generation (RAG).

You can provide relevant context from internal runbooks, architecture documentation, incident reports, and deployment procedures.

This helps the model use your organization's documentation when generating recommendations, although its responses still need validation.

Bedrock Guardrails provides configurable controls for supported use cases, including content filtering and sensitive-information protection.

Guardrails complement application validation and IAM controls; they do not replace them.

AWS Lambda can provide specific functions for collecting metrics, retrieving approved logs, generating reports, or executing an authorized operation.

Keep the AI's recommendations separate from the code that performs the action.

AWS Step Functions can coordinate the process, handle execution states, and support workflows that wait for human approval.

Amazon CloudWatch provides metrics, logs, dashboards, and alarms.

It supplies the operational evidence that the AI-assisted workflow can analyze.

Here's a possible architecture for an AI-assisted incident-triage system:

Production Application | v Amazon CloudWatch Metrics, Logs, Alarms | v AWS Step Functions | v AWS Lambda Collect approved evidence | v Amazon Bedrock Analyze incident and recommend actions | +----> Knowledge Base / Runbooks | v Validate recommendation | v Human approval | v AWS Lambda Execute approved action | v CloudWatch Verify outcome Enter fullscreen mode Exit fullscreen mode This is a reference architecture, not an automatically configured integration. You must implement the relevant API calls, IAM roles, validation logic, and workflow states.

The important design decision is that the model recommends an action, but the application decides whether that action is allowed.

Let's create a small project that identifies symptoms in simulated production telemetry.

For the first version, we'll use deterministic Python rules. This makes the project runnable without AWS credentials or model access.

aws-ai-incident-triage/ └── incident_triage.py Enter fullscreen mode Exit fullscreen mode Step 2: Add the code Create incident_triage.py:

import json incident = { " service ": " checkout-api ", " environment ": " production ", " cpu_utilization_percent ": 92, " memory_utilization_percent ": 78, " http_5xx_count ": 45, " p95_latency_ms ": 2400, " recent_logs ": [ " Database connection pool timeout ", " Request latency exceeded 2000 ms ", " HTTP 503 returned by checkout-api " ] } def investigate ( data ): findings = [] recommendations = [] if data [ " cpu_utilization_percent " ] >= 85: findings. append ( " CPU utilization is elevated. " ) recommendations. append ( " Inspect CPU-intensive requests and task capacity. " ) if data [ " memory_utilization_percent " ] >= 85: findings. append ( " Memory utilization is elevated.

" ) recommendations. append ( " Inspect memory usage and possible memory leaks. " ) if data [ " http_5xx_count " ] >= 10: findings. append ( " HTTP 5xx errors exceed the demo threshold. " ) recommendations. append ( " Inspect application errors and dependency failures.

" ) if data [ " p95_latency_ms " ] >= 1000: findings. append ( " P95 latency exceeds the demo threshold. " ) recommendations. append ( " Investigate slow dependencies and request bottlenecks. " ) if any ( " connection pool timeout " in message. lower () for message in data [ " recent_logs " ] ): findings. append ( " Logs indicate database connection pool timeouts.

" ) recommendations. append ( " Inspect database connections, pool limits, " " and query latency. " ) return { " service ": data [ " service " ], " environment ": data [ " environment " ], " findings ": findings, " recommendations ": recommendations, " proposed_action ": ( " Review service capacity and database connection behavior. " ), " approval_required ": True, " execution_status ": " NOT_EXECUTED " } report = investigate ( incident ) print ( json. dumps ( report, indent = 2 )) Enter fullscreen mode Exit fullscreen mode Step 3: Run it python incident_triage.py Enter fullscreen mode Exit fullscreen mode No third-party Python packages are required for this first version.

{ "service": "checkout-api", "environment": "production", "findings": [ "CPU utilization is elevated.", "HTTP 5xx errors exceed the demo threshold.", "P95 latency exceeds the demo threshold.", "Logs indicate database connection pool timeouts." ], "recommendations": [ "Inspect CPU-intensive requests and task capacity.", "Inspect application errors and dependency failures.", "Investigate slow dependencies and request bottlenecks.", "Inspect database connections, pool limits, and query latency." ], "proposed_action": "Review service capacity and database connection behavior.", "approval_required": true, "execution_status": "NOT_EXECUTED" } Enter fullscreen mode Exit fullscreen mode This output is generated by deterministic rules, not by a live AI model.

The demo identifies symptoms and recommends next steps. It does not restart services, modify infrastructure, or make production changes.

Next, replace the rule-based analysis with a foundation model.

First, configure AWS credentials and choose a model available to your account in your selected Region. Ensure the identity has the required Bedrock permissions.

Install the AWS SDK for Python if needed:

python -m pip install boto3 Enter fullscreen mode Exit fullscreen mode The following example assumes the incident dictionary from the previous section is available:

import json import boto3 bedrock = boto3. client ( " bedrock-runtime ", region_name = " us-east-1 " ) response = bedrock. converse ( modelId = " YOUR_SUPPORTED_MODEL_ID ", system = [ { " text ": ( " You are an incident-triage assistant. " " Use only the supplied evidence. " " Distinguish observed facts from hypotheses. " " Never claim to have executed a remediation. " " Recommend investigation steps and flag uncertainty. " ) } ], messages = [ { " role ": " user ", " content ": [ { " text ": json. dumps ( incident )} ] } ], inferenceConfig = { " maxTokens ": 800, " temperature ": 0 } ) print ( response [ " output " ][ " message " ][ " content " ][ 0 ][ " text " ] ) Enter fullscreen mode Exit fullscreen mode Replace YOUR_SUPPORTED_MODEL_ID with an actual supported model ID.

This code sends the sample incident to Amazon Bedrock and prints the model's response. It does not automatically retrieve CloudWatch telemetry or execute remediation actions.

For a real implementation, add an authorized Lambda integration to collect metrics and logs, validate model output against a defined schema, and pass approved operations into a controlled workflow.

A model can suggest a valid-looking action that is inappropriate for the target environment.

Start with an application-level allowlist:

APPROVED_ACTIONS = { " inspect_logs ", " collect_metrics ", " create_incident_report " } def authorize_action ( action ): if action not in APPROVED_ACTIONS: return { " allowed ": False, " reason ": " Action is not in the approved allowlist. " } return { " allowed ": True, " reason ": " Action is allowed by the demo policy. " } print ( authorize_action ( " inspect_logs " )) print ( authorize_action ( " delete_production_database " )) Enter fullscreen mode Exit fullscreen mode The first action is permitted by this example policy; the second is denied.

This is a basic demonstration, not a complete authorization system. Production code must also validate the target resource, parameters, caller identity, and IAM permissions.

For actions that can affect production availability or data, use a workflow that requires explicit human approval before execution.

Never rely on a prompt alone to enforce a security boundary.

AI adoption should be evaluated against a baseline.

Test the workflow with different cases, including high CPU, database timeouts, missing telemetry, misleading symptoms, and incidents that require more information.

These are evaluation goals, not measured results from the demo.

Before deploying an agentic workflow, plan for:

Bedrock Guardrails can provide useful additional controls, but they are only one layer of the security design.

A practical rollout can follow these steps:

Use Bedrock Agents when model-driven tool selection and multi-step task execution are valuable. A deterministic workflow containing a model call may be easier to secure and operate for simpler use cases.

Agentic AI is not just about connecting a language model to tools. It is about combining useful reasoning with reliable evidence, controlled execution, and clear authorization boundaries.

Amazon Bedrock provides model access, Lambda supplies controlled tools, CloudWatch provides operational evidence, and Step Functions coordinates the workflow.

Our Python demo illustrates the first step: transform telemetry into structured findings and recommendations without executing production changes.

The next step is to connect the workflow to real telemetry, test it against known incidents, and measure whether it improves the investigation process.

My principle for production AI: Let the model recommend, let the application validate, and let authorized people approve consequential changes.