Log Doctor: Letting AI Take the First Look at My Kubernetes Errors Using AWS Bedrock and AWS Opensearch

Description: "Every 15 minutes, a small CronJob reads the error logs, asks an LLM on Amazon Bedrock what's going on, and posts a short diagnosis to Slack. Here's how I built it on EKS, Fluentd and OpenSearch."

If you've ever been on call, you know how this goes.

An alert fires: error rate on orders-api above 5%. You open the log search, filter on level: ERROR, and get back a few hundred lines. You start scrolling. Most of them are the same stack trace repeated over and over. A few are something else. One or two of those might be the real problem, but you won't know until you've read them.

By the time you've worked out "okay, the database is out of connections, and the payment timeouts are probably a side effect", ten minutes have gone. That's ten minutes of reading, not fixing.

What's missing is the step in between, the first pass a senior engineer does without thinking about it. Group the noise, notice which error is the loud one, form a quick theory, and decide where to look first.

That first pass is mostly reading and summarizing repetitive text. That happens to be something large language models are genuinely good at.

I didn't want an AI agent with access to my whole cluster. I wanted something much smaller and simpler:

Every 15 minutes, look at the errors from the last 15 minutes. If there are any, post a short note to Slack, per service: what's happening, the likely cause, and what to check. If there are none, say nothing.

A few rules I set for myself before writing any code:

orders-api / payments-api → Fluentd → OpenSearch → Log Doctor (CronJob) → Amazon Bedrock → Slack Enter fullscreen mode Exit fullscreen mode Everything runs on a stack that's common in AWS shops, and it's all Terraform plus plain Kubernetes manifests:

👉 The code is here: github.com/ranjiniganeshan/log-doctor

My first thought was to fetch a few hundred log lines and give them to the model. But OpenSearch can already do the grouping, and it gives exact counts. Log Doctor runs a single aggregation: errors per service, then top messages per service, with two recent samples of each.

" aggs ": { " services ": { " terms ": { " field ": cfg. service_field, " size ": cfg. max_services, " missing ": " (unknown) " }, " aggs ": { " messages ": { " terms ": { " field ": cfg. message_field, " size ": cfg. max_messages_per_service }, " aggs ": { " first_seen ": { " min ": { " field ": cfg. timestamp_field }}, " last_seen ": { " max ": { " field ": cfg. timestamp_field }}, " samples ": { " top_hits ": { " size ": cfg. samples_per_message, " sort ": [{ cfg. timestamp_field: { " order ": " desc " }}], " _source ": { " includes ": SAMPLE_SOURCE_FIELDS }, } }, }, } }, } }, Enter fullscreen mode Exit fullscreen mode With the defaults, that's at most 10 services, 5 messages per service and 2 samples per message. So I know the maximum size of the prompt, however bad the incident is. The Python code then trims it further:

There's another benefit: the model gets real counts. It's told "25 of these" instead of having to count lines itself.

I spent extra time on this part. Log lines can contain whatever your users typed. If you put them in a prompt and post the answer to Slack, a stranger's text can end up in your on-call channel.

_REDACTIONS = [ ( re. compile ( r " AKIA[0-9A-Z]{16} " ), " [REDACTED_AWS_KEY] " ), ( re. compile ( r " (?i)bearer\s+[a-z0-9._\-]+ " ), " Bearer [REDACTED] " ), ( re. compile ( r " (?i)(password|passwd|secret|token|api[_-]?key)\s*[=:]\s*\S+ " ), r " \1=[REDACTED] " ), ( re. compile ( r " [A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,} " ), " [REDACTED_EMAIL] " ), ( re. compile ( r " https://hooks\.slack\.com/\S+ " ), " [REDACTED_WEBHOOK] " ), ] Enter fullscreen mode Exit fullscreen mode Redaction runs on the messages and samples before they reach the prompt. The system prompt says log content is untrusted and that instructions inside log lines must not be followed. Slack control characters are escaped, so a log line can't get <!channel> pushed to your whole team.

None of these is perfect. Regex only catches the patterns you thought of, and a model can still be tricked. So the most important protection is the design itself: the model has no tools and no credentials, and all it can do is write text for Slack.

If someone does trick it, the worst result is a misleading message.

Run these checks before you create anything. They don't cost anything, and the EC2 command is only a dry run:

export AWS_REGION = ap-south-1 # Confirm you're in the right account aws sts get-caller-identity # Check that you can launch the node type AMI = $( aws ssm get-parameter \ --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \ --query Parameter.Value --output text ) aws ec2 run-instances --dry-run --image-id " $AMI " \ --instance-type t3.medium --count 1 Enter fullscreen mode Exit fullscreen mode The EC2 command should reply with Request would have succeeded, but DryRun flag is set.

💸 This creates billable resources. Estimated cost is in the cost section. Tear it down when you're finished.

Log Doctor uses Bedrock's Converse API, so it works with any Bedrock chat model. List the models you can call directly in your region:

aws bedrock list-foundation-models --region ap-south-1 --by-output-modality TEXT \ --query "modelSummaries[?contains(inferenceTypesSupported,'ON_DEMAND')].modelId" Enter fullscreen mode Exit fullscreen mode I used openai.gpt-oss-20b-1:0, OpenAI's open-weight model. Some models are only offered through cross-region inference profiles, with IDs that start with global. or apac.. Those can send your request to another region, so a policy that limits your account to one region can block them.

Then make a real test call. A model appearing in the list doesn't prove your account can use it.

export AWS_REGION = ap-south-1 aws bedrock-runtime converse --model-id "openai.gpt-oss-20b-1:0" \ --messages '[{"role":"user","content":[{"text":"Reply with OK"}]}]' \ --inference-config '{"maxTokens":800}' Enter fullscreen mode Exit fullscreen mode Third-party models can create an AWS Marketplace subscription on first use. The CronJob's role isn't allowed to do that, which is correct for least privilege, so make this first call with your own admin identity.

This model returns reasoning blocks as well as the answer, and it counts them against maxTokens. Don't set the limit too low. Log Doctor keeps only the text blocks.

At api.slack.com/apps, create an app, turn on Incoming Webhooks, and add one to a channel. Treat the URL like a password, because anyone who has it can post to your channel.

export SLACK_WEBHOOK_URL = 'https://hooks.slack.com/services/...' Enter fullscreen mode Exit fullscreen mode It stays in your shell and goes into a Kubernetes Secret later. Nothing gets written to the repo.

📌 ADD BEFORE PUBLISHING: screenshot of the Incoming Webhooks page, with the URL hidden

cp infra/terraform.tfvars.example infra/terraform.tfvars Enter fullscreen mode Exit fullscreen mode These are the settings I used:

region = "ap-south-1" bedrock_model_id = "openai.gpt-oss-20b-1:0" node_instance_type = "t3.medium" cluster_endpoint_public_access_cidrs = [ "<your-ip>/32" ] Enter fullscreen mode Exit fullscreen mode Then:

make infra-up kubectl get nodes Enter fullscreen mode Exit fullscreen mode That creates 67 resources: a VPC, an EKS cluster with one node, a single-node OpenSearch domain, two ECR repositories and the IAM roles.

For me, the whole apply took 23 minutes. OpenSearch was the slowest part at about 14 minutes, and the EKS control plane took a little over 8.

kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-40-13-187.ap-south-1.compute.internal Ready <none> 17m v1.36.5-eks-6e7947a Enter fullscreen mode Exit fullscreen mode How the permissions work. Neither workload has stored AWS credentials. Each one gets its own role through IRSA, and each role's trust policy is pinned to one exact service account:

I used OpenSearch's IAM-only access policy rather than fine-grained access control because it's the simplest secure option: no master user, no role mappings, no setup step after terraform apply. The trade-off is that Fluentd writes through /_bulk, which names the index in the request body, so its role could write to any index. It still can't read, search or delete. If you need per-index write control, use fine-grained access control.

make build-push Enter fullscreen mode Exit fullscreen mode This builds for linux/amd64, which matters if you're on Apple Silicon: without it, you get exec format error when the pods start on an x86 node.

make slack-secret make deploy make status Enter fullscreen mode Exit fullscreen mode make deploy renders the manifests from the Terraform outputs and applies them in order: namespaces, service accounts, Fluentd, the two services, then the CronJob.

{ "DomainNames": [ { "DomainName": "log-doctor", "EngineType": "OpenSearch" } ] } { "clusters": [ "log-doctor" ] } [ "nat-0c95abf4eb696bfdf" ] Enter fullscreen mode Exit fullscreen mode Two CronJob settings are there to handle failures:

concurrencyPolicy: Forbid # a slow run never overlaps the next one backoffLimit: 0 # a retry could post the same summary twice Enter fullscreen mode Exit fullscreen mode If Bedrock fails, Log Doctor still posts a raw digest of top messages and counts, then exits 1. On-call gets the information without the explanation, and the failed Job stays visible.

kubectl -n logging get pods NAME READY STATUS RESTARTS AGE fluentd-v9dxv 1/1 Running 0 2m2s kubectl -n apps get pods NAME READY STATUS RESTARTS AGE orders-api-5d7fc5f46f-s49kk 1/1 Running 0 2m1s payments-api-754bb84f5c-f7c2v 1/1 Running 0 2m kubectl -n log-doctor get cronjobs,jobs,pods NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/log-doctor */15 * * * * Etc/UTC False 0 <none> 2m Enter fullscreen mode Exit fullscreen mode Demo: breaking things on purpose Now for the fun part. Each step below shows one stage of the pipeline.

1. Cause some trouble. This sends normal traffic plus a burst of errors: 25 database failures and 4 NoneType bugs on orders-api, and 12 timeouts and 3 database failures on payments-api.

make simulate-errors Enter fullscreen mode Exit fullscreen mode { "timestamp": "2026-11-05T15:50:04.398+00:00", "level": "ERROR", "service": "orders-api", "message": "Unhandled error building order confirmation", "trace_id": "3dd9857419094b36a047432aba78a598", "logger": "orders-api", "error": { "type": "TypeError", "message": "'NoneType' object is not subscriptable", "stack_trace": "Traceback (most recent call last): \n File \" /srv/app/main.py \", line 194, in simulate_error \n action() \n File \" /srv/app/main.py \", line 182, in <lambda> \n \" null \": ( \" Unhandled error building order confirmation \", lambda: build_confirmation_email( \" c-9999 \" )), \n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n File \" /srv/app/main.py \", line 146, in build_confirmation_email \n return f \" To: {customer['email']} \" # customer is None for unknown IDs -> TypeError \n ~~~~~~~~^^^^^^^^^ \n TypeError: 'NoneType' object is not subscriptable \n " }, "error_scenario": "null" } Enter fullscreen mode Exit fullscreen mode 2. See exactly what the model will see. Wait 30 seconds for Fluentd to flush, then run this.

It makes no Bedrock call and posts nothing, and it also proves the logs reached OpenSearch.

make dry-run SKIP_LLM = 1 Enter fullscreen mode Exit fullscreen mode ERROR log summary for the last 15 minutes (total errors: 44, services shown: 2). <logs> ## service: orders-api (total errors: 29) - message: "Failed to fetch order from database" count: 25, first_seen: 2026-11-05T15:50:04.846Z, last_seen: 2026-11-05T15:50:04.846Z sample 1: [2026-11-05T15:50:04.846254876+00:00] pod=orders-api-5d7fc5f46f-s49kk trace_id=51c260bd3fab48a793f43dec068802e9 DatabaseConnectionError: connection to server at "orders-db.internal" (10.40.3.21), port 5432 failed: FATAL: remaining connection slots are reserved for non-replication superuser connections... Enter fullscreen mode Exit fullscreen mode 3. Ask the model, but don't post.

make dry-run Enter fullscreen mode Exit fullscreen mode *Shared root cause:* `orders-db` connection pool exhausted – “remaining connection slots are reserved for non‑replication superuser connections” is firing in both services. **Orders‑api** – 29 errors • *What's happening:* 25 DB connection errors plus 4 TypeError failures building confirmation emails. • *Likely cause:* The Postgres pool is saturated, and an unknown customer ID triggers a None dereference. • *What to check:* • run `kubectl exec -it <orders-db-pod> -- psql -c "SELECT * FROM pg_stat_activity WHERE state='active';"` to count active connections. • review `orders-api`’s connection‑pool config (`max_pool_size`, `idle_timeout`) and consider raising `max_connections` or decreasing pool size. • audit `build_confirmation_email` to guard against missing customer records. **Payments‑api** – 15 errors • *What's happening:* 12 upstream timeouts to the payment gateway, 3 DB connection errors. • *Likely cause:* The gateway service is too slow or unresponsive, and the shared DB pool limit is contributing. • *What to check:* • check `payment-gateway` pod CPU/mem and logs for latency spikes (`kubectl logs <gateway-pod>`). • run `curl -v http://payment-gateway.payments.svc.cluster.local:8080/v1/authorize` from a pod to test connectivity. • increase `payments-api`’s HTTP timeout or enable a retry/circuit‑breaker if the gateway is intermittently overloaded. Enter fullscreen mode Exit fullscreen mode 4. The real thing.

make run-now Enter fullscreen mode Exit fullscreen mode

Wait for a window with no errors. The job runs, finds zero errors, and posts nothing.

make infra-down Enter fullscreen mode Exit fullscreen mode If you try it, I'd love to hear what it says about your logs, especially when it gets something wrong. The code is at github.com/ranjiniganeshan/log-doctor.