From Slack to Diagnosis: Building an AI Debugging Agent for Our Kubernetes Platform

Hana Ben Ghezail · July 20, 2026 · 7 min read

AIPlatform Engineering

I’m part of the Platform Engineering team at AutoScout24. We manage the Kubernetes clusters—groups of servers that run and coordinate our applications—that the rest of engineering builds on. Builders deploy their services onto our platform, but they don’t have direct access to query the cluster. That’s intentional: we keep cluster access centralised for security and operational reasons. The side effect is that we become the middleman in debugging ambiguous Kubernetes errors.

A builder’s service starts throwing errors. The monitoring dashboards don’t tell the full story. Someone opens a support thread, then an incident, then pings us directly. By the time we’re involved, we’re spending on-call time tracing through pod logs, Prometheus metrics, and recent deployments to hand back an answer that unblocks them.

This is the story of how we built an AI agent to be that middleman instead.

#The support tax

Every week, questions land in the platform support channel that follow the same pattern: “My service is crashing in our production cluster, pods are restarting. Any idea what’s going on?”

Answering that requires:

1. Looking up which cluster the service runs on
2. Checking pod status and recent events in EKS
3. Checking Grafana for the logs and the error rate / HPA metrics
4. Cross-referencing the last deployment in GitHub Actions

For a platform engineer, this is muscle memory. For a developer on a feature team, it’s a maze and a bottleneck. They can’t query the cluster themselves, so the only path forward is opening a support thread and waiting for the platform team.

The thing is, we already had a support bot that could answer “how do I do X” questions from platform docs and debug CI/CD pipeline failures. It worked well for those cases. A live service outage, though, is a different kind of problem: the answer can be scattered across several systems.

#What we wanted to build

We had two goals:

1. For developers: let them describe a problem in natural language and receive a structured, evidence-based diagnosis.
2. For the platform team: remove ourselves as the bottleneck. A platform engineer should not be required to diagnose a failing service, and the access controls that protect the platform should not mean every production incident needs us in the loop.

A few constraints shaped the design:

• No new interface. Developers already live in our Slack support channel for all platform-related requests. The bot had to meet them there.
• Multi-source. A useful diagnosis needs cluster state and observability data and deployment history. One source alone is usually incomplete.
• Safe. The agent can read. It should never write. No restarts, no rollbacks.
• Access boundaries. Some clusters are off-limits for direct Kubernetes access for compliance reasons. The agent needs to handle that gracefully, not silently fail.

#Architecture overview

The support bot is a multi-agent system. A top-level orchestrator receives Slack messages, decides if they’re actionable, and routes them to the right sub-agent based on the type of question. There are currently four sub-agents: one for querying the platform knowledge base, one for debugging CI/CD pipeline failures, one for creating GitHub documentation issues, and the service debugging agent we recently built.

The service debugging agent handles anything about service health on our Kubernetes clusters. When invoked, it runs through a structured process:

1. Detect the cluster: query Loki for the k8s_cluster_name label scoped to the service namespace. This works even for crashlooping or recently-stopped pods because Loki retains log history.
2. Check the cluster registry: if the cluster is restricted, skip Kubernetes tools and note this in the response. Otherwise, continue.
3. Gather Kubernetes state: pod status, restart counts, events, resource limits.
4. Pull observability data: Prometheus metrics (error rate, HPA), Loki logs, Grafana alerts.
5. Check deployment history: recent GitHub Actions runs for the service.
6. Synthesize: correlate signals across sources and produce a structured diagnosis.

#The tech stack

The agent runs on AWS Bedrock AgentCore and uses Anthropic models for reasoning. The agent framework is Strands Agents, which lets us define tools as plain Python functions and wire them together with a system prompt.

What’s interesting is that the agent uses four different integration patterns depending on the use case for the target system:

Kubernetes — Python client with cross-account IAM We use the official kubernetes Python client, but the clusters live in a different AWS account from the bot. The agent assumes a scoped read-only IAM role in the platform account via AWS STS, gets temporary credentials, and uses those to build the Kubernetes client. The role grants AmazonEKSViewPolicy on specific clusters only, nothing else. There’s no standing access: every call is a fresh credential with a short TTL.

Grafana — MCP server Instead of writing separate REST clients for Prometheus, Loki, and Grafana alerts, we connected to Grafana’s Model Context Protocol server. MCP is a standard that lets AI agents discover and call tools exposed by a server, and Grafana ships one out of the box. We got metrics, logs, and alerting as ready-to-use agent tools without writing a single API client.

GitHub — GitHub App authentication For reading GitHub Actions runs and querying the ArgoCD apps repository, we authenticate as a GitHub App rather than using a personal access token. A GitHub App is a first-class machine identity: it has scoped repository permissions, its credentials don’t expire when someone leaves the team, and the audit trail is clean.

Backstage — REST API Service metadata (team ownership, repository links, on-call info) comes from our internal Backstage catalog via its standard REST API. This gives the agent context about the service and its owner, and helps it find the exact service name used in our infrastructure when the developer does not provide it.

#What was harder than expected

#Cluster discovery

The agent needs to know which cluster a service runs on before it can query anything. We started with Loki. As long as the service has produced any logs recently, a single label query gives us the cluster. It works even for crashlooping pods, and the logs remain available even if the pod has been evicted.

The gap we hit quickly: services that haven’t had Grafana set up yet won’t have any Loki data at all. For those, we fall back to the ArgoCD apps Git repository that stores all ArgoCD Application manifests for the platform, one per service. Each manifest declares which cluster and namespace the service is synced to, so it’s a reliable source of truth for the cluster mapping even for brand-new services with no observability data yet.

#Handling restricted clusters

Some clusters restrict direct Kubernetes access for compliance reasons. Early on, the agent would hit a hard stop when it detected one.

After some iteration, we landed on graceful degradation: skip the Kubernetes tools, continue with everything that’s allowed (Grafana, GitHub), and clearly note in the response that pod-level data isn’t available and why. A developer still gets an interpretation of Prometheus metrics and Loki logs, which can often be sufficient to diagnose the issue.

#Results so far

The support bot has handled 354 support threads in the past month across #platform-engineering-support. The resolution rate target is 80% (measured as fully or partially resolved threads), and the bot has been consistently hitting that bar.

Before we shipped the debugging agent, Kubernetes-related questions were among the hardest for the bot to handle. The bot would either skip them or offer generic kubectl commands that developers could not run themselves. Now, it can actually work through a diagnosis.

We ran end-to-end validation in staging across five scenarios before opening it up:

• High error rate on a live service → Prometheus + Loki correlation surfacing the spike and its timing relative to a recent deploy
• Pod crash loop → K8s events and container logs identifying the root cause
• Deployment failure → GitHub Actions run linked to the symptom window
• Restricted cluster → graceful degradation with Grafana-only data and a clear note about what’s not available and why
• Vague question → agent asks a targeted clarifying question rather than guessing

All five scenarios passed without a platform engineer in the loop.

#What’s next

A few things are on the backlog:

• Feedback loop — developers can already react with :bad_bot: to stop the bot mid-thread, but we’re not doing anything with that signal yet. The next step is closing the loop: tracking which responses get flagged, reviewing them, and using them to improve the agent’s reasoning and tool usage over time.
• Secure access to restricted clusters — right now the agent gracefully skips Kubernetes tools for compliance-restricted clusters. The real fix is a secure mechanism that lets the agent query those clusters in a controlled, auditable way. That’s the next infrastructure piece to design.