AI SRE agents need incident accounts
An AI agent helping during a Kubernetes incident needs a short-lived, scoped operational identity—not a human's kubectl credentials.
The demo is irresistible.
An alert fires. An agent reads the dashboard, follows a trace, searches logs, runs a few kubectl commands, and comes back with a plausible explanation before a human has finished finding the right browser tab. The next slide is always some version of: “and then it fixes the deployment.”
I believe the first half. I even think it will become normal. During an incident, having a tireless assistant that can gather evidence across Kubernetes, logs, traces, tickets, deploy history, and runbooks is genuinely useful.
The second half is where the architecture usually becomes hand-wavy. Giving an agent the same broad production access as the on-call engineer and trusting its good intentions is not an SRE design. It is kubectl with optimism.
The production pattern I want is much more boring: treat the agent as an incident account.
That means a short-lived identity created for one investigation, constrained to the relevant services, read-only by default, routed through a policy-enforcing tool gateway, and fully auditable. If it needs to change production, it crosses a deliberate, human-owned approval boundary.
That is not an obstacle to useful automation. It is what makes useful automation survivable at 03:17.
The agent is reading instructions, not just evidence
Incident response has always involved untrusted or semi-trusted inputs: error messages, customer reports, tickets, dashboards, pasted commands, and the occasional ancient runbook nobody has dared delete.
An LLM agent changes the risk because it can turn what it reads directly into actions. A log line can contain text that looks like a command. A GitHub issue can recommend a dangerous workaround. A tool description can be overly broad. A compromised dependency README can try to redirect the investigation. None of those inputs need to “hack Kubernetes” if they can persuade a highly privileged agent to do it for them.
This is not a claim that models are uniquely reckless. It is an access-control observation. The model should not be the thing we rely on to reject bad instructions, because the model is the component interpreting all those instructions.
The recent Kubernetes-agent security literature makes the point crisply: for an agent, the boundary between data and control collapses. The practical response is infrastructure controls that still work even if the agent follows a malicious prompt. OWASP’s guidance arrives at a compatible rule: give agents their own identity, only the access required for the task, and only for as long as it is needed.
An incident account is a real operational identity
Do not run the agent with Alice’s kubeconfig. Do not create a permanent ai-sre-admin service account and promise to be careful. Do not give it a cloud role that can edit every production resource because that was faster than designing scopes.
Instead, when an incident opens, mint an identity with a name that is boringly explicit, such as incident-2026-10-27-checkout-api. Bind it to the incident ID, severity, affected service, namespace, and a short expiry. Make the default permissions sufficient for diagnosis:
- list and get the relevant workload, pod, event, and replica-set objects;
- read logs, metrics, traces, deployment history, and approved runbooks;
- query the incident ticket and recent pull requests for the affected service;
- create an evidence summary or a suggested remediation plan outside the production control plane.
Notice what is missing: deleting pods, patching deployments, changing autoscaling limits, applying manifests, editing cloud network policy, and sending customer-facing updates. Those are not investigation permissions.
An incident is exactly when teams are tempted to collapse those distinctions. That is also when a bad action is most expensive. The account should expire when the incident closes, or sooner if it is idle. Revoking it should be one operation, not a hunt through a collection of shared credentials.
The scope matters as much as the lifetime. An agent investigating checkout should not casually enumerate every namespace, read every secret, or browse unrelated production repositories. Namespace boundaries are not a complete security model, but they are an excellent place to start making the agent’s world smaller than the whole cluster.
Put a gateway between the agent and every sharp edge
The agent should call a small set of well-defined tools, not inherit a shell with the internet and a directory full of credentials.
For Kubernetes, that can mean a gateway exposing read-only operations with structured arguments: get this deployment, retrieve these pod logs for this time window, compare these two ReplicaSets, or list recent warning events in this namespace. The gateway validates the incident scope and rejects operations that do not belong to it.
Apply the same idea to the rest of the incident surface:
- the observability tool can query approved datasets, not arbitrary internal endpoints;
- the cloud tool can describe the relevant load balancer or database health, not mutate IAM;
- the Git provider can read commits and open a draft remediation, not merge to main;
- the ticketing tool can append an evidence-backed note, not close the incident;
- external egress is restricted to the providers the workflow actually needs.
This is more than wrapping kubectl in friendlier JSON. A gateway is where you can enforce allowlists, validate arguments, redact secrets from outputs, rate-limit expensive searches, and attach every call to the incident record. It also prevents the usual escape hatch: “the tool was read-only, but the agent had a shell anyway.”
The agent needs a sandboxed runtime as well. Docker’s recent move to managed cloud sandboxes is one visible signal that long-running agent work is leaving developer laptops and acquiring policy-bearing execution environments. Good. But sandboxing the process is not a substitute for scoping its identity; it is another layer that limits what a compromised process can do.
Translate the idea into platform controls
None of this requires exotic new Kubernetes primitives. The hard part is deciding that incident access is a first-class product in your platform.
RBAC and workload identity. Create roles for investigation, not just generic view and admin. Issue short-lived credentials to a workload identity associated with the incident, scoped to the service namespaces and resource types it needs. Keep secret reads out unless there is a separately approved, redacted workflow.
Admission policy. Use ValidatingAdmissionPolicy, policy engines, or both to make accidental writes harder. Production changes from an agent-originated identity should fail by default. A write-capable identity should be distinct, short-lived, and activated only after an approver has seen the proposed diff and impact.
Network and runtime isolation. Use a sandbox runtime where it fits, a restrictive network policy, and FQDN-aware egress controls where your stack supports them. The agent should not be able to phone home arbitrary cluster data just because a log-search task happened to receive a clever prompt.
Tool gateway policy. Put the available tools and their argument rules in version control. If the incident account can execute a restart, describe the permitted target, safety checks, and required approval in policy—not only in a prompt that may be ignored at the wrong moment.
A human break-glass transition. When a change is warranted, the agent produces evidence, a proposed command or manifest diff, a rollback plan, and an estimate of blast radius. A human with the right on-call role authorizes the transition. The write happens under a separate identity that records who approved it and why.
The word “human” here is important. Approval cannot mean the same agent calling an approve=true tool after it has convinced itself. That is just autonomous write access wearing a fake moustache.
Auditability is the feature, not paperwork
After a real incident, the useful question is not “did the agent solve it?” It is: can we reconstruct what happened without trusting anyone’s memory?
You should be able to answer, quickly:
- What did the agent read, including the runbooks, tickets, and tool descriptions that could have influenced it?
- Which identity did it use, what was its exact scope, and when did it expire?
- Which tools did it call, with which arguments, and what did they return after redaction?
- Did it attempt an action outside its policy, and was it denied?
- Who approved a privilege escalation or production write, and what change actually reached the API server?
- Can the team replay the investigation from the transcript and audit logs?
This is not bureaucratic overhead. It is how you make the system better after the next incident. It also gives security teams something concrete to review besides a vendor slide saying the agent is “enterprise ready.”
Start smaller than the demo
The first useful AI SRE agent does not need the ability to heal production.
Give it one service, one namespace, read-only access, a tight tool catalog, and an obligation to cite the evidence behind its diagnosis. Let it assemble a timeline, correlate a deploy with an error-rate increase, find the noisy pods, and draft a rollback plan. Measure whether it reduces time spent gathering facts without increasing false confidence.
Then make escalation a separate product decision. Define the tiny set of reversible actions that may eventually be approved: perhaps scaling a known worker pool within a fixed range, restarting a single unhealthy replica, or executing a pre-reviewed rollback. Treat every additional write action as a capability with an owner, a policy, an audit trail, and a failure mode.
An agent that cannot explain what it read, what it was allowed to change, who approved an escalation, and how to replay the incident is not an SRE agent.
It is kubectl with optimism.
References
- Containing the Autonomous Operator: A Defense-in-Depth Framework and Reference Architecture for Securing AI Agents on Kubernetes
- OWASP DevSecOps Guideline: AI Agent and MCP Security
- Docker launches Cloud Sandboxes
- Kubernetes RBAC documentation
- Kubernetes Validating Admission Policy documentation
To test my projects, I use Railway. If you want $20 USD to get started, use this link.