kubernetes ca rotation is the boring outage teams forgot to rehearse

Managed Kubernetes CA rotation turns certificate expiry into a platform lifecycle problem. Providers can automate the safer mechanics, but teams still have to find every kubeconfig, controller, worker node, and automation path that trusts the old root.

Posted Sep 4, 2026

By Paulo Victor Leite Lima Gomes

7 min read

Managed Kubernetes made certificate authority rotation easy enough to ignore.

That is the uncomfortable bit.

Clusters kept serving traffic, teams kept shipping, platform dashboards stayed green, and the root of trust behind the Kubernetes API server became background furniture. Then the calendar kept moving. Amazon EKS clusters have certificate authorities with a ten-year validity period, which means early EKS clusters created in 2018 are approaching their 2028 expiry window. AWS now has an automated CA rotation lifecycle for EKS, and that is good news. It also makes the real work more visible.

The provider can rotate the control-plane trust chain. It cannot tell you every laptop, CI runner, GitOps controller, monitoring job, Terraform module, self-managed node, and forgotten cron script that cached the old one.

This is not a crypto-drama story. It is an inventory story.

the root nobody dashboards

Every Kubernetes cluster has a CA that clients use to trust the API server. Most teams meet it indirectly through a kubeconfig. You run aws eks update-kubeconfig, gcloud container clusters get-credentials, az aks get-credentials, or whatever your platform wrapper hides behind a shorter command, and then you stop thinking about the certificate data embedded in the file.

That abstraction is useful until the root changes.

The problem is not that CA rotation is conceptually hard. The Kubernetes documentation has long described manual CA rotation as a careful sequence of distributing new trust, restarting components, updating clients, rotating serving certificates, and eventually removing the old CA. The problem is that production clusters are not clean diagrams. They are surrounded by years of accumulated clients.

Some clients are obvious:

developer kubeconfigs
CI/CD pipelines
GitOps controllers
cluster bootstrap tooling
observability agents
incident-response scripts
self-managed node groups
Terraform and policy-as-code jobs

Some clients are less obvious:

an old build runner image with a baked kubeconfig
a bastion host used only during incidents
a vendor integration that nobody logs into anymore
a backup job that talks to the API once per day
a local script owned by the person who left two teams ago

Those clients are the outage surface. The CA expiry date just decides when the surface becomes visible.

what AWS changed

AWS announced automated lifecycle management for Amazon EKS certificate authority rotation in August 2026. The important design choice is not simply “EKS can rotate the CA now.” It is that the rotation is staged.

The EKS deep dive describes a dual-trust phase where the cluster trust bundle contains both the outgoing CA and the successor CA. During that window, clients can update their trust without losing access to the API server. AWS also calls out a practical detail: the combined trust bundle is roughly 2.8 KB, which matters because kubeconfigs and embedded CA data tend to move through systems that were never designed as certificate distribution channels.

The clean managed path is:

prepare the successor CA
distribute it in the cluster trust bundle
wait until successor distribution is complete
update external clients and customer-managed nodes
activate the successor CA
retire the outgoing CA after the safety window

EKS Auto Mode and AWS Fargate nodes get handled by AWS. External clients do not. Managed node groups, Karpenter-controlled nodes, self-managed nodes, hybrid nodes, kubeconfigs, pipelines, GitOps controllers, and monitoring tools are still your responsibility.

That boundary is the entire lesson.

Cloud providers can automate the part they own. They cannot automate the trust decisions living outside their operational boundary.

managed does not mean ownerless

This is where managed Kubernetes creates a governance trap. When the provider takes over enough mechanics, the team can mistake reduced toil for removed ownership.

CA rotation is not removed ownership. It is split ownership.

The provider owns control-plane mechanics. The platform team owns the customer side of the trust graph.

That split should change how teams design the platform around the cluster. A serious Kubernetes platform should be able to answer these questions without opening twenty Slack threads:

Which systems have kubeconfigs for this cluster?
Which kubeconfigs embed certificate authority data instead of fetching it dynamically?
Which CI runners can reach the API server?
Which GitOps controllers target this cluster?
Which worker nodes are outside the provider-managed rotation path?
Which external integrations authenticate to the API server?
Which incident tools would fail if the old CA disappeared today?
Who can prove the successor CA is trusted before activation?

If the answer is “we will find out during the rotation,” that is not a plan. That is a test in production with a calendar deadline.

the boring rehearsal

The right mental model for CA rotation is closer to disaster recovery than certificate administration.

Nobody wants a dramatic runbook. The best runbook is dull. It has owners, timestamps, queries, dashboards, rollback expectations, and a list of clients that looks embarrassingly complete.

For EKS, the rehearsal should start before the cluster enters the dangerous part of the rotation. During dual trust, teams should regenerate kubeconfigs, refresh runner images, restart long-lived automation, replace or recycle worker nodes that need it, and prove that all important API clients trust the successor CA.

That proof matters more than the command.

aws eks update-kubeconfig is easy. Knowing where to run it is hard.

A useful platform checklist looks like this:

Generate an inventory of all API clients by cluster.
Separate provider-managed nodes from customer-managed nodes.
Identify clients with embedded CA data.
Update kubeconfigs and runner images during the dual-trust window.
Restart long-lived controllers that do not reload trust bundles.
Rotate or recycle customer-managed nodes before activation.
Verify GitOps, CI/CD, observability, backup, and incident tooling.
Track the successor CA distribution status before moving forward.
Keep a short rollback and pause decision log.
Close the work only after the old CA can disappear without surprising anyone.

The checklist is not glamorous, but this is the work that keeps a certificate lifecycle event from turning into a platform incident.

the multi-cloud clue

The same pattern appears outside AWS.

Azure AKS documents certificate rotation as a cluster operation that can recreate nodes and warns teams to plan for service disruption. Google Kubernetes Engine distributed cloud for VMware also treats CA rotation as an explicit administrative flow with staged updates. Upstream Kubernetes documents manual CA rotation because the primitive itself is not magic: trust has to be distributed, components have to reload it, and clients have to use it.

The provider-specific details differ. The shape of the responsibility does not.

Every Kubernetes platform has two layers of trust lifecycle:

the infrastructure layer, where the cluster and provider know how to issue and serve certificates
the operating layer, where humans and automation have copied trust material into real workflows

Most outages hide in the second layer.

That is why I like AWS making EKS CA rotation more visible now. It turns a far-away expiration date into a platform hygiene conversation. It gives teams a dual-trust window and clearer lifecycle states. It makes the managed mechanics safer.

But it does not absolve the team from the boring part.

root rotation is platform inventory

The worst Kubernetes incidents are rarely caused by one exotic failure. They are caused by a small, reasonable assumption repeated across too many systems.

“The kubeconfig is fine.”

“The runner image is fine.”

“The GitOps controller will reconnect.”

“The monitoring integration probably uses the latest credentials.”

“Nobody uses that bastion anymore.”

CA rotation breaks those assumptions in a very specific way. It asks whether the cluster’s trust root is a managed platform artifact or just a blob that leaked into every corner of the engineering environment.

The difference is ownership.

If kubeconfigs are disposable and discoverable, rotation is routine. If they are scattered and untracked, rotation is archaeology. If node groups are clearly owned, recycling them is normal work. If self-managed nodes are a rumor, activation day becomes an incident bridge. If GitOps and CI are first-class platform clients, they get tested. If they are tribal knowledge, they fail exactly when the old root stops working.

Kubernetes CA rotation is boring infrastructure hygiene. That is the point.

The teams that rehearse it will barely remember it happened. The teams that do not will learn, under pressure, that “managed Kubernetes” still has a customer side of the contract.

sources

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

Kubernetes, Platform Engineering, Security

This post is licensed under CC BY 4.0 by the author.