custom metrics are the real autoscaling contract

Kubernetes has a very simple lie that everyone enjoys for a while:

CPU and memory tell you when to scale.

This is a useful lie. It gets teams started. It makes demos easy. It gives the HorizontalPodAutoscaler something obvious to watch. CPU goes up, replicas go up, everyone nods, the dashboard looks serious.

Then production arrives wearing dirty shoes.

The queue is saturated while CPU is bored. The payment provider is slow but your pods look healthy. The batch job is falling behind because one tenant sent a weird workload. The API is green, but the thing the customer actually cares about is waiting in a backlog somewhere.

Congratulations. You have discovered that autoscaling is not about resources.

Autoscaling is about pressure.

That is why the recent Kubernetes post about building a custom metrics exporter is more interesting than it looks. On the surface, it is a practical walkthrough: expose an application-specific metric, wire it through the custom metrics API, let the HPA consume it.

Fine. Useful.

But the bigger story is that Kubernetes is quietly telling teams something important: if CPU and memory are the only signals your platform exposes, your platform understands the wrong bottlenecks.

cpu is a symptom, not a business signal

CPU is not useless. Please do not turn this into another “metrics are fake” LinkedIn sermon. CPU matters. Memory matters. Saturation matters. If your service is burning cores like a crypto miner in a basement, the autoscaler should probably notice.

But CPU is usually an implementation symptom, not the user-facing problem.

A worker pool can be overloaded because queue depth is growing, even if every worker spends half its time blocked on a downstream API. A payment service can need more capacity because authorization latency is increasing, not because Go suddenly became CPU hungry. An inference service can need smarter routing because time-to-first-token and token throughput are drifting, not because average CPU crossed a nice round number.

This is where custom metrics become the real contract.

Not “scale when the pod is hot.”

“Scale when the system is failing to absorb work.”

Those are different sentences.

The first sentence belongs to infrastructure. The second belongs to the product.

And this is where things get uncomfortable, because platform teams cannot invent the right metrics alone. They can provide the plumbing, adapters, exporters, Prometheus integration, guardrails, naming rules, and examples. But they cannot know that checkout_pending_authorizations matters more than container_cpu_usage_seconds_total on Black Friday.

The application team has to bring domain knowledge.

The platform has to make that knowledge operational.

That is the contract.

custom metrics make ownership visible

I like custom metrics because they force a conversation that teams often avoid.

What does pressure mean for this system?

Not in general. Not in a diagram. In this service.

For a queue worker, pressure might be backlog age. For a fraud system, it might be decisions waiting longer than an SLA. For an AI gateway, it might be tokens per second, cache hit rate, provider throttle responses, or cost per successful answer. For a data pipeline, it might be lag behind the source. For a customer-facing API, it might be request latency split by tenant or endpoint.

Once that metric is used for autoscaling, it stops being decorative observability.

It becomes a control input.

That changes the seriousness of the metric. A dashboard panel can be wrong for months and survive because humans are very good at ignoring dashboards. A scaling metric that is wrong will spend money, hide incidents, create replica storms, or keep the system underprovisioned while everyone wonders why the HPA is “not working.”

Very often the HPA is working exactly as configured.

The configuration is just measuring the wrong reality.

This is why custom metrics need ownership attached to them. Who defines the metric? Who validates it? Who changes thresholds? Who knows whether a spike means demand, a bug, abuse, a downstream outage, or a bad release?

If nobody owns those answers, the autoscaler becomes a very confident random number amplifier.

the adapter is plumbing; the metric is design

The Kubernetes mechanics matter.

You need a custom metrics API adapter, labels that match Kubernetes objects, sane cardinality, scraping, permissions, and failure behavior.

All of that matters.

But the hardest part is deciding which signal deserves to affect capacity.

That is design.

A bad custom metric can be worse than no custom metric. If you scale workers based on queue length alone, you may create a thundering herd against a downstream dependency. If you scale an inference service based only on request count, you ignore prompt length, cache behavior, model selection, and provider throttling. If you scale a batch system on average lag, one large tenant can hide many small tenants having a terrible day.

So the mature pattern is not “add custom metrics everywhere.”

The mature pattern is to treat scaling signals like APIs.

Name them carefully. Document what they mean. Keep cardinality boring. Test them during load. Define what happens when they are missing. Review threshold changes like code.

Annoying, yes. Still cheaper than learning during an incident that your autoscaler has been scaling on a metric nobody trusts.

platform teams should ship metric paths, not magic

The platform team response should not be: “every service team, please become a metrics adapter expert by Friday.”

That is how you get twelve half-working exporters, three naming conventions, and one service that accidentally exposes customer IDs as metric labels because someone copied a tutorial at 1 AM.

The better platform move is paved paths: safe exporter templates, a blessed custom metrics adapter, common examples, cardinality review, and clear ownership.

But do not pretend the platform can decide every threshold.

The useful division of labor is simple: platform owns the measurement path, service teams own the meaning of the signal, SRE owns the failure modes, and finance eventually asks why the graph looks like a staircase made of money.

That last one is not a joke. Autoscaling is spending automation. Every scaling decision is also a cost decision.

If a queue gets deep, should we scale to zero lag no matter the price?

Maybe.

Maybe not.

That is a product decision wearing Kubernetes clothes.

the ai angle makes this louder

AI workloads make this louder.

CPU and memory are comically incomplete for model-serving and agent systems. A model server can look “fine” while users wait too long for first token. An agent workflow can saturate a tool API, a browser pool, a vector database, or a code review queue. A gateway can burn money because requests are routed without understanding cache locality or token volume.

The pressure signal might be latency, tokens, queue age, tool-call failures, provider quota, cache misses, or cost per task.

None of those fit neatly into the old “average CPU utilization” mental model.

This is why custom metrics are going to become less optional. As Kubernetes absorbs more AI, batch, serverless, and specialized workloads, generic resource pressure becomes only the first layer.

Not magical autonomous infrastructure.

Just honest feedback loops.

The boring kind. The useful kind.

the punchline

A custom metrics exporter sounds like plumbing because it is plumbing.

But plumbing is often where the architecture tells the truth.

CPU and memory are easy to measure, but they are not always what the system is actually constrained by. Serious production systems care about backlog, latency, tenant pressure, provider limits, model throughput, payment failures, retry storms, cost, and work that is aging in a queue while the pod looks relaxed.

When you connect those signals to autoscaling, you are not just adding a metric.

You are declaring what pressure means.

That declaration deserves engineering discipline. Ownership. Tests. Documentation. Review. Failure behavior. Cost awareness.

The Kubernetes custom metrics path is useful because it lets platform teams move beyond generic resource watching. But the important question is not “can we expose this metric?”

The important question is:

“Do we trust this signal enough to let it move production?”

If the answer is yes, make it boring, visible, and owned.

If the answer is no, maybe keep it on the dashboard until it grows up.

references

• Kubernetes Blog: Building a Custom Metrics Exporter for Kubernetes
• Kubernetes Documentation: Horizontal Pod Autoscaling

To test my projects, I use Railway. If you want $20 USD to get started, use this link.