Search for a command to run...
The real challenges of running K8s in prod. Operator fatigue, upgrade nightmares, monitoring gaps, and the cost surprises that hit after the honeymoon phase.
Senior Freelancer & Technical Lead
Working as a Golang developer since 2020. Working as a mobile developer since 2013.
Focussed on architecture, testability and clean code. Open minded & product driven. Based in Rhede, available world-wide
No comments yet. Be the first to comment.
How to implement comprehensive software supply chain tracking using Open Component Model that captures dependencies, deployment artifacts, signatures, and version constraints across distributed systems.
A practical framework for evaluating and migrating from deprecated ingress controllers to Gateway API or Envoy Gateway, including risk assessment and zero-downtime migration patterns.
How modern software supply chains use component models to track dependencies, artifacts, and metadata across distributed systems, moving beyond traditional SBOMs to comprehensive delivery tracking.
How to implement comprehensive software supply chain tracking using the Open Component Model that captures dependencies, deployment artifacts, configurations, and runtime bindings across multi-repository environments.
Master React performance optimization through systematic profiling, efficient rendering patterns, and smart bundle splitting strategies for production applications. Learn practical techniques that actually move the performance needle.
Every Kubernetes tutorial follows the same script: spin up a cluster, kubectl apply a deployment, watch your pods come up, celebrate. Maybe throw in an Ingress and a HPA for extra credit. Done — you're running Kubernetes!
No. You're running a demo.
I've been operating Kubernetes clusters in production for years now. For companies ranging from startups to enterprises. And the gap between "I deployed my first pod" and "I'm running a reliable production workload" is enormous. Here's what nobody tells you.
Kubernetes releases a new minor version every four months. Each version is supported for roughly 14 months. That means you're on a treadmill — if you stop upgrading, you fall off support.
Sounds manageable? Here's the reality:
PodSecurityPolicy you rely on? Gone in 1.25. Your Ingress annotations? Half of them changed between beta and GA. Every upgrade is an audit of your entire manifest base.A real-world upgrade looks like this:
# What you think it looks like
kubectl upgrade cluster --version 1.30
# What it actually looks like
# 1. Read the changelog (all of it)
# 2. Check deprecated APIs with pluto/kubent
# 3. Test in staging (you have staging, right?)
# 4. Update all Helm charts for compatibility
# 5. Coordinate with every team that deploys to the cluster
# 6. Pick a maintenance window
# 7. Pray
# 8. Roll back when something breaks
# 9. Debug for 3 hours
# 10. Try again
Budget at least one full day per upgrade for a medium-complexity cluster. Per quarter.
The Kubernetes ecosystem loves operators. Need a database? There's an operator. Message queue? Operator. Certificate management? Operator. Monitoring? You guessed it.
The promise is beautiful: declare your desired state, and the operator reconciles reality to match. The problem is that every operator is its own little control plane with its own bugs, its own upgrade cycle, and its own opinions about how things should work.
I've seen clusters running 15+ operators. Each one:
When something goes wrong — and it will — you're debugging interactions between multiple reconciliation loops. Is the database operator fighting with the backup operator? Is the Istio sidecar injector interfering with your StatefulSet rollout? Good luck grepping those logs.
My rule of thumb: Every operator you add should save more operational effort than it creates. Most don't pass that test. A well-written Helm chart with a CronJob often beats a poorly maintained operator.
Here's a dirty secret: most Kubernetes monitoring setups only tell you what's happening inside Kubernetes. They don't tell you what your users are experiencing.
A typical Prometheus + Grafana setup gives you:
What it doesn't give you out of the box:
The metrics you need to actually operate a production system are application-level metrics and infrastructure-level metrics that go beyond what the Kubernetes API exposes. You need distributed tracing. You need synthetic monitoring. You need to care about the things between your pods and your users.
# What most people monitor
- pod_cpu_usage
- pod_memory_usage
- container_restarts
# What actually matters
- p99_request_latency
- error_rate_by_endpoint
- dns_resolution_time
- certificate_expiry_days
- persistent_volume_iops
- cross_az_network_latency
"Kubernetes will save us money through better resource utilization!" — every pitch deck ever.
Here's what actually happens:
Control plane costs. On EKS, you're paying $0.10/hour just for the API server. That's $73/month before a single pod runs. Multi-cluster? Multiply that.
Node overhead. Each node runs kubelet, kube-proxy, a CNI agent, potentially a logging agent, a monitoring agent, and whatever DaemonSets your platform team decided you need. On a t3.medium, that's easily 30-40% of your node's resources consumed by the platform itself.
The "we need three environments" multiplier. Dev, staging, prod — each gets its own cluster. Each cluster needs its own nodes, its own monitoring, its own ingress. Your cloud bill just tripled.
Load balancer proliferation. Every Service of type LoadBalancer creates a cloud LB. Each costs $15-25/month on AWS. I've seen clusters with 40+ load balancers because nobody bothered setting up a shared Ingress controller.
Persistent volume waste. Volumes that were provisioned for deleted pods, sitting there accruing charges. PVCs that are 100GB because someone copy-pasted a manifest and forgot to adjust.
The real cost of Kubernetes isn't the sticker price — it's the operational overhead. You need at least one person who deeply understands the platform. That person's salary dwarfs your cloud bill.
Kubernetes networking is a layer cake of abstractions, and every layer can break independently:
# This innocent-looking config has bitten me multiple times
apiVersion: v1
kind: Pod
spec:
dnsConfig:
options:
- name: ndots
value: "2" # Override the default 5
# Without this, resolving "api.stripe.com" generates
# 5 failed lookups before the real one succeeds
Yes — if you have the team, the workload, and the operational maturity for it.
Kubernetes makes sense when:
Kubernetes is overkill when:
The honest truth: Kubernetes is infrastructure for building platforms. If you don't need a platform, you don't need Kubernetes. A well-configured VPS with Docker Compose and a CI/CD pipeline will outperform a poorly operated Kubernetes cluster every single time.
Kubernetes is a powerful tool. It's also a complex one that demands respect. If you're going to run it in production, go in with your eyes open:
And if you're a mid-sized company wondering whether Kubernetes is right for you — talk to someone who's operated it, not just someone who sells it. The difference matters.