Kubernetes in Production: What to Check Before You Trust the Cluster

Getting a workload running on Kubernetes is the easy part now. Most teams reach that point within a week of starting. The harder, less visible work is everything that decides whether that cluster behaves predictably at 3am during an incident, six months after the person who set it up has moved to a different project.

Resource requests and limits are not optional metadata

A pod without resource requests can be scheduled anywhere, including onto a node that is already under pressure, and Kubernetes has no basis for making a good decision about it. Without limits, a single misbehaving process can consume enough memory or CPU to affect every other workload sharing that node. This is one of the most common root causes we see in noisy, hard-to-diagnose production incidents, and it is also one of the simplest to prevent.

• Every container should declare CPU and memory requests based on observed usage, not guesses
• Limits should be set deliberately, understanding that a memory limit breach kills the container while a CPU limit only throttles it
• Namespaces handling shared or multi-tenant workloads should also set resource quotas at the namespace level

Liveness and readiness probes need to mean something different

It is common to see both probes pointed at the same shallow endpoint, or worse, at an endpoint that checks downstream dependencies. A liveness probe answers one question: is this process stuck and does it need to be restarted? A readiness probe answers a different one: can this pod currently accept traffic? Conflating them causes two failure patterns that look confusing until you know what to look for: cascading restarts when a downstream dependency has a blip, or traffic being sent to a pod that cannot actually serve it yet.

Namespace boundaries and network policy are usually an afterthought

By default, any pod in a cluster can talk to any other pod. That is convenient during initial setup and becomes a real exposure the moment more than one team or more than one trust level shares the cluster. Namespaces without network policy are an organizational boundary, not a security boundary, and treating them as the same thing is a common and avoidable mistake.

• Define default-deny network policies per namespace, then explicitly allow the traffic that is actually needed
• Separate namespaces by trust level and blast radius, not only by team or product name
• Review RBAC bindings for accumulated permissions nobody remembers granting

Someone needs to own upgrades before an upgrade is forced on you

Kubernetes versions have a support window, and cloud providers eventually force upgrades whether or not a team has planned for one. Clusters that have never been upgraded on a deliberate schedule tend to accumulate deprecated API usage that only surfaces as a break during a forced, urgent upgrade. Testing upgrades against a staging cluster on a regular cadence turns this into routine maintenance instead of a fire drill.

Observability has to answer questions, not just display metrics

A dashboard full of CPU and memory graphs looks reassuring and often is not that useful during an actual incident. The more useful question is whether you can quickly answer: which deployment changed right before this started, which pods are actually serving traffic right now, and what did this specific request path look like end to end. That usually means connecting metrics to logs to traces, not adding another isolated chart.

Secrets management deserves more than the built-in default

Kubernetes Secrets are base64 encoded, not encrypted, unless you have deliberately enabled encryption at rest and configured access carefully. Treating a Kubernetes Secret as equivalent to a properly managed vault is a common misunderstanding that becomes a real problem the first time cluster access is broader than intended, whether through a compromised service account or an overly permissive RBAC role.

A short pre-production checklist

• Every container has explicit CPU and memory requests and limits
• Liveness and readiness probes check different things for a real reason
• Default-deny network policy is in place, with explicit allow rules
• There is an owner and a schedule for cluster and API version upgrades
• Observability connects metrics, logs and traces for the services that matter most
• Secrets are encrypted at rest and access is scoped through RBAC, not shared broadly
• There is a tested runbook for at least the two most likely failure scenarios

None of this is exotic. It is the difference between a cluster that happens to work today and a platform a team can actually operate with confidence during an incident. Most Kubernetes consulting engagements we run start by walking this exact list against a real cluster, because it surfaces the gaps faster than starting from architecture diagrams.