Kubernetes cluster on your own hardware: building it and locking it down

Most companies that ask me about Kubernetes do not need a Kubernetes cluster. That is the first thing I say on the call, and half of those conversations end there, to everyone’s benefit. When it is genuinely the right answer, what follows is engineering rather than a choice between “spin it up tonight” and “buy managed”. Below is how I build a cluster on my own hardware and what I actually lock down in it: not a list of recommendations, but decisions with the price of each one attached.

One warning first. A large share of the Kubernetes advice that search engines return today is not merely dated, it is wrong. The ingress controller in most tutorials has been retired and will never receive another security fix. The flag that supposedly “enables secret encryption” may enable nothing at all. Everything here was checked against primary sources in September 2026, and the links are at the end.

First, the honest question: do you need a Kubernetes cluster

Kubernetes solves a problem most teams do not have: many teams shipping many services onto a shared fleet without having to coordinate by hand. With three services and one team you pay the full price of that decoupling and use a fraction of it.

The most useful thing I can tell you about cluster size sounds backwards: two nodes are strictly worse than one. etcd quorum is a majority, and the fault-tolerance table reads: one node survives zero failures, two nodes survive zero failures, three nodes survive one. The second machine doubles your cost and your failure surface without buying a single survived failure. It buys the feeling of high availability.

So for many clients the honest answer is one node, etcd snapshots and the machine configuration in git. Recovery takes minutes rather than hours, and it is testable. High availability you have never exercised is not high availability, it is a line on a slide. Red Hat ships and supports single-node OpenShift for production at the edge, and Talos turns a single-node cluster on with one configuration key, so you are in respectable company.

In production I recommend separate services, not a cluster

If you have read this far and are still unsure, here is my default answer for production: not a cluster, but a handful of self-contained services with ordinary high availability underneath them. The cleanest form of a service is a statically linked Go binary: it drags no runtime behind it, needs no base image, starts as a systemd unit and is upgraded by replacing a file. Next to it, a floating address through keepalived, or the hypervisor’s own high availability layer. That is the entire stack.

The argument is not fashion and not taste, it is the number of moving parts you have to maintain. A cluster is not one component: it is a container network, an ingress, admission control, a state store, a delivery system and an agent on every node, each with its own upgrade calendar and its own way of failing. Everything described below in this article is the cost of owning that set. A binary with a systemd unit has a failure that fits in your head: the process is alive or it is not, the port is listening or it is not, the file is there or it is not. So does the recovery: put the file back, restart the unit.

A cluster wins exactly where its defining property wins: many teams shipping many services without having to coordinate by hand. While you have a dozen services and one team, you are paying for decoupling you do not use and getting the work of maintaining it in return. That is not caution, it is arithmetic.

And here is the point I press hardest: Kubernetes for developers and Kubernetes for production are two different decisions, and they must not be mixed. In development a cluster is exactly where it belongs: an environment per branch, throwaway namespaces, state nobody will miss, a fast rollback. There its price is repaid in team speed. Production means a separate cluster with separate credentials, a separate network and separate access, not a namespace inside the same one.

The reason is technical rather than organisational. A namespace is a convenience boundary, not a security boundary: admission control is bypassed through the four doors that get their own section below, a get permission on nodes/proxy knows nothing about your namespaces, and one mistake in the delivery system’s policy arrives in all of them at once. The test is simple: if a production service and a developer sandbox can be stopped by the same typo in a manifest, you do not have two environments, you have one.

The distribution decides how much operating system stays yours

Kubernetes distributions do not differ much on features: they all pass conformance, so they can all do the same things. They differ in how much operating system remains your responsibility.

• kubeadm bootstraps the control plane and openly disclaims the rest: machine provisioning and container networking are declared out of scope. Maximum control, maximum work for you.
• k3s runs the whole control plane as goroutines in a single binary. It is a CNCF-certified Kubernetes distribution rather than a stripped-down toy, and it states plainly that it will not modify the host operating system. The binary, incidentally, is 81.5 MB, not the “under forty” that reviews keep repeating.
• RKE2 gives you static pods, CIS profiles behind one setting, and a build with federally validated cryptography. Its own documentation still cites FIPS 140-2 while its competitors have moved to 140-3.
• Talos Linux removes the operating system from your hands entirely: no shell, no SSH, the whole machine declared in one YAML file, upgrades delivered as A/B images.
• OpenShift is a platform plus a support contract. You pay for them in distance from upstream.

That distance deserves a number, because “lightweight distributions always lag” is a dead claim. I checked the release APIs on 18 September 2026. Upstream 1.37 landed on 26 August. k3s and RKE2 both shipped on 14 September, Talos within the same days. Nineteen days. OpenShift 4.22 shipped in July and still carries Kubernetes 1.35: seven months. The gap between nineteen days and seven months is the actual trade: you are not buying lag, you are buying someone to call.

Talos deletes my Ansible role

My default answer to any infrastructure is to describe it in code. The Proxmox cluster that rebuilds itself from a git clone works exactly that way: every manual fix becomes a playbook, and the state lives in a repository rather than in someone’s memory.

Talos answers that with something uncomfortable. Its philosophy page says, in plain words, that machine state is declared in a single manifest, with no scripting and no procedural steps, and with no configuration management tools to maintain. That is not a compliment to my method. It is a challenge to it.

The conclusion I reached does not favour consistency: Ansible is right for a mutable operating system I own, and unnecessary for an operating system I cannot log into. Proxmox is mine. It has a shell, packages and state that drifts, and there the Ansible role pays for itself daily. On Talos there is nothing to drift: no shell, no packages, no way to adjust anything by hand between releases. There is no configuration to automate, because the configuration is the manifest. Two layers, two answers.

That choice has a cost, and it should be named before a reader names it for me. At three in the morning, with a node in a state the API does not describe, Talos gives me no strace, no tcpdump and no way to simply log in and look. What remains is talosctl dmesg, logs and a support bundle. And what also remains is the same decision I already apply on Proxmox: do not repair, reprovision and run the automation again.

Why three nodes and not two, for the second time

I have written this heading before, one floor down, about Corosync quorum. The argument is word for word the same and only the numbers change: a majority is what lets a system decide unambiguously, two nodes cannot form one, and the third breaks the symmetry so the winner is always determined.

The more interesting part is what general articles cannot tell you, because it requires looking at a specific estate. Put a three-node Kubernetes cluster on top of a three-node Proxmox cluster and you get two independent quorum systems that fail together. Losing one physical machine costs one vote in each. That is not a catastrophe: both survive the loss of a single member. It does mean the margins do not add up, they coincide, and maintenance has to be planned as if for one system rather than two.

External etcd on separate machines breaks that coupling. I do not run it: six machines to protect an estate of three is a solution to a problem I do not have.

There is no load balancer in the rack

Almost every Kubernetes guide is written from a cloud, and it shows in one place: a service of type LoadBalancer simply works there. On your own hardware something has to provide it, and this is the most under-written part of the whole subject.

• MetalLB is the best-known answer. It is nine years old and still has not reached version 1.0.
• Cilium address allocation with L2 announcements is in beta, failover takes ten to twenty seconds while ARP settles, one elected node announces the address, and there is no balancing between nodes.
• BGP to a real router is the right answer when the router is yours and you are willing to keep its configuration as part of the infrastructure.

The textbook separately demands HAProxy plus keepalived in front of the API server. Here I make the same move as wiring Thunderbolt instead of buying a ten-gigabit switch: refuse the expensive standard component, but only after understanding why it was recommended. Talos runs a local load balancer to the control plane on every node, and k3s solves the same problem with a client-side balancer inside the agent. Intra-cluster traffic then does not depend on an external box and survives it being switched off. You still want a balancer from the outside, but now it serves administrative access rather than the life of the cluster.

In March 2026 ingress-nginx was retired

This is the rare case where the textbook answer did not get worse, it stopped existing. The ingress-nginx repository is now a read-only archive. The statement from the Steering and Security Response Committees leaves no room for interpretation: there will be no further releases, no bug fixes and no fixes for security vulnerabilities. The same statement describes it as critical infrastructure for roughly half of cloud native environments.

It gets better. InGate, the replacement the network special interest group had officially planned, was retired without ever shipping; its repository is archived too. And the Ingress API itself is now frozen, which is stated on the concept page in the Kubernetes documentation alongside the recommendation to use Gateway.

There is a naming trap here that catches a lot of people: ingress-nginx and NGINX Ingress Controller are two different products. The community project died. F5’s commercial controller is alive and still shipping. If you are reading instructions and cannot tell which of the two they mean, you also cannot tell what advice you just received.

The main thing to know about migrating is that it is not automatic. The official conversion tool states outright that it does not carry over annotations, and annotations are where everything interesting lives: rate limits, external authorisation, path rewrites, configuration snippets. You get the routing skeleton by machine and write the policy by hand.

And one trap makes migrations fail quietly: ingress-nginx routing does not behave the way the manifests read. With regular expressions enabled, matching becomes prefix-based and case-insensitive, so a path such as /[A-Z]{3} also catches lowercase and everything that follows it. Port the rules literally and you get different behaviour from identical files.

The default deny that denies nothing

Kubernetes network policies are a declaration of intent, not a mechanism. The container network enforces them. If your CNI does not implement them, you get the worst possible outcome: kubectl apply succeeds, kubectl get netpol shows the policy, and traffic flows exactly as before. On Flannel that is precisely what happens.

The second thing worth memorising: traffic to and from a pod’s own node is always allowed, and that is written into the specification as an exception rather than being an implementation bug. Which means “we have default deny enabled” is a claim to be tested, not configured. The task is properly stated like this: produce traffic that should be blocked and confirm that it is blocked. Everything else is faith.

One more landmine for anyone reading articles older than a year: cluster-wide administrative policies have been renamed. The resource kind used in most examples no longer exists under its old name.

What sits in etcd is base64, not ciphertext

The Kubernetes documentation puts this undiplomatically: by default the API server stores plain-text representations of resources in etcd, with no at-rest encryption. A Kubernetes Secret is not secure storage, it is an object labelled “handle with care”.

Then come three traps in a row, each of which appears in articles ranking on the first page.

• A configuration that looks like encryption and is not. If identity is listed first among the providers, nothing is encrypted, even though the flag is set and everything starts cleanly.
• The algorithm nearly everyone recommends and the documentation does not. The aescbc provider is marked as not recommended, because CBC is vulnerable to padding oracle attacks.
• Turning it on does not encrypt what is already there. Existing secrets stay in the clear until something physically rewrites them. That is a separate operation, not a consequence of editing the config.

And now the honest part about method. On your own hardware, with no cloud key management service, local encryption is largely theatre, and the documentation says so itself: the keys sit in plain YAML on the same host as etcd, so the protection works against a stolen etcd disk and fails against a compromised host. I still enable it, but I call it what it is: one layer against one scenario. The real answer lies elsewhere and is described separately: secrets are issued for a period rather than stored, and the master keys live in a hardware module.

Policies moved in-tree, and Kyverno is one more service that can go down

The reflex to install Kyverno or Gatekeeper first was formed when Kubernetes had no policy language of its own. It has one now: CEL validating policies have been stable since 1.30 and mutating policies since 1.36, and both are in the default admission plugin list. For simple guardrails an external engine is no longer required.

One addition fits my approach to infrastructure exactly: 1.37 brings admission policies defined by files on disk. They never touch etcd, they load before the API server serves its first request, and no set of permissions can delete them. That closes the bootstrap window every “policy as code” post ignores. The feature is on by default but still in beta, and that is the caveat I take into production with it.

An external engine carries a risk that installation guides never mention. Kyverno configures its webhooks to fail closed by default, which is safer for policy and more dangerous for availability. If the webhook is unreachable while the control plane restarts, nothing gets created. Including Kyverno itself. Getting out means deleting the webhook configuration objects by hand, and you will be doing it at night. In-process policies cannot fail this way, because they execute inside the API server.

If you do install Kyverno, note its own API shift: the resource kind used by every tutorial on the internet is deprecated in favour of the newer CEL-based types. New policies should be written against those from the start.

Four doors around admission control and audit

Admission control runs on the API server. Anything that never reaches the API server never passes through it, and never appears in the audit log either. There are four such doors: static pods, which the kubelet starts from a file on disk; the kubelet’s own API; direct access to etcd; and the container runtime socket on the node.

Of the practical consequences the most underrated is this: get on nodes/proxy is not a read permission. It authorises command execution in any container on that node. It is handed to monitoring agents routinely, “so they can collect metrics”, and it means nothing like what it looks like. Since 1.36 there is a proper way out: fine-grained kubelet authorisation, where the agent receives narrow permissions instead of one broad one.

GitOps is tier zero, not a deployment tool

A tool with permission to change anything in the cluster sits next to the hypervisor in criticality, not next to CI. Two stories from 2026 make the point better than any argument.

In July a remote code execution chain in Argo CD was published: the internal component that renders manifests listens on its port with no authentication, because it is “internal”. Inside a cluster, “internal” means “reachable by any pod”. It was reported back in January 2025. At publication there was no code-level fix; the only public remedy is a change of defaults in the Helm chart, which finally creates network policies. If you installed Argo CD by chart and carried your own values.yaml across versions, you most likely still have none. Checking takes a minute: the chart version, and whether network policy objects exist in the namespace.

In August Flux got its own: a policy installed by default with the tool admitted ingress to the notification controller from every namespace. That closes the argument that a pull model saves you from needing network policies.

I stopped choosing between Argo CD and Flux on a feature table long ago, because there is parity there. The real axis is different: whose code is this, and under which licence does it run inside my cluster. The Flux core is Apache-2.0 within the CNCF, but the entire modern layer around it lives in a vendor organisation under AGPL-3.0. Argo CD’s interface sits inside the CNCF project itself. This is the same kind of reasoning as choosing a distribution: not what wins a review, but what I am willing to host and what I pay for it.

One more thing breaks without warning on upgrade: moving Argo CD to the current branch switches the templating engine to Helm 4 for all your charts, and pinning the old version in the application manifest is simply ignored. Helm 3 receives security fixes only until February 2027.

Secrets: a chain with no standing token

Start with the unpleasant part: every popular approach, sealed secrets, encrypted files in the repository, external secret operators, ends by writing an ordinary plain object into etcd. What differs between them is who can read the source, not whether etcd is clean. The only path that avoids materialising the object needs a CSI driver with an ephemeral volume and a commercial licence.

That has to be accepted deliberately and said out loud. I accept it: encryption at rest sits above it, permissions are separated, and nobody touches the data by hand. The interesting work is in removing standing access.

A capability barely six weeks old makes that possible: the Flux controller can exchange its Kubernetes service account token for a short-lived OpenBao token and decrypt files through the transit engine, with no permanent secret anywhere in the cluster. The key never physically leaves the store, and the store’s own master keys live in a hardware module. It is a continuation of the same principle I already apply one floor down: access is granted for a period, and standing access does not exist.

A warning for anyone running OpenBao with a hardware key: the next major version removes hardware module distribution and the vendor auto-unseal mechanisms from the core, moving them into plugins, and drops the file storage backend entirely. For an installation built around a hardware key that is not an optional upgrade, it is a scheduled breakage to prepare for in advance.

Cheap wins almost nobody turns on

Most hardening guides ask you to edit every application manifest. That works right up to the first team that does not know your rules. Node-level and cluster-level settings are more reliable, because a new service cannot forget them.

• A default syscall restriction profile at kubelet level. One key in the node configuration applies it to every pod, with no manifest edits at all.
• Restricting what a node may change in the API. An admission plugin kubeadm adds for you and hand-built clusters frequently miss.
• Narrowing anonymous access to liveness endpoints instead of the all-or-nothing switch.

And one fact about pod security standards that props up a false sense of safety: enforcement does not apply to Deployments or Jobs, only to the resulting pods. Warnings and audit do work at the workload level; enforcement does not. So “we enabled restricted” does not mean the Deployment will be rejected. It will be accepted, the pod will not be created, and you will find out during the rollout.

What breaks on upgrades

Abstract Kubernetes risk gets discussed endlessly, while what actually breaks is upgrade day. Here is what belongs in the calendar.

• kubeadm client certificates last exactly one year. They renew automatically only during a cluster upgrade. This is the most common way a hand-built cluster dies quietly: it simply stops letting you in a year after installation.
• Mandatory mounting with security labels became default behaviour in 1.37, with an explicit warning that it may break existing workloads.
• Upgrading etcd from 3.5 to 3.6 through the wrong version can resurrect removed members and destroy quorum. You have to land on the fixed release in the 3.5 line first, and only then move on.
• etcd 3.7 deleted every experimental flag. Infrastructure-as-code templates that set them will refuse to start.
• The snapshot commands moved. Restoring from a backup now uses a different utility, so a runbook written before 3.6 is wrong.

Separately, about something that did not happen even though many wrote that it did. Support for the first control group mechanism was not removed in 1.36: I downloaded the changelog and checked. It was deprecated earlier, and the kubelet refuses to start on such a node by default, but nothing was removed. And nftables did not become the default proxy mode, contrary to half the release coverage: it is stable, the old mechanism is still the default, and a different one is the one scheduled for removal.

What I deliberately do not do

• No OpenShift on a small estate. That buys half a year of lag and a licence in exchange for a support contract that does not pay off at this size.
• No external etcd. Six machines to protect three.
• No service mesh on three nodes. What people usually install it for, mutual authentication between services, is now available at the container network layer without injecting a proxy into every pod. Skipping the mesh costs me application-level policy, per-request retries and circuit breaking, and a shared identity across clusters. I do not need those; if you do, that is exactly the reason to install one.
• No management plane above clusters. Good tools exist, and three nodes do not need managing by them.
• No reliance on domain-name egress policy. It is still experimental.
• No proxy mode that is scheduled for removal, even where it is still faster today.

A benchmark is a question generator, not a score

The industry security benchmark for Kubernetes is useful, and it carries a misconception that costs money. The current revision came out in June 2026 and covers versions 1.34 and 1.35. Kubernetes 1.37 shipped in August. Which means there is no benchmark for a current cluster at all, and the scanning tool will score you against a document written for earlier releases.

Hence the right way to use it: take the list of questions, not the final percentage. Whole sections about control-plane file permissions do not apply to a managed cluster, and satisfy themselves on a kubeadm cluster. Teams burn weeks on them because they are chasing a compliance number. Meanwhile a small cluster’s real threat model differs from the one the checklist assumes: on my own hardware there is no cloud metadata service to steal credentials from, and there are things the checklist never mentions.

Stack

• Kubernetes: the orchestrator; 1.37 is current and three minor releases are supported.
• Talos Linux and k3s: two answers to how much operating system stays yours.
• etcd: cluster state; dedicated disks, snapshots, and caution on major upgrades.
• Cilium: container networking, network policy and address allocation on bare metal.
• Gateway API: cluster ingress now that Ingress is frozen; installed separately from core and versioned by you.
• cert-manager: certificates; lifetimes are short, expiry emails are gone, and watching for expiry is now your job.
• OpenBao: secrets and transit encryption, with master keys in a hardware module.
• Flux or Argo CD: delivery from the repository, chosen on licence and stewardship rather than on the interface.
• Ansible: the layer below the cluster, where the operating system is mutable and mine.

Why this matters if you are paying

A Kubernetes cluster is not a line in an infrastructure budget, it is a multi-year commitment. You take on three releases a year, each with a support window of about fourteen months, the last two of which are security only. You take on a component that can be retired, as the ingress controller of half the industry just was. You take on a delivery tool with permission to change everything in the cluster.

Two practical conclusions follow for whoever signs the invoice. First: do not take a cluster if the job can be done more simply, because the money goes on your team’s attention rather than on hardware, and attention is the expensive line. Second: if the cluster already exists, the most valuable spend is not a new security product but a review of what is already running by default. Plain-text secrets in etcd, policies that enforce nothing, network access to the internals of the delivery system, and permissions that read like read access and mean execution: all of it is already switched on, and fixing it costs nothing but time.

Frequently asked questions

Are two nodes more reliable than one?

No, strictly the opposite. Quorum is a majority: one node survives zero failures, two also survive zero, three survive one. The second machine doubles cost and failure surface without buying a single survived failure.

Does Kubernetes encrypt secrets?

Not by default: the API server stores plain-text representations of objects in etcd. Enabling encryption at rest does not touch existing secrets until something rewrites them, and if identity is listed first among the providers, nothing is encrypted at all.

Can I still install ingress-nginx?

No. The project was retired in March 2026, the repository is archived, and there will be no further security fixes. The Ingress API itself is frozen, and the replacement planned by the working group was retired without shipping. F5’s commercial NGINX controller is a different product and it is alive.

Do I need Kyverno if Kubernetes has policies built in?

For simple guardrails, no: CEL validating policies have been stable since 1.30 and mutating policies since 1.36, and both are enabled by default. An external engine adds capability and a new failure mode at the same time: its webhooks are configured by default so that nothing is created while they are unreachable.

What should I use instead of Kubernetes in production?

A handful of self-contained services with ordinary high availability underneath: a statically linked binary, a systemd unit, a floating address through keepalived or the hypervisor’s own high availability layer. A cluster is justified when many teams ship many services without coordinating by hand; with a dozen services and one team you are paying for decoupling you do not use.

Can development and production share one cluster?

No. A namespace is a convenience boundary, not a security boundary: admission control is bypassed by static pods and the kubelet’s own API, a get permission on nodes/proxy knows nothing about namespaces, and one mistake in the delivery policy arrives in all of them at once. Production belongs in a separate cluster with separate credentials, network and access.

What should I check in my cluster first?

Four things: whether encryption at rest is on and identity is not listed first; whether network policies are enforced by your container network or silently ignored; whether network policies exist around the delivery system’s components; and who holds get on nodes/proxy, because that is not read access but command execution on the node.

Sources

• Kubernetes releases and support windows, the 1.37 announcement.
• Encrypting data in etcd: defaults, providers and the limits of local keys.
• The ingress-nginx retirement and the Steering Committee statement.
• The frozen Ingress API and the Gateway API documentation.
• Network policies: CNI dependence and the same-node exception.
• Pod Security Admission: why enforcement does not apply to workload resources.
• CEL admission policies built into Kubernetes.
• Operating etcd: defragmentation, compaction and snapshots.
• CIS Kubernetes Benchmark and kube-bench with its version mapping table.
• The Talos Linux philosophy and the k3s documentation.

Need a consultation?

If you are deciding whether you need a cluster at all, or you already have one and want to know what is open in it by default, book a call. We will look at your case and say honestly where Kubernetes pays for itself and where it costs more than it returns. Related reading: the Proxmox cluster as code, Zero Trust in practice, SIEM on an open core.