Kubernetes Study 1 - node and cluster

I started learning Kubernetes because I have something to build around backup. I've run containers and put a program on a single server, but I didn't know the concepts.

Part 1 covers three words — node · Pod · controller — and the place where I got badly stuck while learning them: the cluster.

A container isn't a lightweight VM

I had this wrong from the start. I thought of a container as a small VM.

A virtual machine is a separate computer with a whole second operating system inside it, so it's heavy and takes tens of seconds to boot. A container is just a process — one process, same as any other program running on my laptop. The kernel limits what that process can see: which file system it sees, whether other processes are visible, and what its IP is. There's only one OS; each process just has a different scope of what it can see.

That's why docker run comes up almost instantly. It isn't booting a new OS; it's starting a process.

One important fact follows. A container that dies stays dead. It's a process, and nothing restarts it. Everything later in this post starts from that fact.

Four moments where one server isn't enough

I liked that the lesson didn't start with the vocabulary. It started with the problems. Say you have containers running on one server, and then this happens.

1. A process dies at 3am. The service is down until someone gets to work. Who restarts it?
2. The whole server dies. A disk failed, or the power went out. Restarting a process doesn't help. How do you move to another server?
3. Traffic grows and you need three servers. Who works out how many go where? What if one sits idle while another is overloaded?
4. You have to ship a new version without downtime. Take one down, bring one up, check it, roll back if it's bad. By hand, every time?

All four are solved by having a person watching. Kubernetes is the program that does that person's work.

Seeing the problems first meant I understood the words without having to memorize them.

The three words

node

One computer that actually runs Pods — a physical server or a cloud VM. To solve problem 2 (the whole server dies) you need more than one machine in the first place, which is why this word exists.

Pod

Almost always the unit that wraps one container. Kubernetes adds this layer because sometimes two containers must run on the same machine, share an IP, and start and stop together. A Pod is the unit for handling such a group as one.

This hit my project directly. A Pod is designed to be disposable. When it dies, it isn't revived; it's discarded and a new one is created, with a different name and a different IP.

So "snapshot a running Pod and restore it as-is" doesn't work from the start. What a backup has to preserve is the description saying "a Pod like this should exist." The running Pod itself can always be recreated. This apparently comes back later as the spec vs status split, but I haven't got there yet.

controller

A program that runs a very simple loop continuously.

1. Read the desired state ("there should be 3 nginx Pods")
2. Look at the current state ("there are only 2")
3. Act to reduce the difference ("create one more")
4. Go back to 1, and keep repeating.

The official docs compare it to a thermostat. Set it to 25 degrees, and it checks the room's current temperature and turns the heating on or off when they differ. It doesn't take a "make it 25 degrees" order once and stop; it keeps checking and keeps reducing the difference.

All four problems above are solved by this. If something dies, it's recreated. If a server drops out, it's recreated on another node. If the count is short, it's filled. For a new version, you change the desired state and the Pods are replaced one at a time. The loop does the work a person used to do.

The exercise had no Kubernetes in it

The first exercise wasn't creating a cluster. It was starting one container with Docker and killing it. It doesn't come back, and nothing restarts it.

The exercise shows what goes wrong without Kubernetes before teaching it, and that worked faster for me than learning controllers from a definition. "A loop that reduces state differences" didn't mean much to me; "I killed it and nothing restarted it" made sense immediately.

Where I got badly stuck

On a quiz question asking what node refers to, I picked "the name for the whole cluster" and got it wrong.

Why I picked it mattered more than getting it wrong, because my next question was: "If a node is one server, where is the cluster installed as a process?"

That question held the whole misconception. I was treating every Kubernetes concept as software installed somewhere. That's probably the natural assumption if your only experience is putting a program on a single server.

There's no process called cluster

Here is everything that is actually installed on the machines.

[Server A]  kube-apiserver · etcd · kube-scheduler · kube-controller-manager · kubelet
[Server B]  kubelet · kube-proxy · containerd
[Server C]  kubelet · kube-proxy · containerd

There's no process named cluster anywhere. A cluster is the name for "the set of machines registered with the same API server."

If server B's kubelet is configured to talk to server A's API server, B belongs to that cluster. Point it at a different API server and B belongs to a different cluster, without reinstalling anything.

That's when it made sense. I had been wondering where the cluster exists, and it turned out to be the name for the registered machines as a group.

How registration works

A machine needs three things to join.

• API server address — who to send requests to
• CA cert hash — how the node trusts the API server
• A credential (bootstrap token) — how the API server trusts the node

Trust goes both ways, so it's three things rather than two. Not any server can claim to be the API server, and not any machine can join.

kubeadm join 10.0.0.5:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:1a2b3c...

Then registration happens. The kubelet authenticates with the token, gets its own client certificate issued (TLS bootstrapping), and uses it to create itself as a Node object on the API server. After that, it periodically updates "still alive, this much capacity left."

So the only thing installed on the machine is the kubelet, and joining means one entry appears in the API server's list.

Leaving works the same way, which I found interesting. kubectl delete node removes that entry and the machine keeps running. If the kubelet is still running, it registers itself again shortly after. Removing the entry from the list and actually removing the machine are separate things.

Is a master node a separate kind of node?

I asked one more thing. I had assumed a master node was a separate kind of machine, and wondered whether "cluster nodes" were their own category.

There's no separate kind. A control plane node runs a kubelet and registers the same way as above; it's just a node. There are only two differences.

• Different processes run on it. kube-apiserver, etcd, kube-scheduler and kube-controller-manager run there as well — and usually even these run as Pods.
• It carries one marker. A taint, node-role.kubernetes.io/control-plane:NoSchedule. It's a rule string telling the scheduler "don't place ordinary apps here," not a physical constraint.

Because it isn't a physical constraint, removing the taint lets apps run on that machine too. A single-node cluster created with kind reportedly removes it, since with only one node you couldn't run any apps otherwise.

master is a retired name; it's now called a control plane node. It doesn't have to be one machine either — production setups usually run three or more because of etcd quorum, though I haven't done that yet so I don't really know.

Part 1 summary

• A container isn't a VM; it's a process with a limited view of the system. When it dies, it stays dead.
• There are four moments where one server isn't enough, and Kubernetes does the work a person used to do in them.
• node is one machine, Pod is the unit that wraps a container, and controller is a loop that reduces the difference between desired and current state.
• A Pod is disposable, so you can't snapshot and restore one. A backup has to preserve the description that says "a Pod like this should exist."

And the things I had wrong:

The last three were the same misconception: I took things that are only names to be something installed. I suspect I'll get stuck the same way with namespaces and Services.

It isn't all sorted out yet, but I think I'm getting a feel for it. When a new word comes up, I'll ask first whether it's a process or a name.

Next part: the API server.