Kubernetes Study 1 - node and cluster

Started learning Kubernetes because I've got something to build around backup. I've run containers and I've put a program on a single server, but I don't know the concepts.

Part 1 is three words. node · Pod · controller. And while learning those I got stuck badly on one thing: the cluster.

A container isn't a lightweight VM

Got this wrong from the start. I thought of a container as a small VM.

A virtual machine is a fake computer. There's a whole other operating system inside it, so it's heavy and takes tens of seconds to boot. But a container is just a process. One process, same as any other program running on my laptop. The kernel just lies to it. "This is the only file system you can see," "there are no other processes," "this is your IP." There's only one OS, and the container is just running with its view blocked.

That's why docker run comes up almost instantly. It isn't booting a new OS, it's starting a process.

One important consequence. A container that dies just stays dead. It's a process. Nobody brings it back. That fact drags everything else along behind it.

Four moments where one server isn't enough

What I liked about the lesson was that it didn't hand over the words first. It showed the awkward situations first. Say you've got containers running on one server, and then this happens.

1. A process dies at 3am. Service is down until you get to work in the morning. Who brings it back?
2. The whole server dies. Disk failed, or power went. Restarting a process doesn't fix that. How do you move to another server?
3. Traffic grows and you need three servers. Who calculates how many go where? What if one sits idle while another is on fire?
4. You have to ship a new version without stopping. Take one down, bring one up, check it, roll back if it's bad. By hand, every time?

All four are solved by having a person on it. Kubernetes is the program that replaces that person for those four things. That's the whole thing, apparently.

I liked this order. Once you see the pain, you don't need to memorize the words.

So three words came out of that

node

One computer that actually runs Pods. Physical server or cloud VM, doesn't matter. To solve problem 2 (the whole server dies) you need more than one machine in the first place, so you need this word.

Pod

Almost always a shell wrapping one container. Why wrap a layer around it instead of dealing with containers directly? Because sometimes two containers have to be on the same machine, share the same IP, and live and die together. Something had to hold that "one body," and that's a Pod.

And here's something that hits my project directly. A Pod is designed to be disposable. When it dies you don't revive it, you throw it away and make a new one. The new Pod has a different name and a different IP.

So "take a snapshot of a running Pod and restore it as-is" doesn't hold up from the start. What a backup has to preserve isn't the Pod itself but the description saying "a Pod like this should exist." Apparently this comes back as the spec vs status split later, but I'm not there yet.

controller

A program running a very simple loop forever.

1. Read the state that should be ("there should be 3 nginx Pods")
2. Look at the state that is ("only 2")
3. Do something to close the gap ("make one more")
4. Back to 1. Forever.

The official docs compare it to a thermostat. Set it to 25 degrees, it looks at what the room is at now, and turns the boiler on or off if there's a gap. It doesn't take a "make it 25 degrees" order once and finish — it keeps watching and keeps closing.

All four problems above get solved by this. Dies, make it again. Server drops out, make it on another node. Short on count, fill it. New version, change the desired state and it swaps them one at a time. A loop taking the place of the person who used to wake up at 3am.

The exercise had no Kubernetes in it

The first exercise wasn't creating a cluster, it was starting one container with Docker and killing it. It doesn't come back. Nobody brings it back.

It's built to make you feel the pain of not having it first, and that landed faster than learning controllers from a definition. "A loop that closes state gaps" doesn't reach me. "I killed it and nobody's bringing it back" does, immediately.

And here's where I got stuck badly

On a quiz question asking what node refers to, I picked "the name for the whole cluster" and got it wrong.

Being wrong mattered less than why. Because the next thing out of my mouth was this.

If a node is one server, then where is the cluster installed as a process?

The whole misconception is in that question. I was treating every Kubernetes concept as software installed somewhere. Probably the model you end up with if the only thing you've done is put a program on a single server.

There's no process called cluster

List only the things actually installed on a machine and this is all of it.

[Server A]  kube-apiserver · etcd · kube-scheduler · kube-controller-manager · kubelet
[Server B]  kubelet · kube-proxy · containerd
[Server C]  kubelet · kube-proxy · containerd

No process named cluster. Go looking and it isn't there. A cluster turned out to be the name for "the set of machines registered with the same API server."

If server B's kubelet is set to look at server A's API server, B belongs to that cluster. Change the config to point somewhere else and B belongs to a different cluster, without a single thing being reinstalled.

That's where it just clicked. I'd been wondering where the cluster exists, and it felt like something like a logical agreement. You just get registered and that counts as being the cluster.

So how does registering work

Three things to put a machine in.

• API server address — who to talk to
• CA cert hash — the node's basis for trusting the API server
• A credential (bootstrap token) — the API server's basis for trusting the node

Trust runs both ways, so three and not two. Not just any server gets to say "I'm the API server," and not just any machine gets to walk in.

kubeadm join 10.0.0.5:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:1a2b3c...

Then comes the registering. The kubelet authenticates with the token, gets its own client cert issued (TLS bootstrapping), and uses that cert to create itself as a Node object on the API server. After that it just keeps refreshing "still alive, this much capacity left."

So the only thing installed on that machine is the kubelet, and joining was one line showing up in a list on the API server.

Leaving is symmetric, which I thought was neat. kubectl delete node erases that line and the machine keeps running fine. If the kubelet is still alive it registers itself again a bit later. Erasing a name and removing a thing are different.

A master node is a role, not a kind

Asked one more thing. I'd been assuming there was a separate master node. Wondered if a "cluster node" was its own kind of thing.

There isn't one. A control plane node runs a kubelet and registered the same way as above. Just a node. Two differences only.

• Different processes run on it. kube-apiserver, etcd, kube-scheduler, kube-controller-manager run there too. Though usually even those are up as Pods.
• It carries one marker. A taint, node-role.kubernetes.io/control-plane:NoSchedule. A rule string handed to the scheduler saying "don't put ordinary apps here," not a physical constraint.

Not a physical constraint means take it off and apps run on that machine too. A single-node cluster made with kind apparently drops that taint. With one node, not being able to place apps would leave you with nothing.

master is a retired name and it's called a control plane node now. Doesn't have to be one machine either — production runs three or more because of etcd quorum, though I haven't done that so I don't really know.

Part 1, wrapped up

• A container isn't a VM, it's a process with its view blocked. Dies and stays dead.
• There are four moments where one server isn't enough, and Kubernetes takes the place of the person who used to be stuck on them.
• node is one machine, Pod is a shell around a container, controller is a loop closing the gap between desired and current state.
• Pod is disposable, so snapshotting one and restoring it doesn't work. What a backup preserves is the description, not the Pod.

And the things I had wrong.

The bottom three were all the same misconception. Names, not things, and I took them for things. Feels like I'll trip on the same spot when I get to namespaces and Services.

Not like it's all sorted yet. But separate from being confused, I think I'm getting a feel for it. When a new word shows up, first thing to ask is whether it's a process or a name.

Next part is the API server.