Low Orbit Flux Logo 2 D

Kubernetes - Kubeadm - Cluster Setup


Make sure you have a unique MAC and product_uuid:

ip link

Enable bridged traffic:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

Also with IPv6:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

Load settings:

sudo sysctl --system

Remove Swap

Disabling / removing swap space is absolutely required.


sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab


sudo rm /swap.img

Install Kubeadm / Kubelet / Kubectl

Install these:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

Download key:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Update repo:

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install and pin versions:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Cgroup Driver

cgroup driver options:

Check kublet cgroup driver????????

Check system cgroup version ( v1 or v2 ):

grep cgroup /proc/filesystems

If you see cgroup2 in the output it is available:

nodev   cgroup
nodev   cgroup2

Kubernetes Cgroup Driver

Explicitly set cgroup driver to systemd with minimal config:

# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.21.0
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Use this config:

kubeadm init --config kubeadm-config.yaml

Docker Cgroup Driver

Check the Docker service and see which driver it is useing:

systemctl status docker

Edit the service config and add options to the ExecStart linke:

sudo vi /lib/systemd/system/docker.service

Replace this line:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 

With this:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

Restart Docker:

systemctl daemon-reload
systemctl restart docker

Verify the cgroup driver in use by Docker:

docker info|grep -i cgroup

Want to see this:

 Cgroup Driver: systemd
 Cgroup Version: 1

Not this:

 Cgroup Driver: cgroupfs
 Cgroup Version: 1

Fix Issues

Fix problems caused by Docker restarting:

sudo systemctl restart docker.service
sudo kubeadm reset


Your first server, later a VIP for your cluster:

sudo su -
echo cluster-endpoint >> /etc/hosts

Initialize the control plane:

sudo su -
sudo kubeadm init --control-plane-endpoint=cluster-endpoint 

You could alternatively use this but it doesn’t seem necessary:

sudo kubeadm init --control-plane-endpoint=cluster-endpoint --pod-network-cidr=

As a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Two of many network options:

You will generally use a command that looks like this to setup your pod network:

kubectl apply -f podnetwork.yaml

I choose to use Flannel and this is what I used:

sudo kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Check that CoreDNS is working. This means that networking is working.

kubectl get pods --all-namespaces

Output should look like this:

user1@swan1:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   coredns-64897985d-ppfwj         1/1     Running   0          24m
kube-system   coredns-64897985d-txh7j         1/1     Running   0          24m
kube-system   etcd-swan1                      1/1     Running   1          25m
kube-system   kube-apiserver-swan1            1/1     Running   1          25m
kube-system   kube-controller-manager-swan1   1/1     Running   1          25m
kube-system   kube-flannel-ds-gzqmq           1/1     Running   0          71s
kube-system   kube-proxy-dm4ln                1/1     Running   0          24m
kube-system   kube-scheduler-swan1            1/1     Running   1          25m

SKIP THIS ( probably ) - Turn off control plane isolation so that you can run pods on the same node as the control plane.

kubectl taint nodes --all node-role.kubernetes.io/master-

Worker Node

Get the token from the control plane host if you don’t have it:

kubeadm token list

Create a new token if it is expired ( after 24 hours )

kubeadm token create

Get the cert hash if you don’t have it:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'

On a worker node:

sudo su - 
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Watch control plane pods start up:

kubectl get pod -n kube-system -w

In case you need to re-upload the certs after 2 hour expiration:

sudo kubeadm init phase upload-certs --upload-certs

Join other control plane nodes:

sudo kubeadm join --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07

Join workers nodes:

sudo kubeadm join --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

See doc for external etcd setup or manual cert distribution:


Rebalance CoreDNS after another control-plane node has joined:

kubectl -n kube-system rollout restart deployment coredns 

Get Kubectl on client host:

scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes

Access API server here: http://localhost:8001/api/v1

Deprovision cluster:

Drain and reset:

kubectl drain <node name> --delete-emptydir-data --force --ignore-daemonsets
kubeadm reset

Clean IPTables:

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C

Delete node and reset:

kubectl delete node <node name>
sudo kubeadm reset

HA Cluster

HA Cluster:



sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" --upload-certs

–upload-certs will upload certs for other nodes in the server so that you don’t need to do it manually


kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

kubectl proxy  

Access the proxy like this: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/


Checking Kublet:

systemctl status kubelet
journalctl -xeu kubelet

Not working:

kubectl port-forward service/mongo 28015:27017