Kubernetes - Kubeadm - Cluster Setup

Prerequisites:

2 GB RAM
2 CPUs
Swap disabled
unique MAC and product_uuid
bridged traffic is enabled
Open required ports
install a container runtime ( Docker, CRI-O, or another )
remove swap

Make sure you have a unique MAC and product_uuid:


ip link
/sys/class/dmi/id/product_uuid

Enable bridged traffic:


cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

Also with IPv6:


cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

Load settings:


sudo sysctl --system

Remove Swap

Disabling / removing swap space is absolutely required.

Required:


sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

Optional:


sudo rm /swap.img

Install Kubeadm / Kubelet / Kubectl

Install these:


sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

Download key:


sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Update repo:


echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install and pin versions:


sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Cgroup Driver

cgroup driver options:

cgroupfs
systemd - we want this one ( default in v1.22 )

Check kublet cgroup driver????????

Check system cgroup version ( v1 or v2 ):


grep cgroup /proc/filesystems

If you see cgroup2 in the output it is available:


nodev   cgroup
nodev   cgroup2

Kubernetes Cgroup Driver

Explicitly set cgroup driver to systemd with minimal config:


# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.21.0
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

Use this config:


kubeadm init --config kubeadm-config.yaml

Docker Cgroup Driver

Check the Docker service and see which driver it is useing:

 
systemctl status docker

Edit the service config and add options to the ExecStart linke:


sudo vi /lib/systemd/system/docker.service

Replace this line:


ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

With this:


ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

Restart Docker:


systemctl daemon-reload
systemctl restart docker

Verify the cgroup driver in use by Docker:


docker info|grep -i cgroup

Want to see this:


 Cgroup Driver: systemd
 Cgroup Version: 1

Not this:


 Cgroup Driver: cgroupfs
 Cgroup Version: 1

Fix Issues

Fix problems caused by Docker restarting:


sudo systemctl restart docker.service
sudo kubeadm reset

Setup

Use the –control-plane-endpoint option if you want HA later.

Your first server, later a VIP for your cluster:


sudo su -
echo 192.168.3.214 cluster-endpoint >> /etc/hosts

Initialize the control plane:


sudo su -
sudo kubeadm init --control-plane-endpoint=cluster-endpoint

You could alternatively use this but it doesn’t seem necessary:


sudo kubeadm init --control-plane-endpoint=cluster-endpoint --pod-network-cidr=192.168.123.0/16

As a regular user:


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Two of many network options:

Flannel
Calico

You will generally use a command that looks like this to setup your pod network:


kubectl apply -f podnetwork.yaml

I choose to use Flannel and this is what I used:


sudo kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Check that CoreDNS is working. This means that networking is working.


kubectl get pods --all-namespaces

Output should look like this:


user1@swan1:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   coredns-64897985d-ppfwj         1/1     Running   0          24m
kube-system   coredns-64897985d-txh7j         1/1     Running   0          24m
kube-system   etcd-swan1                      1/1     Running   1          25m
kube-system   kube-apiserver-swan1            1/1     Running   1          25m
kube-system   kube-controller-manager-swan1   1/1     Running   1          25m
kube-system   kube-flannel-ds-gzqmq           1/1     Running   0          71s
kube-system   kube-proxy-dm4ln                1/1     Running   0          24m
kube-system   kube-scheduler-swan1            1/1     Running   1          25m

SKIP THIS ( probably ) - Turn off control plane isolation so that you can run pods on the same node as the control plane.


kubectl taint nodes --all node-role.kubernetes.io/master-

Worker Node

Get the token from the control plane host if you don’t have it:


kubeadm token list

Create a new token if it is expired ( after 24 hours )


kubeadm token create

Get the cert hash if you don’t have it:


openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'

On a worker node:

Install a runtime ( Docker )
Install kubeadm ( from install section above )
Join the cluster


sudo su - 
kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Watch control plane pods start up:


kubectl get pod -n kube-system -w

In case you need to re-upload the certs after 2 hour expiration:


sudo kubeadm init phase upload-certs --upload-certs

Join other control plane nodes:


sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07

Join workers nodes:


sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866

See doc for external etcd setup or manual cert distribution:

Doc 1
Doc 2

Other

Rebalance CoreDNS after another control-plane node has joined:


kubectl -n kube-system rollout restart deployment coredns

Get Kubectl on client host:


scp root@<control-plane-host>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes

Access API server here: http://localhost:8001/api/v1

Deprovision cluster:

Drain and reset:


kubectl drain <node name> --delete-emptydir-data --force --ignore-daemonsets
kubeadm reset

Clean IPTables:


iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm -C

Delete node and reset:


kubectl delete node <node name>
sudo kubeadm reset

HA Cluster

HA Cluster:

minimum of three control planes
Stacked etcd topology
- stacking etcd with other control plane components
- default
- more simple
External etcd topology
- double the hosts
- less risk

Steps:

install container runtime
install packages mentioned above
Create Loadbalancer VIP for kube-apiserver
initialize

initilaize:


sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" --upload-certs

–upload-certs

will upload certs for other nodes in the server so that you don’t need to do it manually

Dashboard


kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.0/aio/deploy/recommended.yaml

kubectl proxy

Access the proxy like this: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/

Checking Kublet:


systemctl status kubelet
journalctl -xeu kubelet

Not working:


kubectl port-forward service/mongo 28015:27017

References

Official Documentation