Adding New Nodes to an Existing Kubernetes Cluster

One of the common "day two" tasks for a Kubernetes deployment is to scale the cluster capacity by adding new nodes. Since the way that you scale your cluster depends on the tools that were used initially during the cluster bootstrapping, there are a couple of ways to provision new machines. This article shows the steps that we follow at Oak-Tree to add nodes to an existing cluster.

We deploy our Kubernetes clusters on top of Ubuntu 20.04 using Ansible and kubeadm.

• Ansible is an automation framework that makes it easy to create a readable description of state in a file called a playbook. The framework then reads your playbook and makes changes to target systems so that they match the desired configuration. This is an approach to infrastructure management called "Infrastructure as Code."
• kubeadm is a tool built to make it is easy to initialize and join new machines to a Kubernetes cluster.

Using Ansible and kubeadm together, it is possible for Oak-Tree to provision and join new machines to an existing cluster quickly, usually in minutes. Here are the steps we follow:

1. Use ssh-copy-id to install the master SSH key for the cluster administrative user to the new machine's $HOME/.ssh/authorized_keys file. This facilitates automated, passwordless logins and single sign-on using the SSH protocol. ssh-copy-id is part of the OpenSSH utilities installed with the openssh-server system package on Ubuntu.
2. If using a machine as a gateway to handle load balancing for the cluster, update the firewall rules for the new node so that it can communicate with the gateway.
3. Install dependencies and set the initial state of the new node.
4. Install kubeadm, kubelet, and kubectl.
5. Create a join command using kubeadm and run the join command of each of the new nodes that need to be added.

In the remainder of this article, we'll look at steps three, four, and five in more detail.

Step 3: Install Dependencies and Set Initial State

To ensure that our environment is well defined as possible, Oak-Tree has created a set of Ansible playbooks to describe the dependencies and components required by our preferred configuration of Kubernetes. These playbooks allow us to declare the packages and other dependencies we need to run Kubernetes, and (when needed) install the dependencies for GPU accelerated machine learning and CUDA.

The playbooks are first run when we setup the cluster, and must be executed for new nodes before we can use kubeadm to join the new machines. To do that:

1. Create a special file called the "inventory," which we call nodes by convention. This file includes a list of all the masters and workers in our cluster, and is the set of machines for which instructions in the playbook will run.
2. To the nodes inventory file, include a list of the new machines.
3. Then run the kube-deps.yaml playbook.

Step 4: Install kubeadm , kubectl, and kubelet

Once dependencies have been installed, the new nodes need to have kubeadm, kubectl, and kubelet installed. These three applications provide the system daemon and administrative tools required to join the system to the cluster.

The method of installation will depend on the version needed and the operating system version, and may be installed as a Debian package or via Snap. The Ubuntu community maintains current and a number older versions all three tools. Refer to the documentation for "Canonical Kubernetes" for details. More recent versions of the tools are only available as Snap packages.

Step 5: Join the New Nodes

When the kube-deps.yaml playbook finishes executing, the next step is to create a join command and execute that command for each of the new machines that needs to be added. Note: the kubeadm command will be generated on one of the current master nodes and executed on the new machine.

Worker Nodes

To create the command for worker nodes, use kubeadm token create:

This will return the command that needs to be executed on the new worker node.

Master Nodes

The process for adding master nodes is slightly more involved. In order for the machine to configure itself as part of the control plane, it needs a special certificate and key that is passed to the kubeadm token create using the --certificate-key option. To that end, joining a new master involves three steps: create the certificate-key, create the join command, join the new node.

The command in the listing below below will generate the certificate-key and output the value to the terminal:

Grab the output of the key generation command and include it in the kubeadm token create command:

As with the worker node, this will generate a command that needs to be run on the new master.