Module 06 — Provision the 5-VM Cluster

The Docker section ran everything on a single VM. Kubernetes is a distributed system — it needs multiple machines. In this module you provision 5 Ubuntu VMs using Vagrant and VirtualBox, configure host-only networking with static IPs, set up SSH access between all nodes, and verify full mesh connectivity.

By the end you will have this network:

VM	Hostname	IP	vCPUs	RAM	Role
Load Balancer	lb	192.168.56.20	1	512 MB	HAProxy (API load balancer)
Control Plane 1	cp1	192.168.56.21	2	2 GB	etcd, kube-apiserver, scheduler, controller-manager
Control Plane 2	cp2	192.168.56.22	2	2 GB	etcd, kube-apiserver, scheduler, controller-manager
Worker 1	worker1	192.168.56.23	2	2 GB	containerd, kubelet, kube-proxy
Worker 2	worker2	192.168.56.24	2	2 GB	containerd, kubelet, kube-proxy

Total host resources: 9 vCPUs, 8.5 GB RAM, ~90 GB disk. Make sure your machine can spare this alongside the 3 Fundamentals VMs (which can be powered off during this track).

---

1. Why This Layout

Two control planes, not one

A single control plane is a single point of failure. With two, etcd maintains quorum and the API server stays available if one node goes down. HAProxy on the lb VM distributes API requests across both.

Two workers

Two workers let you test pod scheduling, anti-affinity rules, and rolling updates across nodes. One worker cannot demonstrate any of that.

Separate load balancer

The lb VM is lightweight (512 MB). It runs only HAProxy — keeping it separate from the control plane means you can restart HAProxy without affecting etcd or the API server.

Network layout

All 5 VMs share the 192.168.56.0/24 host-only network. They also have NAT adapters for internet access (downloading binaries, pulling images). The registry from Module 05 is reachable at 192.168.56.12:5000 on this same network.

Host Machine (your Mac)
│
├── vboxnet0 (192.168.56.0/24)
│   ├── lb        .20
│   ├── cp1       .21
│   ├── cp2       .22
│   ├── worker1   .23
│   ├── worker2   .24
│   │
│   ├── db-server  .11  (Fundamentals — can be off)
│   ├── app-server .12  (registry at :5000)
│   └── web-server .13  (Fundamentals — can be off)
│
└── NAT (internet access for each VM)

---

2. Prerequisites

Vagrant

Vagrant automates VM provisioning through a declarative Vagrantfile. Install it if you haven't already:

macOS (Homebrew):

brew install --cask vagrant

Linux (apt):

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vagrant

Verify:

vagrant --version

Expected: Vagrant 2.x.x

VirtualBox host-only network

If you completed the Fundamentals track, vboxnet0 already exists on 192.168.56.1. Verify:

VBoxManage list hostonlyifs | grep -A 3 "vboxnet0"

If it does not exist, create it:

VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.56.1 --netmask 255.255.255.0
VBoxManage dhcpserver remove --ifname vboxnet0 2>/dev/null || true

Checkpoint: vagrant --version returns 2.x. VBoxManage list hostonlyifs shows vboxnet0 at 192.168.56.1.

---

3. Create the Vagrantfile

Create a directory for the Kubernetes cluster configuration:

mkdir -p ~/k8s-cluster
cd ~/k8s-cluster

~/k8s-cluster/Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# ── VM Definitions ────────────────────────────────
NODES = [
  { name: "lb",      ip: "192.168.56.20", cpus: 1, memory: 512  },
  { name: "cp1",     ip: "192.168.56.21", cpus: 2, memory: 2048 },
  { name: "cp2",     ip: "192.168.56.22", cpus: 2, memory: 2048 },
  { name: "worker1", ip: "192.168.56.23", cpus: 2, memory: 2048 },
  { name: "worker2", ip: "192.168.56.24", cpus: 2, memory: 2048 },
]

# ── /etc/hosts entries for all nodes ──────────────
HOSTS_ENTRIES = NODES.map { |n| "#{n[:ip]} #{n[:name]}" }.join("\n")

# ── Shared provisioning script ────────────────────
$provision_script = <<-'SCRIPT'
#!/bin/bash
set -euo pipefail

# Add all cluster nodes to /etc/hosts
cat >> /etc/hosts <<EOF

# Kubernetes cluster nodes
HOSTS_PLACEHOLDER
# Fundamentals track (registry)
192.168.56.12 app-server
EOF

# Enable password authentication for SSH (for inter-node access)
sed -i 's/^#\?PasswordAuthentication .*/PasswordAuthentication yes/' /etc/ssh/sshd_config
# Also handle the sshd_config.d drop-in that Ubuntu 22.04+ may use
find /etc/ssh/sshd_config.d/ -name '*.conf' -exec \
  sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' {} + 2>/dev/null || true
systemctl restart ssh

# Disable swap (Kubernetes requires this)
swapoff -a
sed -i '/\sswap\s/d' /etc/fstab

# Load required kernel modules
cat > /etc/modules-load.d/k8s.conf <<EOF2
overlay
br_netfilter
EOF2
modprobe overlay
modprobe br_netfilter

# Set required sysctl parameters (persist across reboots)
cat > /etc/sysctl.d/k8s.conf <<EOF3
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF3
sysctl --system > /dev/null 2>&1

# Install common packages
apt-get update -qq
apt-get install -y -qq \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg \
  socat \
  conntrack \
  ipset > /dev/null

echo "Provisioning complete for $(hostname)"
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"

  NODES.each do |node|
    config.vm.define node[:name] do |n|
      n.vm.hostname = node[:name]
      n.vm.network "private_network", ip: node[:ip]

      n.vm.provider "virtualbox" do |vb|
        vb.name   = node[:name]
        vb.cpus   = node[:cpus]
        vb.memory = node[:memory]
        vb.linked_clone = true
      end

      # Run the provisioning script with hosts entries injected
      n.vm.provision "shell",
        inline: $provision_script.gsub("HOSTS_PLACEHOLDER", HOSTS_ENTRIES)
    end
  end
end

What the Vagrantfile does

1. Defines 5 VMs from the NODES array — name, IP, CPU, and memory for each
2. Uses ubuntu/jammy64 — Ubuntu 22.04 LTS base box (downloaded automatically on first run)
3. Host-only networking — private_network assigns the static IP on vboxnet0
4. Linked clones — vb.linked_clone = true creates VMs faster by sharing the base disk image
5. Shell provisioner — runs on each VM after creation to:
   • Add all node hostnames to /etc/hosts
   • Enable SSH password authentication
   • Disable swap (Kubernetes requirement)
   • Load kernel modules (overlay, br_netfilter) for container networking
   • Set sysctl parameters for iptables bridge traffic and IP forwarding
   • Install packages needed by Kubernetes components

Checkpoint: cat ~/k8s-cluster/Vagrantfile shows all 5 VM definitions.

---

4. Provision the VMs

From the ~/k8s-cluster directory:

cd ~/k8s-cluster
vagrant up

This takes 10–15 minutes on the first run. Vagrant:

1. Downloads the ubuntu/jammy64 box (once, then cached)
2. Creates 5 VMs in VirtualBox
3. Configures networking on each
4. Runs the provisioning script on each

Watch the output — each VM shows its provisioning progress. When complete:

==> lb: Provisioning complete for lb
==> cp1: Provisioning complete for cp1
==> cp2: Provisioning complete for cp2
==> worker1: Provisioning complete for worker1
==> worker2: Provisioning complete for worker2

Verify VMs are running

vagrant status

Expected output:

Current machine states:

lb                        running (virtualbox)
cp1                       running (virtualbox)
cp2                       running (virtualbox)
worker1                   running (virtualbox)
worker2                   running (virtualbox)

All five should show running.

Checkpoint: vagrant status shows all 5 VMs running.

---

5. Access the VMs

Vagrant provides SSH access to each VM by name:

vagrant ssh lb

vagrant ssh cp1

Inside a VM, verify the hostname and IP:

hostname
ip addr show enp0s8 | grep "inet "

Type exit to return to your host.

Configure SSH from your Mac

For convenience outside of vagrant ssh, add the VMs to your SSH config. From your Mac:

cd ~/k8s-cluster

cat >> ~/.ssh/config <<'EOF'

# Kubernetes cluster VMs
Host lb
  HostName 192.168.56.20
  User vagrant
  IdentityFile ~/k8s-cluster/.vagrant/machines/lb/virtualbox/private_key

Host cp1
  HostName 192.168.56.21
  User vagrant
  IdentityFile ~/k8s-cluster/.vagrant/machines/cp1/virtualbox/private_key

Host cp2
  HostName 192.168.56.22
  User vagrant
  IdentityFile ~/k8s-cluster/.vagrant/machines/cp2/virtualbox/private_key

Host worker1
  HostName 192.168.56.23
  User vagrant
  IdentityFile ~/k8s-cluster/.vagrant/machines/worker1/virtualbox/private_key

Host worker2
  HostName 192.168.56.24
  User vagrant
  IdentityFile ~/k8s-cluster/.vagrant/machines/worker2/virtualbox/private_key
EOF

Now you can SSH directly:

ssh cp1

Checkpoint: ssh cp1 logs you into Control Plane 1 without a password prompt.

---

6. Verify /etc/hosts

The provisioner added all node hostnames to /etc/hosts on every VM. Verify on any node:

ssh cp1 "cat /etc/hosts"

You should see entries like:

192.168.56.20 lb
192.168.56.21 cp1
192.168.56.22 cp2
192.168.56.23 worker1
192.168.56.24 worker2
192.168.56.12 app-server

This means any node can resolve any other node by hostname. Test it:

ssh cp1 "ping -c 1 worker2"

PING worker2 (192.168.56.24) 56(84) bytes of data.
64 bytes from worker2 (192.168.56.24): icmp_seq=1 ttl=64 time=0.5 ms

Checkpoint: ssh cp1 "getent hosts worker2" returns 192.168.56.24 worker2.

---

7. Set Up SSH Key-Based Access Between Nodes

Kubernetes components on different nodes need to communicate, and you will frequently SSH between nodes for configuration. Generate a shared SSH key pair and distribute it to all nodes.

7.1 Generate a key pair on your Mac

ssh-keygen -t ed25519 -f ~/k8s-cluster/cluster-key -N "" -C "k8s-cluster"

This creates cluster-key (private) and cluster-key.pub (public) with no passphrase.

7.2 Distribute the public key to all nodes

for node in lb cp1 cp2 worker1 worker2; do
  ssh "$node" "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
  scp ~/k8s-cluster/cluster-key.pub "${node}:~/.ssh/"
  ssh "$node" "cat ~/.ssh/cluster-key.pub >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
done

7.3 Distribute the private key to all nodes

Each node needs the private key to SSH into the other nodes:

for node in lb cp1 cp2 worker1 worker2; do
  scp ~/k8s-cluster/cluster-key "${node}:~/.ssh/"
  ssh "$node" "chmod 600 ~/.ssh/cluster-key"
done

7.4 Configure SSH on each node to use the cluster key

for node in lb cp1 cp2 worker1 worker2; do
  ssh "$node" 'cat > ~/.ssh/config <<EOF
Host lb cp1 cp2 worker1 worker2
  User vagrant
  IdentityFile ~/.ssh/cluster-key
  StrictHostKeyChecking no
EOF
chmod 600 ~/.ssh/config'
done

7.5 Test inter-node SSH

From cp1, SSH to worker1:

ssh cp1 "ssh worker1 hostname"

Expected output:

worker1

Test a full mesh from your Mac:

for src in lb cp1 cp2 worker1 worker2; do
  for dst in lb cp1 cp2 worker1 worker2; do
    if [ "$src" != "$dst" ]; then
      result=$(ssh "$src" "ssh $dst hostname" 2>/dev/null)
      echo "$src -> $dst: $result"
    fi
  done
done

Every line should show the correct destination hostname.

Checkpoint: ssh cp1 "ssh worker1 hostname" returns worker1. All nodes can SSH to each other without passwords.

---

8. Verify Provisioning

Confirm the provisioning script set up each node correctly.

8.1 Swap is disabled

for node in lb cp1 cp2 worker1 worker2; do
  echo -n "$node swap: "
  ssh "$node" "swapon --show | wc -l"
done

Every node should show 0 (no swap partitions active).

8.2 Kernel modules are loaded

ssh cp1 "lsmod | grep -E 'overlay|br_netfilter'"

Expected output:

br_netfilter           ...
overlay                ...

8.3 Sysctl parameters are set

ssh cp1 "sysctl net.bridge.bridge-nf-call-iptables net.ipv4.ip_forward"

Expected output:

net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

8.4 Required packages are installed

ssh cp1 "which curl socat conntrack"

All three paths should be returned.

Checkpoint: All 5 VMs have swap disabled, kernel modules loaded, sysctl parameters set, and required packages installed.

---

9. Verify Full Mesh Connectivity

9.1 Ping every node from every node

for src in lb cp1 cp2 worker1 worker2; do
  echo "--- From $src ---"
  ssh "$src" 'for dst in lb cp1 cp2 worker1 worker2; do
    if [ "$(hostname)" != "$dst" ]; then
      ping -c 1 -W 1 "$dst" > /dev/null 2>&1 && echo "  -> $dst: OK" || echo "  -> $dst: FAIL"
    fi
  done'
done

Every line should show OK.

9.2 Verify internet access

for node in lb cp1 cp2 worker1 worker2; do
  echo -n "$node internet: "
  ssh "$node" "ping -c 1 -W 2 8.8.8.8 > /dev/null 2>&1 && echo OK || echo FAIL"
done

All should show OK.

9.3 Verify registry reachability

The private registry from Module 05 runs on app-server (192.168.56.12:5000). Verify the cluster VMs can reach it:

for node in cp1 worker1 worker2; do
  echo -n "$node -> registry: "
  ssh "$node" "curl -sk https://192.168.56.12:5000/v2/ -o /dev/null -w '%{http_code}' 2>/dev/null || echo UNREACHABLE"
done

If the registry is running with TLS, you should see 401 (unauthorized — which means the registry is reachable and TLS works; authentication is expected to fail without credentials). If the registry is not running, start it on app-server first.

Checkpoint: All nodes can ping each other, reach the internet, and connect to the registry on 192.168.56.12:5000.

---

10. Vagrant Lifecycle Commands

You will use these throughout the Kubernetes modules:

Command	What it does
vagrant up	Create and start all VMs (or start if already created)
vagrant halt	Gracefully shut down all VMs
vagrant halt cp1	Shut down a specific VM
vagrant up cp1	Start a specific VM
vagrant ssh cp1	SSH into a VM
vagrant status	Show VM states
vagrant provision	Re-run the provisioning script on all VMs
vagrant destroy -f	Delete all VMs and their disks (irreversible)

Saving resources

When you are done for the day, halt the VMs instead of leaving them running:

cd ~/k8s-cluster
vagrant halt

Resume the next day:

vagrant up

The VMs retain their state — no reprovisioning needed.

---

11. Troubleshooting

vagrant up fails with "VBoxManage: error: The host-only adapter"

The host-only network vboxnet0 does not exist or has the wrong IP. Create/fix it:

VBoxManage hostonlyif create
VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.56.1 --netmask 255.255.255.0

Then retry vagrant up.

VM stuck at "Waiting for machine to boot"

The VM is not getting an IP. Common causes:

1. VirtualBox kernel modules not loaded (macOS after update): sudo kextload -b org.virtualbox.kext.VBoxDrv or reinstall VirtualBox
2. Conflicting DHCP server: VBoxManage dhcpserver remove --ifname vboxnet0
3. Insufficient host resources: close other VMs or applications

"SSH authentication failed" during provisioning

Vagrant's default box credentials may have changed. Try:

vagrant destroy -f <vm-name>
vagrant up <vm-name>

Nodes cannot resolve each other by hostname

The /etc/hosts entries may be missing. Re-run provisioning:

vagrant provision

Or manually add the entries on the affected node:

ssh <node> "sudo tee -a /etc/hosts <<'EOF'
192.168.56.20 lb
192.168.56.21 cp1
192.168.56.22 cp2
192.168.56.23 worker1
192.168.56.24 worker2
EOF"

"Cannot allocate memory" when starting VMs

The 5 VMs need ~8.5 GB RAM total. Free up memory by:

1. Halting the Fundamentals VMs: VBoxManage controlvm db-server poweroff (repeat for app-server and web-server)
2. Closing memory-heavy applications on your Mac
3. Reducing the VM memory in the Vagrantfile (not recommended — control planes need 2 GB for etcd + API server)

---

12. What You Have Now

Capability	Verification Command
5 VMs running	vagrant status — all running
Static IPs on host-only network	ssh cp1 "ip addr show enp0s8 | grep inet"
Hostname resolution between nodes	ssh cp1 "ping -c 1 worker2"
SSH key-based access between nodes	ssh cp1 "ssh worker1 hostname"
Swap disabled	ssh cp1 "swapon --show | wc -l" — returns 0
Kernel modules loaded	ssh cp1 "lsmod | grep br_netfilter"
IP forwarding enabled	ssh cp1 "sysctl net.ipv4.ip_forward" — returns 1
Registry reachable from cluster	ssh worker1 "curl -sk https://192.168.56.12:5000/v2/"

The cluster infrastructure is ready. All 5 VMs can reach each other by hostname, have the kernel parameters Kubernetes needs, and can access the container registry on app-server.

---

Next up: Module 07 — Certificate Authority & TLS — create a Certificate Authority and generate TLS certificates for every Kubernetes component.