OpenStack Cluster Bootstrap

Bootstrap the baremetal OpenStack cluster — three dedicated servers (lucy, makise, quinn) running NixOS, kubeadm, and kube-vip. This cluster hosts the OpenStack control plane.

INFO

This is a manual bootstrap on baremetal hardware. There is no Cluster API involvement — the cluster is self-contained and not managed by any external orchestrator.

For the CAPI-managed management cluster, see Management Cluster Bootstrap.

┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│ 1. Build ISO     │───>│ 2. Install NixOS │───>│ 3. Bootstrap K8s │───>│ 4. Install Flux  │
│    (Hephaestus)  │    │    (3 nodes)     │    │    (kubeadm)     │    │    → Yaook/OpenStack │
└──────────────────┘    └──────────────────┘    └──────────────────┘    └──────────────────┘

Prerequisites

• Access to the Hephaestus repository
• USB drive for ISO installation
• SSH access to lucy after OS install
• Root privileges (sudo) on all nodes

---

Step 1: Build the NixOS ISO

git clone git@github.com:RPCU/hephaestus.git
cd hephaestus
devenv shell
build-iso --argstr partition root --argstr cloud false

This produces a bootable ISO in ./output/ with the root partition layout:

Partition	Size	Purpose
ESP	512M	EFI boot
/var	200G	Data
/nix	200G	Nix store
/	100G	Root LV

TIP

Other partition layouts are available in installer/partitions/: root-grub (BIOS), small, test.

Step 2: Install NixOS on All Nodes

Repeat for each node (lucy, makise, quinn):

1. Write ISO to USB:

   dd if=./output/*.iso of=/dev/sdX bs=4M status=progress

2. Boot the node from USB (UEFI mode required)

3. The ISO auto-logs in as nixos and runs the installer:

   • Detects the first unused disk
   • Wipes disk signatures (wipefs)
   • Partitions and formats (disko)
   • Runs nixos-install
   • Reboots

4. After first boot, ginx runs automatically:

   • Pulls the latest hephaestus repo
   • Applies colmena apply-local --sudo
   • Reboots into the final configuration

Step 3: Verify OS Installation

SSH into each node and confirm NixOS is running:

ssh user@lucy    && sudo hostnamectl   # lucy
ssh user@makise  && sudo hostnamectl   # makise
ssh user@quinn   && sudo hostnamectl   # quinn

The nodes auto-update via the ginx agent every 60 seconds.

---

Step 4: Bootstrap Kubernetes (kubeadm)

4.1 Initialize the first control plane (lucy)

ssh user@lucy
sudo initKubeadm

The initKubeadm script (pre-generated by NixOS at /etc/kubernetes/kubeadm/bootstrap.yaml):

1. Deploys kube-vip static pods (VIP 10.0.0.5)
2. Runs kubeadm init with the pre-generated config
3. Sets up kubectl for root and the calling user
4. Outputs the join command — copy this for the next step

4.2 Join the remaining control plane nodes

Run the join command from step 4.1 on makise and quinn:

# On makise
ssh user@makise
sudo joinCPKubeadm <TOKEN> <CERT_KEY>

# On quinn
ssh user@quinn
sudo joinCPKubeadm <TOKEN> <CERT_KEY>

4.3 Verify the cluster

kubectl get nodes

Expected — 3 nodes in Ready:

NAME     STATUS   ROLES           AGE   VERSION
lucy     Ready    control-plane   ...   v1.35.4
makise   Ready    control-plane   ...   v1.35.4
quinn    Ready    control-plane   ...   v1.35.4

---

Step 5: Install Cilium (CNI)

Cilium replaces kube-proxy (skipped during kubeadm init) and provides eBPF networking + L2 LoadBalancer.

helm repo add cilium https://helm.cilium.io/
helm repo update
helm upgrade --install cilium cilium/cilium -n kube-system \
  --create-namespace \
  -f ./infrastructure/cilium/values.yaml \
  --version 1.18.6

Wait for rollout:

kubectl -n kube-system rollout status daemonset/cilium --timeout=300s

WARNING

socketLB.hostNamespaceOnly: true is set for this cluster — required for nested KVM/QEMU VMs to function correctly.

---

Step 6: Install Flux (GitOps)

6.1 Flux Operator

kustomize build infrastructure/fluxcd/operator/ | kubectl apply -f -
kubectl wait --for=condition=Available deployment/flux-operator -n flux-system --timeout=180s

6.2 Flux Instance

kustomize build clusters/openstack/fluxcd/ | kubectl apply -f -
kubectl wait --for=condition=Ready fluxinstance/flux -n flux-system --timeout=180s

Flux now watches clusters/openstack/ in the Argus repository.

---

Step 7: Verify Full Reconciliation

kubectl get kustomization -n flux-system

Reconciliation order:

flux-operator → fluxcd
├─> cilium
├─> cert-manager → cert-manager-issuer
├─> trust-manager
├─> gateway-api → kgateway-crds → kgateway
├─> crossplane
├─> external-secrets
├─> rook (setup → configs)
└─> yaook-operator (CRDs → operators) → yaook (OpenStack services)

Monitor:

kubectl get kustomization -n flux-system -w
kubectl get helmrelease -A -w

Once complete, the full OpenStack control plane is running with services at *.rpcu.vpn.

---

Troubleshooting

NixOS ISO boots to login but doesn't install

Run manually: sudo installer

Node doesn't come up after install

systemctl status ginx
sudo osupdate   # force a one-shot update

Cilium pods not starting

Verify kube-proxy was skipped during kubeadm init:

kubectl -n kube-system get cm kube-proxy -o yaml | grep -A2 mode
# Should show "mode: "" or be absent