Logo Wael's Digital Garden

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster#

Talos runs great on ProxmoxVE. Make sure to add the system extension siderolabs/qemu-guest-agent when producing the Talos - Generate ISO and Installer with Custom Extensions via Image Factory. This enhances VM manageability from Proxmox (e.g., proper shutdowns, IP reporting in Proxmox UI).

The required steps for a new Kubernetes Cluster are:

  1. Generate an ISO and Download it on ProxmoxVE.
  2. Setup the Control Plane of Kubernetes.
  3. Create one Talos VM for each Talos Control Plane node.
    • Tip: Kubernetes recommends an odd number of nodes. Typically one node for development and three nodes for production for high availability.
  4.  
  5.  

Generate an ISO and Download it on ProxmoxVE#

  1. Generate an ISO by following Talos - Generate ISO and Installer with Custom Extensions via Image Factory.
  2. Tip: Pin the Talos version in your image factory command for reproducible builds.
  3. Download the ISO on ProxmoxVE

Create Talos VM(s)#

  1. Create a new VM, and be sure to follow the Talos Linux - System Requirements.
  2. Use advanced configuration and disable memory ballooning, that will not be useful with Kubernetes and could in fact be harmful.
  3. Choose OVMF over SeaBIOS as it loads a nicer console font by default. See Proxmox VM - UEFI vs SeaBIOS Consideration.
  4. CPU Type: Set CPU type to host in Proxmox VM options for best performance, unless live migration to different CPU hosts is a strict requirement.
  5. Network Adapter: Use virtio (paravirtualized) for the network device model for best performance.
  6. Resource Allocation: Plan CPU cores and RAM for each VM based on expected load. For control planes: typically 2+ cores, 4GB+ RAM. Workers depend on workload.
  7. (Optional) VM Template: After setting up and testing one worker VM (before applying Talos config), consider converting it to a Proxmox template for faster worker node deployment. You'd still need to boot with the ISO and apply the Talos config.
  8. Gather the NIC Mac Address from the VM Hardware page, this is needed to setup a static IP on the router.
  9. Add the node to the router with fixed IP following UniFi - Add DHCP Reservation for MAC.
  10. Start the VM
  11. Follow the heading below for configuring the cluster.

Configure the cluster#

  1. Write the controlplane-patch.yaml and worker-patch.yaml with the contents below.
  2. Generate the secrets with talosctl gen secrets -o secrets.yaml
  3. Security: Store secrets.yaml securely. It contains sensitive PKI data. Consider encrypting it at rest (e.g., using GPG, HashiCorp Vault, or similar). Back it up to a secure location.
  4. Generate the configuration with the command talosctl gen config --with-secrets secrets.yaml --config-patch-control-plane @controlplane-patch.yaml --config-patch-worker @worker-patch.yaml <cluster name> https://<IP_of_first_node>:6443 --output-dir . --install-image <url-to-installer-image-from-factory>
  5.  
    • Note: <IP_of_first_node> in the URL for talosctl gen config is used as the initial endpoint for the generated talosconfig and as one of the default cert SANs. Your controlplane-patch.yaml correctly overrides/supplements apiServer.certSANs and defines controlPlane.endpoint to your VIP (<VIP>), which is good practice.
  6. The command generates controlplane.yaml, worker.yaml, and talosconfig. The .yaml files are the machine configs for your nodes.
  7. For each control plane node, apply the config with the command and wait for the node to become ready talosctl apply-config --insecure --file controlplane.yaml --nodes <IP>
  8. First node should start saying "waiting for talosctl bootstrap"
  9. Other nodes are ready as soon you see the number of machines increment on the first node.
  10. Worker plane nodes can be applied all at once with the command talosctl apply-config --insecure --file worker.yaml --nodes <IP> --nodes <IP>
  11. Configure talosconfig with the IP of the first node. Do not add other nodes yet (see Talos - Bootstrap - x509 Unknown Authority)
  12. talosctl config endpoint <IP_of_one_control_plane_node>
  13. talosctl config node <IP_of_one_control_plane_node>
  14. Either copy talosconfig to ~/.talos/config or export TALOSCONFIG environment variable to point to its location.
  15. Security: The talosconfig file contains client certificates that grant administrative access to your cluster. Protect it accordingly.
  16. Bootstrap the cluster with talosctl bootstrap
  17. Wait for talosctl health -n <IP> for each IP to report healthy.
  18. Generate the proper kube config with the command talosctl kubeconfig. This command merges with ~/.kube/config if one is found and automatically sets the context to the cluster so it's ready to use.
  19. Important: After bootstrap and verifying health, update your talosconfig to include all control plane node endpoints for high availability when using talosctl: talosctl config endpoint <CP1_IP>,<CP2_IP>,<CP3_IP> --nodes <CP1_IP>,<CP2_IP>,<CP3_IP> (Or edit the talosconfig YAML file directly to update contexts.<your-cluster>.endpoints and contexts.<your-cluster>.nodes). This ensures talosctl can reach the Talos API even if one control plane node is down.
  20. Run kubectl get node -o wide to confirm things are working fine.
  21. Ensure no node is tainted as uninitialized, use the following command: kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINTS:.spec.taints
  22. Talos Linux - Cloud Controller Manager (or CCM)

controlplane-patch.yaml

cluster:
  apiServer:
    certSANs:
      - <IP of node 1> # CP-01
      - <IP of node 2> # CP-02
      - <IP of node 3> # CP-03
      - <VIP>          # VIP
      - <DNS>          # VIP
  controlPlane:
    endpoint: https://<DNS>:6443
machine:
  certSANs:
    - <IP of node 1> # CP-01
    - <IP of node 2> # CP-02
    - <IP of node 3> # CP-03
    - <DNS for IP of node 1>
    - <DNS for IP of node 2>
    - <DNS for IP of node 3>
    - <VIP>          # VIP
    - <DNS>          # VIP
  features:
    kubernetesTalosAPIAccess:
      enabled: true
      allowedRoles:
        - os:reader
      allowedKubernetesNamespaces:
        - kube-system
  kubelet:
    # TODO: Adjust this to install CCM.
    extraArgs: {}
      #cloud-provider: external
      #rotate-server-certificates: true
  network:
    interfaces:
      - deviceSelector:
          physical: true
        dhcp: true
        vip:
          ip: <VIP>
  time:
    disabled: false
    servers:
      - time.cloudflare.com
      - pool.ntp.org

worker-patch.yaml

machine:
  kubelet:
    # TODO: Adjust this to install CCM.
    extraArgs: {}
      #cloud-provider: external
      #rotate-server-certificates: true
  time:
    disabled: false
    servers:
      - time.cloudflare.com
      - pool.ntp.org
  network:
    interfaces:
      - deviceSelector:
          physical: true
        dhcp: true

Related#