Using k3d on Ubuntu 24.04 with rootless Podman

Using k3d on Ubuntu 24.04 with rootless Podman

k3d creates Kubernetes clusters using k3s in Docker containers. To use k3d with rootless Podman on Ubuntu 24.04, you must configure cgroup delegation, increase system file watcher limits, and enable the Podman socket for Docker API compatibility.

Prerequisites

Ensure the following are installed:

• Podman configured for rootless operation
• k3d (install via curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash)
• kubectl (for interacting with the cluster)

Step 1: Enable cgroup CPU, CPUSET, and I/O delegation

Rootless Podman requires cgroup delegation to manage resources in user services.

Create /etc/systemd/system/user@.service.d/delegate.conf (requires sudo) with:

[Service]
Delegate=cpu cpuset io memory pids

After creating the file, reload the systemd configuration:

sudo systemctl daemon-reload

Note: You must log out and log back in (or reboot) for these changes to take effect for your user session.

Step 2: Enable Podman socket

Enable the Podman socket to provide a Docker-compatible API for k3d.

Run:

systemctl --user enable --now podman.socket

Step 3: Export socket locations

Set environment variables to direct k3d to the Podman socket. DOCKER_HOST is used by the k3d CLI to talk to the Podman socket, while DOCKER_SOCK is often used by scripts or tools that expect a direct path to the socket file.

Export:

export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"
export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock

To set these variables automatically at login, add the export commands to your shell profile (e.g., ~/.bashrc or ~/.zshrc).

Step 4: Increase inotify resource limits

When running multiple Kubernetes nodes (like a server and multiple agents) as rootless containers on a single host, they aggressively consume the kernel’s file watcher (inotify) resources.

If you leave the default limits in place, the Kubelet’s cAdvisor will quickly hit a ceiling and silently crash (inotify_init: too many open files). Because the agents fail to initialize, k3d will hang indefinitely while waiting for the cluster to report a “Ready” status.

To fix this permanently, increase the limits by creating /etc/sysctl.d/99-k3d.conf (requires sudo):

fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288

Apply the new limits immediately without rebooting:

sudo sysctl -p /etc/sysctl.d/99-k3d.conf

Step 5: Create the k3d cluster

Create the cluster with user namespace support for rootless Podman. The feature gates ensure the Kubelet handles the User Namespace environment correctly without crashing due to missing host-level privileges.

Run:

k3d cluster create --servers 1 --agents 3 --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@server:*' --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@agent:*'

Verification

Verify the cluster:

k3d cluster list
kubectl get nodes

Troubleshooting

AppArmor and User Namespaces

Ubuntu 24.04 restricts unprivileged user namespaces, which can prevent rootless Podman from creating containers. If you encounter errors related to user namespaces, you may need to adjust the AppArmor configuration.

A temporary workaround is to disable the restriction:

sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

For a permanent fix, ensure you are using a version of Podman that includes the necessary AppArmor profiles, or create a profile that allows userns.

Cgroup Version

Ensure your system is using cgroup v2, which is the default on Ubuntu 24.04. You can verify this with:

podman info | grep "cgroup version"

It should output cgroup version: v2.