Skip to main content

Step isolation

Written by Aron Day

Every workflow step runs code, whether your team wrote it or an AI agent did. 3B runs each step in its own gVisor sandbox, which gives the step its own kernel instead of the node’s. A step can’t see other steps, the node’s files, or 3B’s credentials, and its network traffic goes only where your egress rules allow.

This page explains what that needs from your nodes, which setup to pick for your cluster, and what each one trades off.

Why 3B needs user namespaces

Starting a sandbox takes administrator powers inside Linux, such as mounting filesystems and setting up network interfaces. A common way to get those powers inside Kubernetes is a privileged pod, which gives the container root control of the node. The 3B pods that run workflow code are never privileged.

Instead, 3B creates a Linux user namespace, a private set of user IDs where it has administrator powers that only apply to what it creates there. It starts the sandboxes inside that namespace, and those powers don’t reach the node. The api and worker containers drop every capability except SETUID, SETGID, and SETFCAP, and none of them need CAP_SYS_ADMIN or a host network capability. They do run with seccomp and AppArmor unconfined, because the default profiles block the namespace and mount calls that starting a sandbox needs. gVisor then applies its own seccomp filter to each sandbox, and 3B applies another to the processes inside it.

So the node’s kernel must let unprivileged processes create user namespaces:

  • user.max_user_namespaces and user.max_net_namespaces must be above 0.

  • On Debian, kernel.unprivileged_userns_clone must be 1 where it exists.

  • On Ubuntu 23.10 and newer, kernel.apparmor_restrict_unprivileged_userns must be 0.

Most distributions allow this by default. Ubuntu 23.10 and newer, EKS Auto Mode, and Bottlerocket turn it off, and the chart can turn it back on for you, as described below. The single-server installer sets all of these for you.

Pick a setup

3B runs on standard Kubernetes clusters, including EKS, GKE Standard, and OKE, as long as the nodes allow user namespaces or the cluster lets one privileged DaemonSet turn them on.

Your cluster

What to set

Nodes allow user namespaces and can run user-namespaced pods

Nothing. This is the default and what we recommend.

Nodes allow user namespaces, but the cluster can’t run user-namespaced pods

sandboxd.hostUsers: true

The node kernel blocks user namespaces, and the cluster allows privileged pods

usernsbox.enabled: true, or your own node configuration

The node kernel blocks user namespaces, and the cluster allows no privileged pods

Your own node configuration, or a dedicated server

Step 5 of the Helm installation guide checks the first row with a throwaway pod before you install.

Default

With sandboxd.hostUsers: false, Kubernetes gives each api and worker pod its own user namespace. Root inside the pod is an unprivileged user on the node. Code that escaped both its gVisor sandbox and the container would still have no special rights on the node. This is the most isolated setup.

It needs nodes that can run user-namespaced pods:

  • Kubernetes 1.33 or newer, or 1.29 to 1.32 with the UserNamespacesSupport feature gate on.

  • containerd 2.x.

  • A Linux kernel with idmapped mount support, 6.3 or newer on most node images.

When the cluster can’t run user-namespaced pods

An older Kubernetes version or container runtime, or an admission policy, can rule out user-namespaced pods even when the kernel allows user namespaces. For those clusters, set:

sandboxd:
  hostUsers: true

The pods then share the node’s user namespace. 3B still creates its own user namespace inside each pod and runs every step in gVisor, with the same sandbox, egress rules, and capabilities as the default. The pods are not privileged.

The gVisor sandbox is unchanged, but you give up one of the layers behind it. Root inside the api and worker containers is now root on the node, though still limited to those three capabilities. Code would have to escape gVisor and then the container before that mattered, but if it did, it would reach the node as root instead of as an unprivileged user.

This setting removes the Kubernetes and container runtime requirements. It doesn’t remove the kernel requirements, because 3B creates user namespaces itself.

When the node kernel blocks user namespaces

Set the sysctls listed above on every node that runs the api and worker pods, including autoscaled and replacement nodes. Use whatever your platform already uses to configure nodes, such as a launch template, node bootstrap script, or machine config. 3B doesn’t care how they get set, only that they are.

If your cluster allows privileged pods, the chart can do it for you:

usernsbox:
  enabled: true

This runs a small DaemonSet whose privileged init container sets those sysctls when it starts on each node, then exits. Nothing else in it runs privileged, and the api and worker pods stay unprivileged. This is how EKS Auto Mode and Bottlerocket nodes run 3B. The sysctls apply to the whole node, so other workloads there can create user namespaces too. Scope it with usernsbox.nodeSelector if only some node pools run 3B.

We don’t offer a setting that runs the api or worker pods privileged. It wouldn’t help when user.max_user_namespaces is 0, which blocks root too. Where it would help, usernsbox gets the same result and keeps privilege out of the pods that run workflow code.

When the cluster allows neither

If the nodes block user namespaces, the cluster allows no privileged pods, and the node configuration can’t change, you can try these alternatives:

Did this answer your question?