How and when to scale

3B scales out by adding replicas of whichever service is under pressure: more workers to run more executions at once, more api and public pods to serve more traffic. When every worker slot is busy, new executions queue until one frees up. The system overview describes each service. This page covers Kubernetes (Helm) deployments. To scale the single-server install, resize the VM as described in sizing.

Match the symptom to the service:

Watch pod CPU and memory from your cluster’s normal metrics pipeline. CPU-based alerts on worker, api, and public are a reasonable first line. Richer operational metrics will ship in a future release.

Each scalable service exposes a replicas value in the Helm chart:

Start with worker.replicas: set a baseline that covers your busiest sustained periods rather than scaling reactively from one replica. Each worker pod and its sandbox sidecar together request about 1.2 vCPU and 2.5 GB of memory, more than any other pod in a 3B deployment, so plan node capacity accordingly.

For autoscaling, a CPU-based HorizontalPodAutoscaler on worker, api, and public works with the chart’s Deployments; a target around 50% average CPU is a good starting point. Let it scale out quickly, but give it a long scale-down stabilization window (around 15 minutes) so it removes replicas slowly.

Note: CPU can stay low on saturated workers when steps spend their time waiting on external services. If executions sit pending while worker CPU looks calm, the workers are still the bottleneck — raise worker.replicas directly.

Note: the other components stay at one replica. postgres and nginx are singletons, and the chart enforces a single blobstore replica. If storage runs short, raise blobstore.storage.size instead. For a highly available database, use an external Postgres as described in the Helm installation guide.

When the container runtime exposes the container’s own cgroup root read-write, each workflow step runs under cgroup v2 limits of 1 vCPU and 2 GiB of memory. Installs leave the runtime class empty by default, so this enforcement is off: sandboxd logs that cgroup enforcement is disabled and continues without per-step compute or memory limits.

We recommend turning it on when you scale workers. With per-step limits, each worker’s capacity is predictable, and one heavy step can’t starve the other steps on the same node. To enable it, configure a containerd runtime handler with writable cgroups on every worker node, then set sandboxd.runtimeClassName to that handler’s name:

The runtime class name must match the handler configured on the nodes. (On the single-server install, pass this as an extra values file via EXTRA_VALUES_FILE in /opt/3b/.env.)