How and when to scale | Tines 3B Docs

3B scales out by adding replicas of whichever service is under pressure: more workers to run more executions at once, more api and edge pods to serve more traffic. When every worker slot is busy, new executions queue until one frees up. The system overview describes each service. This page covers Kubernetes (Helm) deployments. To scale the single-server install, resize the VM as described in sizing.

When to scale

Match the symptom to the service:

Symptom	Service under pressure
Executions sit pending before they start	`worker`
The UI feels slow; API calls lag	`api`
Webhooks respond slowly or time out	`edge`

Watch pod CPU and memory from your cluster’s normal metrics pipeline, and page on them. See CPU and memory for thresholds, Postgres and Blobstore disk for the storage alarms, and what to monitor for how Metrics Server fits next to a real alerting pipeline.

How to scale

Each scalable service exposes a replicas value in the Helm chart:

To handle	Raise
More concurrent executions	`worker.replicas`
More UI and API traffic	`api.replicas`
More webhook traffic	`edge.replicas`

Start with worker.replicas: set a baseline that covers your busiest sustained periods rather than scaling reactively from one replica. Each worker pod and its sandbox sidecar together request about 1.2 vCPU and 2.5 GiB of memory, so plan node capacity accordingly.

For autoscaling, a CPU-based HorizontalPodAutoscaler on worker, api, and edge works with the chart’s Deployments; a target around 50% average CPU is a good starting point. Let it scale out quickly, but give it a long scale-down stabilization window (around 15 minutes) so it removes replicas slowly.

Note: CPU can stay low on saturated workers when steps spend their time waiting on external services. If executions sit pending while worker CPU looks calm, the workers are still the bottleneck — raise worker.replicas directly.

Note: The other components stay at one replica. postgres and nginx are singletons, and the chart enforces a single blobstore replica. If storage runs short, expand the Blobstore volume as described in Blobstore disk. For a highly available database, use an external Postgres as described in the Helm installation guide.

For Blobstore CPU or memory pressure, raise blobstore.resources. The serving defaults request 2 CPU and 4 GiB of memory, with a 4 GiB memory limit and no CPU limit. Increase the memory request and limit together when the workload needs more memory, and check that a node can fit the pod. See sizing for cluster capacity.

Per-step resource limits

When the container runtime exposes the container’s own cgroup root read-write, each workflow step runs under cgroup v2 limits of 1 vCPU and 2 GiB of memory. Installs leave the runtime class empty by default, so this enforcement is off: sandboxd logs that cgroup enforcement is disabled and continues without per-step compute or memory limits.

We recommend turning it on when you scale workers. Per-step limits bound each execution’s CPU and memory consumption. Configure a dedicated containerd runtime handler with writable cgroups on every node eligible to run API or worker pods. Both services use sandboxd, so both need the same node setup when this option is enabled.

For containerd 2.x using configuration version 3, merge this handler into the node’s containerd configuration:

[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.sandboxd-runc]
runtime_type = "io.containerd.runc.v2"
cgroup_writable = true

[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.sandboxd-runc.options]
SystemdCgroup = true

The handler uses ordinary runc. It gives each container a writable view of its own cgroup subtree on cgroup v2; sandboxd creates execution limits inside that subtree and launches the bundled gVisor runtime internally. It does not select gVisor as the pod runtime or require a wrapper script. Keep the writable-cgroup setting on this dedicated handler, with the chart’s user namespaces and container resource limits in place.

On k3s, extend the containerd configuration template at /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl: start with {{ template "base" . }}, followed by the TOML above. Configure nodes through your normal maintenance process before scheduling pods with the new handler. Editing k3s’s generated config.toml is not persistent.

Then set the shared Helm value:

sandboxd:
  runtimeClassName: sandboxd-runc

The chart creates the matching RuntimeClass and selects it for API and worker pods. A RuntimeClass selects a node’s handler; it does not install or configure that handler. Existing installations that already set this value must also prepare API-eligible nodes before upgrading. Self-hosted keeps this option empty by default.

On the single-server install, pass this as an extra values file via EXTRA_VALUES_FILE in /opt/3b/.env. After rollout, verify sandboxd no longer reports disabled cgroup enforcement and that a test execution receives the expected CPU and memory limits.