Skip to main content

Enable 3B autoscaling on K3s

Written by Aron Day

BETA You’re responsible for monitoring and scaling your self-hosted 3B deployment. Use this guide at your own risk and test changes before applying them in production.

The autoscaling.enabled setting currently supports only KEDA. Install and verify KEDA before setting it to true in 3B v10.2.0 or later.

As demand grows, KEDA can add workers and Edge pods. This example sets a maximum of three workers and two Edge pods. Check your K3s node’s capacity before enabling autoscaling.

Install monitoring on K3s and KEDA first. Autoscaling uses the existing Collector and Prometheus in monitoring and creates scaling rules alongside the workloads in 3b.

1. Check the node and choose replica limits

Run the checks on the K3s server. Use sudo k3s kubectl if the kubeconfig requires elevated access.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
cd /opt/3b
kubectl -n monitoring get service otel-collector monitoring-kube-prometheus-prometheus
kubectl -n 3b get deployment edge worker
kubectl -n 3b get hpa,scaledobjects.keda.sh
kubectl wait --for=condition=Available apiservice/v1beta1.external.metrics.k8s.io --timeout=2m
kubectl top nodes
kubectl describe nodes
df -h /var/lib/kubelet/pods /var/lib/rancher/k3s
  • Both services show ClusterIP (Collector on 4318, Prometheus on 9090).

  • Edge and worker show 1/1 ready by default. The external metrics API reports condition met.

  • Remove any non-KEDA autoscaler that targets Edge or worker.

  • With monitoring installed, three default workers need more than 8 vCPUs on a single node.

2. Enable autoscaling in the bootstrap configuration

Enable autoscaling when traffic is low. Edge and workers can briefly drop to one replica each when KEDA takes over. With the default rollout, webhooks can fail while Kubernetes replaces the only ready Edge pod.

Open /opt/3b/.env with vi, vim, or nano, for example sudo nano /opt/3b/.env.

  • If EXTRA_VALUES_FILE is set, merge these values into the file it names.

  • Otherwise, add EXTRA_VALUES_FILE=/opt/3b/autoscaling.yaml to .env. Open the new file (for example, sudo nano /opt/3b/autoscaling.yaml), paste the YAML below, and save it.

autoscaling:
  enabled: true
  otlpMetricsEndpoint: http://otel-collector.monitoring.svc:4318/v1/metrics
  prometheusUrl: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090
edge:
  maxReplicas: 2
worker:
  maxReplicas: 3

On a single node with edge.smtpHostPort set, use edge.maxReplicas: 1.

If you use an HTTP proxy, add otel-collector.monitoring.svc to NO_PROXY. Keep VERSION at your installed release and apply the values.

sudo /opt/3b/setup.sh

setup.sh runs helm upgrade --install and waits for rollout. To raise the worker limit later, change worker.maxReplicas in the same file and rerun the installer.

How 3B decides how many pods to run

Edge and the orchestrator send gauges every 10 seconds. KEDA queries Prometheus for your stack label. Metrics Server supplies CPU readings.

Target

Signal

Scaling rule

Edge

CPU usage of the edge container

Target 50% of requested CPU, averaged across Edge pods.

Edge

edge_admission_utilization

Execution and response each target 50% of concurrent permits. KEDA averages the last minute per pod, then across pods.

Worker

orchestrator_worker_slots_wanted

Occupied slots plus runnable demand, capped per tenant. KEDA uses the largest of each orchestrator’s latest samples from the last 30 seconds.

Edge takes the highest of its three recommendations. Workers target ceil(slots wanted / slots per worker) + 1, within the configured limits. With one slot per worker, zero slots wanted gives one worker, one gives two workers, and two or more gives three.

Both HPAs keep the highest recommendation for 15 minutes. Workers scale in by at most 25% per 15 minutes, rounded up to whole pods. Missing metrics block scale-in. Kubernetes HPA algorithm

For multiple Prometheus replicas or scheduled worker prewarming, see How and when to scale.

3. Verify autoscaling

kubectl -n 3b wait --for=condition=Ready scaledobject/edge scaledobject/worker --timeout=5m
kubectl -n 3b get hpa

Expect condition met for both autoscalers and numbers under TARGETS. The example scales workers from 1 to 3 and Edge from 1 to 2.

Start the watch, then launch two overlapping runs of a workflow with a 60-second Shell step.

kubectl -n 3b get deployment worker --watch

3/3 under READY means all three workers are ready. Press Ctrl+C to stop. Troubleshoot scaling or return to fixed replicas

Did this answer your question?