Skip to main content

Common questions about 3B autoscaling

Written by Aron Day

BETA You’re responsible for monitoring and scaling your self-hosted 3B deployment. Use this guide at your own risk and test changes before applying them in production.

The autoscaling.enabled setting currently supports only KEDA. Install and verify KEDA before setting it to true in 3B v10.2.0 or later.

Use your 3B cluster context on K3s or Kubernetes with Helm. Substitute your workload namespace for 3b if needed.

Do I need KEDA to enable autoscaling?

Yes. You set autoscaling.enabled: true in the 3B Helm values. The chart creates KEDA scaling rules for Edge and workers. If Kubernetes doesn’t recognize KEDA’s ScaledObject resource, the 3B install or upgrade fails with no matches for kind "ScaledObject". Install and verify KEDA first.

If only the resource definition is installed, Helm can accept the rules, but the KEDA operator must be running to create the autoscalers. If KEDA’s metrics API goes down later, existing workers keep processing executions. New work waits when their slots fill.

For three fixed workers without KEDA, leave autoscaling.enabled: false, set worker.replicas: 3, and reapply your Helm values. This updates both the pod count and the orchestrator’s execution limit. Changing the pod count alone with kubectl scale or a separate autoscaler leaves that limit in place.

How do I see why a replica was added?

In Grafana, open Dashboards, then KEDA overview. Compare Worker slots wanted with 3B replicas managed by KEDA at the time of the change. To sign in, see Grafana on K3s or Grafana on Kubernetes with Helm. Add the KEDA dashboard if it’s missing.

The HorizontalPodAutoscaler (HPA) shows current and target values under Metrics. Its SuccessfulRescale events explain recent changes.

kubectl -n 3b describe hpa keda-hpa-edge keda-hpa-worker

Edge targets 50% for CPU and for each of its two permit pools. Metrics Server measures CPU against Edge’s request. The Prometheus gauge edge_admission_utilization measures execution and response permits in use. Kubernetes takes the highest recommendation. For workers, orchestrator_worker_slots_wanted counts occupied slots plus runnable work, capped per tenant.

Why do I have one worker if the maximum is three?

With one slot per worker, 3B asks for one spare. At zero slots wanted, expect one worker. One slot calls for two workers, and two or more call for three. At idle, 1/1 (avg) on the worker HPA and ACTIVE=False on the Edge ScaledObject are normal with valid metrics.

Run two workflows that hold a slot at the same time. Watch for 3/3 under READY, then press Ctrl+C.

kubectl -n 3b get deployment worker --watch

Why does an HPA target show unknown?

The HPA events identify the failed metric. Check Metrics Server for Edge CPU, or KEDA’s API and the installed Prometheus queries for external metrics.

kubectl -n 3b top pods
kubectl get apiservice v1beta1.external.metrics.k8s.io
kubectl -n 3b get scaledobject edge worker \
  -o jsonpath='{range .items[*]}{range .spec.triggers[?(@.type=="prometheus")]}{.metadata.query}{"\n"}{end}{end}'

top should return pod usage and the APIService should show AVAILABLE=True. In Grafana Explore, select Prometheus and run each printed query. Zero is valid at idle. An empty result means KEDA has no sample.

Check for export errors with kubectl -n monitoring logs deployment/otel-collector --tail=50. If you restrict pod traffic, allow Edge and the orchestrator to reach the Collector on 4318, and the Collector and KEDA to reach Prometheus on 9090. With valid samples, expect ScalingActive=True and ValidMetricFound.

Why did Kubernetes request three workers if only two are ready?

For a Pending worker, read its FailedScheduling event. It names the unmet requirement, such as too little CPU or memory on an eligible node.

kubectl -n 3b get deployment worker
kubectl -n 3b get pods -l app=worker
kubectl -n 3b get events --field-selector reason=FailedScheduling

On K3s, check the additional capacity for three workers. On a multi-node cluster, check room on eligible nodes and whether your node pool can grow.

Why are extra workers still running after the workflows finish?

Edge and worker keep the highest recommendation from the last 15 minutes. The HPA reports ScaleDownStabilized while it holds that count. Workers can drop by at most 25% per 15 minutes, rounded up to whole pods, so going from three to one can take multiple intervals.

How do I go back to a fixed number of workers?

Set autoscaling.enabled: false and choose your worker.replicas and edge.replicas counts. Reapply your values through K3s setup or Helm upgrade.

Did this answer your question?