Skip to main content

Install monitoring on K3s

Written by Aron Day

BETA Whoever manages your K3s installation or cluster is responsible for monitoring and scaling 3B. Use this guide at your own risk and test changes before applying them in production.

You can use your existing monitoring tools. For the 3B metrics available with autoscaling on or off and suggested alert thresholds, see What to monitor.

If you’re setting up monitoring for the first time, you can use Prometheus to store metrics, Grafana for dashboards, and the OpenTelemetry Collector to receive 3B metrics. All three stay private in monitoring. Autoscaling reuses the Collector and Prometheus.

If you’ve already installed Grafana with this guide, skip to step 7 to sign in.

1. Check capacity and add the Helm repositories

For a proof of concept, budget 1 extra CPU core, 2 GB of RAM, and 50 GiB of SSD space for monitoring, including the 20 GiB Prometheus volume. This is on top of the 8 vCPUs, 32 GB of RAM, and 500 GB SSD in the 3B sizing guide.

Run steps 1–6 in one Bash shell on the K3s server with kubectl and Helm. If reading the kubeconfig requires sudo, use sudo k3s kubectl and sudo -H /usr/local/bin/helm --kubeconfig "$KUBECONFIG" throughout, including for helm repo add.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get nodes
kubectl -n 3b get deployment,statefulset
kubectl get storageclass local-path

Look for a Ready node, matching ready and desired replicas, and the local-path storage class.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts --force-update
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts --force-update

Helm should list both repositories as added. Allow outbound access to the upstream chart and image registries. Tines will provide these charts and images later.

2. Install Prometheus and Grafana

Paste each Helm install block into your terminal, including the closing YAML line.

Prometheus keeps up to seven days of samples and expires older blocks at 15 GiB. The 20 GiB claim leaves room for the write-ahead log and compaction. This setup disables Alertmanager, which routes Prometheus alerts.

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --version 93.4.1 --namespace monitoring --create-namespace \
  --wait --timeout 10m --values - <<'YAML'
alertmanager:
  enabled: false
kubeScheduler:
  enabled: false
kubeControllerManager:
  enabled: false
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false
grafana:
  enabled: true
  service:
    type: ClusterIP
  resources:
    requests: {cpu: 100m, memory: 192Mi}
  sidecar:
    resources:
      requests: {cpu: 25m, memory: 64Mi}
prometheusOperator:
  resources:
    requests: {cpu: 100m, memory: 128Mi}
  prometheusConfigReloader:
    resources:
      requests: {cpu: 10m, memory: 32Mi}
kube-state-metrics:
  resources:
    requests: {cpu: 50m, memory: 64Mi}
prometheus-node-exporter:
  hostNetwork: false
  resources:
    requests: {cpu: 50m, memory: 32Mi}
prometheus:
  service:
    type: ClusterIP
  prometheusSpec:
    replicas: 1
    enableOTLPReceiver: true
    retention: 7d
    retentionSize: 15GiB
    resources:
      requests: {cpu: 250m, memory: 1Gi}
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 20Gi
YAML

After Helm reports STATUS: deployed, wait for the Prometheus pod and volume.

prom_sts=prometheus-monitoring-kube-prometheus-prometheus
kubectl -n monitoring wait --for=create "statefulset/$prom_sts" --timeout=10m
kubectl -n monitoring rollout status "statefulset/$prom_sts" --timeout=10m
kubectl -n monitoring get pvc

Expect one ready pod and a Bound claim showing 20Gi, RWO, and local-path.

3. Install the OpenTelemetry Collector

Send OTLP/HTTP metrics to http://otel-collector.monitoring.svc:4318/v1/metrics. The Collector appends /v1/metrics to its Prometheus exporter URL.

helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
  --version 0.180.0 --namespace monitoring \
  --wait --timeout 10m --values - <<'YAML'
fullnameOverride: otel-collector
mode: deployment
replicaCount: 1
image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
command:
  name: otelcol-k8s
service:
  type: ClusterIP
ports:
  otlp-http: {enabled: true}
  otlp: {enabled: false}
  jaeger-compact: {enabled: false}
  jaeger-thrift: {enabled: false}
  jaeger-grpc: {enabled: false}
  zipkin: {enabled: false}
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 256Mi}
config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: ${env:MY_POD_IP}:4318
        grpc: null
    jaeger: null
    prometheus: null
    zipkin: null
  exporters:
    debug: null
    otlp_http:
      endpoint: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090/api/v1/otlp
  service:
    pipelines:
      logs: null
      traces: null
      metrics:
        receivers: [otlp]
        exporters: [otlp_http]
YAML

kubectl -n monitoring get deployment/otel-collector service/otel-collector

Expect STATUS: deployed, 1/1 ready, and a ClusterIP Service on 4318/TCP.

4. Check readiness and the private endpoints

kubectl -n monitoring get pods,svc
kubectl -n 3b get deployment,statefulset

Expect Running pods with all containers ready. Grafana, Prometheus, and the Collector should show ClusterIP with no external IP. Compare 3B replica counts with step 1.

Check usage under your usual load. Use sudo df if the storage directory is restricted.

kubectl top pods -n monitoring
df -h /var/lib/rancher/k3s/storage

Expect CPU and memory per monitoring pod. Watch available disk space over the first seven days.

kubectl get --raw '/api/v1/namespaces/monitoring/services/http:monitoring-grafana:http-web/proxy/api/health'

Grafana should return JSON with "database": "ok" and a version.

5. Query 3B pod memory and scrape health

This helper runs promtool in the Prometheus pod and waits for the first samples.

prom_query() {
  local rows attempt
  for attempt in {1..30}; do
    rows=$(kubectl -n monitoring exec statefulset/prometheus-monitoring-kube-prometheus-prometheus -c prometheus -- \
      /bin/promtool query instant http://localhost:9090 "$1") || return 1
    if [[ -n "$rows" ]]; then
      printf '%s\n' "$rows"
      return 0
    fi
    sleep 2
  done
  echo 'Prometheus returned no samples after 30 queries.' >&2
  return 1
}

prom_query 'sum by (pod) (container_memory_working_set_bytes{job="kubelet",metrics_path="/metrics/cadvisor",namespace="3b",container!="",container!="POD"}) / 1024^2'
prom_query 'min by (job) (up)'

The first query returns memory in MiB per sampled 3B pod, including {pod="postgres-0"} on a stock install. The second returns scrape health by job. Expect {job="kubelet"} => 1 when all its targets are up, or 0 if any scrape failed.

6. Send a gauge through the Collector

Send one sample to the Collector using Python 3 on the K3s server.

collector_ip=$(kubectl -n monitoring get service otel-collector -o jsonpath='{.spec.clusterIP}')
sent_at=$(date +%s)
python3 - "$collector_ip" "$sent_at" <<'PY'
import json
import sys
from http.client import HTTPConnection

point = {"timeUnixNano": sys.argv[2] + "000000000", "asInt": "1"}
metric = {"name": "lab.monitoring.check", "gauge": {"dataPoints": [point]}}
body = {"resourceMetrics": [{"scopeMetrics": [{"metrics": [metric]}]}]}
conn = HTTPConnection(sys.argv[1], 4318, timeout=10)
conn.request("POST", "/v1/metrics", json.dumps(body), {"Content-Type": "application/json"})
response = conn.getresponse()
print(response.status, response.read().decode())
sys.exit(0 if response.status == 200 else 1)
PY

Expect HTTP 200 and zero rejected points, usually {"partialSuccess":{}}. Check Prometheus for the same sample.

prom_query "lab_monitoring_check == 1 and timestamp(lab_monitoring_check) == $sent_at"

Expect lab_monitoring_check => 1 @[...]. Prometheus converts dots to underscores. The timestamp selects the sample you just sent. Prometheus OTLP guide

7. How do I access Grafana?

Use the workstation where you’ll open the browser and its kubeconfig for K3s. Without cluster access, ask your operator for the private Grafana URL and sign-in.

kubectl config current-context
kubectl -n monitoring get service monitoring-grafana

Confirm the K3s context and port 80/TCP. With permission to read the Secret, get the initial password for admin and start the port-forward.

kubectl -n monitoring get secret monitoring-grafana -o go-template='{{index .data "admin-password" | base64decode}}{{"\n"}}'
kubectl -n monitoring port-forward --address 127.0.0.1 service/monitoring-grafana 3000:80

At Forwarding from 127.0.0.1:3000, open http://127.0.0.1:3000 and sign in. Under Dashboards, open Kubernetes / Compute Resources / Namespace (Pods) and select 3b. Keep the terminal open until you’re done, then press Ctrl+C.

Common questions

Why can’t I reach Grafana in my browser?

If port 3000 is busy, change 3000:80 to 3001:80 and open http://127.0.0.1:3001. With the forward running, /api/health should return "database": "ok". If it fails, check Grafana from the server in step 4.

Where do I run a Prometheus query?

In Grafana’s Explore, select Prometheus, enter a query from step 5 in Code mode, and select Run query.

Why does my dashboard show no 3B pods?

Select namespace 3b and a time range when the pods were running. If the memory query in step 5 is empty too, check the kubelet job with the scrape health query.

Where is the autoscaling dashboard?

The KEDA guide adds KEDA overview to this Grafana. Scaling data appears after you enable 3B autoscaling.

Did this answer your question?