Skip to main content

Install a monitoring stack on Kubernetes with Helm

Written by Aron Day

BETA You’re responsible for monitoring and scaling your self-hosted 3B deployment. Use this guide at your own risk and test changes before applying them in production.

Install Prometheus, Grafana, and the OpenTelemetry Collector as private ClusterIP Services in monitoring. If your cluster already has monitoring, go to Monitor a 3B Helm release.

What do I need before installing?

Use Bash on a workstation with Helm, kubectl, and permission to install cluster-wide resources, including Prometheus Operator custom resource definitions.

kubectl config current-context
helm list --all-namespaces
kubectl get storageclass

Confirm the 3B cluster context and check with your team for monitoring managed outside Helm. The default StorageClass must support a 20 GiB ReadWriteOnce claim. To select another, set storageClassName beside accessModes below. Kubernetes storage classes

On one node, the stack requests 660m CPU and 1,664 MiB of RAM. Add 50m CPU and 32 MiB per additional node running node-exporter. Prometheus usage depends on time series and query load.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts --force-update
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts --force-update

Install Prometheus and Grafana

Paste each Helm install block into your terminal, including the closing YAML line.

Prometheus keeps up to seven days of kubelet and workload metrics with a 15 GiB retention cap. This setup disables Alertmanager and skips control-plane scrape jobs commonly unavailable on managed clusters.

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  --version 93.4.1 --namespace monitoring --create-namespace \
  --wait --timeout 10m --values - <<'YAML'
alertmanager:
  enabled: false
kubeScheduler:
  enabled: false
kubeControllerManager:
  enabled: false
kubeProxy:
  enabled: false
kubeEtcd:
  enabled: false
grafana:
  enabled: true
  service:
    type: ClusterIP
  resources:
    requests: {cpu: 100m, memory: 192Mi}
  sidecar:
    resources:
      requests: {cpu: 25m, memory: 64Mi}
prometheusOperator:
  resources:
    requests: {cpu: 100m, memory: 128Mi}
  prometheusConfigReloader:
    resources:
      requests: {cpu: 10m, memory: 32Mi}
kube-state-metrics:
  resources:
    requests: {cpu: 50m, memory: 64Mi}
prometheus-node-exporter:
  hostNetwork: false
  resources:
    requests: {cpu: 50m, memory: 32Mi}
prometheus:
  service:
    type: ClusterIP
  prometheusSpec:
    replicas: 1
    enableOTLPReceiver: true
    retention: 7d
    retentionSize: 15GiB
    resources:
      requests: {cpu: 250m, memory: 1Gi}
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 20Gi
YAML
prom_sts=prometheus-monitoring-kube-prometheus-prometheus
kubectl -n monitoring wait --for=create "statefulset/$prom_sts" --timeout=10m
kubectl -n monitoring rollout status "statefulset/$prom_sts" --timeout=10m
kubectl -n monitoring get pvc

Expect one ready Prometheus pod and a Bound 20 GiB claim. For a Pending claim, check provisioning events with kubectl -n monitoring describe pvc.

Install the OpenTelemetry Collector

Prometheus scrapes pod metrics immediately. Once you enable autoscaling, 3B also sends scaling gauges to http://otel-collector.monitoring.svc:4318/v1/metrics for the Collector to forward to Prometheus.

helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
  --version 0.180.0 --namespace monitoring \
  --wait --timeout 10m --values - <<'YAML'
fullnameOverride: otel-collector
mode: deployment
replicaCount: 1
image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
command:
  name: otelcol-k8s
service:
  type: ClusterIP
ports:
  otlp-http: {enabled: true}
  otlp: {enabled: false}
  jaeger-compact: {enabled: false}
  jaeger-thrift: {enabled: false}
  jaeger-grpc: {enabled: false}
  zipkin: {enabled: false}
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 256Mi}
config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: ${env:MY_POD_IP}:4318
        grpc: null
    jaeger: null
    prometheus: null
    zipkin: null
  exporters:
    debug: null
    otlp_http:
      endpoint: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090/api/v1/otlp
  service:
    pipelines:
      logs: null
      traces: null
      metrics:
        receivers: [otlp]
        exporters: [otlp_http]
YAML

How do I check the install and open Grafana?

kubectl -n monitoring get deployment/otel-collector
kubectl -n monitoring get service monitoring-grafana monitoring-kube-prometheus-prometheus otel-collector
kubectl -n monitoring exec "statefulset/$prom_sts" -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 'count(kube_pod_info{namespace="3b"})'

Expect 1/1 ready and ClusterIP Services for Grafana (80), Prometheus (9090), and the Collector (4318). After the first kube-state-metrics scrape, expect a positive pod count. Replace 3b if your namespace differs.

For the private Grafana URL and credentials, see Monitor a 3B Helm release. To scale workers and Edge pods, see Autoscale a 3B Helm release.

Did this answer your question?