Replace Prometheus with VictoriaMetrics (victoria-metrics-single + vmagent)

Lower RAM/disk for the same metric volume on an 8GB single node — VM's own compression is the entire reason it exists as a project. Speaks Prometheus's own query API (/api/v1/query), so nothing downstream needed to change beyond the URL it points at: same PromQL, same kubernetes-nodes-cadvisor-sourced container_cpu_usage_seconds_total / container_memory_working_set_bytes toolshed's metrics work (docs/PRODUCT-ARCHITECTURE.md step 5) already targets. Three releases, matching the one-release-per-component convention already used everywhere in this repo, since victoria-metrics-single has no bundled scraper/exporter subcharts the way the Prometheus chart did: - victoria-metrics-single: the TSDB + query engine. 3Gi/local-path, 7-day retention, resources capped at 512Mi — same trim reasoning as the Prometheus server it replaces. - vmagent: the scraper. Its default scrape_configs already includes kubernetes-nodes-cadvisor and kubernetes-service-endpoints (the chart's own comment says "COPY from Prometheus helm chart") — nothing to override there, only remoteWrite pointed at victoria-metrics-single and trimmed resources. - node-exporter: pulled out of the removed Prometheus chart's bundled subchart into its own standalone release, since victoria-metrics-single has no equivalent. Same trim as before (32Mi/64Mi), same prometheus.io/scrape annotation vmagent's default scrape config already looks for. Found and fixed while vendoring: helm-templates/victoria-metrics-single already existed in this repo — a leftover GKE-targeted vendored copy from the original Meesho monorepo import (commit b8575bb), with its own templates/values.yaml for a different chart entirely. `mkdir -p` on an already-existing directory silently did nothing, so the first vendoring attempt left that old tree sitting alongside the new one and rendered two ServiceAccounts/Services/StatefulSets with colliding names. Removed outright rather than adapted, same reasoning as the ~35 charts devops-helm-charts removed in its own prune — see this chart's own Chart.yaml for the full note. Verified with `helm template` against the real charts and these values for all three releases individually, and again for the whole generic-argo-apps-chart appSpec list (11 Applications render, including the three new ones and nothing orphaned from the old prometheus entry). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
2026-09-06 08:38:34 +05:30
parent 6677c9e315
commit 04e7ef2f82
37 changed files with 126 additions and 2465 deletions
@@ -0,0 +1,16 @@
+prometheus-node-exporter:
+  # Same resource trim this ran with as a subchart of the (now removed)
+  # prometheus server chart. One DaemonSet pod, hostNetwork, light enough
+  # that this is the whole footprint regardless of which TSDB scrapes it.
+  resources:
+    requests:
+      cpu: 20m
+      memory: 32Mi
+    limits:
+      memory: 64Mi
+
+  # rbac.pspEnabled defaults false already (no PodSecurityPolicy on this
+  # cluster) — nothing to override here. The chart's own Service carries
+  # `prometheus.io/scrape: true` by default, which is what vmagent's
+  # already-enabled kubernetes-service-endpoints job picks up — no
+  # separate scrape config needed on either side.
@@ -1,66 +0,0 @@
-prometheus:
-  # alertmanager/kube-state-metrics/node-exporter/pushgateway are all
-  # enabled by default in this chart. None of the four are needed for
-  # what actually consumes this deployment — toolshed's per-app CPU/memory
-  # come straight from kubelet's own cAdvisor endpoint (the
-  # kubernetes-nodes-cadvisor scrape job below, built into the server
-  # itself) — but node-exporter earns its keep separately: this box is an
-  # 8GB single node already near its ceiling, and node-exporter is the one
-  # thing here that would actually say so (host disk/memory/load, not
-  # per-container usage cAdvisor already covers). One DaemonSet pod on one
-  # node, hostNetwork rather than a Service — light enough to justify on
-  # its own.
-  #
-  # alertmanager/kube-state-metrics/pushgateway stay off: nothing here
-  # defines alerting rules to route (alertmanager), nothing needs
-  # deployment/rollout-status metrics yet as opposed to raw resource usage
-  # (kube-state-metrics), and nothing is a short-lived batch job pushing
-  # its own metrics (pushgateway). Each is a real pod this node has no
-  # spare room for — turn one on when something concrete needs it, not
-  # preemptively.
-  alertmanager:
-    enabled: false
-  kube-state-metrics:
-    enabled: false
-  prometheus-node-exporter:
-    enabled: true
-    resources:
-      requests:
-        cpu: 20m
-        memory: 32Mi
-      limits:
-        memory: 64Mi
-  prometheus-pushgateway:
-    enabled: false
-
-  server:
-    persistentVolume:
-      # local-path-provisioner, this cluster's default StorageClass —
-      # installed right after Cilium precisely because kubeadm ships no
-      # default (unlike k3s). 3Gi rather than the chart's 8Gi default:
-      # `retention: 7d` below on one small cluster's worth of series
-      # comfortably fits, and the node has no room to spare. Not
-      # resizable in place with this provisioner, so sized deliberately
-      # rather than grown later.
-      size: 3Gi
-      storageClass: local-path
-
-    # 7 days, not the chart's 15-day default — a homelab whose entire
-    # purpose is proving a CI/CD pipeline has no use for a month of
-    # historical series, and every extra day is disk this node does not
-    # have spare.
-    retention: 7d
-
-    resources:
-      requests:
-        cpu: 50m
-        memory: 128Mi
-      limits:
-        memory: 512Mi
-
-  # The default kubernetes-nodes-cadvisor job (scheme https, bearer token
-  # from the pod's own ServiceAccount, metrics_path /metrics/cadvisor) is
-  # left exactly as the chart ships it — this is what
-  # container_cpu_usage_seconds_total and container_memory_working_set_bytes
-  # come from, and toolshed's metrics connection (internal/metrics)
-  # queries exactly those two, summed by namespace.
@@ -0,0 +1,24 @@
+victoria-metrics-single:
+  server:
+    # 7 days, not the chart's month-long default — a homelab whose entire
+    # purpose is proving a CI/CD pipeline has no use for that much
+    # history, and every extra day is disk this node does not have spare.
+    retentionPeriod: "7d"
+
+    persistentVolume:
+      # local-path-provisioner, this cluster's default StorageClass —
+      # installed right after Cilium precisely because kubeadm ships no
+      # default (unlike k3s). 3Gi, not the chart's 16Gi default: VM's own
+      # compression is the whole reason it replaced Prometheus here, and
+      # this cluster's metric volume at a 7-day retention comfortably
+      # fits well inside that. Not resizable in place with this
+      # provisioner, so sized deliberately rather than grown later.
+      storageClassName: local-path
+      size: 3Gi
+
+    resources:
+      requests:
+        cpu: 50m
+        memory: 128Mi
+      limits:
+        memory: 512Mi
@@ -0,0 +1,28 @@
+victoria-metrics-agent:
+  # Ships to victoria-metrics-single. The write path is /api/v1/write,
+  # the same endpoint any Prometheus remote_write client uses — this is
+  # the one config line that actually wires the two components together;
+  # the chart names don't imply it on their own.
+  #
+  # Cluster-internal Service DNS: this call happens from a pod, so
+  # CoreDNS resolves it. Same reasoning as everywhere else in this
+  # project that a Service name (not an Ingress hostname) is correct
+  # here — see claude.md's registry-hostname convention for the one case
+  # where cluster DNS specifically does NOT work (a node-level pull),
+  # which does not apply to this pod-to-pod write.
+  remoteWrite:
+    - url: http://victoria-metrics-single-server.monitoring.svc.cluster.local:8428/api/v1/write
+
+  # config.scrape_configs is left at the chart's own default — it already
+  # includes kubernetes-nodes-cadvisor (kubelet's cAdvisor endpoint,
+  # comment literally says "COPY from Prometheus helm chart") and
+  # kubernetes-service-endpoints (the prometheus.io/scrape annotation
+  # convention node-exporter's Service carries). Nothing here needs a
+  # scrape target the chart doesn't already define out of the box.
+
+  resources:
+    requests:
+      cpu: 25m
+      memory: 64Mi
+    limits:
+      memory: 192Mi