Provisioned rather than built by hand in Grafana's UI — same reasoning
as the datasource: survives a pod restart, and a git diff shows what
changed. Four rows: ingress (Envoy total RPS + connections + response
class breakdown), service level (CPU/memory by namespace, filterable
via a $namespace template variable, plus a current-usage table), cluster
utilization (used vs actual node capacity, not an assumed limit), and
total resources (cores/memory/pods/disk).
Two scrape gaps found and fixed to make this possible, both in vmagent:
- Contour's own ingress Envoy (projectcontour namespace — the actual
data plane for everything routed through this homelab, hostPort
80/443) was not being scraped at all. Confirmed live: Cilium's
separate embedded Envoy (kube-system, its own L7 policy proxy) was
already flowing, via the annotation-based kubernetes-pods job — which
is what first showed envoy_* metrics existed in this cluster at all —
but Contour's Envoy carries no such annotation. Added an explicit job
targeting the projectcontour namespace by container port (8002, the
official chart's fixed Envoy metrics port) rather than guessing at
pod labels this cluster's auto-detected object names may not match.
- node-exporter, deployed two commits ago, was never actually being
scraped either: confirmed live that kubernetes-service-endpoints
(role: endpointslice, keyed on the Service's scrape annotation — where
that chart puts it) finds nothing in this cluster at all, not merely
down. Rather than chase why, added the same fix as Envoy: target the
pod directly by its declared container port (9100).
Verified against the live deployment (queried through vmui) before
writing a single panel: envoy_http_downstream_rq_total,
envoy_http_downstream_rq_xx, container_cpu_usage_seconds_total,
container_memory_working_set_bytes, machine_cpu_cores and
machine_memory_bytes all confirmed present with real data. The one
exception is the "Disk free" panel, which depends on the node-exporter
scrape fix landing in this same change — noted in the values file's own
comment as unverified until it actually deploys.
Also verified with `helm template`: the dashboard JSON round-trips
through the YAML values file and the chart's own ConfigMap templating
intact (19 panels both times), and vmagent's scrape_configs list still
carries all 8 chart defaults plus both new jobs — nothing lost by using
extraScrapeConfigs instead of overriding the full list by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
Grafana: real dashboards over VictoriaMetrics. Provisioned rather than
clicked through: a VictoriaMetrics datasource (type: prometheus — VM
speaks that query API, which is the whole point of it existing) is
baked into the chart's own datasource-provisioning config, so a fresh
install has it working with no manual setup. Admin credentials from
Vault via ExternalSecret (secretstores/grafana-admin-credentials.yaml
in devops-infra-argo-config), same pattern as every other admin
credential in this project — never plaintext in this repo. 1Gi
local-path PVC for dashboards/Grafana's own state (VictoriaMetrics
holds the actual metric data, not this). Dual LAN+Tailscale Ingress
hosts, same convention as everything externally reachable here.
Found and fixed while vendoring: helm-templates/grafana already held a
fully-vendored old Grafana chart (v6.58.7, appVersion 10.0.3) from the
original Meesho monorepo import (commit b8575bb) — generic production
config (fullnameOverride: grafana-infra-prd, GKE-shaped RBAC/PSP
defaults) unrelated to this homelab, same class of leftover as
victoria-metrics-single's collision two commits ago. Removed and
re-vendored fresh (10.5.15) as a thin wrapper, matching every other
official-chart component in this repo now.
vmui: VictoriaMetrics' own built-in UI (ad-hoc PromQL + graphs, no
saved dashboards — what Grafana is for) is served on the same
pod/port, so exposing it cost one ingress block on the
victoria-metrics-single values already committed. No new component,
no new RAM.
Verified with `helm template` against the real charts for both
components individually (Grafana: admin env vars correctly reference
the ExternalSecret's keys, datasource ConfigMap renders the intended
VictoriaMetrics URL, PVC/resources/ingress hosts all match; vmui:
ingress renders both hostnames pointing at the existing Service's named
http port) and again for the whole generic-argo-apps-chart appSpec
list — 12 Applications render, including grafana.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wajog7nELA3i8JWTjxYGHF
