Coding Guideline — Observability stack overrides

Conventions for editing values for the observability charts vendored in this repo:

victoria-metrics-cluster and victoria-metrics-cluster-latest
victoria-metrics-agent and victoria-metrics-agent-latest
vmalert, victoria-metrics-operator
mimir-distributed
loki, loki-distributed
tempo, tempo-distributed
grafana
opentelemetry-collector and opentelemetry-collector-latest
kube-state-metrics, node-exporter, metrics-server
kube-prometheus-stack (Prometheus alert rules)

The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as data-pipeline changes, not static config.

Versioned siblings

*-latest siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See wiki/analyses/ADR-A2-blue-green-sibling-pattern.md and ../platform/procedures/blue-green-chart-migration.md.

Cardinality discipline

Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:

New extraLabels / externalLabels on victoria-metrics-agent.
New relabel_configs that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
New scrape targets in additionalScrapeConfigs.

If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.

PromQL / alert-rule validation

Alert rules live in kube-prometheus-stack, vmalert, and Mimir ruler config. Before merging:

Render with helm template:

helm template prom helm-templates/kube-prometheus-stack \
  -f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml

Extract the PrometheusRule objects and validate PromQL with promtool check rules <file.yaml> (Prometheus tooling) or vmalert -dryRun -rule=<file.yaml> for vmalert-specific rules.
Confirm the rule's for: window and severity label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.

Retention and tenancy

VictoriaMetrics vmstorage.retentionPeriod is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC resources.requests.storage.
Loki and Mimir multi-tenancy is keyed on the X-Scope-OrgID header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
Tempo trace retention is set via compactor.compaction.block_retention. Default is short (24h–48h); long retention is opt-in per cluster.

Grafana dashboards & datasources

Datasources are declared in the cluster's grafana/custom-values.yaml under datasources.datasources.yaml. Pin URLs to in-cluster Service DNS, not external endpoints.
Dashboards bundled via dashboardProviders reference ConfigMaps; uniqueness of uid matters for panel-image links and alert dashboards.

Validation checklist (always)

Before raising a PR that touches an observability chart:

helm template renders without error.
PromQL in any new alert rule passes promtool check rules.
No new high-cardinality label is added without a bound.
Retention / PVC sizing has not been changed silently.
If touching a *-latest sibling, the matching Argo Application points at it. See argocd.md.

Schema references

Per-cluster scheduling fields (must be rewritten, never copy-pasted): ../../platform/schemas/custom-values-schema.md
Raw manifest sidecars (e.g., external-dns-services/*.yaml for Grafana ingress): ../../platform/schemas/raw-manifest-sidecar-schema.md
Storage backing for vmstorage, loki chunks, tempo blocks: ../../platform/schemas/storageclass-priorityclass-schema.md

4.6 KiB Raw Blame History Unescape Escape