> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.

# Coding Guideline — Observability stack overrides

Conventions for editing values for the observability charts vendored in this repo:

- `victoria-metrics-cluster` and `victoria-metrics-cluster-latest`
- `victoria-metrics-agent` and `victoria-metrics-agent-latest`
- `vmalert`, `victoria-metrics-operator`
- `mimir-distributed`
- `loki`, `loki-distributed`
- `tempo`, `tempo-distributed`
- `grafana`
- `opentelemetry-collector` and `opentelemetry-collector-latest`
- `kube-state-metrics`, `node-exporter`, `metrics-server`
- `kube-prometheus-stack` (Prometheus alert rules)

The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as *data-pipeline* changes, not static config.

## Versioned siblings

`*-latest` siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See [`wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [../platform/procedures/blue-green-chart-migration.md](../../platform/procedures/blue-green-chart-migration.md).

## Cardinality discipline

Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:

- New `extraLabels` / `externalLabels` on `victoria-metrics-agent`.
- New `relabel_configs` that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
- New scrape targets in `additionalScrapeConfigs`.

If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.

## PromQL / alert-rule validation

Alert rules live in `kube-prometheus-stack`, `vmalert`, and Mimir ruler config. Before merging:

1. Render with `helm template`:
   ```
   helm template prom helm-templates/kube-prometheus-stack \
     -f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml
   ```
2. Extract the `PrometheusRule` objects and validate PromQL with `promtool check rules <file.yaml>` (Prometheus tooling) or `vmalert -dryRun -rule=<file.yaml>` for vmalert-specific rules.
3. Confirm the rule's `for:` window and `severity` label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.

## Retention and tenancy

- VictoriaMetrics `vmstorage.retentionPeriod` is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC `resources.requests.storage`.
- Loki and Mimir multi-tenancy is keyed on the `X-Scope-OrgID` header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
- Tempo trace retention is set via `compactor.compaction.block_retention`. Default is short (24h–48h); long retention is opt-in per cluster.

## Grafana dashboards & datasources

- Datasources are declared in the cluster's `grafana/custom-values.yaml` under `datasources.datasources.yaml`. Pin URLs to in-cluster Service DNS, not external endpoints.
- Dashboards bundled via `dashboardProviders` reference ConfigMaps; uniqueness of `uid` matters for panel-image links and alert dashboards.

## Validation checklist (always)

Before raising a PR that touches an observability chart:

- [ ] `helm template` renders without error.
- [ ] PromQL in any new alert rule passes `promtool check rules`.
- [ ] No new high-cardinality label is added without a bound.
- [ ] Retention / PVC sizing has not been changed silently.
- [ ] If touching a `*-latest` sibling, the matching Argo Application points at it. See [argocd.md](./argocd.md).

## Schema references

- Per-cluster scheduling fields (must be rewritten, never copy-pasted): [../../platform/schemas/custom-values-schema.md](../../platform/schemas/custom-values-schema.md)
- Raw manifest sidecars (e.g., `external-dns-services/*.yaml` for Grafana ingress): [../../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md)
- Storage backing for `vmstorage`, `loki` chunks, `tempo` blocks: [../../platform/schemas/storageclass-priorityclass-schema.md](../../platform/schemas/storageclass-priorityclass-schema.md)

## See also

- [helm-values.md](./helm-values.md) — values-file conventions
- [argocd.md](./argocd.md) — Argo CD interaction model
- [../escalation-matrix.md](../escalation-matrix.md)
- [../../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md)