79 lines
4.6 KiB
Markdown
79 lines
4.6 KiB
Markdown
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
|
||
|
||
# Coding Guideline — Observability stack overrides
|
||
|
||
Conventions for editing values for the observability charts vendored in this repo:
|
||
|
||
- `victoria-metrics-cluster` and `victoria-metrics-cluster-latest`
|
||
- `victoria-metrics-agent` and `victoria-metrics-agent-latest`
|
||
- `vmalert`, `victoria-metrics-operator`
|
||
- `mimir-distributed`
|
||
- `loki`, `loki-distributed`
|
||
- `tempo`, `tempo-distributed`
|
||
- `grafana`
|
||
- `opentelemetry-collector` and `opentelemetry-collector-latest`
|
||
- `kube-state-metrics`, `node-exporter`, `metrics-server`
|
||
- `kube-prometheus-stack` (Prometheus alert rules)
|
||
|
||
The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as *data-pipeline* changes, not static config.
|
||
|
||
## Versioned siblings
|
||
|
||
`*-latest` siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See [`wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [../platform/procedures/blue-green-chart-migration.md](../../platform/procedures/blue-green-chart-migration.md).
|
||
|
||
## Cardinality discipline
|
||
|
||
Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:
|
||
|
||
- New `extraLabels` / `externalLabels` on `victoria-metrics-agent`.
|
||
- New `relabel_configs` that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
|
||
- New scrape targets in `additionalScrapeConfigs`.
|
||
|
||
If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.
|
||
|
||
## PromQL / alert-rule validation
|
||
|
||
Alert rules live in `kube-prometheus-stack`, `vmalert`, and Mimir ruler config. Before merging:
|
||
|
||
1. Render with `helm template`:
|
||
```
|
||
helm template prom helm-templates/kube-prometheus-stack \
|
||
-f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml
|
||
```
|
||
2. Extract the `PrometheusRule` objects and validate PromQL with `promtool check rules <file.yaml>` (Prometheus tooling) or `vmalert -dryRun -rule=<file.yaml>` for vmalert-specific rules.
|
||
3. Confirm the rule's `for:` window and `severity` label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.
|
||
|
||
## Retention and tenancy
|
||
|
||
- VictoriaMetrics `vmstorage.retentionPeriod` is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC `resources.requests.storage`.
|
||
- Loki and Mimir multi-tenancy is keyed on the `X-Scope-OrgID` header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
|
||
- Tempo trace retention is set via `compactor.compaction.block_retention`. Default is short (24h–48h); long retention is opt-in per cluster.
|
||
|
||
## Grafana dashboards & datasources
|
||
|
||
- Datasources are declared in the cluster's `grafana/custom-values.yaml` under `datasources.datasources.yaml`. Pin URLs to in-cluster Service DNS, not external endpoints.
|
||
- Dashboards bundled via `dashboardProviders` reference ConfigMaps; uniqueness of `uid` matters for panel-image links and alert dashboards.
|
||
|
||
## Validation checklist (always)
|
||
|
||
Before raising a PR that touches an observability chart:
|
||
|
||
- [ ] `helm template` renders without error.
|
||
- [ ] PromQL in any new alert rule passes `promtool check rules`.
|
||
- [ ] No new high-cardinality label is added without a bound.
|
||
- [ ] Retention / PVC sizing has not been changed silently.
|
||
- [ ] If touching a `*-latest` sibling, the matching Argo Application points at it. See [argocd.md](./argocd.md).
|
||
|
||
## Schema references
|
||
|
||
- Per-cluster scheduling fields (must be rewritten, never copy-pasted): [../../platform/schemas/custom-values-schema.md](../../platform/schemas/custom-values-schema.md)
|
||
- Raw manifest sidecars (e.g., `external-dns-services/*.yaml` for Grafana ingress): [../../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md)
|
||
- Storage backing for `vmstorage`, `loki` chunks, `tempo` blocks: [../../platform/schemas/storageclass-priorityclass-schema.md](../../platform/schemas/storageclass-priorityclass-schema.md)
|
||
|
||
## See also
|
||
|
||
- [helm-values.md](./helm-values.md) — values-file conventions
|
||
- [argocd.md](./argocd.md) — Argo CD interaction model
|
||
- [../escalation-matrix.md](../escalation-matrix.md)
|
||
- [../../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md)
|