added repo

2026-08-26 03:39:42 +05:30
parent 45c25a95af
commit b8575bb8b9
6889 changed files with 1217125 additions and 0 deletions
@@ -0,0 +1,78 @@
+> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
+
+# Coding Guideline — Observability stack overrides
+
+Conventions for editing values for the observability charts vendored in this repo:
+
+- `victoria-metrics-cluster` and `victoria-metrics-cluster-latest`
+- `victoria-metrics-agent` and `victoria-metrics-agent-latest`
+- `vmalert`, `victoria-metrics-operator`
+- `mimir-distributed`
+- `loki`, `loki-distributed`
+- `tempo`, `tempo-distributed`
+- `grafana`
+- `opentelemetry-collector` and `opentelemetry-collector-latest`
+- `kube-state-metrics`, `node-exporter`, `metrics-server`
+- `kube-prometheus-stack` (Prometheus alert rules)
+
+The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as *data-pipeline* changes, not static config.
+
+## Versioned siblings
+
+`*-latest` siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See [`wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [../platform/procedures/blue-green-chart-migration.md](../../platform/procedures/blue-green-chart-migration.md).
+
+## Cardinality discipline
+
+Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:
+
+- New `extraLabels` / `externalLabels` on `victoria-metrics-agent`.
+- New `relabel_configs` that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
+- New scrape targets in `additionalScrapeConfigs`.
+
+If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.
+
+## PromQL / alert-rule validation
+
+Alert rules live in `kube-prometheus-stack`, `vmalert`, and Mimir ruler config. Before merging:
+
+1. Render with `helm template`:
+   ```
+   helm template prom helm-templates/kube-prometheus-stack \
+     -f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml
+   ```
+2. Extract the `PrometheusRule` objects and validate PromQL with `promtool check rules <file.yaml>` (Prometheus tooling) or `vmalert -dryRun -rule=<file.yaml>` for vmalert-specific rules.
+3. Confirm the rule's `for:` window and `severity` label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.
+
+## Retention and tenancy
+
+- VictoriaMetrics `vmstorage.retentionPeriod` is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC `resources.requests.storage`.
+- Loki and Mimir multi-tenancy is keyed on the `X-Scope-OrgID` header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
+- Tempo trace retention is set via `compactor.compaction.block_retention`. Default is short (24h–48h); long retention is opt-in per cluster.
+
+## Grafana dashboards & datasources
+
+- Datasources are declared in the cluster's `grafana/custom-values.yaml` under `datasources.datasources.yaml`. Pin URLs to in-cluster Service DNS, not external endpoints.
+- Dashboards bundled via `dashboardProviders` reference ConfigMaps; uniqueness of `uid` matters for panel-image links and alert dashboards.
+
+## Validation checklist (always)
+
+Before raising a PR that touches an observability chart:
+
+- [ ] `helm template` renders without error.
+- [ ] PromQL in any new alert rule passes `promtool check rules`.
+- [ ] No new high-cardinality label is added without a bound.
+- [ ] Retention / PVC sizing has not been changed silently.
+- [ ] If touching a `*-latest` sibling, the matching Argo Application points at it. See [argocd.md](./argocd.md).
+
+## Schema references
+
+- Per-cluster scheduling fields (must be rewritten, never copy-pasted): [../../platform/schemas/custom-values-schema.md](../../platform/schemas/custom-values-schema.md)
+- Raw manifest sidecars (e.g., `external-dns-services/*.yaml` for Grafana ingress): [../../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md)
+- Storage backing for `vmstorage`, `loki` chunks, `tempo` blocks: [../../platform/schemas/storageclass-priorityclass-schema.md](../../platform/schemas/storageclass-priorityclass-schema.md)
+
+## See also
+
+- [helm-values.md](./helm-values.md) — values-file conventions
+- [argocd.md](./argocd.md) — Argo CD interaction model
+- [../escalation-matrix.md](../escalation-matrix.md)
+- [../../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md)