devops-infra-helm-charts-gcp/docs/platform/procedures/modify-observability-config.md

> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.

# Procedure — Modify observability stack config

> **Layer:** Layer 1 — values diff + PR.
> **Blast radius:** one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU).
> **Approval:** observability owner (`siddharth.pal@meesho.com`) + cluster owner.

This procedure covers edits to the cluster-by-cluster observability override files:

| Override path | What it controls |
|---------------|------------------|
| `helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml` | Scrape targets, relabel rules, remote-write tenancy. |
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmstorage retention, vmselect/vminsert sizing, ruler config. |
| `helm-overrides/<cluster>/mimir/custom-values.yaml` (also `mimir-distributed`) | Mimir distributor/ingester/ruler config and tenancy. |
| `helm-overrides/<cluster>/loki/custom-values.yaml` (also `loki-distributed`) | Loki retention, ingestion limits, multi-tenant config. |
| `helm-overrides/<cluster>/tempo/custom-values.yaml` (also `tempo-distributed`) | Tempo block retention, distributor sizing. |
| `helm-overrides/<cluster>/grafana/custom-values.yaml` | Datasources, dashboard providers, plugins. |
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert alerting rules and notifier config. |
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` (or `kube-prometheus-stack`) | Prometheus alert rules, scrape config, retention. |

All of these are **versioned-sibling-aware** — confirm whether the cluster's Argo Application points at `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`, etc., before editing. See [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md) for stack conventions.

---

## When to use

- Adding/removing scrape targets, relabel rules, recording rules.
- Changing retention (`vmstorage.retentionPeriod`, Loki `retention_period`, Tempo `compaction.block_retention`).
- Adding/removing/tuning alert rules in `vmalert` or `kube-prometheus-stack`.
- Adding/removing Grafana datasources, dashboards, plugins.
- Tuning ingester/distributor sizing for Mimir/Loki/Tempo.

Do **not** use this procedure for:

- **Bumping the chart version** of an observability tool — use [update-chart-version.md](update-chart-version.md).
- **Onboarding a brand-new observability tool** to a cluster — use [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
- **A blue-green migration** between sibling charts (`-latest`) — use [blue-green-chart-migration.md](blue-green-chart-migration.md).

---

## Inputs

| Input | Example |
|-------|---------|
| Target cluster | `k8s-supply-prd-ase1` |
| Target chart | `victoria-metrics-cluster` |
| Versioned-sibling target (if applicable) | `victoria-metrics-cluster-latest` |
| Change kind | `retention bump`, `new scrape target`, `new alert rule`, `dashboard add` |
| Promql expression(s) touched (if any) | `sum(rate(...)) by (job) > 0.05` |
| Approval ticket | CMR-1234 (if BU policy requires) |

---

## Pre-conditions

- [ ] The cluster directory and chart override exist.
- [ ] The cluster's Argo `Application` points at the chart you're editing (and not its sibling). Check `github.com/Meesho/devops-infra-argo-config`.
- [ ] You have the previous PR diff for context (most observability edits are touching a known knob).
- [ ] You have access to PromQL/promtool (or vmalert binary) locally for rule validation.

---

## Steps

### 1. Confirm which sibling the cluster runs

```bash
# In the sister repo
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*
```

Find the `Application` whose `spec.source.path` points at this repo. Note whether it's `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`. **Editing the wrong one is silent** — the file diff merges, but no cluster picks it up.

### 2. Read the current values

```bash
yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less
```

Note current retention, scrape targets, and any inline alert rules.

### 3. Author the change

Edit `helm-overrides/<cluster>/<chart>/custom-values.yaml`. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.

#### Cardinality discipline

If the change adds a label, scrape target, or `relabel_configs` rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See [observability.md §Cardinality discipline](../../global/coding-guidelines/observability.md).

#### Retention changes

Increasing `vmstorage.retentionPeriod`, Loki `retention_period`, or Tempo `block_retention` grows the backing PVC. Confirm `persistence.size` (or `vmstorage.persistentVolume.size`) has been bumped to match — otherwise vmstorage runs out of disk silently.

#### Alert-rule edits

If the rule's `for:` window or `severity` label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.

### 4. Render the chart locally

```bash
helm template <release> helm-templates/<chart> \
  -f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
```

The render must succeed. If it errors, fix the values before continuing.

### 5. Validate any PromQL touched

For Prometheus/`kube-prometheus-stack`:

```bash
# Extract PrometheusRule objects from rendered output
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml

# Validate
promtool check rules /tmp/rules.yaml
```

For vmalert:

```bash
# Extract VMRule (or ConfigMap with rules)
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml

# vmalert dry-run
vmalert -dryRun -rule=/tmp/vmrules.yaml
```

If `promtool` / `vmalert` is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.

### 6. (Grafana) Validate datasource URL is in-cluster

Datasources should point at in-cluster Service DNS (e.g. `http://victoria-metrics-cluster-vmselect:8481`), not external endpoints. **Never** pin a Grafana datasource at `*.meeshogcp.in`, `prd.meesho.int`, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.

### 7. Open the PR

```bash
git checkout -b obs/<cluster>-<chart>-<short-desc>
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
git commit -m "obs(<cluster>/<chart>): <short desc>"
git push origin obs/<cluster>-<chart>-<short-desc>
gh pr create --base main
```

### PR description template

```markdown
## Summary
<one-line: what changed and why>

## Cluster × chart
- Cluster: `<cluster>`
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`

## Validation
- [ ] `helm template` renders cleanly
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
- [ ] Cardinality bounded (no unbounded label introduced)
- [ ] Retention/PVC headroom confirmed (if retention changed)
- [ ] PagerDuty routing unchanged (if alert rule changed)

## Approvers
- Observability owner: <handle>
- Cluster owner: <handle>

## Sister repo
- N/A (no Application change required)
```

### 8. After merge

Argo CD on the target cluster reconciles. Most observability charts use **manual sync** (see [observability.md](../../global/coding-guidelines/observability.md)) — open the cluster's Argo CD UI, find the Application, click **Sync**. Watch:

```bash
kubectl --context=<ctx> -n <observability-ns> get pods -w
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50
```

If a metric goes missing post-sync → [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).

---

## Anti-patterns

1. **Editing the wrong sibling.** The Argo Application points at `*-latest`; you edit the stable chart's values. Diff merges, nothing applies.
2. **Adding `pod_name` / `request_id` / `user_id` as a label** without aggregation — explodes cardinality.
3. **Bumping retention without bumping PVC.** vmstorage runs out of disk; ingest fails silently.
4. **Silent PagerDuty re-route.** Changing `severity:` from `warning` to `critical` (or vice versa) without coordinating on-call.
5. **Inlining production hostnames** as Grafana datasource URLs — see [SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3.
6. **Cross-cluster normalising** — touching every cluster's `custom-values.yaml` in one PR. Surgical only; one cluster per PR.

---

## Rollback

- Revert the values PR. Argo CD will re-render with the previous values; click Sync.
- For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the *config* but not the *data*.

---

## Related

- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
- Procedure: [modify-alert-rules.md](modify-alert-rules.md) — alert-only edits.
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
- ADR: [../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
- Wiki: [../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md](../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md).