gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b8575bb8b9a28e3c2f6deef59947edc4a74ff357
devops-infra-helm-charts-gcp/docs/platform/procedures/modify-observability-config.md
T
Your Name b8575bb8b9 added repo
2026-08-26 03:39:42 +05:30

9.3 KiB
Raw Blame History

Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.

Procedure — Modify observability stack config

Layer: Layer 1 — values diff + PR. Blast radius: one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU). Approval: observability owner (siddharth.pal@meesho.com) + cluster owner.

This procedure covers edits to the cluster-by-cluster observability override files:

Override path What it controls
helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml Scrape targets, relabel rules, remote-write tenancy.
helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml vmstorage retention, vmselect/vminsert sizing, ruler config.
helm-overrides/<cluster>/mimir/custom-values.yaml (also mimir-distributed) Mimir distributor/ingester/ruler config and tenancy.
helm-overrides/<cluster>/loki/custom-values.yaml (also loki-distributed) Loki retention, ingestion limits, multi-tenant config.
helm-overrides/<cluster>/tempo/custom-values.yaml (also tempo-distributed) Tempo block retention, distributor sizing.
helm-overrides/<cluster>/grafana/custom-values.yaml Datasources, dashboard providers, plugins.
helm-overrides/<cluster>/vmalert/custom-values.yaml vmalert alerting rules and notifier config.
helm-overrides/<cluster>/prometheus/custom-values.yaml (or kube-prometheus-stack) Prometheus alert rules, scrape config, retention.

All of these are versioned-sibling-aware — confirm whether the cluster's Argo Application points at victoria-metrics-cluster or victoria-metrics-cluster-latest, etc., before editing. See ../../global/coding-guidelines/observability.md for stack conventions.

When to use

  • Adding/removing scrape targets, relabel rules, recording rules.
  • Changing retention (vmstorage.retentionPeriod, Loki retention_period, Tempo compaction.block_retention).
  • Adding/removing/tuning alert rules in vmalert or kube-prometheus-stack.
  • Adding/removing Grafana datasources, dashboards, plugins.
  • Tuning ingester/distributor sizing for Mimir/Loki/Tempo.

Do not use this procedure for:

Inputs

Input Example
Target cluster k8s-supply-prd-ase1
Target chart victoria-metrics-cluster
Versioned-sibling target (if applicable) victoria-metrics-cluster-latest
Change kind retention bump, new scrape target, new alert rule, dashboard add
Promql expression(s) touched (if any) sum(rate(...)) by (job) > 0.05
Approval ticket CMR-1234 (if BU policy requires)

Pre-conditions

  • The cluster directory and chart override exist.
  • The cluster's Argo Application points at the chart you're editing (and not its sibling). Check github.com/Meesho/devops-infra-argo-config.
  • You have the previous PR diff for context (most observability edits are touching a known knob).
  • You have access to PromQL/promtool (or vmalert binary) locally for rule validation.

Steps

1. Confirm which sibling the cluster runs

# In the sister repo
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*

Find the Application whose spec.source.path points at this repo. Note whether it's victoria-metrics-cluster or victoria-metrics-cluster-latest. Editing the wrong one is silent — the file diff merges, but no cluster picks it up.

2. Read the current values

yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less

Note current retention, scrape targets, and any inline alert rules.

3. Author the change

Edit helm-overrides/<cluster>/<chart>/custom-values.yaml. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.

Cardinality discipline

If the change adds a label, scrape target, or relabel_configs rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See observability.md §Cardinality discipline.

Retention changes

Increasing vmstorage.retentionPeriod, Loki retention_period, or Tempo block_retention grows the backing PVC. Confirm persistence.size (or vmstorage.persistentVolume.size) has been bumped to match — otherwise vmstorage runs out of disk silently.

Alert-rule edits

If the rule's for: window or severity label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.

4. Render the chart locally

helm template <release> helm-templates/<chart> \
  -f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml

The render must succeed. If it errors, fix the values before continuing.

5. Validate any PromQL touched

For Prometheus/kube-prometheus-stack:

# Extract PrometheusRule objects from rendered output
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml

# Validate
promtool check rules /tmp/rules.yaml

For vmalert:

# Extract VMRule (or ConfigMap with rules)
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml

# vmalert dry-run
vmalert -dryRun -rule=/tmp/vmrules.yaml

If promtool / vmalert is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.

6. (Grafana) Validate datasource URL is in-cluster

Datasources should point at in-cluster Service DNS (e.g. http://victoria-metrics-cluster-vmselect:8481), not external endpoints. Never pin a Grafana datasource at *.meeshogcp.in, prd.meesho.int, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.

7. Open the PR

git checkout -b obs/<cluster>-<chart>-<short-desc>
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
git commit -m "obs(<cluster>/<chart>): <short desc>"
git push origin obs/<cluster>-<chart>-<short-desc>
gh pr create --base main

PR description template

## Summary
<one-line: what changed and why>

## Cluster × chart
- Cluster: `<cluster>`
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`

## Validation
- [ ] `helm template` renders cleanly
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
- [ ] Cardinality bounded (no unbounded label introduced)
- [ ] Retention/PVC headroom confirmed (if retention changed)
- [ ] PagerDuty routing unchanged (if alert rule changed)

## Approvers
- Observability owner: <handle>
- Cluster owner: <handle>

## Sister repo
- N/A (no Application change required)

8. After merge

Argo CD on the target cluster reconciles. Most observability charts use manual sync (see observability.md) — open the cluster's Argo CD UI, find the Application, click Sync. Watch:

kubectl --context=<ctx> -n <observability-ns> get pods -w
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50

If a metric goes missing post-sync → ../runbooks/metrics-gap.md.

Anti-patterns

  1. Editing the wrong sibling. The Argo Application points at *-latest; you edit the stable chart's values. Diff merges, nothing applies.
  2. Adding pod_name / request_id / user_id as a label without aggregation — explodes cardinality.
  3. Bumping retention without bumping PVC. vmstorage runs out of disk; ingest fails silently.
  4. Silent PagerDuty re-route. Changing severity: from warning to critical (or vice versa) without coordinating on-call.
  5. Inlining production hostnames as Grafana datasource URLs — see SANCTITY_RULES.md R3.
  6. Cross-cluster normalising — touching every cluster's custom-values.yaml in one PR. Surgical only; one cluster per PR.

Rollback

  • Revert the values PR. Argo CD will re-render with the previous values; click Sync.
  • For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the config but not the data.

Related

Reference in New Issue View Git Blame Copy Permalink