added repo
This commit is contained in:
Your Name
@@ -0,0 +1,210 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify observability stack config
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU).
|
||||
> **Approval:** observability owner (`siddharth.pal@meesho.com`) + cluster owner.
|
||||
|
||||
This procedure covers edits to the cluster-by-cluster observability override files:
|
||||
|
||||
| Override path | What it controls |
|
||||
|---------------|------------------|
|
||||
| `helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml` | Scrape targets, relabel rules, remote-write tenancy. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmstorage retention, vmselect/vminsert sizing, ruler config. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` (also `mimir-distributed`) | Mimir distributor/ingester/ruler config and tenancy. |
|
||||
| `helm-overrides/<cluster>/loki/custom-values.yaml` (also `loki-distributed`) | Loki retention, ingestion limits, multi-tenant config. |
|
||||
| `helm-overrides/<cluster>/tempo/custom-values.yaml` (also `tempo-distributed`) | Tempo block retention, distributor sizing. |
|
||||
| `helm-overrides/<cluster>/grafana/custom-values.yaml` | Datasources, dashboard providers, plugins. |
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert alerting rules and notifier config. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` (or `kube-prometheus-stack`) | Prometheus alert rules, scrape config, retention. |
|
||||
|
||||
All of these are **versioned-sibling-aware** — confirm whether the cluster's Argo Application points at `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`, etc., before editing. See [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md) for stack conventions.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing scrape targets, relabel rules, recording rules.
|
||||
- Changing retention (`vmstorage.retentionPeriod`, Loki `retention_period`, Tempo `compaction.block_retention`).
|
||||
- Adding/removing/tuning alert rules in `vmalert` or `kube-prometheus-stack`.
|
||||
- Adding/removing Grafana datasources, dashboards, plugins.
|
||||
- Tuning ingester/distributor sizing for Mimir/Loki/Tempo.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- **Bumping the chart version** of an observability tool — use [update-chart-version.md](update-chart-version.md).
|
||||
- **Onboarding a brand-new observability tool** to a cluster — use [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
||||
- **A blue-green migration** between sibling charts (`-latest`) — use [blue-green-chart-migration.md](blue-green-chart-migration.md).
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `victoria-metrics-cluster` |
|
||||
| Versioned-sibling target (if applicable) | `victoria-metrics-cluster-latest` |
|
||||
| Change kind | `retention bump`, `new scrape target`, `new alert rule`, `dashboard add` |
|
||||
| Promql expression(s) touched (if any) | `sum(rate(...)) by (job) > 0.05` |
|
||||
| Approval ticket | CMR-1234 (if BU policy requires) |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The cluster directory and chart override exist.
|
||||
- [ ] The cluster's Argo `Application` points at the chart you're editing (and not its sibling). Check `github.com/Meesho/devops-infra-argo-config`.
|
||||
- [ ] You have the previous PR diff for context (most observability edits are touching a known knob).
|
||||
- [ ] You have access to PromQL/promtool (or vmalert binary) locally for rule validation.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Confirm which sibling the cluster runs
|
||||
|
||||
```bash
|
||||
# In the sister repo
|
||||
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*
|
||||
```
|
||||
|
||||
Find the `Application` whose `spec.source.path` points at this repo. Note whether it's `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`. **Editing the wrong one is silent** — the file diff merges, but no cluster picks it up.
|
||||
|
||||
### 2. Read the current values
|
||||
|
||||
```bash
|
||||
yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less
|
||||
```
|
||||
|
||||
Note current retention, scrape targets, and any inline alert rules.
|
||||
|
||||
### 3. Author the change
|
||||
|
||||
Edit `helm-overrides/<cluster>/<chart>/custom-values.yaml`. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.
|
||||
|
||||
#### Cardinality discipline
|
||||
|
||||
If the change adds a label, scrape target, or `relabel_configs` rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See [observability.md §Cardinality discipline](../../global/coding-guidelines/observability.md).
|
||||
|
||||
#### Retention changes
|
||||
|
||||
Increasing `vmstorage.retentionPeriod`, Loki `retention_period`, or Tempo `block_retention` grows the backing PVC. Confirm `persistence.size` (or `vmstorage.persistentVolume.size`) has been bumped to match — otherwise vmstorage runs out of disk silently.
|
||||
|
||||
#### Alert-rule edits
|
||||
|
||||
If the rule's `for:` window or `severity` label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.
|
||||
|
||||
### 4. Render the chart locally
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
The render must succeed. If it errors, fix the values before continuing.
|
||||
|
||||
### 5. Validate any PromQL touched
|
||||
|
||||
For Prometheus/`kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
# Extract PrometheusRule objects from rendered output
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
|
||||
# Validate
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
# Extract VMRule (or ConfigMap with rules)
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
|
||||
# vmalert dry-run
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
If `promtool` / `vmalert` is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.
|
||||
|
||||
### 6. (Grafana) Validate datasource URL is in-cluster
|
||||
|
||||
Datasources should point at in-cluster Service DNS (e.g. `http://victoria-metrics-cluster-vmselect:8481`), not external endpoints. **Never** pin a Grafana datasource at `*.meeshogcp.in`, `prd.meesho.int`, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.
|
||||
|
||||
### 7. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b obs/<cluster>-<chart>-<short-desc>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "obs(<cluster>/<chart>): <short desc>"
|
||||
git push origin obs/<cluster>-<chart>-<short-desc>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<one-line: what changed and why>
|
||||
|
||||
## Cluster × chart
|
||||
- Cluster: `<cluster>`
|
||||
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
|
||||
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`
|
||||
|
||||
## Validation
|
||||
- [ ] `helm template` renders cleanly
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
|
||||
- [ ] Cardinality bounded (no unbounded label introduced)
|
||||
- [ ] Retention/PVC headroom confirmed (if retention changed)
|
||||
- [ ] PagerDuty routing unchanged (if alert rule changed)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
|
||||
## Sister repo
|
||||
- N/A (no Application change required)
|
||||
```
|
||||
|
||||
### 8. After merge
|
||||
|
||||
Argo CD on the target cluster reconciles. Most observability charts use **manual sync** (see [observability.md](../../global/coding-guidelines/observability.md)) — open the cluster's Argo CD UI, find the Application, click **Sync**. Watch:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> -n <observability-ns> get pods -w
|
||||
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50
|
||||
```
|
||||
|
||||
If a metric goes missing post-sync → [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Editing the wrong sibling.** The Argo Application points at `*-latest`; you edit the stable chart's values. Diff merges, nothing applies.
|
||||
2. **Adding `pod_name` / `request_id` / `user_id` as a label** without aggregation — explodes cardinality.
|
||||
3. **Bumping retention without bumping PVC.** vmstorage runs out of disk; ingest fails silently.
|
||||
4. **Silent PagerDuty re-route.** Changing `severity:` from `warning` to `critical` (or vice versa) without coordinating on-call.
|
||||
5. **Inlining production hostnames** as Grafana datasource URLs — see [SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3.
|
||||
6. **Cross-cluster normalising** — touching every cluster's `custom-values.yaml` in one PR. Surgical only; one cluster per PR.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the values PR. Argo CD will re-render with the previous values; click Sync.
|
||||
- For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the *config* but not the *data*.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
- Procedure: [modify-alert-rules.md](modify-alert-rules.md) — alert-only edits.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- ADR: [../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- Wiki: [../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md](../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md).
|
||||
Reference in New Issue
Block a user