> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.

# Runbook — Metrics gap on a cluster / namespace

> **Type:** Decision tree.
> **Entry symptom:** "Grafana panels are blank for `<cluster>` / `<namespace>` / `<service>`," or an alert that should be firing isn't, or VictoriaMetrics shows `no data`.
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.

The observability path on Meesho's GKE fleet is:

```
workload Pod (exposes /metrics)
   └→ victoria-metrics-agent (vmagent) scrapes
       └→ remote_write to victoria-metrics-cluster (vmstorage)
           └→ vmselect ← Grafana / vmalert query
```

A metrics gap can be at any hop. Walk this tree top-down — the most common root cause is hop 1 (scrape config).

---

## Entry — gather context

```bash
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<workload-namespace-where-metric-is-missing>
METRIC=<metric-name-or-job-label>

# Confirm the cluster runs VM agent + cluster
ls helm-overrides/$CLUSTER | grep -E '^victoria-metrics-(agent|cluster)'

# Confirm pods are healthy
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
```

If pods are missing/CrashLooping → that's the gap. Skip to §5.

---

## Decision tree

```text
START
  │
  ├── Is the metric known to be emitted by the workload?
  │     ├── No  → §0 — Workload not emitting; out of scope (app team)
  │     └── Yes →
  │
  ├── §1 — Is vmagent scraping the workload?
  │     ├── No  → fix scrape config (Layer 1)
  │     └── Yes →
  │
  ├── §2 — Are scrape targets healthy (status=up)?
  │     ├── Down → fix endpoint reachability (Layer 2 / Layer 1)
  │     └── Up →
  │
  ├── §3 — Are relabel rules dropping the metric?
  │     ├── Yes → adjust relabel_configs (Layer 1)
  │     └── No →
  │
  ├── §4 — Is remote_write succeeding?
  │     ├── No  → vmagent → vmstorage path broken (Layer 2 / Layer 1)
  │     └── Yes →
  │
  ├── §5 — Is vmstorage healthy and ingesting?
  │     ├── No  → vmstorage outage (Layer 2)
  │     └── Yes →
  │
  └── §6 — Is the Grafana datasource / tenant correct?
        └── Misrouted query → fix datasource URL or tenant header (Layer 1)
```

---

## §0 — Workload not emitting

Out of scope for this repo. Confirm with:

```bash
# Port-forward and curl /metrics directly
kubectl --context=$CTX -n $NS port-forward <pod> 9090:<metrics-port> &
curl -s localhost:9090/metrics | grep -i "$METRIC"
```

If `/metrics` is empty or doesn't contain `$METRIC` → app team. Stop.

---

## §1 — Is vmagent scraping the workload?

```bash
# vmagent UI exposes /api/v1/targets
kubectl --context=$CTX -n monitoring port-forward svc/victoria-metrics-agent 8429:8429 &
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.labels.namespace=="'$NS'")'
```

If no targets for `$NS`:

- Look at `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml`.
- Check the `additionalScrapeConfigs` (or `config.scrape_configs`) for a job that matches the workload's labels / namespace selector.
- Common cause: a `kubernetes_sd_configs` `namespaces.names` filter excludes `$NS`.
- Common cause: a missing `Pod`/`Service`/`PodMonitor` annotation `prometheus.io/scrape: "true"` on the workload.

**Layer 1 fix** (if the gap is in the scrape config): edit `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml` per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).

**Layer 2 fix** (if the gap is on the workload — missing annotation): hand off to app team.

---

## §2 — Targets healthy?

```bash
# In the same vmagent /targets output
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up") | {labels, lastError}'
```

If targets are `down`:

| `lastError` | Cause | Fix |
|-------------|-------|-----|
| `connection refused` | Workload not listening on declared port | App team — Layer 2. |
| `i/o timeout` | NetworkPolicy / firewall blocking vmagent → workload | Check `NetworkPolicy` in `$NS`. Often a NetworkPolicy allowing only intra-namespace traffic and not vmagent's namespace. **Layer 2** — recommend the workload team allow vmagent. |
| `x509: certificate signed by unknown authority` | mTLS misconfigured | App team — Layer 2. |
| `404 Not Found` | Wrong path (default `/metrics` vs custom) | Add `metrics_path:` in scrape config. **Layer 1.** |

---

## §3 — Relabel rules dropping the metric?

```bash
yq e '.config.scrape_configs[].metric_relabel_configs, .config.scrape_configs[].relabel_configs' \
  helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml
```

Look for:

- `action: drop` rules with regexes matching `$METRIC`.
- `action: keep` rules whose regex *excludes* `$METRIC`.
- `action: labeldrop` removing a label the query uses.

Test in vmagent's UI under the *Targets* tab — it shows the labels post-relabel.

**Layer 1 fix:** loosen the relabel rule. Re-PR per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).

---

## §4 — Remote_write succeeding?

```bash
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent | grep -i 'remote_write\|error\|failed' | tail -20
```

| Symptom | Cause | Fix |
|---------|-------|-----|
| `429 Too Many Requests` from vmstorage | vmstorage ingest saturated | Scale vmstorage / vminsert (Layer 1 — see [observability.md](../../global/coding-guidelines/observability.md)). |
| `connection refused` to vmstorage URL | vmstorage Service down or wrong URL | Verify `remoteWrite.url` in vmagent values matches the live `vmstorage` Service DNS. Layer 1. |
| `out of bounds timestamp` | Clock skew on vmagent's node | Layer 2 — node time sync. |
| `series limit exceeded` | Cardinality bomb on vmstorage tenant | Layer 1 — drop the offending label. See [observability.md §Cardinality](../../global/coding-guidelines/observability.md). |

---

## §5 — vmstorage healthy?

```bash
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
kubectl --context=$CTX -n monitoring describe pod vmstorage-0 | tail -30
kubectl --context=$CTX -n monitoring exec vmstorage-0 -- df -h /storage
```

Failure modes:

| Symptom | Layer | Fix |
|---------|-------|-----|
| Pod CrashLooping with `out of disk` | Layer 1 | Bump `persistence.size`. Note: PVC growth requires the StorageClass to support `allowVolumeExpansion: true`. See [../schemas/storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md). |
| Pod CrashLooping with retention/index errors | Layer 2 | Escalate — may need data-side intervention. |
| Pod Pending | Layer 1 | Scheduling issue. See [pod-pending-scheduling.md](pod-pending-scheduling.md) and [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md). |
| Pod Running but vmselect can't reach it | Layer 1 / 2 | Verify the headless Service and StatefulSet pod-DNS records. |

---

## §6 — Grafana datasource / tenant correct?

```bash
yq e '.datasources.datasources.yaml.datasources[] | select(.name == "*VictoriaMetrics*" or .type == "prometheus")' \
  helm-overrides/$CLUSTER/grafana/custom-values.yaml
```

| Sub-check | Action |
|-----------|--------|
| `url:` points at the correct in-cluster vmselect Service DNS | If wrong, **Layer 1** — fix the values per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md). |
| `httpHeaderName1: X-Scope-OrgID` (multi-tenant clusters only) | If the cluster runs multi-tenant VM, the tenant header must be set. Layer 1 fix. |
| Datasource `url:` points at an external `*.meeshogcp.in` host | Forbidden — see [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3. Repoint at in-cluster Service DNS. |

---

## Remediation summary

| Hop | Likely fix | Layer | Procedure |
|-----|-----------|-------|-----------|
| §1 scrape | Add scrape config / fix selector | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §2 endpoint | Workload-side / NetworkPolicy | 2 | Hand off to app team |
| §3 relabel | Loosen drop rule | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §4 remote_write | Scale vmstorage / fix URL | 1/2 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §5 vmstorage | PVC grow / scheduling fix | 1/2 | [pod-pending-scheduling.md](pod-pending-scheduling.md) |
| §6 datasource | Fix Grafana datasource | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |

---

## Escalation triggers

- §5 vmstorage outage with no obvious values-side fix → platform team (observability owner).
- Multi-cluster simultaneous metrics gap → platform team — cluster-level / control-plane issue.
- Cardinality explosion impacting vmstorage stability → platform team + workload owner — joint fix.

---

## Done conditions

- The query that was returning `no data` returns the expected series in Grafana Explore.
- No remote_write errors in vmagent logs for at least 5 minutes post-fix.
- Alerts that depend on the metric have transitioned from `pending` / silent back to expected state.

---

## Related

- Procedure: [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
- Procedure: [../procedures/modify-alert-rules.md](../procedures/modify-alert-rules.md) — if the gap is "alert silent" not "metric missing."
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — for §5 vmstorage scheduling failures.
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if a sync didn't take.