added repo
This commit is contained in:
Your Name
@@ -0,0 +1,231 @@
|
||||
# Runbook — Argo CD Sync Failure (infra release)
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** an Argo CD `Application` for an infra release is `OutOfSync`, errored, or stuck `Progressing`.
|
||||
> **Layer:** mostly Layer 1 (read state, propose YAML diff). Some branches are Layer 2 (advisory).
|
||||
|
||||
This runbook handles sync failures for infra Applications routed by `github.com/Meesho/devops-infra-argo-config` and rendering against this repo's `helm-overrides/<cluster>/<app>/`.
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
APP=<release> # e.g. argocd, contour-internal-0
|
||||
CLUSTER=<cluster-name> # e.g. k8s-supply-prd-ase1
|
||||
NS=$(argocd app get $APP -o json | jq -r '.spec.destination.namespace')
|
||||
PROJECT=$(argocd app get $APP -o json | jq -r '.spec.project')
|
||||
|
||||
argocd app get $APP # headline
|
||||
argocd app get $APP -o json | jq -r '.status.conditions[]?'
|
||||
argocd app get $APP -o json | jq -r '.status.operationState.message // empty'
|
||||
```
|
||||
|
||||
Note which Argo CD instance you're hitting — most infra Applications live in a per-cluster Argo CD install.
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── Is `argocd app get $APP` known to this Argo at all?
|
||||
│
|
||||
├── NO → §1 — Application not found
|
||||
│
|
||||
└── YES → What's the symptom?
|
||||
│
|
||||
├── Sync failed with an error message → §2 — Errored sync
|
||||
├── Sync stuck `Progressing` for >5 min → §3 — Stuck progressing
|
||||
├── App is `OutOfSync` but Sync hasn't run → §4 — OutOfSync only
|
||||
└── `Synced`+`Healthy` but workload bad → §5 — Wrong workload (leave runbook)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Application not found
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Are you on the right Argo CD instance? | Most v2 infra apps live in per-cluster Argo CDs. |
|
||||
| Was the app deboarded recently? | `git -C <argo-config> log --diff-filter=D -- 'apps/<cluster>/<app>*'` and `git log --diff-filter=D -- 'helm-overrides/<cluster>/<app>/'`. |
|
||||
| Did the values directory land on `main`? | `git log --all -- 'helm-overrides/<cluster>/<app>/'`. |
|
||||
|
||||
If the file *should* exist on `main` but the `Application` resource isn't created → **Layer 2** — escalate to the platform team. The cluster's Argo CD bootstrap (`ApplicationSet`) may not be picking up the path.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Errored sync (read the error message)
|
||||
|
||||
### §2a — `repository not accessible / authentication required`
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Check `spec.source.repoURL` is `github.com/Meesho/devops-infra-helm-charts.git`. |
|
||||
| If yes, the credentials in Argo CD's repo-list need refreshing. **Layer 2** — recommend platform team rotates credentials. |
|
||||
|
||||
### §2b — `path 'X' does not exist in repo Y`
|
||||
|
||||
```text
|
||||
path 'helm-overrides/k8s-supply-prd-ase1/argocd' does not exist
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| `ls helm-overrides/<cluster>/<app>/` on `main`. |
|
||||
| If absent → either the values-side PR wasn't merged, or the path was typo'd in the sister-repo `Application`. **Layer 1** — open a fix PR (sister repo). |
|
||||
| If a blue-green migration just landed: the `Application` may be pointing at the **old** chart path that was retired. **Layer 1** — repoint the `Application` to the new sibling path. |
|
||||
|
||||
### §2c — `Helm template error` / `values file not found`
|
||||
|
||||
```text
|
||||
open helm-overrides/<cluster>/<app>/custom-values.yaml: no such file or directory
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Verify the file exists on `main`: `git ls-tree origin/main -- helm-overrides/<cluster>/<app>/custom-values.yaml`. |
|
||||
| If absent → onboarding is incomplete. **Layer 1** — open the missing values-side PR. |
|
||||
|
||||
### §2d — `unable to render manifests` / `template error`
|
||||
|
||||
Helm template error inside the chart (missing required value, type mismatch).
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Reproduce locally: `helm template <app> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml`. |
|
||||
| Determine whether the fix is in this repo (rare — usually values shape changed) or `helm-templates/<chart>` (more common after a chart bump). **Layer 1**. |
|
||||
| If the error is `Cannot use existing release: ...` — see §2h. |
|
||||
|
||||
### §2e — `cluster not found / dial tcp ... no route to host`
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| **Layer 2** — escalate to platform team. Cluster API server unreachable, or cluster-secret stale in Argo CD. |
|
||||
| Do **not** edit `spec.destination.{server,name}` to redirect; that masks the underlying cluster issue. |
|
||||
|
||||
### §2f — `forbidden: ...` / admission webhook deny
|
||||
|
||||
```text
|
||||
admission webhook "validate.kyverno.svc-fail" denied the request
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Look at the rule that denied (Kyverno? PSP? OPA? GKE Autopilot policy?). |
|
||||
| Often the chart's manifest violates a cluster policy (e.g. `runAsUser: 0`, missing `securityContext`). |
|
||||
| **Layer 1** — fix in chart values; pair with the policy team if the policy is wrong. |
|
||||
| GKE Autopilot specifically denies many privileged settings — read the deny message carefully. |
|
||||
|
||||
### §2g — `webhook errored: ... cert-manager / external-secrets / kyverno`
|
||||
|
||||
A webhook that should validate the new resource is itself unhealthy.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| `kubectl get pods -n cert-manager` (or the relevant operator's namespace) — is it running? |
|
||||
| **Layer 2** — recommend recovering the webhook before re-syncing this app. |
|
||||
|
||||
### §2h — `cannot patch ... immutable field`
|
||||
|
||||
Most often: `Deployment.spec.selector` or `StatefulSet.volumeClaimTemplates`. A chart bump that changes labels.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Read the upstream changelog to confirm. |
|
||||
| **Layer 2** — recommend deleting the old `Deployment` / `StatefulSet` (with the workload owner) so the chart can recreate it. **Do not delete blindly** — for `StatefulSet`, the PVCs survive but the rollout is disruptive. |
|
||||
| For systemic immutable-field changes across a chart bump, this is a sign the bump should have been a [blue-green migration](../procedures/blue-green-chart-migration.md). Roll back, plan the migration. |
|
||||
|
||||
### §2i — `dependent CRD ... not installed`
|
||||
|
||||
The chart needs a CRD that doesn't exist yet on the cluster.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Check whether the chart includes the CRD in `templates/crds/` (most upstream charts ship CRDs). |
|
||||
| If yes: the chart's `helm template` may not include CRDs by default — Argo CD has `IncludeCRDs` semantics; check the `Application`'s `helm.skipCrds` setting. |
|
||||
| If the CRD is supposed to come from a different chart (`cert-manager`, `kube-prometheus-stack`): **Layer 2** — sync that chart first. |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Stuck `Progressing` for > 5 minutes
|
||||
|
||||
The sync started but resources aren't reconciling.
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `argocd app get $APP --refresh` shows resource-level status. | Look for `Progressing` resources. |
|
||||
| Is a Deployment failing to roll out? | `kubectl rollout status deploy/<name> -n $NS`. If yes → see [pod-pending-scheduling.md](pod-pending-scheduling.md) or [ingress-down.md](ingress-down.md). |
|
||||
| Is a Job hung? | `kubectl describe job/<name> -n $NS`. Old `Job`s sometimes block syncs (Helm pre-/post-install hooks). |
|
||||
| Is a `PreSync`/`PostSync` hook hanging? | `kubectl get pods -n $NS -l argocd.argoproj.io/hook=PostSync`. |
|
||||
|
||||
If the workload itself is the problem, leave this runbook.
|
||||
|
||||
---
|
||||
|
||||
## §4 — `OutOfSync` only (no error, sync hasn't run)
|
||||
|
||||
Argo CD sees a diff between git and the cluster. **Most infra apps are intentionally manual-sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)).
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is this expected? (e.g. you just merged a PR.) | Click Sync. |
|
||||
| Diff suspicious? (e.g. someone `kubectl edit`-ed.) | `argocd app diff $APP`. If out-of-band edit happened, the GitOps contract was violated; recommend reverting the manual change or capturing it in a PR. |
|
||||
| Diff has sat for > 1 day? | Notify the app owner — manual-sync apps rot if no one clicks. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — `Synced` and `Healthy` but workload misbehaving
|
||||
|
||||
Argo thinks all is fine; the workload is broken. Not a sync failure. **Leave this runbook.**
|
||||
|
||||
| Symptom | Where to go |
|
||||
|---------|-------------|
|
||||
| Pods crashlooping | [pod-pending-scheduling.md](pod-pending-scheduling.md) §3 |
|
||||
| Ingress 5xx | [ingress-down.md](ingress-down.md) |
|
||||
| Specific feature broken | App-team playbook |
|
||||
|
||||
---
|
||||
|
||||
## §6 — Special: blue-green migration in flight
|
||||
|
||||
If this app is in a `<chart>` ↔ `<chart>-<variant>` migration:
|
||||
|
||||
- Confirm which variant the `Application` points at (check `spec.source.path`).
|
||||
- The chart name may have changed in the new variant; release-name pinning via `fullnameOverride` may be required to keep the same Service DNS during cutover.
|
||||
- A failed sync mid-migration is the trigger to roll back (`spec.source.path` ← old) and Sync, not to push forward.
|
||||
- Read [blue-green-chart-migration.md](../procedures/blue-green-chart-migration.md) before deciding.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | Action | Escalate to |
|
||||
|---------|--------|-------------|
|
||||
| §1 + bootstrap looks healthy | Investigate further | App owner |
|
||||
| §2a (repo auth) | Confirm allowed repoURL; rotate creds | Platform team |
|
||||
| §2b (path missing) | Fix in this repo or sister repo | App owner |
|
||||
| §2c, §2d (helm render) | Reproduce; fix values or chart | App owner / platform team |
|
||||
| §2e (cluster unreachable) | Don't edit destination | Platform team |
|
||||
| §2f (admission webhook) | Fix in chart values | App + policy team |
|
||||
| §2g (webhook unhealthy) | Recover the webhook first | Platform team |
|
||||
| §2h (immutable field) | Probably needs blue-green | Platform team |
|
||||
| §3 (stuck > 30 min) | Check pod events; consider workload rollback | App owner |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `argocd app get $APP` shows `Synced` + `Healthy`.
|
||||
- The PR or manual fix that resolved it is on `main`.
|
||||
- If the failure was caused by a regression, a postmortem / RCA is scheduled.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
|
||||
- Runbook: [ingress-down.md](ingress-down.md).
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- Boundaries: [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md).
|
||||
@@ -0,0 +1,197 @@
|
||||
# Runbook — Ingress (Contour) is down on a cluster
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** services on a cluster are returning 5xx, not reachable, or DNS doesn't resolve to working endpoints.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl actions). Layer 1 only when the fix is a values diff in this repo.
|
||||
|
||||
Contour is the ingress for almost every BU cluster, and most clusters run **multiple Contour releases** — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. The matrix is in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Read it before editing any Contour values.**
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
|
||||
# Which Contour instances does this cluster run?
|
||||
ls helm-overrides/$CLUSTER | grep '^contour'
|
||||
|
||||
# Pod state for each Contour
|
||||
for c in $(ls helm-overrides/$CLUSTER | grep '^contour'); do
|
||||
echo "=== $c ==="
|
||||
kubectl --context=$CTX get pods -n projectcontour -l app.kubernetes.io/instance=$c -o wide 2>/dev/null \
|
||||
|| kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c -o wide
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── Which Contour instance is affected?
|
||||
│
|
||||
├── External (`contour-external*`) → §A — North-south traffic
|
||||
├── Internal (`contour-internal-*`) → §B — East-west traffic
|
||||
└── Both / unsure → §C — Cluster-wide (worst case)
|
||||
```
|
||||
|
||||
For each branch:
|
||||
|
||||
```text
|
||||
└── What's the failure shape?
|
||||
│
|
||||
├── Pods Pending → §1 — Scheduling failure
|
||||
├── Pods CrashLooping → §2 — Contour boot failure
|
||||
├── Pods Running but no endpoints → §3 — Service / load-balancer unhealthy
|
||||
├── HTTPS responses are 5xx → §4 — Backend / cert / config error
|
||||
└── DNS doesn't resolve → §5 — external-dns / DNS plumbing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Contour pods Pending
|
||||
|
||||
This is almost always a scheduling-key mismatch. **The single most common Contour incident.**
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <contour-pod>
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common causes:
|
||||
|
||||
| Reason | Fix in |
|
||||
|--------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint <key>=<value>` | Tolerations in `helm-overrides/<cluster>/<contour-instance>/custom-values.yaml`. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | `nodeSelector` in the values. Same fix. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) had no available compute class` (GKE Autopilot) | Either the `ComputeClass` resource is missing, or `nodeSelector: cloud.google.com/compute-class: <X>` references a class that doesn't exist. Check `helm-overrides/<cluster>/<contour-instance>/computeclass/*-cc.yaml`. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu/memory` | Node-pool autoscaler is at max. **Layer 2** — escalate to cluster owner / platform team. |
|
||||
|
||||
The fix is always to bring the values' `nodeSelector` / `tolerations` into agreement with the cluster's actual node-pool topology. Never guess — read the matrix and the cluster's other apps.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Contour pods CrashLooping
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <contour-pod> -c contour --previous
|
||||
kubectl --context=$CTX logs <contour-pod> -c envoy --previous
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `failed to load TLS certificates` | Missing `Secret`, expired cert, wrong key | `cert-manager` / `external-secrets`. **Layer 2.** |
|
||||
| `error parsing config: invalid HTTPProxy` | A user `HTTPProxy` is malformed and Contour refuses to load | The user's namespace. **Layer 2** — recommend `kubectl get httpproxy -A` to find the offender, then fix in the consuming team's repo. |
|
||||
| `bind: address already in use` | Two Contour pods on the same node fighting for the host port | Pod anti-affinity in values. **Layer 1.** |
|
||||
| `error: ratelimit service ... unavailable` | `contour-rate-limit` sidecar / external service is down | Operator action. **Layer 2.** |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Contour Running but no endpoints / LB unhealthy
|
||||
|
||||
The Contour pods are healthy, but downstream LB / DNS / Service routing is broken.
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get svc -n projectcontour
|
||||
kubectl --context=$CTX get endpoints -n projectcontour
|
||||
kubectl --context=$CTX describe svc <contour-svc> -n projectcontour
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Service` of type `LoadBalancer` has no `EXTERNAL-IP` | GCP LB provisioning failed. Check the `Service` annotations match the cluster's LB pattern. **Layer 2** — escalate. |
|
||||
| `Endpoints` empty | The Contour pods aren't matching the `Service` selector. Often a label drift between values and the chart's defaults. **Layer 1** — fix selectors. |
|
||||
| `Service` annotations mention `cloud.google.com/load-balancer-type: Internal` but external traffic is expected | Wrong Contour instance / Service annotation. **Layer 1.** |
|
||||
| `Health checks failing` on the GCP LB | Backend pods aren't ready. See §4. |
|
||||
|
||||
---
|
||||
|
||||
## §4 — HTTPS responses are 5xx
|
||||
|
||||
Don't curl the production endpoint yourself — that violates [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md). Instead, recommend the user check from a controlled vantage:
|
||||
|
||||
```bash
|
||||
# From inside the cluster
|
||||
kubectl --context=$CTX run -it --rm curl-test --image=curlimages/curl --restart=Never -- \
|
||||
curl -v https://<service>.<ns>.svc.cluster.local
|
||||
|
||||
# Contour access logs
|
||||
kubectl --context=$CTX logs <envoy-pod> -c envoy | tail -50
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `503 no_healthy_upstream` | Backend pods all unhealthy | App's pod readiness — see [pod-pending-scheduling.md](pod-pending-scheduling.md). **Layer 2.** |
|
||||
| `502 upstream connect error` | Backend connection refused / TLS mismatch | App config / mTLS. **Layer 2.** |
|
||||
| `404 route not found` | No matching `HTTPProxy`/`Ingress` | The user's `HTTPProxy` is missing or has the wrong host/path. **Layer 2.** |
|
||||
| `500` from the app | Application error | App-team playbook. **Out of scope for this runbook.** |
|
||||
|
||||
---
|
||||
|
||||
## §5 — DNS doesn't resolve
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is `external-dns` running on this cluster? | `kubectl --context=$CTX get pods -n external-dns`. |
|
||||
| Are there `Service` resources with `external-dns.alpha.kubernetes.io/hostname` annotations? | `kubectl --context=$CTX get svc -A -o json \| jq '.items[] \| select(.metadata.annotations."external-dns.alpha.kubernetes.io/hostname")'`. |
|
||||
| Did `external-dns` reconcile recently? | `kubectl --context=$CTX logs deploy/external-dns -n external-dns \| tail -30`. |
|
||||
| Is the Cloud DNS zone wired up? | **Layer 2** — out of scope; escalate to platform team. |
|
||||
|
||||
---
|
||||
|
||||
## §C — Cluster-wide ingress outage
|
||||
|
||||
If both internal and external Contour are degraded simultaneously:
|
||||
|
||||
1. **Stop. Don't iterate values fixes.** This is incident-grade.
|
||||
2. **Layer 2 — escalate to platform team immediately.**
|
||||
3. Check whether a recent merge in this repo or the sister repo correlates: `git log --since='2 hours ago' -- helm-overrides/$CLUSTER/contour*` and the same in `devops-infra-argo-config`.
|
||||
4. If a recent merge is implicated: revert it, click Sync to roll back to the previous state.
|
||||
5. If no recent merge: cluster-level issue (node pool, network policy, GCP LB) — out of scope for this repo.
|
||||
|
||||
---
|
||||
|
||||
## When this repo *is* the right place to fix
|
||||
|
||||
A Contour outage traces back to `devops-infra-helm-charts` only when:
|
||||
|
||||
1. **`nodeSelector` / `tolerations` / `computeClass`** were copied from the wrong cluster.
|
||||
2. **A chart bump** introduced an immutable-selector change or removed a values key.
|
||||
3. **A blue-green migration** was cutover prematurely (Application points at `<contour>-vX.Y.Z` but values are still on the old shape).
|
||||
4. **`fullnameOverride`** was changed (very rare, but catastrophic).
|
||||
|
||||
For 80%+ of ingress incidents, the fix is **outside** this repo (cluster issues, app-side issues, GCP LB, DNS).
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 (pending — scheduling) | Yourself with the values fix | Cluster owner if node pool full |
|
||||
| §2 (CrashLoop — TLS) | cert-manager team | Security if cert source unknown |
|
||||
| §2 (CrashLoop — invalid HTTPProxy) | Consuming team | Platform if Contour itself is broken |
|
||||
| §3 (no endpoints / LB) | Cluster owner | Platform team |
|
||||
| §4 (5xx) | App team | Platform team if Contour-side |
|
||||
| §5 (DNS) | Platform team | — |
|
||||
| §C (cluster-wide outage) | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl get pods -n projectcontour` shows N/N Ready for the affected instances.
|
||||
- Synthetic probes from the app team return expected status codes.
|
||||
- If the root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,233 @@
|
||||
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Runbook — Metrics gap on a cluster / namespace
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** "Grafana panels are blank for `<cluster>` / `<namespace>` / `<service>`," or an alert that should be firing isn't, or VictoriaMetrics shows `no data`.
|
||||
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
|
||||
|
||||
The observability path on Meesho's GKE fleet is:
|
||||
|
||||
```
|
||||
workload Pod (exposes /metrics)
|
||||
└→ victoria-metrics-agent (vmagent) scrapes
|
||||
└→ remote_write to victoria-metrics-cluster (vmstorage)
|
||||
└→ vmselect ← Grafana / vmalert query
|
||||
```
|
||||
|
||||
A metrics gap can be at any hop. Walk this tree top-down — the most common root cause is hop 1 (scrape config).
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<workload-namespace-where-metric-is-missing>
|
||||
METRIC=<metric-name-or-job-label>
|
||||
|
||||
# Confirm the cluster runs VM agent + cluster
|
||||
ls helm-overrides/$CLUSTER | grep -E '^victoria-metrics-(agent|cluster)'
|
||||
|
||||
# Confirm pods are healthy
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
|
||||
```
|
||||
|
||||
If pods are missing/CrashLooping → that's the gap. Skip to §5.
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
├── Is the metric known to be emitted by the workload?
|
||||
│ ├── No → §0 — Workload not emitting; out of scope (app team)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §1 — Is vmagent scraping the workload?
|
||||
│ ├── No → fix scrape config (Layer 1)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §2 — Are scrape targets healthy (status=up)?
|
||||
│ ├── Down → fix endpoint reachability (Layer 2 / Layer 1)
|
||||
│ └── Up →
|
||||
│
|
||||
├── §3 — Are relabel rules dropping the metric?
|
||||
│ ├── Yes → adjust relabel_configs (Layer 1)
|
||||
│ └── No →
|
||||
│
|
||||
├── §4 — Is remote_write succeeding?
|
||||
│ ├── No → vmagent → vmstorage path broken (Layer 2 / Layer 1)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §5 — Is vmstorage healthy and ingesting?
|
||||
│ ├── No → vmstorage outage (Layer 2)
|
||||
│ └── Yes →
|
||||
│
|
||||
└── §6 — Is the Grafana datasource / tenant correct?
|
||||
└── Misrouted query → fix datasource URL or tenant header (Layer 1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §0 — Workload not emitting
|
||||
|
||||
Out of scope for this repo. Confirm with:
|
||||
|
||||
```bash
|
||||
# Port-forward and curl /metrics directly
|
||||
kubectl --context=$CTX -n $NS port-forward <pod> 9090:<metrics-port> &
|
||||
curl -s localhost:9090/metrics | grep -i "$METRIC"
|
||||
```
|
||||
|
||||
If `/metrics` is empty or doesn't contain `$METRIC` → app team. Stop.
|
||||
|
||||
---
|
||||
|
||||
## §1 — Is vmagent scraping the workload?
|
||||
|
||||
```bash
|
||||
# vmagent UI exposes /api/v1/targets
|
||||
kubectl --context=$CTX -n monitoring port-forward svc/victoria-metrics-agent 8429:8429 &
|
||||
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.labels.namespace=="'$NS'")'
|
||||
```
|
||||
|
||||
If no targets for `$NS`:
|
||||
|
||||
- Look at `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml`.
|
||||
- Check the `additionalScrapeConfigs` (or `config.scrape_configs`) for a job that matches the workload's labels / namespace selector.
|
||||
- Common cause: a `kubernetes_sd_configs` `namespaces.names` filter excludes `$NS`.
|
||||
- Common cause: a missing `Pod`/`Service`/`PodMonitor` annotation `prometheus.io/scrape: "true"` on the workload.
|
||||
|
||||
**Layer 1 fix** (if the gap is in the scrape config): edit `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml` per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
|
||||
**Layer 2 fix** (if the gap is on the workload — missing annotation): hand off to app team.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Targets healthy?
|
||||
|
||||
```bash
|
||||
# In the same vmagent /targets output
|
||||
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up") | {labels, lastError}'
|
||||
```
|
||||
|
||||
If targets are `down`:
|
||||
|
||||
| `lastError` | Cause | Fix |
|
||||
|-------------|-------|-----|
|
||||
| `connection refused` | Workload not listening on declared port | App team — Layer 2. |
|
||||
| `i/o timeout` | NetworkPolicy / firewall blocking vmagent → workload | Check `NetworkPolicy` in `$NS`. Often a NetworkPolicy allowing only intra-namespace traffic and not vmagent's namespace. **Layer 2** — recommend the workload team allow vmagent. |
|
||||
| `x509: certificate signed by unknown authority` | mTLS misconfigured | App team — Layer 2. |
|
||||
| `404 Not Found` | Wrong path (default `/metrics` vs custom) | Add `metrics_path:` in scrape config. **Layer 1.** |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Relabel rules dropping the metric?
|
||||
|
||||
```bash
|
||||
yq e '.config.scrape_configs[].metric_relabel_configs, .config.scrape_configs[].relabel_configs' \
|
||||
helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml
|
||||
```
|
||||
|
||||
Look for:
|
||||
|
||||
- `action: drop` rules with regexes matching `$METRIC`.
|
||||
- `action: keep` rules whose regex *excludes* `$METRIC`.
|
||||
- `action: labeldrop` removing a label the query uses.
|
||||
|
||||
Test in vmagent's UI under the *Targets* tab — it shows the labels post-relabel.
|
||||
|
||||
**Layer 1 fix:** loosen the relabel rule. Re-PR per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
|
||||
---
|
||||
|
||||
## §4 — Remote_write succeeding?
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent | grep -i 'remote_write\|error\|failed' | tail -20
|
||||
```
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `429 Too Many Requests` from vmstorage | vmstorage ingest saturated | Scale vmstorage / vminsert (Layer 1 — see [observability.md](../../global/coding-guidelines/observability.md)). |
|
||||
| `connection refused` to vmstorage URL | vmstorage Service down or wrong URL | Verify `remoteWrite.url` in vmagent values matches the live `vmstorage` Service DNS. Layer 1. |
|
||||
| `out of bounds timestamp` | Clock skew on vmagent's node | Layer 2 — node time sync. |
|
||||
| `series limit exceeded` | Cardinality bomb on vmstorage tenant | Layer 1 — drop the offending label. See [observability.md §Cardinality](../../global/coding-guidelines/observability.md). |
|
||||
|
||||
---
|
||||
|
||||
## §5 — vmstorage healthy?
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
|
||||
kubectl --context=$CTX -n monitoring describe pod vmstorage-0 | tail -30
|
||||
kubectl --context=$CTX -n monitoring exec vmstorage-0 -- df -h /storage
|
||||
```
|
||||
|
||||
Failure modes:
|
||||
|
||||
| Symptom | Layer | Fix |
|
||||
|---------|-------|-----|
|
||||
| Pod CrashLooping with `out of disk` | Layer 1 | Bump `persistence.size`. Note: PVC growth requires the StorageClass to support `allowVolumeExpansion: true`. See [../schemas/storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md). |
|
||||
| Pod CrashLooping with retention/index errors | Layer 2 | Escalate — may need data-side intervention. |
|
||||
| Pod Pending | Layer 1 | Scheduling issue. See [pod-pending-scheduling.md](pod-pending-scheduling.md) and [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md). |
|
||||
| Pod Running but vmselect can't reach it | Layer 1 / 2 | Verify the headless Service and StatefulSet pod-DNS records. |
|
||||
|
||||
---
|
||||
|
||||
## §6 — Grafana datasource / tenant correct?
|
||||
|
||||
```bash
|
||||
yq e '.datasources.datasources.yaml.datasources[] | select(.name == "*VictoriaMetrics*" or .type == "prometheus")' \
|
||||
helm-overrides/$CLUSTER/grafana/custom-values.yaml
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `url:` points at the correct in-cluster vmselect Service DNS | If wrong, **Layer 1** — fix the values per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md). |
|
||||
| `httpHeaderName1: X-Scope-OrgID` (multi-tenant clusters only) | If the cluster runs multi-tenant VM, the tenant header must be set. Layer 1 fix. |
|
||||
| Datasource `url:` points at an external `*.meeshogcp.in` host | Forbidden — see [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3. Repoint at in-cluster Service DNS. |
|
||||
|
||||
---
|
||||
|
||||
## Remediation summary
|
||||
|
||||
| Hop | Likely fix | Layer | Procedure |
|
||||
|-----|-----------|-------|-----------|
|
||||
| §1 scrape | Add scrape config / fix selector | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §2 endpoint | Workload-side / NetworkPolicy | 2 | Hand off to app team |
|
||||
| §3 relabel | Loosen drop rule | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §4 remote_write | Scale vmstorage / fix URL | 1/2 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §5 vmstorage | PVC grow / scheduling fix | 1/2 | [pod-pending-scheduling.md](pod-pending-scheduling.md) |
|
||||
| §6 datasource | Fix Grafana datasource | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
|
||||
---
|
||||
|
||||
## Escalation triggers
|
||||
|
||||
- §5 vmstorage outage with no obvious values-side fix → platform team (observability owner).
|
||||
- Multi-cluster simultaneous metrics gap → platform team — cluster-level / control-plane issue.
|
||||
- Cardinality explosion impacting vmstorage stability → platform team + workload owner — joint fix.
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- The query that was returning `no data` returns the expected series in Grafana Explore.
|
||||
- No remote_write errors in vmagent logs for at least 5 minutes post-fix.
|
||||
- Alerts that depend on the metric have transitioned from `pending` / silent back to expected state.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
- Procedure: [../procedures/modify-alert-rules.md](../procedures/modify-alert-rules.md) — if the gap is "alert silent" not "metric missing."
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — for §5 vmstorage scheduling failures.
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if a sync didn't take.
|
||||
@@ -0,0 +1,192 @@
|
||||
# Runbook — Pod Pending / wrong-node scheduling
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** an infra workload's pods are `Pending` indefinitely, or scheduling onto the wrong node pool.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.
|
||||
|
||||
The single most common values-side bug in this repo is **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster**. ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md))
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
APP=<release>
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<namespace>
|
||||
|
||||
kubectl --context=$CTX get pods -n $NS -o wide
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40 # Events: section
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── What state are the pods in?
|
||||
│
|
||||
├── Pending — never scheduled → §1 — Pending pods
|
||||
├── ContainerCreating long → §1 — Pending pods
|
||||
├── Running but on the WRONG node pool → §2 — Wrong-pool scheduling
|
||||
├── ImagePullBackOff / ErrImagePull → §3 — Image pull
|
||||
├── CrashLoopBackOff → §4 — Crash loop
|
||||
├── Running but PVC unbound → §5 — Storage
|
||||
└── Running fine → leave runbook
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Pending pods
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common patterns:
|
||||
|
||||
| Reason | Diagnosis | Fix in |
|
||||
|--------|-----------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}` | Pod has wrong toleration or no toleration. | `helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | Pod's `nodeSelector` doesn't match any node label. | Values `nodeSelector:`. **Layer 1.** |
|
||||
| `0/N nodes available: ... had no available compute class` (Autopilot) | `cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster. | (a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu` / `insufficient memory` | Node-pool autoscaler at max, or pod's `requests` too high. | Either reduce `resources.requests`, or **Layer 2** — escalate to cluster owner to raise node-pool max. |
|
||||
| `0/N nodes available: pod has unbound immediate PersistentVolumeClaims` | PVC is `Pending`. | See §5. |
|
||||
| `0/N nodes available: didn't tolerate node-pressure taint` | Node has `node.kubernetes.io/disk-pressure` etc. | **Layer 2** — cluster-level issue. |
|
||||
| `volume "X" not found` | PVC bound to a non-existent PV. | See §5. |
|
||||
|
||||
**The diagnostic for "wrong cluster's scheduling values":**
|
||||
|
||||
```bash
|
||||
# What does the pod's nodeSelector say?
|
||||
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
|
||||
| yq e '.spec.nodeSelector'
|
||||
|
||||
# What labels do the cluster's nodes actually have?
|
||||
kubectl --context=$CTX get nodes --show-labels | head -3
|
||||
|
||||
# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
|
||||
grep -h 'dedicated:\|cloud.google.com/compute-class' \
|
||||
helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head
|
||||
```
|
||||
|
||||
If the values use `dedicated:` but the cluster only has `cloud.google.com/compute-class:` keys (or vice versa), the values were copied from a sibling cluster. **Author the values from scratch** using the cluster's own key style.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Running but on the wrong node pool
|
||||
|
||||
The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| What `nodeSelector` does the pod actually have? | `kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`. |
|
||||
| Where is it running? | `kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels. |
|
||||
| Is the values-side `nodeSelector` too permissive? | If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect. |
|
||||
|
||||
The fix is to make the `nodeSelector` selective enough that only the intended pool matches. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §3 — Image pull failing
|
||||
|
||||
```text
|
||||
ErrImagePull / ImagePullBackOff
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is the image pinned to Meesho's GAR mirror? | `kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's [SANCTITY_RULES R11](../../global/SANCTITY_RULES.md) violation. **Layer 1** — fix the image reference. |
|
||||
| Does the tag exist in the registry? | Out-of-band check (registry UI). |
|
||||
| Is the registry-pull credential present on the cluster? | `kubectl get secret -n $NS \| grep gcr-pull`. **Layer 2** if missing. |
|
||||
|
||||
The image tag is set in `image.repository` / `image.tag` of `custom-values.yaml`. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §4 — CrashLoopBackOff
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <pod> -n $NS --previous
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
| Pattern | Likely cause | Fix in |
|
||||
|---------|--------------|--------|
|
||||
| Application stack trace, missing config | App expects an env var / file that isn't there | `custom-values.yaml` config section. **Layer 1.** |
|
||||
| `connection refused` to a dependency | Dependency not up; or wrong DNS | Dependency team. **Layer 2.** |
|
||||
| `exit code 137` | OOMKilled — `kubectl describe` confirms | Bump `resources.limits.memory` in values. **Layer 1.** |
|
||||
| `permission denied` on a file | Volume mount / `securityContext` | Values. **Layer 1.** |
|
||||
| `existing Secret <X> not found` | `existingSecret:` references a secret that doesn't exist | Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. **Layer 1.** |
|
||||
| Init container failed | Init logs explain | `kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — Running but PVC unbound
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get pvc -n $NS
|
||||
kubectl --context=$CTX describe pvc <pvc> -n $NS
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Events: Failed to provision volume with StorageClass "<X>"` | The StorageClass doesn't exist on this cluster. | `ls manifests/storageclass/<X>.yaml`. If absent, fix `persistence.storageClass:` in values to a real class. **Layer 1.** |
|
||||
| `Events: ProvisioningFailed: googleapi: Error 403` | CSI driver lacks IAM permission. **Layer 2.** | Platform / IAM team. |
|
||||
| PVC `Pending` with no events | StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled. | Schedule the pod (resolve §1 first). |
|
||||
| PVC bound but pod can't mount | Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`). | Either set replicas to 1, or use a `StatefulSet` chart variant, or use a Filestore-backed `ReadWriteMany` class. **Layer 1.** |
|
||||
|
||||
---
|
||||
|
||||
## §6 — When this repo *is* the right place to fix
|
||||
|
||||
For these cases, the fix is a values diff in `helm-overrides/<cluster>/<app>/custom-values.yaml`:
|
||||
|
||||
1. Wrong-cluster `nodeSelector` / `tolerations` / `computeClass`.
|
||||
2. Image tag pointing outside the GAR mirror.
|
||||
3. `resources.requests` / `limits` mis-sized (OOMKill, throttling).
|
||||
4. `persistence.storageClass` referencing a non-existent class.
|
||||
5. `existingSecret:` referencing a non-existent secret.
|
||||
6. `replicaCount` set on an HPA-managed release.
|
||||
|
||||
For these cases, the fix is **outside** this repo:
|
||||
|
||||
- Node pool full → cluster-owner / Terraform.
|
||||
- Image not in GAR → image-mirror automation / build pipeline.
|
||||
- CSI provisioning failures → platform / IAM team.
|
||||
- App-internal crashes (config, dependencies) → app team.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 — wrong scheduling values | Yourself with values fix | Cluster owner if topology is unclear |
|
||||
| §1 — node pool full | Cluster owner | Platform team if quota raise needed |
|
||||
| §3 — image pull (mirror miss) | Yourself with values fix | Platform if mirror push is missing |
|
||||
| §4 — OOMKilled | Yourself with `resources.limits` bump | App team if root cause is leak |
|
||||
| §4 — secret missing | Yourself with `ExternalSecret` add | Security if cluster-level `SecretStore` is missing |
|
||||
| §5 — CSI provision failure | Platform team | — |
|
||||
| §C — cluster-wide scheduling failure | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl rollout status deploy/<name> -n $NS` returns "successfully rolled out".
|
||||
- `kubectl get pods -n $NS` shows N/N Ready for the expected replica count.
|
||||
- Pods scheduled onto the **intended** node pool (verify `kubectl get pods -o wide` shows the right node names).
|
||||
- If root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- Runbook: [ingress-down.md](ingress-down.md).
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md).
|
||||
@@ -0,0 +1,251 @@
|
||||
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Runbook — Vault unavailable / External Secrets failing to render
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** Pods are CrashLooping referencing a missing `Secret`, or `kubectl describe externalsecret` shows `SecretSyncedError`, or `kubectl get secrets <name>` returns NotFound for a name an `ExternalSecret` should be creating.
|
||||
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
|
||||
>
|
||||
> **Important Layer 3 boundary:** **Vault HA itself is Layer 3.** Vault runs in production-only with no lower environment, so the agent must NOT attempt server-side fixes (unsealing, leader-election toggling, raft config changes, restoring from snapshot). Those are platform-team / security-team operations. The agent's role here is *diagnose, narrow root cause, and escalate*.
|
||||
|
||||
---
|
||||
|
||||
## Architecture refresher
|
||||
|
||||
```
|
||||
workload Pod (mounts Secret <name>)
|
||||
↑ created by
|
||||
External Secrets Operator (ESO) controller
|
||||
│
|
||||
└─ reads SecretStore / ClusterSecretStore
|
||||
│
|
||||
├─ kind: gcpsm → GCP Secret Manager
|
||||
│ └─ auth via Workload Identity (KSA → GSA binding)
|
||||
└─ kind: vault → Vault HA cluster
|
||||
└─ auth via Kubernetes auth (KSA token review)
|
||||
```
|
||||
|
||||
Most clusters in the fleet use **GCP Secret Manager** as the primary backend (via Workload Identity), with Vault HA as the secondary for legacy services. Some clusters use Vault as the primary. Confirm before troubleshooting:
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/external-secrets/
|
||||
yq e '.spec.provider' helm-overrides/<cluster>/external-secrets/*.yaml 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<namespace-of-the-failing-workload>
|
||||
ES=<external-secret-name>
|
||||
|
||||
# ESO controller status
|
||||
kubectl --context=$CTX -n external-secrets get pods
|
||||
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets | tail -50
|
||||
|
||||
# The failing ExternalSecret
|
||||
kubectl --context=$CTX -n $NS describe externalsecret $ES
|
||||
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.status'
|
||||
|
||||
# The SecretStore / ClusterSecretStore it references
|
||||
STORE=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.name}')
|
||||
KIND=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.kind}')
|
||||
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status, .spec.provider'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
├── §1 — Is ESO controller running and healthy?
|
||||
│ ├── No → §1a — ESO outage
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §2 — Is the SecretStore / ClusterSecretStore Ready?
|
||||
│ ├── No → §2a — Store config / auth broken
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §3 — Does the ExternalSecret reference a real remote key?
|
||||
│ ├── No → §3a — Bad spec.data[].remoteRef.key (Layer 1 likely)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §4 — Is the auth path working? (WI binding or Vault K8s auth)
|
||||
│ ├── No → §4a — Identity binding (Layer 2/3)
|
||||
│ └── Yes →
|
||||
│
|
||||
└── §5 — Is the upstream backend healthy?
|
||||
├── GCP Secret Manager 5xx → §5a — escalate to platform
|
||||
└── Vault unsealed / leader OK? → §5b — Vault outage (Layer 3)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — ESO controller health
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n external-secrets get pods
|
||||
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=100 | grep -iE 'error|failed|panic'
|
||||
```
|
||||
|
||||
| Symptom | Cause | Layer | Action |
|
||||
|---------|-------|-------|--------|
|
||||
| Pods CrashLooping | Bad chart upgrade or RBAC misconfig | 1 | Check `helm-overrides/<cluster>/external-secrets/custom-values.yaml`. Last bump? Revert. |
|
||||
| Pods Pending | Scheduling — wrong nodeSelector | 1 | See [pod-pending-scheduling.md](pod-pending-scheduling.md). |
|
||||
| Pods Running, no logs about reconcile | Cluster-watch RBAC missing | 2 | Escalate. |
|
||||
| `Forbidden` errors on CRD list | RBAC on the CRDs | 1 / 2 | Verify chart values' `rbac.create: true`. |
|
||||
|
||||
---
|
||||
|
||||
## §2 — SecretStore / ClusterSecretStore status
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status'
|
||||
```
|
||||
|
||||
Look for `conditions[].status` and `conditions[].message`.
|
||||
|
||||
| Status / message | Cause | Layer | Action |
|
||||
|------------------|-------|-------|--------|
|
||||
| `Ready: False, ValidationFailed` | Store spec invalid | 1 | Fix the `SecretStore` YAML in `helm-overrides/<cluster>/external-secrets/`. |
|
||||
| `Ready: False, InvalidProviderConfig` | Provider block malformed | 1 | Validate `spec.provider.gcpsm.projectID` / `spec.provider.vault.server`. |
|
||||
| `Ready: False, AuthFailed` (gcpsm) | Workload Identity binding broken | 2/3 | §4 below. |
|
||||
| `Ready: False, AuthFailed` (vault) | KSA token review fails | 2/3 | §4 below. |
|
||||
| `Ready: True` | Store OK; problem is elsewhere | — | Continue to §3. |
|
||||
|
||||
---
|
||||
|
||||
## §3 — ExternalSecret spec validity
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.spec.data, .spec.dataFrom'
|
||||
```
|
||||
|
||||
For each `remoteRef.key`:
|
||||
|
||||
- **GCP SM:** the key is the secret name in the project. Verify it exists:
|
||||
```bash
|
||||
gcloud secrets list --project=<gcp-project> --filter="name:<key>"
|
||||
```
|
||||
(Read-only — no write to GCP SM from the agent.)
|
||||
|
||||
- **Vault:** the key is the path under the engine. Cannot directly verify without Vault access; rely on the ESO controller's reconcile error message.
|
||||
|
||||
If the controller logs say `secret not found in backend` → the `remoteRef.key` is wrong. **Layer 1 fix:** correct the key in the `ExternalSecret` YAML.
|
||||
|
||||
---
|
||||
|
||||
## §4 — Auth path
|
||||
|
||||
### §4a — GCP Secret Manager (Workload Identity)
|
||||
|
||||
```bash
|
||||
# The ServiceAccount the ESO controller (or this ExternalSecret's pod) runs as
|
||||
kubectl --context=$CTX -n external-secrets get sa external-secrets -o yaml | yq e '.metadata.annotations'
|
||||
|
||||
# Should have:
|
||||
# iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
| Sub-check | Layer | Action |
|
||||
|-----------|-------|--------|
|
||||
| KSA missing the `iam.gke.io/gcp-service-account` annotation | 1 | Add via `helm-overrides/<cluster>/external-secrets/custom-values.yaml` `serviceAccount.annotations`. |
|
||||
| GSA exists but no IAM binding to KSA | 2 | Escalate to platform — IAM is out of repo. |
|
||||
| GSA lacks `roles/secretmanager.secretAccessor` on the project | 2 | Escalate to platform. |
|
||||
|
||||
### §4b — Vault Kubernetes auth
|
||||
|
||||
| Sub-check | Layer | Action |
|
||||
|-----------|-------|--------|
|
||||
| `SecretStore.spec.provider.vault.auth.kubernetes.role` references a Vault role | — | Read-only; verify against existing working stores on the same cluster. |
|
||||
| ESO logs say `permission denied` from Vault | 3 | The role/policy on the Vault server is wrong. **Cannot fix from this repo.** Escalate to security / Vault platform team. |
|
||||
| ESO logs say `Vault is sealed` | 3 | **Vault HA outage. Do not attempt to unseal.** Escalate immediately. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — Backend health
|
||||
|
||||
### §5a — GCP Secret Manager
|
||||
|
||||
GCP SM is a managed service. 5xx from it is rare and is a GCP-side incident. Action: escalate to platform; check GCP status dashboard. **No fix in this repo.**
|
||||
|
||||
### §5b — Vault HA
|
||||
|
||||
Vault HA on Meesho's fleet runs in production-only, no lower env. It is **Layer 3** — agent does not write to or operate Vault. Diagnostic *read* of pod state is OK; mutation is not.
|
||||
|
||||
```bash
|
||||
# Diagnostic only
|
||||
kubectl --context=$CTX -n vault get pods
|
||||
kubectl --context=$CTX -n vault logs <vault-pod> | tail -50
|
||||
# Look for: "core: Vault is sealed", "leader election", "raft", panic stacks
|
||||
```
|
||||
|
||||
| Observed | Action |
|
||||
|----------|--------|
|
||||
| Some Vault pods sealed (HA quorum still up) | Escalate to security team. **Do not unseal.** |
|
||||
| All Vault pods sealed (full outage) | Page security team. ESO will surface stale data only as long as the in-memory cache holds. |
|
||||
| Leader-election thrashing | Escalate; could be a network/raft issue. |
|
||||
| Vault pod Pending | Scheduling — see [pod-pending-scheduling.md](pod-pending-scheduling.md). Even here, restart of a Vault pod requires a security-team-led unseal afterwards. |
|
||||
|
||||
---
|
||||
|
||||
## Mitigations while Vault is down
|
||||
|
||||
If a workload's `ExternalSecret` is failing because Vault is unavailable, options are limited:
|
||||
|
||||
1. **Wait** — ESO caches the last-rendered Secret value. Pods that already mounted continue. New pod scheduling fails until Vault returns.
|
||||
2. **Switch the `ExternalSecret` to GCP SM** if the same secret exists there (most do, with Vault as legacy). Layer 1 PR — change `secretStoreRef.name` to the GCP SM store. **Coordinate with security** before doing this in an outage.
|
||||
3. **Hand-create the Secret as a temporary `kubectl apply`** — out of scope for the agent. This is incident response by a human operator.
|
||||
|
||||
The agent should **not** auto-cut option 2 without explicit human approval — switching the source of truth for a secret has security implications.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 (ESO down) | DevOps on-call | Platform team |
|
||||
| §2 (Store config) | DevOps on-call | Layer 1 PR + reviewer |
|
||||
| §3 (Bad remoteRef) | DevOps on-call | Layer 1 PR + reviewer |
|
||||
| §4a (WI binding) | Platform team | — |
|
||||
| §4b (Vault role/policy) | Security team | — |
|
||||
| §5a (GCP SM 5xx) | Platform team | GCP support |
|
||||
| §5b (Vault outage) | **Security team — page** | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `ExternalSecret` `status.conditions[type=Ready].status == True`.
|
||||
- The downstream `Secret` exists with expected keys.
|
||||
- Workload pods consuming the Secret are Running.
|
||||
- ESO logs are quiet for at least 5 minutes after the fix.
|
||||
|
||||
---
|
||||
|
||||
## What this repo can and cannot fix
|
||||
|
||||
| Fix kind | Layer | This repo? |
|
||||
|----------|-------|-----------|
|
||||
| `ExternalSecret` / `SecretStore` YAML edits | 1 | Yes — values PR. |
|
||||
| ESO chart values (sizing, RBAC, metrics) | 1 | Yes. |
|
||||
| Workload-Identity KSA annotation | 1 | Yes (in chart values' `serviceAccount.annotations`). |
|
||||
| GSA IAM bindings on GCP | 2 | No — platform team. |
|
||||
| Vault server-side role/policy | 3 | **Never** — security team. |
|
||||
| Vault unseal / raft / leader | 3 | **Never** — security team. |
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Schema: [../schemas/raw-manifest-sidecar-schema.md §ExternalSecret](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — if any of the above pods are Pending.
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if the `ExternalSecret` itself didn't sync.
|
||||
- Boundaries: [../../global/AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md), [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md).
|
||||
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
Reference in New Issue
Block a user