# Runbook — Pod Pending / wrong-node scheduling

> **Type:** Decision tree.
> **Entry symptom:** an infra workload's pods are `Pending` indefinitely, or scheduling onto the wrong node pool.
> **Layer:** mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.

The single most common values-side bug in this repo is **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster**. ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md))

---

## Entry — gather context

```bash
APP=<release>
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace>

kubectl --context=$CTX get pods -n $NS -o wide
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40        # Events: section
```

---

## Decision tree

```text
START
  │
  └── What state are the pods in?
        │
        ├── Pending — never scheduled                  → §1 — Pending pods
        ├── ContainerCreating long                     → §1 — Pending pods
        ├── Running but on the WRONG node pool         → §2 — Wrong-pool scheduling
        ├── ImagePullBackOff / ErrImagePull            → §3 — Image pull
        ├── CrashLoopBackOff                           → §4 — Crash loop
        ├── Running but PVC unbound                    → §5 — Storage
        └── Running fine                               → leave runbook
```

---

## §1 — Pending pods

```bash
kubectl --context=$CTX describe pod <pod> -n $NS
```

Read the `Events:` section. Common patterns:

| Reason | Diagnosis | Fix in |
|--------|-----------|--------|
| `0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}` | Pod has wrong toleration or no toleration. | `helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. **Layer 1.** |
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | Pod's `nodeSelector` doesn't match any node label. | Values `nodeSelector:`. **Layer 1.** |
| `0/N nodes available: ... had no available compute class` (Autopilot) | `cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster. | (a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. **Layer 1.** |
| `0/N nodes available: insufficient cpu` / `insufficient memory` | Node-pool autoscaler at max, or pod's `requests` too high. | Either reduce `resources.requests`, or **Layer 2** — escalate to cluster owner to raise node-pool max. |
| `0/N nodes available: pod has unbound immediate PersistentVolumeClaims` | PVC is `Pending`. | See §5. |
| `0/N nodes available: didn't tolerate node-pressure taint` | Node has `node.kubernetes.io/disk-pressure` etc. | **Layer 2** — cluster-level issue. |
| `volume "X" not found` | PVC bound to a non-existent PV. | See §5. |

**The diagnostic for "wrong cluster's scheduling values":**

```bash
# What does the pod's nodeSelector say?
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
  | yq e '.spec.nodeSelector'

# What labels do the cluster's nodes actually have?
kubectl --context=$CTX get nodes --show-labels | head -3

# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
grep -h 'dedicated:\|cloud.google.com/compute-class' \
  helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head
```

If the values use `dedicated:` but the cluster only has `cloud.google.com/compute-class:` keys (or vice versa), the values were copied from a sibling cluster. **Author the values from scratch** using the cluster's own key style.

---

## §2 — Running but on the wrong node pool

The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).

| Sub-check | Action |
|-----------|--------|
| What `nodeSelector` does the pod actually have? | `kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`. |
| Where is it running? | `kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels. |
| Is the values-side `nodeSelector` too permissive? | If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect. |

The fix is to make the `nodeSelector` selective enough that only the intended pool matches. **Layer 1.**

---

## §3 — Image pull failing

```text
ErrImagePull / ImagePullBackOff
```

| Sub-check | Action |
|-----------|--------|
| Is the image pinned to Meesho's GAR mirror? | `kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's [SANCTITY_RULES R11](../../global/SANCTITY_RULES.md) violation. **Layer 1** — fix the image reference. |
| Does the tag exist in the registry? | Out-of-band check (registry UI). |
| Is the registry-pull credential present on the cluster? | `kubectl get secret -n $NS \| grep gcr-pull`. **Layer 2** if missing. |

The image tag is set in `image.repository` / `image.tag` of `custom-values.yaml`. **Layer 1.**

---

## §4 — CrashLoopBackOff

```bash
kubectl --context=$CTX logs <pod> -n $NS --previous
kubectl --context=$CTX describe pod <pod> -n $NS
```

| Pattern | Likely cause | Fix in |
|---------|--------------|--------|
| Application stack trace, missing config | App expects an env var / file that isn't there | `custom-values.yaml` config section. **Layer 1.** |
| `connection refused` to a dependency | Dependency not up; or wrong DNS | Dependency team. **Layer 2.** |
| `exit code 137` | OOMKilled — `kubectl describe` confirms | Bump `resources.limits.memory` in values. **Layer 1.** |
| `permission denied` on a file | Volume mount / `securityContext` | Values. **Layer 1.** |
| `existing Secret <X> not found` | `existingSecret:` references a secret that doesn't exist | Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. **Layer 1.** |
| Init container failed | Init logs explain | `kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`. |

---

## §5 — Running but PVC unbound

```bash
kubectl --context=$CTX get pvc -n $NS
kubectl --context=$CTX describe pvc <pvc> -n $NS
```

| Sub-check | Action |
|-----------|--------|
| `Events: Failed to provision volume with StorageClass "<X>"` | The StorageClass doesn't exist on this cluster. | `ls manifests/storageclass/<X>.yaml`. If absent, fix `persistence.storageClass:` in values to a real class. **Layer 1.** |
| `Events: ProvisioningFailed: googleapi: Error 403` | CSI driver lacks IAM permission. **Layer 2.** | Platform / IAM team. |
| PVC `Pending` with no events | StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled. | Schedule the pod (resolve §1 first). |
| PVC bound but pod can't mount | Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`). | Either set replicas to 1, or use a `StatefulSet` chart variant, or use a Filestore-backed `ReadWriteMany` class. **Layer 1.** |

---

## §6 — When this repo *is* the right place to fix

For these cases, the fix is a values diff in `helm-overrides/<cluster>/<app>/custom-values.yaml`:

1. Wrong-cluster `nodeSelector` / `tolerations` / `computeClass`.
2. Image tag pointing outside the GAR mirror.
3. `resources.requests` / `limits` mis-sized (OOMKill, throttling).
4. `persistence.storageClass` referencing a non-existent class.
5. `existingSecret:` referencing a non-existent secret.
6. `replicaCount` set on an HPA-managed release.

For these cases, the fix is **outside** this repo:

- Node pool full → cluster-owner / Terraform.
- Image not in GAR → image-mirror automation / build pipeline.
- CSI provisioning failures → platform / IAM team.
- App-internal crashes (config, dependencies) → app team.

---

## Escalation matrix

| Symptom | First responder | Escalate to |
|---------|-----------------|-------------|
| §1 — wrong scheduling values | Yourself with values fix | Cluster owner if topology is unclear |
| §1 — node pool full | Cluster owner | Platform team if quota raise needed |
| §3 — image pull (mirror miss) | Yourself with values fix | Platform if mirror push is missing |
| §4 — OOMKilled | Yourself with `resources.limits` bump | App team if root cause is leak |
| §4 — secret missing | Yourself with `ExternalSecret` add | Security if cluster-level `SecretStore` is missing |
| §5 — CSI provision failure | Platform team | — |
| §C — cluster-wide scheduling failure | Platform team — pager | — |

---

## Done conditions

- `kubectl rollout status deploy/<name> -n $NS` returns "successfully rolled out".
- `kubectl get pods -n $NS` shows N/N Ready for the expected replica count.
- Pods scheduled onto the **intended** node pool (verify `kubectl get pods -o wide` shows the right node names).
- If root cause was in this repo, the fix is on `main` and synced.

---

## Related

- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
- Runbook: [ingress-down.md](ingress-down.md).
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md).