Runbook — Pod Pending / wrong-node scheduling

Type: Decision tree. Entry symptom: an infra workload's pods are Pending indefinitely, or scheduling onto the wrong node pool. Layer: mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.

The single most common values-side bug in this repo is nodeSelector / tolerations / computeClass copied from the wrong cluster. (SANCTITY_RULES R5)

Entry — gather context

APP=<release>
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace>

kubectl --context=$CTX get pods -n $NS -o wide
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40        # Events: section

Decision tree

START
  │
  └── What state are the pods in?
        │
        ├── Pending — never scheduled                  → §1 — Pending pods
        ├── ContainerCreating long                     → §1 — Pending pods
        ├── Running but on the WRONG node pool         → §2 — Wrong-pool scheduling
        ├── ImagePullBackOff / ErrImagePull            → §3 — Image pull
        ├── CrashLoopBackOff                           → §4 — Crash loop
        ├── Running but PVC unbound                    → §5 — Storage
        └── Running fine                               → leave runbook

§1 — Pending pods

kubectl --context=$CTX describe pod <pod> -n $NS

Read the Events: section. Common patterns:

Reason	Diagnosis	Fix in
`0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}`	Pod has wrong toleration or no toleration.	`helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. Layer 1.
`0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector`	Pod's `nodeSelector` doesn't match any node label.	Values `nodeSelector:`. Layer 1.
`0/N nodes available: ... had no available compute class` (Autopilot)	`cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster.	(a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. Layer 1.
`0/N nodes available: insufficient cpu` / `insufficient memory`	Node-pool autoscaler at max, or pod's `requests` too high.	Either reduce `resources.requests`, or Layer 2 — escalate to cluster owner to raise node-pool max.
`0/N nodes available: pod has unbound immediate PersistentVolumeClaims`	PVC is `Pending`.	See §5.
`0/N nodes available: didn't tolerate node-pressure taint`	Node has `node.kubernetes.io/disk-pressure` etc.	Layer 2 — cluster-level issue.
`volume "X" not found`	PVC bound to a non-existent PV.	See §5.

The diagnostic for "wrong cluster's scheduling values":

# What does the pod's nodeSelector say?
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
  | yq e '.spec.nodeSelector'

# What labels do the cluster's nodes actually have?
kubectl --context=$CTX get nodes --show-labels | head -3

# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
grep -h 'dedicated:\|cloud.google.com/compute-class' \
  helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head

If the values use dedicated: but the cluster only has cloud.google.com/compute-class: keys (or vice versa), the values were copied from a sibling cluster. Author the values from scratch using the cluster's own key style.

§2 — Running but on the wrong node pool

The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).

Sub-check	Action
What `nodeSelector` does the pod actually have?	`kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`.
Where is it running?	`kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels.
Is the values-side `nodeSelector` too permissive?	If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect.

The fix is to make the nodeSelector selective enough that only the intended pool matches. Layer 1.

§3 — Image pull failing

ErrImagePull / ImagePullBackOff

Sub-check	Action
Is the image pinned to Meesho's GAR mirror?	`kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's SANCTITY_RULES R11 violation. Layer 1 — fix the image reference.
Does the tag exist in the registry?	Out-of-band check (registry UI).
Is the registry-pull credential present on the cluster?	`kubectl get secret -n $NS \| grep gcr-pull`. Layer 2 if missing.

The image tag is set in image.repository / image.tag of custom-values.yaml. Layer 1.

§4 — CrashLoopBackOff

kubectl --context=$CTX logs <pod> -n $NS --previous
kubectl --context=$CTX describe pod <pod> -n $NS

Pattern	Likely cause	Fix in
Application stack trace, missing config	App expects an env var / file that isn't there	`custom-values.yaml` config section. Layer 1.
`connection refused` to a dependency	Dependency not up; or wrong DNS	Dependency team. Layer 2.
`exit code 137`	OOMKilled — `kubectl describe` confirms	Bump `resources.limits.memory` in values. Layer 1.
`permission denied` on a file	Volume mount / `securityContext`	Values. Layer 1.
`existing Secret <X> not found`	`existingSecret:` references a secret that doesn't exist	Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. Layer 1.
Init container failed	Init logs explain	`kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`.

§5 — Running but PVC unbound

kubectl --context=$CTX get pvc -n $NS
kubectl --context=$CTX describe pvc <pvc> -n $NS

Sub-check	Action
`Events: Failed to provision volume with StorageClass "<X>"`	The StorageClass doesn't exist on this cluster.
`Events: ProvisioningFailed: googleapi: Error 403`	CSI driver lacks IAM permission. Layer 2.
PVC `Pending` with no events	StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled.
PVC bound but pod can't mount	Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`).

§6 — When this repo is the right place to fix

For these cases, the fix is a values diff in helm-overrides/<cluster>/<app>/custom-values.yaml:

Wrong-cluster nodeSelector / tolerations / computeClass.
Image tag pointing outside the GAR mirror.
resources.requests / limits mis-sized (OOMKill, throttling).
persistence.storageClass referencing a non-existent class.
existingSecret: referencing a non-existent secret.
replicaCount set on an HPA-managed release.

For these cases, the fix is outside this repo:

Node pool full → cluster-owner / Terraform.
Image not in GAR → image-mirror automation / build pipeline.
CSI provisioning failures → platform / IAM team.
App-internal crashes (config, dependencies) → app team.

Escalation matrix

Symptom	First responder	Escalate to
§1 — wrong scheduling values	Yourself with values fix	Cluster owner if topology is unclear
§1 — node pool full	Cluster owner	Platform team if quota raise needed
§3 — image pull (mirror miss)	Yourself with values fix	Platform if mirror push is missing
§4 — OOMKilled	Yourself with `resources.limits` bump	App team if root cause is leak
§4 — secret missing	Yourself with `ExternalSecret` add	Security if cluster-level `SecretStore` is missing
§5 — CSI provision failure	Platform team	—
§C — cluster-wide scheduling failure	Platform team — pager	—

Done conditions

kubectl rollout status deploy/<name> -n $NS returns "successfully rolled out".
kubectl get pods -n $NS shows N/N Ready for the expected replica count.
Pods scheduled onto the intended node pool (verify kubectl get pods -o wide shows the right node names).
If root cause was in this repo, the fix is on main and synced.

Reference: contour-nodeselector-tolerations-summary.md.
Runbook: argocd-sync-failure.md.
Runbook: ingress-down.md.
Schema: custom-values-schema.md, storageclass-priorityclass-schema.md.

9.2 KiB Raw Blame History