added repo
This commit is contained in:
Your Name
@@ -0,0 +1,192 @@
|
||||
# Runbook — Pod Pending / wrong-node scheduling
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** an infra workload's pods are `Pending` indefinitely, or scheduling onto the wrong node pool.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.
|
||||
|
||||
The single most common values-side bug in this repo is **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster**. ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md))
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
APP=<release>
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<namespace>
|
||||
|
||||
kubectl --context=$CTX get pods -n $NS -o wide
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40 # Events: section
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── What state are the pods in?
|
||||
│
|
||||
├── Pending — never scheduled → §1 — Pending pods
|
||||
├── ContainerCreating long → §1 — Pending pods
|
||||
├── Running but on the WRONG node pool → §2 — Wrong-pool scheduling
|
||||
├── ImagePullBackOff / ErrImagePull → §3 — Image pull
|
||||
├── CrashLoopBackOff → §4 — Crash loop
|
||||
├── Running but PVC unbound → §5 — Storage
|
||||
└── Running fine → leave runbook
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Pending pods
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common patterns:
|
||||
|
||||
| Reason | Diagnosis | Fix in |
|
||||
|--------|-----------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}` | Pod has wrong toleration or no toleration. | `helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | Pod's `nodeSelector` doesn't match any node label. | Values `nodeSelector:`. **Layer 1.** |
|
||||
| `0/N nodes available: ... had no available compute class` (Autopilot) | `cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster. | (a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu` / `insufficient memory` | Node-pool autoscaler at max, or pod's `requests` too high. | Either reduce `resources.requests`, or **Layer 2** — escalate to cluster owner to raise node-pool max. |
|
||||
| `0/N nodes available: pod has unbound immediate PersistentVolumeClaims` | PVC is `Pending`. | See §5. |
|
||||
| `0/N nodes available: didn't tolerate node-pressure taint` | Node has `node.kubernetes.io/disk-pressure` etc. | **Layer 2** — cluster-level issue. |
|
||||
| `volume "X" not found` | PVC bound to a non-existent PV. | See §5. |
|
||||
|
||||
**The diagnostic for "wrong cluster's scheduling values":**
|
||||
|
||||
```bash
|
||||
# What does the pod's nodeSelector say?
|
||||
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
|
||||
| yq e '.spec.nodeSelector'
|
||||
|
||||
# What labels do the cluster's nodes actually have?
|
||||
kubectl --context=$CTX get nodes --show-labels | head -3
|
||||
|
||||
# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
|
||||
grep -h 'dedicated:\|cloud.google.com/compute-class' \
|
||||
helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head
|
||||
```
|
||||
|
||||
If the values use `dedicated:` but the cluster only has `cloud.google.com/compute-class:` keys (or vice versa), the values were copied from a sibling cluster. **Author the values from scratch** using the cluster's own key style.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Running but on the wrong node pool
|
||||
|
||||
The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| What `nodeSelector` does the pod actually have? | `kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`. |
|
||||
| Where is it running? | `kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels. |
|
||||
| Is the values-side `nodeSelector` too permissive? | If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect. |
|
||||
|
||||
The fix is to make the `nodeSelector` selective enough that only the intended pool matches. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §3 — Image pull failing
|
||||
|
||||
```text
|
||||
ErrImagePull / ImagePullBackOff
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is the image pinned to Meesho's GAR mirror? | `kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's [SANCTITY_RULES R11](../../global/SANCTITY_RULES.md) violation. **Layer 1** — fix the image reference. |
|
||||
| Does the tag exist in the registry? | Out-of-band check (registry UI). |
|
||||
| Is the registry-pull credential present on the cluster? | `kubectl get secret -n $NS \| grep gcr-pull`. **Layer 2** if missing. |
|
||||
|
||||
The image tag is set in `image.repository` / `image.tag` of `custom-values.yaml`. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §4 — CrashLoopBackOff
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <pod> -n $NS --previous
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
| Pattern | Likely cause | Fix in |
|
||||
|---------|--------------|--------|
|
||||
| Application stack trace, missing config | App expects an env var / file that isn't there | `custom-values.yaml` config section. **Layer 1.** |
|
||||
| `connection refused` to a dependency | Dependency not up; or wrong DNS | Dependency team. **Layer 2.** |
|
||||
| `exit code 137` | OOMKilled — `kubectl describe` confirms | Bump `resources.limits.memory` in values. **Layer 1.** |
|
||||
| `permission denied` on a file | Volume mount / `securityContext` | Values. **Layer 1.** |
|
||||
| `existing Secret <X> not found` | `existingSecret:` references a secret that doesn't exist | Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. **Layer 1.** |
|
||||
| Init container failed | Init logs explain | `kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — Running but PVC unbound
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get pvc -n $NS
|
||||
kubectl --context=$CTX describe pvc <pvc> -n $NS
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Events: Failed to provision volume with StorageClass "<X>"` | The StorageClass doesn't exist on this cluster. | `ls manifests/storageclass/<X>.yaml`. If absent, fix `persistence.storageClass:` in values to a real class. **Layer 1.** |
|
||||
| `Events: ProvisioningFailed: googleapi: Error 403` | CSI driver lacks IAM permission. **Layer 2.** | Platform / IAM team. |
|
||||
| PVC `Pending` with no events | StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled. | Schedule the pod (resolve §1 first). |
|
||||
| PVC bound but pod can't mount | Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`). | Either set replicas to 1, or use a `StatefulSet` chart variant, or use a Filestore-backed `ReadWriteMany` class. **Layer 1.** |
|
||||
|
||||
---
|
||||
|
||||
## §6 — When this repo *is* the right place to fix
|
||||
|
||||
For these cases, the fix is a values diff in `helm-overrides/<cluster>/<app>/custom-values.yaml`:
|
||||
|
||||
1. Wrong-cluster `nodeSelector` / `tolerations` / `computeClass`.
|
||||
2. Image tag pointing outside the GAR mirror.
|
||||
3. `resources.requests` / `limits` mis-sized (OOMKill, throttling).
|
||||
4. `persistence.storageClass` referencing a non-existent class.
|
||||
5. `existingSecret:` referencing a non-existent secret.
|
||||
6. `replicaCount` set on an HPA-managed release.
|
||||
|
||||
For these cases, the fix is **outside** this repo:
|
||||
|
||||
- Node pool full → cluster-owner / Terraform.
|
||||
- Image not in GAR → image-mirror automation / build pipeline.
|
||||
- CSI provisioning failures → platform / IAM team.
|
||||
- App-internal crashes (config, dependencies) → app team.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 — wrong scheduling values | Yourself with values fix | Cluster owner if topology is unclear |
|
||||
| §1 — node pool full | Cluster owner | Platform team if quota raise needed |
|
||||
| §3 — image pull (mirror miss) | Yourself with values fix | Platform if mirror push is missing |
|
||||
| §4 — OOMKilled | Yourself with `resources.limits` bump | App team if root cause is leak |
|
||||
| §4 — secret missing | Yourself with `ExternalSecret` add | Security if cluster-level `SecretStore` is missing |
|
||||
| §5 — CSI provision failure | Platform team | — |
|
||||
| §C — cluster-wide scheduling failure | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl rollout status deploy/<name> -n $NS` returns "successfully rolled out".
|
||||
- `kubectl get pods -n $NS` shows N/N Ready for the expected replica count.
|
||||
- Pods scheduled onto the **intended** node pool (verify `kubectl get pods -o wide` shows the right node names).
|
||||
- If root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- Runbook: [ingress-down.md](ingress-down.md).
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md).
|
||||
Reference in New Issue
Block a user