175 lines
6.8 KiB
Markdown
175 lines
6.8 KiB
Markdown
# Skill — `diagnose-scheduling`
|
||
|
||
> **Layer:** mostly Layer 2 (advisory — produces diagnosis and recommended action). Layer 1 only when the recommendation is "open this PR with this diff."
|
||
> **Scope:** infra workloads stuck `Pending`, scheduling onto wrong nodes, or with PVCs unbound.
|
||
|
||
This skill walks the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree and outputs a structured diagnosis + recommendation.
|
||
|
||
---
|
||
|
||
## When to use
|
||
|
||
- "Why is `<release>` Pending on `<cluster>`?"
|
||
- "Pods for `<app>` are scheduling on the wrong node pool."
|
||
- "PVC for `<release>` is stuck Pending."
|
||
|
||
Do **not** use this skill to:
|
||
|
||
- Apply a fix without a separate explicit request (use `onboard-app` / a manual values PR).
|
||
- Run `kubectl delete pod` / `kubectl drain` / any cluster mutation.
|
||
|
||
---
|
||
|
||
## Input
|
||
|
||
Required:
|
||
|
||
```yaml
|
||
release: <release-name-or-app> # e.g. kube-state-metrics, contour-internal-0
|
||
cluster: <cluster-folder> # e.g. k8s-supply-prd-ase1
|
||
```
|
||
|
||
Optional but useful:
|
||
|
||
```yaml
|
||
namespace: <ns> # if not standard
|
||
symptom: <free-text> # what the user is seeing
|
||
```
|
||
|
||
---
|
||
|
||
## Steps (deterministic walk)
|
||
|
||
### Step 1 — Identify the values file
|
||
|
||
```bash
|
||
release=<release>
|
||
cluster=<cluster>
|
||
|
||
ls helm-overrides/$cluster/$release/custom-values.yaml 2>/dev/null \
|
||
|| find helm-overrides/$cluster -maxdepth 2 -name 'custom-values.yaml' -path "*${release}*"
|
||
```
|
||
|
||
If nothing: surface "no override file found for `$release` on `$cluster` — is it onboarded? does the release name match the directory?" and stop.
|
||
|
||
### Step 2 — Read the values' scheduling block
|
||
|
||
```bash
|
||
yq e '{nodeSelector: .nodeSelector, tolerations: .tolerations, affinity: .affinity}' \
|
||
helm-overrides/$cluster/$release/custom-values.yaml
|
||
```
|
||
|
||
### Step 3 — Determine the cluster's actual scheduling profile
|
||
|
||
```bash
|
||
# Sample 3 sibling apps to see the cluster's key style
|
||
for f in $(ls helm-overrides/$cluster/*/custom-values.yaml 2>/dev/null | head -3); do
|
||
echo "--- $f ---"
|
||
yq e '.nodeSelector' "$f"
|
||
done
|
||
```
|
||
|
||
Identify whether the cluster uses `dedicated:` keys or `cloud.google.com/compute-class:` keys.
|
||
|
||
### Step 4 — Run the live-cluster checks (if accessible)
|
||
|
||
```bash
|
||
NS=${namespace:-$(yq e '.namespace // "default"' helm-overrides/$cluster/$release/custom-values.yaml)}
|
||
CTX=<kubectl-context-for-$cluster>
|
||
|
||
kubectl --context=$CTX -n $NS get pods -o wide
|
||
kubectl --context=$CTX -n $NS describe pod <pending-pod> | tail -30
|
||
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20
|
||
kubectl --context=$CTX -n $NS get pvc
|
||
```
|
||
|
||
If you can't reach the cluster: mark step 4 "unknown — needs operator with cluster access" and produce a partial diagnosis from steps 1–3.
|
||
|
||
### Step 5 — Walk the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree
|
||
|
||
For each node, record:
|
||
|
||
- The check performed.
|
||
- The observed value.
|
||
- Which branch you take.
|
||
|
||
### Step 6 — Produce the structured report
|
||
|
||
```markdown
|
||
## Diagnosis: <release> on <cluster>
|
||
|
||
**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
|
||
**Namespace:** `<ns>`
|
||
**Pod state:** `Pending` / `Running on wrong node` / `PVC Pending` / ...
|
||
|
||
### Values-side checks
|
||
- [✅/⚠️/❌] `nodeSelector` key style matches cluster: `<dedicated|compute-class>`
|
||
- [✅/⚠️/❌] `nodeSelector` value matches an existing pool / ComputeClass on this cluster
|
||
- [✅/⚠️/❌] `tolerations` cover the node taints
|
||
- [✅/⚠️/❌] `resources.requests` reasonable for cluster's pool sizes
|
||
- [✅/⚠️/❌] `persistence.storageClass` (if set) exists in `manifests/storageclass/`
|
||
- [✅/⚠️/❌] `existingSecret` (if set) exists per the cluster's `external-secrets/`
|
||
|
||
### Cluster-side checks (if accessible)
|
||
- Pod events (top 5):
|
||
```
|
||
<events>
|
||
```
|
||
- Node availability summary:
|
||
```
|
||
<kubectl get nodes -o wide top 5>
|
||
```
|
||
|
||
### Root cause hypothesis
|
||
<one paragraph — e.g. "Values use `cloud.google.com/compute-class: contour-internal-0-cc`,
|
||
but `k8s-supply-prd-ase1` is a standard GKE cluster (uses `dedicated:` keys). Pods cannot
|
||
match any node. Almost certainly copied from `k8s-central-prd-ase1`'s values.">
|
||
|
||
### Recommended next step
|
||
- **Layer 1 fix (PR):** "Open a PR rewriting `nodeSelector` and `tolerations` in `helm-overrides/<cluster>/<release>/custom-values.yaml` from scratch using sibling apps on `<cluster>` as reference. Specifically: replace `cloud.google.com/compute-class: contour-internal-0-cc` with `dedicated: <pool>`."
|
||
- **Layer 2 advisory:** (if needed) "Recommend cluster owner increase node-pool max from N to M to relieve resource pressure."
|
||
- **Layer 3 refusal:** (if applicable) "Cannot infer the right pool name from this repo alone — needs cluster owner to confirm."
|
||
|
||
### References
|
||
- [pod-pending-scheduling.md §1](../../docs/platform/runbooks/pod-pending-scheduling.md)
|
||
- [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) (if Contour)
|
||
- [SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md)
|
||
```
|
||
|
||
---
|
||
|
||
## Output
|
||
|
||
A single markdown report (the structured shape above). The skill does **not**:
|
||
|
||
- Open a PR (use the matching procedure / `onboard-app` skill).
|
||
- Mutate the cluster.
|
||
- Click Sync.
|
||
|
||
If the recommendation is a Layer 1 PR, name the procedure (e.g. "follow [pod-pending-scheduling §1](../../docs/platform/runbooks/pod-pending-scheduling.md) — values rewrite") rather than blind-generating the diff.
|
||
|
||
---
|
||
|
||
## Gotchas
|
||
|
||
1. **A pod scheduled-but-on-wrong-node** is harder to diagnose than a Pending pod. Always check `kubectl get pod <pod> -o wide` to see the actual node.
|
||
2. **PVC `Pending` is sometimes a chained failure** — the pod that would consume it is `Pending` because of scheduling, and the PVC's `WaitForFirstConsumer` mode means it won't bind until a pod is scheduled. Resolve scheduling first.
|
||
3. **Some clusters have `taints` that look like keys but are values** — read the actual node label/taint, not the values file's interpretation.
|
||
4. **`db-*` dataplane clusters** have minimal sibling apps to compare against. Be extra careful.
|
||
5. **Cluster-level issues** (CNI broken, kubelet wedged, node-pool quota) look like scheduling failures from inside the values. Always note the cluster-side context if uncertain.
|
||
|
||
---
|
||
|
||
## Layer constraint
|
||
|
||
Mostly Layer 2. Output a diagnosis; don't execute. If the diagnosis points at a Layer 1 fix, recommend the matching procedure or `onboard-app` skill; don't open the PR as a side effect.
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
|
||
- Reference: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).
|