Skill — `diagnose-scheduling`

Layer: mostly Layer 2 (advisory — produces diagnosis and recommended action). Layer 1 only when the recommendation is "open this PR with this diff." Scope: infra workloads stuck Pending, scheduling onto wrong nodes, or with PVCs unbound.

This skill walks the pod-pending-scheduling.md decision tree and outputs a structured diagnosis + recommendation.

When to use

"Why is <release> Pending on <cluster>?"
"Pods for <app> are scheduling on the wrong node pool."
"PVC for <release> is stuck Pending."

Do not use this skill to:

Apply a fix without a separate explicit request (use onboard-app / a manual values PR).
Run kubectl delete pod / kubectl drain / any cluster mutation.

Input

Required:

release:    <release-name-or-app>     # e.g. kube-state-metrics, contour-internal-0
cluster:    <cluster-folder>          # e.g. k8s-supply-prd-ase1

Optional but useful:

namespace:  <ns>                      # if not standard
symptom:    <free-text>               # what the user is seeing

Steps (deterministic walk)

Step 1 — Identify the values file

release=<release>
cluster=<cluster>

ls helm-overrides/$cluster/$release/custom-values.yaml 2>/dev/null \
  || find helm-overrides/$cluster -maxdepth 2 -name 'custom-values.yaml' -path "*${release}*"

If nothing: surface "no override file found for $release on $cluster — is it onboarded? does the release name match the directory?" and stop.

Step 2 — Read the values' scheduling block

yq e '{nodeSelector: .nodeSelector, tolerations: .tolerations, affinity: .affinity}' \
  helm-overrides/$cluster/$release/custom-values.yaml

Step 3 — Determine the cluster's actual scheduling profile

# Sample 3 sibling apps to see the cluster's key style
for f in $(ls helm-overrides/$cluster/*/custom-values.yaml 2>/dev/null | head -3); do
  echo "--- $f ---"
  yq e '.nodeSelector' "$f"
done

Identify whether the cluster uses dedicated: keys or cloud.google.com/compute-class: keys.

Step 4 — Run the live-cluster checks (if accessible)

NS=${namespace:-$(yq e '.namespace // "default"' helm-overrides/$cluster/$release/custom-values.yaml)}
CTX=<kubectl-context-for-$cluster>

kubectl --context=$CTX -n $NS get pods -o wide
kubectl --context=$CTX -n $NS describe pod <pending-pod> | tail -30
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20
kubectl --context=$CTX -n $NS get pvc

If you can't reach the cluster: mark step 4 "unknown — needs operator with cluster access" and produce a partial diagnosis from steps 1–3.

Step 5 — Walk the pod-pending-scheduling.md decision tree

For each node, record:

The check performed.
The observed value.
Which branch you take.

Step 6 — Produce the structured report

## Diagnosis: <release> on <cluster>

**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
**Namespace:** `<ns>`
**Pod state:** `Pending` / `Running on wrong node` / `PVC Pending` / ...

### Values-side checks
- [✅/⚠️/❌] `nodeSelector` key style matches cluster: `<dedicated|compute-class>`
- [✅/⚠️/❌] `nodeSelector` value matches an existing pool / ComputeClass on this cluster
- [✅/⚠️/❌] `tolerations` cover the node taints
- [✅/⚠️/❌] `resources.requests` reasonable for cluster's pool sizes
- [✅/⚠️/❌] `persistence.storageClass` (if set) exists in `manifests/storageclass/`
- [✅/⚠️/❌] `existingSecret` (if set) exists per the cluster's `external-secrets/`

### Cluster-side checks (if accessible)
- Pod events (top 5):

``` - Node availability summary: ``` ```

Root cause hypothesis

Recommended next step

Layer 1 fix (PR): "Open a PR rewriting nodeSelector and tolerations in helm-overrides/<cluster>/<release>/custom-values.yaml from scratch using sibling apps on <cluster> as reference. Specifically: replace cloud.google.com/compute-class: contour-internal-0-cc with dedicated: <pool>."
Layer 2 advisory: (if needed) "Recommend cluster owner increase node-pool max from N to M to relieve resource pressure."
Layer 3 refusal: (if applicable) "Cannot infer the right pool name from this repo alone — needs cluster owner to confirm."

References

pod-pending-scheduling.md §1
contour-nodeselector-tolerations-summary.md (if Contour)
SANCTITY_RULES R5


---

## Output

A single markdown report (the structured shape above). The skill does **not**:

- Open a PR (use the matching procedure / `onboard-app` skill).
- Mutate the cluster.
- Click Sync.

If the recommendation is a Layer 1 PR, name the procedure (e.g. "follow [pod-pending-scheduling §1](../../docs/platform/runbooks/pod-pending-scheduling.md) — values rewrite") rather than blind-generating the diff.

---

## Gotchas

1. **A pod scheduled-but-on-wrong-node** is harder to diagnose than a Pending pod. Always check `kubectl get pod <pod> -o wide` to see the actual node.
2. **PVC `Pending` is sometimes a chained failure** — the pod that would consume it is `Pending` because of scheduling, and the PVC's `WaitForFirstConsumer` mode means it won't bind until a pod is scheduled. Resolve scheduling first.
3. **Some clusters have `taints` that look like keys but are values** — read the actual node label/taint, not the values file's interpretation.
4. **`db-*` dataplane clusters** have minimal sibling apps to compare against. Be extra careful.
5. **Cluster-level issues** (CNI broken, kubelet wedged, node-pool quota) look like scheduling failures from inside the values. Always note the cluster-side context if uncertain.

---

## Layer constraint

Mostly Layer 2. Output a diagnosis; don't execute. If the diagnosis points at a Layer 1 fix, recommend the matching procedure or `onboard-app` skill; don't open the PR as a side effect.

---

## Related

- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
- Reference: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).

6.8 KiB Raw Blame History Unescape Escape

Skill — diagnose-scheduling