# Skill — `diagnose-scheduling`

> **Layer:** mostly Layer 2 (advisory — produces diagnosis and recommended action). Layer 1 only when the recommendation is "open this PR with this diff."
> **Scope:** infra workloads stuck `Pending`, scheduling onto wrong nodes, or with PVCs unbound.

This skill walks the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree and outputs a structured diagnosis + recommendation.

---

## When to use

- "Why is `<release>` Pending on `<cluster>`?"
- "Pods for `<app>` are scheduling on the wrong node pool."
- "PVC for `<release>` is stuck Pending."

Do **not** use this skill to:

- Apply a fix without a separate explicit request (use `onboard-app` / a manual values PR).
- Run `kubectl delete pod` / `kubectl drain` / any cluster mutation.

---

## Input

Required:

```yaml
release:    <release-name-or-app>     # e.g. kube-state-metrics, contour-internal-0
cluster:    <cluster-folder>          # e.g. k8s-supply-prd-ase1
```

Optional but useful:

```yaml
namespace:  <ns>                      # if not standard
symptom:    <free-text>               # what the user is seeing
```

---

## Steps (deterministic walk)

### Step 1 — Identify the values file

```bash
release=<release>
cluster=<cluster>

ls helm-overrides/$cluster/$release/custom-values.yaml 2>/dev/null \
  || find helm-overrides/$cluster -maxdepth 2 -name 'custom-values.yaml' -path "*${release}*"
```

If nothing: surface "no override file found for `$release` on `$cluster` — is it onboarded? does the release name match the directory?" and stop.

### Step 2 — Read the values' scheduling block

```bash
yq e '{nodeSelector: .nodeSelector, tolerations: .tolerations, affinity: .affinity}' \
  helm-overrides/$cluster/$release/custom-values.yaml
```

### Step 3 — Determine the cluster's actual scheduling profile

```bash
# Sample 3 sibling apps to see the cluster's key style
for f in $(ls helm-overrides/$cluster/*/custom-values.yaml 2>/dev/null | head -3); do
  echo "--- $f ---"
  yq e '.nodeSelector' "$f"
done
```

Identify whether the cluster uses `dedicated:` keys or `cloud.google.com/compute-class:` keys.

### Step 4 — Run the live-cluster checks (if accessible)

```bash
NS=${namespace:-$(yq e '.namespace // "default"' helm-overrides/$cluster/$release/custom-values.yaml)}
CTX=<kubectl-context-for-$cluster>

kubectl --context=$CTX -n $NS get pods -o wide
kubectl --context=$CTX -n $NS describe pod <pending-pod> | tail -30
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20
kubectl --context=$CTX -n $NS get pvc
```

If you can't reach the cluster: mark step 4 "unknown — needs operator with cluster access" and produce a partial diagnosis from steps 1–3.

### Step 5 — Walk the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree

For each node, record:

- The check performed.
- The observed value.
- Which branch you take.

### Step 6 — Produce the structured report

```markdown
## Diagnosis: <release> on <cluster>

**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
**Namespace:** `<ns>`
**Pod state:** `Pending` / `Running on wrong node` / `PVC Pending` / ...

### Values-side checks
- [✅/⚠️/❌] `nodeSelector` key style matches cluster: `<dedicated|compute-class>`
- [✅/⚠️/❌] `nodeSelector` value matches an existing pool / ComputeClass on this cluster
- [✅/⚠️/❌] `tolerations` cover the node taints
- [✅/⚠️/❌] `resources.requests` reasonable for cluster's pool sizes
- [✅/⚠️/❌] `persistence.storageClass` (if set) exists in `manifests/storageclass/`
- [✅/⚠️/❌] `existingSecret` (if set) exists per the cluster's `external-secrets/`

### Cluster-side checks (if accessible)
- Pod events (top 5):
  ```
  <events>
  ```
- Node availability summary:
  ```
  <kubectl get nodes -o wide top 5>
  ```

### Root cause hypothesis
<one paragraph — e.g. "Values use `cloud.google.com/compute-class: contour-internal-0-cc`,
but `k8s-supply-prd-ase1` is a standard GKE cluster (uses `dedicated:` keys). Pods cannot
match any node. Almost certainly copied from `k8s-central-prd-ase1`'s values.">

### Recommended next step
- **Layer 1 fix (PR):** "Open a PR rewriting `nodeSelector` and `tolerations` in `helm-overrides/<cluster>/<release>/custom-values.yaml` from scratch using sibling apps on `<cluster>` as reference. Specifically: replace `cloud.google.com/compute-class: contour-internal-0-cc` with `dedicated: <pool>`."
- **Layer 2 advisory:** (if needed) "Recommend cluster owner increase node-pool max from N to M to relieve resource pressure."
- **Layer 3 refusal:** (if applicable) "Cannot infer the right pool name from this repo alone — needs cluster owner to confirm."

### References
- [pod-pending-scheduling.md §1](../../docs/platform/runbooks/pod-pending-scheduling.md)
- [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) (if Contour)
- [SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md)
```

---

## Output

A single markdown report (the structured shape above). The skill does **not**:

- Open a PR (use the matching procedure / `onboard-app` skill).
- Mutate the cluster.
- Click Sync.

If the recommendation is a Layer 1 PR, name the procedure (e.g. "follow [pod-pending-scheduling §1](../../docs/platform/runbooks/pod-pending-scheduling.md) — values rewrite") rather than blind-generating the diff.

---

## Gotchas

1. **A pod scheduled-but-on-wrong-node** is harder to diagnose than a Pending pod. Always check `kubectl get pod <pod> -o wide` to see the actual node.
2. **PVC `Pending` is sometimes a chained failure** — the pod that would consume it is `Pending` because of scheduling, and the PVC's `WaitForFirstConsumer` mode means it won't bind until a pod is scheduled. Resolve scheduling first.
3. **Some clusters have `taints` that look like keys but are values** — read the actual node label/taint, not the values file's interpretation.
4. **`db-*` dataplane clusters** have minimal sibling apps to compare against. Be extra careful.
5. **Cluster-level issues** (CNI broken, kubelet wedged, node-pool quota) look like scheduling failures from inside the values. Always note the cluster-side context if uncertain.

---

## Layer constraint

Mostly Layer 2. Output a diagnosis; don't execute. If the diagnosis points at a Layer 1 fix, recommend the matching procedure or `onboard-app` skill; don't open the PR as a side effect.

---

## Related

- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
- Reference: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).