devops-infra-helm-charts-gcp/skills/infra/diagnose-deployment.md

> Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.

# Skill — `diagnose-deployment`

> **Layer:** Layer 2 (advisory) — read-only diagnosis. **No writes, no Sync, no PRs.**
> **Scope:** infra deployments managed via this repo on a target cluster.

This skill produces a structured diagnosis for a single infra deployment / Helm release, walking the right runbook decision tree based on the observed symptom. It outputs a hypothesis + the recommended next-action procedure or skill — and stops. The agent does not chain into a write.

---

## When to Use

Triggers like:

- "What's wrong with `<release>` on `<cluster>`?"
- "Diagnose `<release>` — pods are restarting."
- "Why is `<release>` showing OutOfSync in Argo?"
- "Service `<svc>` returning 503 on `<cluster>` — check the infra side."

Do **not** use this skill for:

- Cluster-wide health — use [check-cluster-health.md](check-cluster-health.md).
- Application code bugs — out of scope, hand off to app team.
- Direct fix application — invoke the procedure / skill the diagnosis points at, separately.

---

## Input

Required:

```yaml
release:    <name>                    # e.g. mimir, contour-internal-0, kube-state-metrics
cluster:    <cluster-folder>          # e.g. k8s-supply-prd-ase1
```

Optional:

```yaml
namespace:  <ns>                      # if non-default for the chart
symptom:    <free-text>               # what the user observed
kube_context: <ctx>
```

---

## Steps

### Step 1 — Locate the values file

```bash
ls helm-overrides/<cluster>/<release>/custom-values.yaml 2>/dev/null \
  || find helm-overrides/<cluster> -maxdepth 2 -name 'custom-values.yaml' -path "*<release>*"
```

If nothing: surface "no override file found for `<release>` on `<cluster>` — is it onboarded? does the directory name match the release?" Stop.

### Step 2 — Pull live state

```bash
CTX=<kube_context_or_inferred>
NS=<namespace_or_inferred>

# Argo Application
kubectl --context=$CTX -n argocd get application | grep <release>
kubectl --context=$CTX -n argocd get application <release>-<cluster> -o yaml | yq e '.status'

# Pods
kubectl --context=$CTX -n $NS get pods -l app.kubernetes.io/instance=<release> -o wide

# Recent events
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20

# Logs (last 100 lines)
kubectl --context=$CTX -n $NS logs -l app.kubernetes.io/instance=<release> --tail=100 --all-containers 2>/dev/null | tail -50
```

### Step 3 — Classify the symptom

Walk this matrix to pick the right runbook:

| Observed | Branch |
|----------|--------|
| Argo Application `OutOfSync` or `SyncFailed` | §A → [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md) |
| Pods `Pending` or scheduled on wrong node | §B → [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) |
| Pods `CrashLoopBackOff` referencing missing Secret | §C → [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md) |
| Ingress 5xx / 503 / no endpoints | §D → [ingress-down.md](../../docs/platform/runbooks/ingress-down.md) |
| Metrics gap / Grafana blank / silent alert | §E → [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md) |
| Pods `Running` but app erroring | §F — out of scope (app team) |

Multiple symptoms? Pick the most-upstream (sync first, then scheduling, then secrets, then ingress, then metrics).

### Step 4 — Walk the chosen runbook

For the chosen branch, walk its decision tree to the lowest leaf. Record at each node:

- Check performed.
- Observed value (from step 2's data).
- Branch taken.

For runbooks the agent can fully resolve from kubectl output (e.g. `pod-pending-scheduling.md §1` — taint mismatch), name the leaf. For runbooks needing data the agent doesn't have (Vault server-side, GCP IAM), mark the leaf "needs operator with [X] access" and stop.

### Step 5 — Cross-check the values file

Pull the relevant values keys for the symptom:

```bash
yq e '{
  image: .image,
  resources: .resources,
  nodeSelector: .nodeSelector,
  tolerations: .tolerations,
  persistence: .persistence,
  serviceAccount: .serviceAccount,
  existingSecret: .existingSecret
}' helm-overrides/<cluster>/<release>/custom-values.yaml
```

Look for:
- Image tag suspicious (`latest`, recent bump?).
- nodeSelector key style mismatches the cluster (Autopilot vs standard).
- existingSecret references a Secret that step 2 showed missing.

### Step 6 — Recent merges

```bash
git log --since='48 hours ago' --oneline -- helm-overrides/<cluster>/<release>/
```

If a recent merge is implicated (timing matches the symptom), the diagnosis names the merge and recommends revert as the first remediation.

### Step 7 — Produce the report

```markdown
# Diagnosis: <release> on <cluster>

**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
**Namespace:** `<ns>`
**Inferred kube-context:** `<ctx>`
**Symptom (user-reported):** `<symptom>`

## Live state (snapshot)
- Argo Application: `<sync-status> / <health-status>`
- Pods: `<count Ready / count Total>`, states: `<list>`
- Recent events (relevant):
  ```
  <events>
  ```
- Pod logs (relevant):
  ```
  <logs excerpt>
  ```

## Symptom classification
**Branch chosen:** §<X> — <runbook name>
**Why:** <one-line>

## Decision-tree walk
1. <node> — checked: `<X>` — observed: `<Y>` → branch `<Z>`
2. <node> — checked: `<X>` — observed: `<Y>` → branch `<Z>`
...

## Values-file cross-check
- [✅/⚠️/❌] image.tag: `<v>` — <ok / suspicious>
- [✅/⚠️/❌] nodeSelector key style matches cluster: `<dedicated|compute-class>`
- [✅/⚠️/❌] existingSecret: `<name>` — <exists / missing on cluster>
- [✅/⚠️/❌] persistence.storageClass: `<sc>` — <exists / missing in manifests/storageclass/>

## Recent merges (last 48h)
- <commit list or "none — pre-existing condition">

## Root-cause hypothesis
<one paragraph>

## Recommended next-action
- **Procedure / skill to invoke:** `<path>`
- **Layer:** 1 (values PR) | 2 (advisory — operator action) | 3 (refusal — out of repo scope)
- **Rough diff intent (if Layer 1):** "Edit `helm-overrides/<cluster>/<release>/custom-values.yaml` to <change>." (Do not generate the diff in this skill.)

## Cannot resolve from this repo
<list anything that needs cluster owner / platform / security access>
```

---

## Pattern Reference

- Decision-tree branches map 1:1 to the runbook section headers (`§A` ↔ `argocd-sync-failure.md`, etc.).
- The "two-page rule": every diagnosis cites at most two pages — one runbook (the branch) and one procedure/skill (the recommended next-action). Long chains imply the agent should stop and ask.

---

## Gotchas (Layer constraints, common mistakes)

1. **Read-only.** No writes. No PR. If the user says "now fix it," respond with "invoking [procedure/skill]" and stop in this skill — chain to the next as a fresh invocation.
2. **Stop at the first solid hypothesis.** Don't keep walking trees once one fits. Surface the hypothesis with a confidence note and the recommended next-action.
3. **`OutOfSync` is sometimes intentional** during a blue-green or manual-sync window. Cross-check with the cluster owner before declaring a problem.
4. **`Running but erroring` is app-team territory.** Don't grep app logs for application bugs — surface the pod is `Running` and hand off.
5. **Don't curl production endpoints.** All probes are kubectl-internal or via an in-cluster curl Pod ([SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md)).
6. **Vault outage triggers diagnosis stop, not fix.** §C ends at "escalate to security team" — Vault HA is Layer 3.
7. **Multi-Contour confusion.** A 5xx is from one Contour instance, not all six. Always identify which contour-* release routes the path before declaring "ingress is down."
8. **Recent-merge blame is correlation, not causation.** Surface the timing; don't auto-recommend revert without the user confirming the user-visible symptom started after the merge.

---

## Layer constraint

Layer 2 (read-only advisory). Output a diagnosis report; don't execute. The recommendation ALWAYS names a separate procedure / skill — do not silently slide into Layer 1 from this skill.

---

## Related

- Skill: [check-cluster-health.md](check-cluster-health.md) — broader, cluster-wide read-only.
- Skill: [diagnose-scheduling.md](diagnose-scheduling.md) — narrower, scheduling-only.
- Skill: [add-infra-tool.md](add-infra-tool.md) — Layer 1 follow-up if onboarding gap.
- Skill: [bump-chart-version.md](bump-chart-version.md) — Layer 1 follow-up if chart-version gap.
- Runbook: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
- Runbook: [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md).
- Runbook: [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md).
- Boundaries: [../../docs/global/AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md), [../../docs/global/SANCTITY_RULES.md](../../docs/global/SANCTITY_RULES.md).