devops-infra-helm-charts-gcp/skills/infra/check-cluster-health.md

Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.

Skill — check-cluster-health

Layer: Layer 2 (advisory) — read-only health summary. No writes, no PRs. Scope: the infra surface managed via devops-infra-helm-charts and observed on the live cluster.

This skill produces a structured, read-only health report for a target cluster. It does NOT mutate state, open PRs, or recommend specific values diffs as a side effect — when an issue is found, it points at the procedure / skill to invoke next, then stops.

---

When to Use

Triggers like:

• "Check the health of <cluster>."
• "Is k8s-supply-prd-ase1 healthy?"
• "Pre-incident sanity check on <cluster>."
• "Status report for cluster owners."

Do not use this skill for:

• Diagnosing a single app / service — use diagnose-deployment.md or diagnose-scheduling.md.
• Fixing anything — handoff to the matching procedure.
• Checking workload-cluster app health — that's app-team scope.

---

Input

Required:

cluster:    <cluster-folder>          # e.g. k8s-supply-prd-ase1

Optional:

kube_context: <ctx>                   # if different from inferred default
sections:                             # default = all
  - argo_apps
  - vm_agent_ingest
  - contour
  - eso
  - pvc
  - node_pool

---

Steps

Step 1 — Confirm the cluster directory exists

ls helm-overrides/<cluster>/ >/dev/null

If not: surface "no override directory for <cluster> — is the name correct? See 02-cluster-fleet.md for the fleet list." Stop.

Step 2 — Inventory expected apps

ls helm-overrides/<cluster>/ | sort

This is the agent's expected app list. Compare against actual Argo CD Applications on the cluster.

Step 3 — Argo CD Applications status

CTX=<kube_context_or_inferred>
kubectl --context=$CTX -n argocd get applications -o json | \
  jq -r '.items[] | select(.spec.destination.name == "<cluster>" or .spec.destination.server == "https://<cluster>-api") |
         {name: .metadata.name, sync: .status.sync.status, health: .status.health.status, msg: .status.conditions}'

Tally:

• Synced + Healthy: count
• OutOfSync: list names
• Degraded: list names + health.message
• Missing: apps in step 2 but not in this list (orphaned overrides)

Step 4 — VictoriaMetrics agent ingest

kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent --tail=50 | grep -iE 'error|failed|429' | head -10

Surface:

• Pod count and Ready ratio.
• Recent remote_write errors (if any).
• If 429 Too Many Requests from vmstorage in the last hour — flag as "ingest saturation; see metrics-gap.md §4."

Step 5 — Contour pod readiness

for c in $(ls helm-overrides/<cluster>/ | grep '^contour'); do
  echo "=== $c ==="
  kubectl --context=$CTX -n projectcontour get pods -l app.kubernetes.io/instance=$c 2>/dev/null \
    || kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c
done

Surface per-Contour-instance:

• Ready/Total.
• Any Pending / CrashLoopBackOff → flag with handoff to ingress-down.md.

Step 6 — External Secrets controller

kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=50 | grep -iE 'error|failed' | head -10

Sample a few ExternalSecrets for Ready: True:

kubectl --context=$CTX get externalsecret -A -o json | \
  jq -r '.items[] | {ns: .metadata.namespace, name: .metadata.name,
                     ready: (.status.conditions[]? | select(.type=="Ready") | .status)}' | head -10

Surface:

• ESO controller pod state.
• Count of Ready: False ExternalSecrets across the cluster.
• Any SecretSyncError in last hour → handoff to vault-unavailable.md.

Step 7 — PVC binding

kubectl --context=$CTX get pvc -A | awk '$4 != "Bound" && NR > 1'

Surface unbound PVCs (Pending / Lost) with namespace + name. Cross-reference storageclass-priorityclass-schema.md.

Step 8 — Node pool capacity

kubectl --context=$CTX get nodes -o wide
kubectl --context=$CTX top nodes 2>/dev/null  # may fail if metrics-server is the thing that's down
kubectl --context=$CTX describe nodes | grep -E 'Allocated resources|Taints' | head -40

Surface:

• Node count.
• Any node NotReady.
• Any node-pool with allocated CPU/memory >85% (capacity-limited).

Step 9 — Recent merges to this cluster's overrides

git log --since='24 hours ago' --oneline -- helm-overrides/<cluster>/

Surface recent merges — context for any "regression after deploy" hypotheses.

---

Output

Structured markdown report:

# Cluster Health: <cluster>

**Generated:** <UTC timestamp>
**Inferred kube-context:** <ctx>
**Expected app inventory (from helm-overrides/):** N apps

## Argo CD Applications
- Synced + Healthy: X / N
- OutOfSync: <list or "none">
- Degraded: <list with health.message or "none">
- Orphaned (in repo, not in cluster): <list or "none">

## VictoriaMetrics agent
- Pods: X/Y Ready
- Recent ingest errors (last 50 log lines): <count, pattern>
- Saturation flag: <yes — see metrics-gap.md §4 | no>

## Contour ingress
| Instance | Ready/Total | Notes |
|----------|-------------|-------|
| contour-external | 3/3 | OK |
| contour-internal-0 | 2/3 | 1 Pending — see ingress-down.md §1 |

## External Secrets Operator
- Controller pods: X/Y Ready
- Failing ExternalSecrets: <count> (samples: <list>)
- Vault outage indicator: <yes — see vault-unavailable.md | no>

## PVC binding
- Unbound PVCs: <list or "none">

## Node pool
- Nodes: X (Y Ready)
- Capacity flags: <pool=foo at 88% CPU | none>

## Recent overrides merges (24h)
- <commit list or "none">

---

## Recommended next actions
- <if any issue surfaced, point at the runbook/skill — no fixes here>
- <e.g. "1 Contour pod Pending on contour-internal-0 — invoke skill: diagnose-scheduling">

---

Pattern Reference

• Argo CD Application status fields: spec.destination.name, status.sync.status, status.health.status, status.conditions.
• Healthy steady-state baselines vary per cluster — for a baseline, consult 02-cluster-fleet.md and the cluster's prior-week historical snapshot if one exists.
• Multi-Contour topology — see contour-nodeselector-tolerations-summary.md.

---

Gotchas (Layer constraints, common mistakes)

1. Read-only. This skill never writes. If the user asks "now fix it" inline, refuse the inline fix: hand off to the matching procedure / skill (diagnose-scheduling, diagnose-deployment, add-infra-tool, etc.). The agent does not chain a write into this skill's session.
2. Don't curl production endpoints as part of health checks. No curl prd.meeshogcp.in, curl prd.meesho.int, etc. — see SANCTITY_RULES R3. All checks are kubectl-internal or in-cluster curl Pods.
3. db-* dataplane clusters have a minimal app surface (kube-state-metrics, victoria-metrics-agent) — most sections of this report will be N/A. Don't flag the absence as "Degraded."
4. k8s-shared-int-ase1 is the only non-prod cluster — issues there are not pages, they're warnings.
5. kubectl top nodes may fail if metrics-server is the failure. Don't trust its absence as "no data; OK."
6. Argo CD Application count drift between repo overrides and live Applications can be intentional during a blue-green migration — do not auto-flag as a problem; surface and let the user judge.
7. kube-context inference is brittle. If the user hasn't specified kube_context, ask before running kubectl against the wrong cluster.

---

Layer constraint

Layer 2 (read-only advisory). Output a report; never write, never PR, never Sync. If an issue is found, name the runbook/skill to invoke next; do not chain.

---

Related

• Runbook: argocd-sync-failure.md.
• Runbook: ingress-down.md.
• Runbook: pod-pending-scheduling.md.
• Runbook: metrics-gap.md.
• Runbook: vault-unavailable.md.
• Skill: diagnose-deployment.md — when narrowing to a single app.
• Skill: diagnose-scheduling.md — when narrowing to scheduling.
• Boundaries: ../../docs/global/AGENT_BOUNDARIES.md, ../../docs/global/SANCTITY_RULES.md.