gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b8575bb8b9a28e3c2f6deef59947edc4a74ff357
devops-infra-helm-charts-gcp/skills/infra/diagnose-deployment.md
T
Your Name b8575bb8b9 added repo
2026-08-26 03:39:42 +05:30

9.0 KiB
Raw Blame History

Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.

Skill — diagnose-deployment

Layer: Layer 2 (advisory) — read-only diagnosis. No writes, no Sync, no PRs. Scope: infra deployments managed via this repo on a target cluster.

This skill produces a structured diagnosis for a single infra deployment / Helm release, walking the right runbook decision tree based on the observed symptom. It outputs a hypothesis + the recommended next-action procedure or skill — and stops. The agent does not chain into a write.

When to Use

Triggers like:

  • "What's wrong with <release> on <cluster>?"
  • "Diagnose <release> — pods are restarting."
  • "Why is <release> showing OutOfSync in Argo?"
  • "Service <svc> returning 503 on <cluster> — check the infra side."

Do not use this skill for:

  • Cluster-wide health — use check-cluster-health.md.
  • Application code bugs — out of scope, hand off to app team.
  • Direct fix application — invoke the procedure / skill the diagnosis points at, separately.

Input

Required:

release:    <name>                    # e.g. mimir, contour-internal-0, kube-state-metrics
cluster:    <cluster-folder>          # e.g. k8s-supply-prd-ase1

Optional:

namespace:  <ns>                      # if non-default for the chart
symptom:    <free-text>               # what the user observed
kube_context: <ctx>

Steps

Step 1 — Locate the values file

ls helm-overrides/<cluster>/<release>/custom-values.yaml 2>/dev/null \
  || find helm-overrides/<cluster> -maxdepth 2 -name 'custom-values.yaml' -path "*<release>*"

If nothing: surface "no override file found for <release> on <cluster> — is it onboarded? does the directory name match the release?" Stop.

Step 2 — Pull live state

CTX=<kube_context_or_inferred>
NS=<namespace_or_inferred>

# Argo Application
kubectl --context=$CTX -n argocd get application | grep <release>
kubectl --context=$CTX -n argocd get application <release>-<cluster> -o yaml | yq e '.status'

# Pods
kubectl --context=$CTX -n $NS get pods -l app.kubernetes.io/instance=<release> -o wide

# Recent events
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20

# Logs (last 100 lines)
kubectl --context=$CTX -n $NS logs -l app.kubernetes.io/instance=<release> --tail=100 --all-containers 2>/dev/null | tail -50

Step 3 — Classify the symptom

Walk this matrix to pick the right runbook:

Observed Branch
Argo Application OutOfSync or SyncFailed §A → argocd-sync-failure.md
Pods Pending or scheduled on wrong node §B → pod-pending-scheduling.md
Pods CrashLoopBackOff referencing missing Secret §C → vault-unavailable.md
Ingress 5xx / 503 / no endpoints §D → ingress-down.md
Metrics gap / Grafana blank / silent alert §E → metrics-gap.md
Pods Running but app erroring §F — out of scope (app team)

Multiple symptoms? Pick the most-upstream (sync first, then scheduling, then secrets, then ingress, then metrics).

Step 4 — Walk the chosen runbook

For the chosen branch, walk its decision tree to the lowest leaf. Record at each node:

  • Check performed.
  • Observed value (from step 2's data).
  • Branch taken.

For runbooks the agent can fully resolve from kubectl output (e.g. pod-pending-scheduling.md §1 — taint mismatch), name the leaf. For runbooks needing data the agent doesn't have (Vault server-side, GCP IAM), mark the leaf "needs operator with [X] access" and stop.

Step 5 — Cross-check the values file

Pull the relevant values keys for the symptom:

yq e '{
  image: .image,
  resources: .resources,
  nodeSelector: .nodeSelector,
  tolerations: .tolerations,
  persistence: .persistence,
  serviceAccount: .serviceAccount,
  existingSecret: .existingSecret
}' helm-overrides/<cluster>/<release>/custom-values.yaml

Look for:

  • Image tag suspicious (latest, recent bump?).
  • nodeSelector key style mismatches the cluster (Autopilot vs standard).
  • existingSecret references a Secret that step 2 showed missing.

Step 6 — Recent merges

git log --since='48 hours ago' --oneline -- helm-overrides/<cluster>/<release>/

If a recent merge is implicated (timing matches the symptom), the diagnosis names the merge and recommends revert as the first remediation.

Step 7 — Produce the report

# Diagnosis: <release> on <cluster>

**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
**Namespace:** `<ns>`
**Inferred kube-context:** `<ctx>`
**Symptom (user-reported):** `<symptom>`

## Live state (snapshot)
- Argo Application: `<sync-status> / <health-status>`
- Pods: `<count Ready / count Total>`, states: `<list>`
- Recent events (relevant):
``` - Pod logs (relevant): ``` ```

Symptom classification

Branch chosen: § — Why:

Decision-tree walk

  1. — checked: <X> — observed: <Y> → branch <Z>
  2. — checked: <X> — observed: <Y> → branch <Z> ...

Values-file cross-check

  • [✅/⚠️/❌] image.tag: <v> — <ok / suspicious>
  • [✅/⚠️/❌] nodeSelector key style matches cluster: <dedicated|compute-class>
  • [✅/⚠️/❌] existingSecret: <name> — <exists / missing on cluster>
  • [✅/⚠️/❌] persistence.storageClass: <sc> — <exists / missing in manifests/storageclass/>

Recent merges (last 48h)

  • <commit list or "none — pre-existing condition">

Root-cause hypothesis

Recommended next-action

  • Procedure / skill to invoke: <path>
  • Layer: 1 (values PR) | 2 (advisory — operator action) | 3 (refusal — out of repo scope)
  • Rough diff intent (if Layer 1): "Edit helm-overrides/<cluster>/<release>/custom-values.yaml to ." (Do not generate the diff in this skill.)

Cannot resolve from this repo

<list anything that needs cluster owner / platform / security access>


---

## Pattern Reference

- Decision-tree branches map 1:1 to the runbook section headers (`§A` ↔ `argocd-sync-failure.md`, etc.).
- The "two-page rule": every diagnosis cites at most two pages — one runbook (the branch) and one procedure/skill (the recommended next-action). Long chains imply the agent should stop and ask.

---

## Gotchas (Layer constraints, common mistakes)

1. **Read-only.** No writes. No PR. If the user says "now fix it," respond with "invoking [procedure/skill]" and stop in this skill — chain to the next as a fresh invocation.
2. **Stop at the first solid hypothesis.** Don't keep walking trees once one fits. Surface the hypothesis with a confidence note and the recommended next-action.
3. **`OutOfSync` is sometimes intentional** during a blue-green or manual-sync window. Cross-check with the cluster owner before declaring a problem.
4. **`Running but erroring` is app-team territory.** Don't grep app logs for application bugs — surface the pod is `Running` and hand off.
5. **Don't curl production endpoints.** All probes are kubectl-internal or via an in-cluster curl Pod ([SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md)).
6. **Vault outage triggers diagnosis stop, not fix.** §C ends at "escalate to security team" — Vault HA is Layer 3.
7. **Multi-Contour confusion.** A 5xx is from one Contour instance, not all six. Always identify which contour-* release routes the path before declaring "ingress is down."
8. **Recent-merge blame is correlation, not causation.** Surface the timing; don't auto-recommend revert without the user confirming the user-visible symptom started after the merge.

---

## Layer constraint

Layer 2 (read-only advisory). Output a diagnosis report; don't execute. The recommendation ALWAYS names a separate procedure / skill — do not silently slide into Layer 1 from this skill.

---

## Related

- Skill: [check-cluster-health.md](check-cluster-health.md) — broader, cluster-wide read-only.
- Skill: [diagnose-scheduling.md](diagnose-scheduling.md) — narrower, scheduling-only.
- Skill: [add-infra-tool.md](add-infra-tool.md) — Layer 1 follow-up if onboarding gap.
- Skill: [bump-chart-version.md](bump-chart-version.md) — Layer 1 follow-up if chart-version gap.
- Runbook: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
- Runbook: [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md).
- Runbook: [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md).
- Boundaries: [../../docs/global/AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md), [../../docs/global/SANCTITY_RULES.md](../../docs/global/SANCTITY_RULES.md).
Reference in New Issue View Git Blame Copy Permalink