added repo
This commit is contained in:
Your Name
@@ -0,0 +1,252 @@
|
||||
> Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Skill — `add-infra-tool`
|
||||
|
||||
> **Layer:** Layer 1 — agent generates diff(s) and opens PR(s) in this repo and (separately) in the sister repo. Reviewer + Argo CD UI Sync click are the human gates.
|
||||
> **Scope:** the values-side slice in `devops-infra-helm-charts` plus the matching Argo `Application` slice in `github.com/Meesho/devops-infra-argo-config`.
|
||||
|
||||
This skill is the agent-callable form for adding a new infrastructure tool to a target cluster. It parameterises the [onboard-app-to-cluster.md](../../docs/platform/procedures/onboard-app-to-cluster.md) procedure and pairs the values-side PR with a sister-repo PR for the Argo CD `Application`.
|
||||
|
||||
For graduating an incubator tool fleet-wide, see [../../docs/platform/schemas/incubator-values-schema.md](../../docs/platform/schemas/incubator-values-schema.md).
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
Triggers like:
|
||||
|
||||
- "Add `<tool>` to `<cluster>`."
|
||||
- "Bring up `<tool>` on `k8s-shared-int-ase1` for trial."
|
||||
- "Onboard a new infra tool — chart already exists in `helm-templates/`."
|
||||
- "Land an incubator deploy of `<tool>` on the integration cluster."
|
||||
|
||||
Do **not** use this skill for:
|
||||
|
||||
- Onboarding a service workload — services live in their own repos, not here.
|
||||
- Adding a route to an existing tool — see [add-contour-route.md](../../docs/platform/procedures/add-contour-route.md).
|
||||
- Bumping an existing tool's chart version — use [bump-chart-version.md](bump-chart-version.md).
|
||||
- Adding a brand-new cluster — use the [onboard-new-cluster.md](../../docs/platform/procedures/onboard-new-cluster.md) procedure.
|
||||
- Editing observability / alert rules — [diagnose-deployment.md](diagnose-deployment.md) routes to the right procedure.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
tool: <name> # e.g. cert-manager, keda, kyverno
|
||||
target_cluster: <cluster-folder> # e.g. k8s-shared-int-ase1
|
||||
chart_source: existing | new-vendored # is helm-templates/<tool>/ already present?
|
||||
release_name: <name> # often == tool
|
||||
workload_namespace: <ns>
|
||||
image_tag: <pinned-tag> # NEVER 'latest'
|
||||
resources:
|
||||
cpu_request: <e.g. 250m>
|
||||
memory_request: <e.g. 512Mi>
|
||||
cpu_limit: <e.g. 1>
|
||||
memory_limit: <e.g. 2Gi>
|
||||
node_pool_key: dedicated | cloud.google.com/compute-class
|
||||
node_pool_value: <pool-or-cc-name>
|
||||
```
|
||||
|
||||
Optional:
|
||||
|
||||
```yaml
|
||||
incubator: <bool> # adds top-of-file comment, disables autoscaling defaults
|
||||
needs_external_dns: <bool>
|
||||
needs_external_secret: <bool>
|
||||
needs_compute_class: <bool> # GKE Autopilot only
|
||||
priority_class: <name-in-manifests/priorityclass/<cluster>/>
|
||||
persistence:
|
||||
enabled: <bool>
|
||||
storage_class: <name>
|
||||
size: <e.g. 100Gi>
|
||||
sister_repo_app_name: <name> # default: <release_name>-<target_cluster>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Verify pre-conditions
|
||||
|
||||
```bash
|
||||
# Chart exists (chart_source: existing)
|
||||
ls helm-templates/<tool>/Chart.yaml
|
||||
|
||||
# Cluster exists
|
||||
ls helm-overrides/<target_cluster>/
|
||||
|
||||
# Tool not already onboarded here
|
||||
[ ! -d helm-overrides/<target_cluster>/<tool>/ ]
|
||||
|
||||
# StorageClass exists if persistence.enabled
|
||||
ls manifests/storageclass/<storage_class>.yaml
|
||||
|
||||
# PriorityClass exists if priority_class set
|
||||
ls manifests/priorityclass/<target_cluster>/<priority_class>.yaml
|
||||
```
|
||||
|
||||
If `chart_source: new-vendored`, the chart must already be in `helm-templates/<tool>/`. If not, that is a separate (chart-vendoring) PR — fail fast and ask the user to land that first. **Do not** vendor the chart inside this skill.
|
||||
|
||||
### 2. Read the cluster's scheduling profile
|
||||
|
||||
Sample 3 sibling apps on the **same** cluster:
|
||||
|
||||
```bash
|
||||
for f in $(ls helm-overrides/<target_cluster>/*/custom-values.yaml | grep -v contour | head -3); do
|
||||
echo "--- $f ---"
|
||||
yq e '.nodeSelector, .tolerations' "$f"
|
||||
done
|
||||
```
|
||||
|
||||
Confirm `node_pool_key` matches the cluster's actual style (`dedicated:` for standard GKE; `cloud.google.com/compute-class:` for Autopilot — `k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`). Mismatch → fail fast.
|
||||
|
||||
### 3. Generate `helm-overrides/<target_cluster>/<tool>/custom-values.yaml`
|
||||
|
||||
Skeleton:
|
||||
|
||||
```yaml
|
||||
# Incubator: cluster=<target_cluster>, owner=<owner>, graduation-target=<date> # only if incubator=true
|
||||
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<tool>
|
||||
tag: <image_tag>
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
replicaCount: 1 # incubator default; raise after load profile
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: <cpu_request>
|
||||
memory: <memory_request>
|
||||
limits:
|
||||
cpu: <cpu_limit>
|
||||
memory: <memory_limit>
|
||||
|
||||
nodeSelector:
|
||||
<node_pool_key>: <node_pool_value>
|
||||
tolerations:
|
||||
- key: <node_pool_key>
|
||||
value: <node_pool_value>
|
||||
effect: NoSchedule
|
||||
|
||||
# if persistence.enabled:
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: <storage_class>
|
||||
size: <size>
|
||||
accessModes: [ReadWriteOnce]
|
||||
|
||||
# if priority_class set:
|
||||
priorityClassName: <priority_class>
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: <release_name>
|
||||
annotations: {} # add iam.gke.io/gcp-service-account if WI binding needed
|
||||
```
|
||||
|
||||
### 4. Sidecar manifests (conditional)
|
||||
|
||||
- `needs_compute_class: true` → write `helm-overrides/<target_cluster>/<tool>/computeclass/<node_pool_value>.yaml`. `metadata.name` MUST equal `<node_pool_value>`. See [raw-manifest-sidecar-schema.md §ComputeClass](../../docs/platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
- `needs_external_dns: true` → write `helm-overrides/<target_cluster>/<tool>/external-dns-services/<svc>.yaml`. **Skip in incubator deploys.** See [raw-manifest-sidecar-schema.md §Service for external-dns binding](../../docs/platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
- `needs_external_secret: true` → write `helm-overrides/<target_cluster>/external-secrets/<release_name>.yaml`. Reference an existing `SecretStore` / `ClusterSecretStore`.
|
||||
|
||||
### 5. Validate
|
||||
|
||||
```bash
|
||||
yamllint helm-overrides/<target_cluster>/<tool>/custom-values.yaml
|
||||
|
||||
helm template <release_name> helm-templates/<tool> \
|
||||
-f helm-overrides/<target_cluster>/<tool>/custom-values.yaml > /tmp/render.yaml
|
||||
|
||||
# Optional: dry-run sidecar manifests
|
||||
kubectl --context=<ctx> --dry-run=server -f helm-overrides/<target_cluster>/<tool>/computeclass/ apply 2>/dev/null || true
|
||||
```
|
||||
|
||||
`helm template` must succeed.
|
||||
|
||||
### 6. Open the values-side PR (this repo)
|
||||
|
||||
```bash
|
||||
git checkout -b add-tool/<tool>-on-<target_cluster>
|
||||
git add helm-overrides/<target_cluster>/<tool>/
|
||||
git commit -m "Add <tool> to <target_cluster>"
|
||||
git push origin add-tool/<tool>-on-<target_cluster>
|
||||
gh pr create --base main --title "Add <tool> to <target_cluster>"
|
||||
```
|
||||
|
||||
### 7. Open the sister-repo PR (Argo Application)
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`, draft an Application:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: <sister_repo_app_name>
|
||||
namespace: argocd
|
||||
spec:
|
||||
destination:
|
||||
name: <target_cluster>
|
||||
namespace: <workload_namespace>
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
path: helm-overrides/<target_cluster>/<tool>
|
||||
helm:
|
||||
valueFiles: [custom-values.yaml]
|
||||
syncPolicy:
|
||||
syncOptions: [CreateNamespace=true]
|
||||
# NO automated.{prune,selfHeal} — manual sync per ADR-A5
|
||||
```
|
||||
|
||||
PR-link the values-side PR in the description; PR-link the sister-repo PR back in the values-side PR description.
|
||||
|
||||
### 8. Hand off — do NOT Sync from this skill
|
||||
|
||||
Argo CD UI Sync is a human gate. The skill stops at "two PRs open with green CI." The user clicks Sync after both merge.
|
||||
|
||||
---
|
||||
|
||||
## Pattern Reference
|
||||
|
||||
- Recent onboardings to compare against: `git log --oneline --grep='Onboard\|Add' -i -- helm-overrides/ | head -10` and inspect the diffs.
|
||||
- For Contour onboardings (multi-instance, scheduling-heavy): always cross-reference [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||||
- For incubator-style first deploys: see [../../docs/platform/schemas/incubator-values-schema.md](../../docs/platform/schemas/incubator-values-schema.md).
|
||||
|
||||
---
|
||||
|
||||
## Gotchas (Layer constraints, common mistakes)
|
||||
|
||||
1. **Per-cluster scheduling is non-portable.** Never copy `nodeSelector` / `tolerations` / `computeClass` from another cluster — author from scratch using same-cluster siblings ([SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md)).
|
||||
2. **Vanilla-chart edits are forks.** If `helm-templates/<tool>/` is vanilla upstream, do not edit `templates/` or `values.yaml` to add knobs ([NEVER-DO](../../CLAUDE.md)). Wrap with a Meesho chart or upstream-PR.
|
||||
3. **Never bypass TruffleHog.** No `--no-verify`, no `git commit -n`, no removing the hook. Real secrets via `ExternalSecret`.
|
||||
4. **`fullnameOverride` is load-bearing.** Set deliberately or omit; never change later.
|
||||
5. **`image.tag: latest` is forbidden.** Always pin.
|
||||
6. **Sister-repo PR is mandatory.** Without an `Application`, the values do nothing on the cluster. Do not "ship just the values."
|
||||
7. **Manual sync is the default for infra apps** — do not set `automated.{prune,selfHeal}: true` to "make it easier" ([ADR-A5](../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)).
|
||||
8. **`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1` are GKE Autopilot** — they use `cloud.google.com/compute-class:` not `dedicated:`.
|
||||
9. **`db-*` clusters have minimal sibling apps** to compare scheduling against. Confirm with cluster owner before guessing.
|
||||
10. **No production hostnames** as readiness probes / values URLs ([SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md)).
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Layer 1. Open both PRs; do not merge them; do not Sync. Reviewer + Argo CD UI Sync click are the human gates.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [../../docs/platform/procedures/onboard-app-to-cluster.md](../../docs/platform/procedures/onboard-app-to-cluster.md).
|
||||
- Procedure: [../../docs/platform/procedures/onboard-new-cluster.md](../../docs/platform/procedures/onboard-new-cluster.md) — for brand-new clusters.
|
||||
- Schema: [../../docs/platform/schemas/custom-values-schema.md](../../docs/platform/schemas/custom-values-schema.md).
|
||||
- Schema: [../../docs/platform/schemas/incubator-values-schema.md](../../docs/platform/schemas/incubator-values-schema.md).
|
||||
- Schema: [../../docs/platform/schemas/raw-manifest-sidecar-schema.md](../../docs/platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
- Skill: [onboard-app.md](onboard-app.md) — the existing peer skill (older variant; this skill supersedes for incubator-aware inputs).
|
||||
- Boundaries: [../../docs/global/AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).
|
||||
@@ -0,0 +1,157 @@
|
||||
# Skill — `bump-chart-version`
|
||||
|
||||
> **Layer:** Layer 1 — agent generates diff and opens PR; platform team reviews; per-cluster Sync is the deploy.
|
||||
> **Scope:** updating `helm-templates/<chart>/Chart.yaml` `dependencies[].version` and refreshing `Chart.lock`.
|
||||
|
||||
This skill is the agent-callable form of [docs/platform/procedures/update-chart-version.md](../../docs/platform/procedures/update-chart-version.md).
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
Triggers like:
|
||||
|
||||
- "Bump `<chart>` from `<old>` to `<new>`."
|
||||
- "Upgrade `argo-cd` to 7.8.0."
|
||||
- "Apply CVE patch to `<chart>` — bump to <new>."
|
||||
|
||||
Do **not** use this skill for:
|
||||
|
||||
- Major-version bumps with breaking template changes (use `blue-green-chart-migration` procedure).
|
||||
- Forking a chart (use `fork-upstream-chart` procedure).
|
||||
- Bumping a chart's local version when it's already a fork (the `version:` field at the top of `Chart.yaml`, not `dependencies[].version`).
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
chart: <chart-name> # must exist in helm-templates/
|
||||
old_version: <semver> # current dependencies[].version
|
||||
new_version: <semver> # target
|
||||
```
|
||||
|
||||
Optional:
|
||||
|
||||
```yaml
|
||||
representative_cluster: <cluster-folder> # for the test render; if omitted, skill picks one
|
||||
sample_helm_diff: <bool> # if true, run helm diff against a live cluster (requires kube context)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps (deterministic)
|
||||
|
||||
1. **Pre-conditions.**
|
||||
```bash
|
||||
ls helm-templates/<chart>/Chart.yaml
|
||||
yq e '.dependencies[0].version' helm-templates/<chart>/Chart.yaml # confirm == <old_version>
|
||||
```
|
||||
|
||||
2. **Read the changelog.** Output a one-line note with the changelog URL or an explicit "READ THE CHANGELOG: <url>" message. The skill does not auto-fetch; the user must confirm they've read it. If the user hasn't, **stop**.
|
||||
|
||||
3. **Update `Chart.yaml`.**
|
||||
```bash
|
||||
sed -i.bak "s/^ version: <old_version>$/ version: <new_version>/" \
|
||||
helm-templates/<chart>/Chart.yaml
|
||||
rm helm-templates/<chart>/Chart.yaml.bak
|
||||
```
|
||||
(Or use `yq` with a path expression — depending on `dependencies[]` shape.)
|
||||
|
||||
4. **Refresh `Chart.lock`.**
|
||||
```bash
|
||||
helm dependency update helm-templates/<chart>
|
||||
```
|
||||
On failure: surface the error and stop. Common causes: new version doesn't exist, repo URL changed, network issue.
|
||||
|
||||
5. **Render against the representative cluster.**
|
||||
```bash
|
||||
sibling=${representative_cluster:-$(find helm-overrides -maxdepth 2 -type d -name '<chart>' \
|
||||
| grep -v '^helm-overrides/db-' | head -1)}
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" > /tmp/render.yaml
|
||||
```
|
||||
On failure: surface the error and **roll back** (`git checkout helm-templates/<chart>/Chart.yaml helm-templates/<chart>/Chart.lock`); stop and report. The bump is incompatible with current values.
|
||||
|
||||
6. **(If `sample_helm_diff: true`)**
|
||||
```bash
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<sibling>/<chart>/custom-values.yaml \
|
||||
--kube-context=<context>
|
||||
```
|
||||
Capture the diff and put it in the PR body.
|
||||
|
||||
7. **Open the PR.**
|
||||
```bash
|
||||
git checkout -b chart-bump/<chart>-<new_version>
|
||||
git add helm-templates/<chart>/Chart.yaml helm-templates/<chart>/Chart.lock
|
||||
git add helm-templates/<chart>/charts/ # if subchart .tgz refreshed
|
||||
git commit -m "chart bump: <chart> <old_version> -> <new_version>"
|
||||
git push origin chart-bump/<chart>-<new_version>
|
||||
gh pr create --base main --title "chart bump: <chart> <old_version> -> <new_version>"
|
||||
```
|
||||
|
||||
PR body (heredoc):
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
Bumps `<chart>` from `<old_version>` to `<new_version>`.
|
||||
|
||||
- Upstream changelog: <url>
|
||||
- Procedure: `docs/platform/procedures/update-chart-version.md`
|
||||
- Skill: `skills/infra/bump-chart-version.md`
|
||||
|
||||
## Affected clusters
|
||||
<list from `grep -rl '<chart>' helm-overrides | sort -u`>
|
||||
|
||||
## Validation
|
||||
- `helm dependency update` succeeded
|
||||
- Render against `<representative_cluster>`'s overrides: clean
|
||||
- (if sample_helm_diff) diff against live: <one-line summary>
|
||||
|
||||
## Approver
|
||||
Platform team
|
||||
|
||||
## CMR
|
||||
<ticket if applicable>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
A PR diff with:
|
||||
|
||||
- `helm-templates/<chart>/Chart.yaml` updated (version line)
|
||||
- `helm-templates/<chart>/Chart.lock` refreshed
|
||||
- `helm-templates/<chart>/charts/*.tgz` (if the subchart was re-pulled)
|
||||
|
||||
The skill does **not**:
|
||||
|
||||
- Sync any cluster (manual per-cluster Sync after merge).
|
||||
- Update any `helm-overrides/<cluster>/<app>/custom-values.yaml` to handle a values-shape change. If the bump requires that, **stop and surface a follow-up task** — don't bundle.
|
||||
- Open per-cluster Sync recommendations as separate work items.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
1. **`Chart.yaml` without `Chart.lock` is a no-op.** Argo CD reads the lockfile.
|
||||
2. **The skill must roll back on render failure.** Otherwise a half-committed bump leaves the repo in a broken state.
|
||||
3. **For wrapper charts whose `dependencies[]` has multiple entries**, the skill must operate on the right one. Don't bump the wrong dep.
|
||||
4. **Major-version bumps are not this skill's job.** If `new_version` is a major increment (`7.x → 8.x`), refuse and recommend [blue-green-chart-migration](../../docs/platform/procedures/blue-green-chart-migration.md).
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Layer 1. Open the PR; do not merge it; do not click Sync on any cluster.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](../../docs/platform/procedures/update-chart-version.md).
|
||||
- Procedure: [blue-green-chart-migration.md](../../docs/platform/procedures/blue-green-chart-migration.md).
|
||||
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
@@ -0,0 +1,249 @@
|
||||
> Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Skill — `check-cluster-health`
|
||||
|
||||
> **Layer:** Layer 2 (advisory) — read-only health summary. **No writes, no PRs.**
|
||||
> **Scope:** the infra surface managed via `devops-infra-helm-charts` and observed on the live cluster.
|
||||
|
||||
This skill produces a structured, read-only health report for a target cluster. It does NOT mutate state, open PRs, or recommend specific values diffs as a side effect — when an issue is found, it points at the procedure / skill to invoke next, then stops.
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
Triggers like:
|
||||
|
||||
- "Check the health of `<cluster>`."
|
||||
- "Is `k8s-supply-prd-ase1` healthy?"
|
||||
- "Pre-incident sanity check on `<cluster>`."
|
||||
- "Status report for cluster owners."
|
||||
|
||||
Do **not** use this skill for:
|
||||
|
||||
- Diagnosing a single app / service — use [diagnose-deployment.md](diagnose-deployment.md) or [diagnose-scheduling.md](diagnose-scheduling.md).
|
||||
- Fixing anything — handoff to the matching procedure.
|
||||
- Checking workload-cluster app health — that's app-team scope.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
cluster: <cluster-folder> # e.g. k8s-supply-prd-ase1
|
||||
```
|
||||
|
||||
Optional:
|
||||
|
||||
```yaml
|
||||
kube_context: <ctx> # if different from inferred default
|
||||
sections: # default = all
|
||||
- argo_apps
|
||||
- vm_agent_ingest
|
||||
- contour
|
||||
- eso
|
||||
- pvc
|
||||
- node_pool
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Confirm the cluster directory exists
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/ >/dev/null
|
||||
```
|
||||
|
||||
If not: surface "no override directory for `<cluster>` — is the name correct? See [02-cluster-fleet.md](../../claude/02-cluster-fleet.md) for the fleet list." Stop.
|
||||
|
||||
### Step 2 — Inventory expected apps
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/ | sort
|
||||
```
|
||||
|
||||
This is the agent's expected app list. Compare against actual Argo CD Applications on the cluster.
|
||||
|
||||
### Step 3 — Argo CD Applications status
|
||||
|
||||
```bash
|
||||
CTX=<kube_context_or_inferred>
|
||||
kubectl --context=$CTX -n argocd get applications -o json | \
|
||||
jq -r '.items[] | select(.spec.destination.name == "<cluster>" or .spec.destination.server == "https://<cluster>-api") |
|
||||
{name: .metadata.name, sync: .status.sync.status, health: .status.health.status, msg: .status.conditions}'
|
||||
```
|
||||
|
||||
Tally:
|
||||
- Synced + Healthy: count
|
||||
- OutOfSync: list names
|
||||
- Degraded: list names + `health.message`
|
||||
- Missing: apps in step 2 but not in this list (orphaned overrides)
|
||||
|
||||
### Step 4 — VictoriaMetrics agent ingest
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
|
||||
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent --tail=50 | grep -iE 'error|failed|429' | head -10
|
||||
```
|
||||
|
||||
Surface:
|
||||
- Pod count and Ready ratio.
|
||||
- Recent remote_write errors (if any).
|
||||
- If `429 Too Many Requests` from vmstorage in the last hour — flag as "ingest saturation; see [metrics-gap.md §4](../../docs/platform/runbooks/metrics-gap.md)."
|
||||
|
||||
### Step 5 — Contour pod readiness
|
||||
|
||||
```bash
|
||||
for c in $(ls helm-overrides/<cluster>/ | grep '^contour'); do
|
||||
echo "=== $c ==="
|
||||
kubectl --context=$CTX -n projectcontour get pods -l app.kubernetes.io/instance=$c 2>/dev/null \
|
||||
|| kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c
|
||||
done
|
||||
```
|
||||
|
||||
Surface per-Contour-instance:
|
||||
- Ready/Total.
|
||||
- Any Pending / CrashLoopBackOff → flag with handoff to [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
|
||||
|
||||
### Step 6 — External Secrets controller
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n external-secrets get pods
|
||||
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=50 | grep -iE 'error|failed' | head -10
|
||||
```
|
||||
|
||||
Sample a few `ExternalSecret`s for `Ready: True`:
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get externalsecret -A -o json | \
|
||||
jq -r '.items[] | {ns: .metadata.namespace, name: .metadata.name,
|
||||
ready: (.status.conditions[]? | select(.type=="Ready") | .status)}' | head -10
|
||||
```
|
||||
|
||||
Surface:
|
||||
- ESO controller pod state.
|
||||
- Count of `Ready: False` ExternalSecrets across the cluster.
|
||||
- Any `SecretSyncError` in last hour → handoff to [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md).
|
||||
|
||||
### Step 7 — PVC binding
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get pvc -A | awk '$4 != "Bound" && NR > 1'
|
||||
```
|
||||
|
||||
Surface unbound PVCs (Pending / Lost) with namespace + name. Cross-reference [storageclass-priorityclass-schema.md](../../docs/platform/schemas/storageclass-priorityclass-schema.md).
|
||||
|
||||
### Step 8 — Node pool capacity
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get nodes -o wide
|
||||
kubectl --context=$CTX top nodes 2>/dev/null # may fail if metrics-server is the thing that's down
|
||||
kubectl --context=$CTX describe nodes | grep -E 'Allocated resources|Taints' | head -40
|
||||
```
|
||||
|
||||
Surface:
|
||||
- Node count.
|
||||
- Any node `NotReady`.
|
||||
- Any node-pool with allocated CPU/memory >85% (capacity-limited).
|
||||
|
||||
### Step 9 — Recent merges to this cluster's overrides
|
||||
|
||||
```bash
|
||||
git log --since='24 hours ago' --oneline -- helm-overrides/<cluster>/
|
||||
```
|
||||
|
||||
Surface recent merges — context for any "regression after deploy" hypotheses.
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
Structured markdown report:
|
||||
|
||||
```markdown
|
||||
# Cluster Health: <cluster>
|
||||
|
||||
**Generated:** <UTC timestamp>
|
||||
**Inferred kube-context:** <ctx>
|
||||
**Expected app inventory (from helm-overrides/):** N apps
|
||||
|
||||
## Argo CD Applications
|
||||
- Synced + Healthy: X / N
|
||||
- OutOfSync: <list or "none">
|
||||
- Degraded: <list with health.message or "none">
|
||||
- Orphaned (in repo, not in cluster): <list or "none">
|
||||
|
||||
## VictoriaMetrics agent
|
||||
- Pods: X/Y Ready
|
||||
- Recent ingest errors (last 50 log lines): <count, pattern>
|
||||
- Saturation flag: <yes — see metrics-gap.md §4 | no>
|
||||
|
||||
## Contour ingress
|
||||
| Instance | Ready/Total | Notes |
|
||||
|----------|-------------|-------|
|
||||
| contour-external | 3/3 | OK |
|
||||
| contour-internal-0 | 2/3 | 1 Pending — see ingress-down.md §1 |
|
||||
|
||||
## External Secrets Operator
|
||||
- Controller pods: X/Y Ready
|
||||
- Failing ExternalSecrets: <count> (samples: <list>)
|
||||
- Vault outage indicator: <yes — see vault-unavailable.md | no>
|
||||
|
||||
## PVC binding
|
||||
- Unbound PVCs: <list or "none">
|
||||
|
||||
## Node pool
|
||||
- Nodes: X (Y Ready)
|
||||
- Capacity flags: <pool=foo at 88% CPU | none>
|
||||
|
||||
## Recent overrides merges (24h)
|
||||
- <commit list or "none">
|
||||
|
||||
---
|
||||
|
||||
## Recommended next actions
|
||||
- <if any issue surfaced, point at the runbook/skill — no fixes here>
|
||||
- <e.g. "1 Contour pod Pending on contour-internal-0 — invoke skill: diagnose-scheduling">
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern Reference
|
||||
|
||||
- Argo CD Application status fields: `spec.destination.name`, `status.sync.status`, `status.health.status`, `status.conditions`.
|
||||
- Healthy steady-state baselines vary per cluster — for a baseline, consult [02-cluster-fleet.md](../../claude/02-cluster-fleet.md) and the cluster's prior-week historical snapshot if one exists.
|
||||
- Multi-Contour topology — see [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||||
|
||||
---
|
||||
|
||||
## Gotchas (Layer constraints, common mistakes)
|
||||
|
||||
1. **Read-only.** This skill never writes. If the user asks "now fix it" inline, **refuse the inline fix**: hand off to the matching procedure / skill (`diagnose-scheduling`, `diagnose-deployment`, `add-infra-tool`, etc.). The agent does not chain a write into this skill's session.
|
||||
2. **Don't curl production endpoints** as part of health checks. No `curl prd.meeshogcp.in`, `curl prd.meesho.int`, etc. — see [SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md). All checks are kubectl-internal or in-cluster `curl` Pods.
|
||||
3. **`db-*` dataplane clusters** have a minimal app surface (`kube-state-metrics`, `victoria-metrics-agent`) — most sections of this report will be N/A. Don't flag the absence as "Degraded."
|
||||
4. **`k8s-shared-int-ase1`** is the only non-prod cluster — issues there are not pages, they're warnings.
|
||||
5. **`kubectl top nodes` may fail** if `metrics-server` is the failure. Don't trust its absence as "no data; OK."
|
||||
6. **Argo CD Application count drift** between repo overrides and live Applications can be intentional during a blue-green migration — do not auto-flag as a problem; surface and let the user judge.
|
||||
7. **kube-context inference is brittle.** If the user hasn't specified `kube_context`, ask before running `kubectl` against the wrong cluster.
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Layer 2 (read-only advisory). Output a report; never write, never PR, never Sync. If an issue is found, name the runbook/skill to invoke next; do not chain.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Runbook: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
|
||||
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
|
||||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||||
- Runbook: [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md).
|
||||
- Runbook: [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md).
|
||||
- Skill: [diagnose-deployment.md](diagnose-deployment.md) — when narrowing to a single app.
|
||||
- Skill: [diagnose-scheduling.md](diagnose-scheduling.md) — when narrowing to scheduling.
|
||||
- Boundaries: [../../docs/global/AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md), [../../docs/global/SANCTITY_RULES.md](../../docs/global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,224 @@
|
||||
> Per AI Blitz Plan §skills.infra. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Skill — `diagnose-deployment`
|
||||
|
||||
> **Layer:** Layer 2 (advisory) — read-only diagnosis. **No writes, no Sync, no PRs.**
|
||||
> **Scope:** infra deployments managed via this repo on a target cluster.
|
||||
|
||||
This skill produces a structured diagnosis for a single infra deployment / Helm release, walking the right runbook decision tree based on the observed symptom. It outputs a hypothesis + the recommended next-action procedure or skill — and stops. The agent does not chain into a write.
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
Triggers like:
|
||||
|
||||
- "What's wrong with `<release>` on `<cluster>`?"
|
||||
- "Diagnose `<release>` — pods are restarting."
|
||||
- "Why is `<release>` showing OutOfSync in Argo?"
|
||||
- "Service `<svc>` returning 503 on `<cluster>` — check the infra side."
|
||||
|
||||
Do **not** use this skill for:
|
||||
|
||||
- Cluster-wide health — use [check-cluster-health.md](check-cluster-health.md).
|
||||
- Application code bugs — out of scope, hand off to app team.
|
||||
- Direct fix application — invoke the procedure / skill the diagnosis points at, separately.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
release: <name> # e.g. mimir, contour-internal-0, kube-state-metrics
|
||||
cluster: <cluster-folder> # e.g. k8s-supply-prd-ase1
|
||||
```
|
||||
|
||||
Optional:
|
||||
|
||||
```yaml
|
||||
namespace: <ns> # if non-default for the chart
|
||||
symptom: <free-text> # what the user observed
|
||||
kube_context: <ctx>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Locate the values file
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/<release>/custom-values.yaml 2>/dev/null \
|
||||
|| find helm-overrides/<cluster> -maxdepth 2 -name 'custom-values.yaml' -path "*<release>*"
|
||||
```
|
||||
|
||||
If nothing: surface "no override file found for `<release>` on `<cluster>` — is it onboarded? does the directory name match the release?" Stop.
|
||||
|
||||
### Step 2 — Pull live state
|
||||
|
||||
```bash
|
||||
CTX=<kube_context_or_inferred>
|
||||
NS=<namespace_or_inferred>
|
||||
|
||||
# Argo Application
|
||||
kubectl --context=$CTX -n argocd get application | grep <release>
|
||||
kubectl --context=$CTX -n argocd get application <release>-<cluster> -o yaml | yq e '.status'
|
||||
|
||||
# Pods
|
||||
kubectl --context=$CTX -n $NS get pods -l app.kubernetes.io/instance=<release> -o wide
|
||||
|
||||
# Recent events
|
||||
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20
|
||||
|
||||
# Logs (last 100 lines)
|
||||
kubectl --context=$CTX -n $NS logs -l app.kubernetes.io/instance=<release> --tail=100 --all-containers 2>/dev/null | tail -50
|
||||
```
|
||||
|
||||
### Step 3 — Classify the symptom
|
||||
|
||||
Walk this matrix to pick the right runbook:
|
||||
|
||||
| Observed | Branch |
|
||||
|----------|--------|
|
||||
| Argo Application `OutOfSync` or `SyncFailed` | §A → [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md) |
|
||||
| Pods `Pending` or scheduled on wrong node | §B → [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) |
|
||||
| Pods `CrashLoopBackOff` referencing missing Secret | §C → [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md) |
|
||||
| Ingress 5xx / 503 / no endpoints | §D → [ingress-down.md](../../docs/platform/runbooks/ingress-down.md) |
|
||||
| Metrics gap / Grafana blank / silent alert | §E → [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md) |
|
||||
| Pods `Running` but app erroring | §F — out of scope (app team) |
|
||||
|
||||
Multiple symptoms? Pick the most-upstream (sync first, then scheduling, then secrets, then ingress, then metrics).
|
||||
|
||||
### Step 4 — Walk the chosen runbook
|
||||
|
||||
For the chosen branch, walk its decision tree to the lowest leaf. Record at each node:
|
||||
|
||||
- Check performed.
|
||||
- Observed value (from step 2's data).
|
||||
- Branch taken.
|
||||
|
||||
For runbooks the agent can fully resolve from kubectl output (e.g. `pod-pending-scheduling.md §1` — taint mismatch), name the leaf. For runbooks needing data the agent doesn't have (Vault server-side, GCP IAM), mark the leaf "needs operator with [X] access" and stop.
|
||||
|
||||
### Step 5 — Cross-check the values file
|
||||
|
||||
Pull the relevant values keys for the symptom:
|
||||
|
||||
```bash
|
||||
yq e '{
|
||||
image: .image,
|
||||
resources: .resources,
|
||||
nodeSelector: .nodeSelector,
|
||||
tolerations: .tolerations,
|
||||
persistence: .persistence,
|
||||
serviceAccount: .serviceAccount,
|
||||
existingSecret: .existingSecret
|
||||
}' helm-overrides/<cluster>/<release>/custom-values.yaml
|
||||
```
|
||||
|
||||
Look for:
|
||||
- Image tag suspicious (`latest`, recent bump?).
|
||||
- nodeSelector key style mismatches the cluster (Autopilot vs standard).
|
||||
- existingSecret references a Secret that step 2 showed missing.
|
||||
|
||||
### Step 6 — Recent merges
|
||||
|
||||
```bash
|
||||
git log --since='48 hours ago' --oneline -- helm-overrides/<cluster>/<release>/
|
||||
```
|
||||
|
||||
If a recent merge is implicated (timing matches the symptom), the diagnosis names the merge and recommends revert as the first remediation.
|
||||
|
||||
### Step 7 — Produce the report
|
||||
|
||||
```markdown
|
||||
# Diagnosis: <release> on <cluster>
|
||||
|
||||
**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
|
||||
**Namespace:** `<ns>`
|
||||
**Inferred kube-context:** `<ctx>`
|
||||
**Symptom (user-reported):** `<symptom>`
|
||||
|
||||
## Live state (snapshot)
|
||||
- Argo Application: `<sync-status> / <health-status>`
|
||||
- Pods: `<count Ready / count Total>`, states: `<list>`
|
||||
- Recent events (relevant):
|
||||
```
|
||||
<events>
|
||||
```
|
||||
- Pod logs (relevant):
|
||||
```
|
||||
<logs excerpt>
|
||||
```
|
||||
|
||||
## Symptom classification
|
||||
**Branch chosen:** §<X> — <runbook name>
|
||||
**Why:** <one-line>
|
||||
|
||||
## Decision-tree walk
|
||||
1. <node> — checked: `<X>` — observed: `<Y>` → branch `<Z>`
|
||||
2. <node> — checked: `<X>` — observed: `<Y>` → branch `<Z>`
|
||||
...
|
||||
|
||||
## Values-file cross-check
|
||||
- [✅/⚠️/❌] image.tag: `<v>` — <ok / suspicious>
|
||||
- [✅/⚠️/❌] nodeSelector key style matches cluster: `<dedicated|compute-class>`
|
||||
- [✅/⚠️/❌] existingSecret: `<name>` — <exists / missing on cluster>
|
||||
- [✅/⚠️/❌] persistence.storageClass: `<sc>` — <exists / missing in manifests/storageclass/>
|
||||
|
||||
## Recent merges (last 48h)
|
||||
- <commit list or "none — pre-existing condition">
|
||||
|
||||
## Root-cause hypothesis
|
||||
<one paragraph>
|
||||
|
||||
## Recommended next-action
|
||||
- **Procedure / skill to invoke:** `<path>`
|
||||
- **Layer:** 1 (values PR) | 2 (advisory — operator action) | 3 (refusal — out of repo scope)
|
||||
- **Rough diff intent (if Layer 1):** "Edit `helm-overrides/<cluster>/<release>/custom-values.yaml` to <change>." (Do not generate the diff in this skill.)
|
||||
|
||||
## Cannot resolve from this repo
|
||||
<list anything that needs cluster owner / platform / security access>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern Reference
|
||||
|
||||
- Decision-tree branches map 1:1 to the runbook section headers (`§A` ↔ `argocd-sync-failure.md`, etc.).
|
||||
- The "two-page rule": every diagnosis cites at most two pages — one runbook (the branch) and one procedure/skill (the recommended next-action). Long chains imply the agent should stop and ask.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas (Layer constraints, common mistakes)
|
||||
|
||||
1. **Read-only.** No writes. No PR. If the user says "now fix it," respond with "invoking [procedure/skill]" and stop in this skill — chain to the next as a fresh invocation.
|
||||
2. **Stop at the first solid hypothesis.** Don't keep walking trees once one fits. Surface the hypothesis with a confidence note and the recommended next-action.
|
||||
3. **`OutOfSync` is sometimes intentional** during a blue-green or manual-sync window. Cross-check with the cluster owner before declaring a problem.
|
||||
4. **`Running but erroring` is app-team territory.** Don't grep app logs for application bugs — surface the pod is `Running` and hand off.
|
||||
5. **Don't curl production endpoints.** All probes are kubectl-internal or via an in-cluster curl Pod ([SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md)).
|
||||
6. **Vault outage triggers diagnosis stop, not fix.** §C ends at "escalate to security team" — Vault HA is Layer 3.
|
||||
7. **Multi-Contour confusion.** A 5xx is from one Contour instance, not all six. Always identify which contour-* release routes the path before declaring "ingress is down."
|
||||
8. **Recent-merge blame is correlation, not causation.** Surface the timing; don't auto-recommend revert without the user confirming the user-visible symptom started after the merge.
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Layer 2 (read-only advisory). Output a diagnosis report; don't execute. The recommendation ALWAYS names a separate procedure / skill — do not silently slide into Layer 1 from this skill.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Skill: [check-cluster-health.md](check-cluster-health.md) — broader, cluster-wide read-only.
|
||||
- Skill: [diagnose-scheduling.md](diagnose-scheduling.md) — narrower, scheduling-only.
|
||||
- Skill: [add-infra-tool.md](add-infra-tool.md) — Layer 1 follow-up if onboarding gap.
|
||||
- Skill: [bump-chart-version.md](bump-chart-version.md) — Layer 1 follow-up if chart-version gap.
|
||||
- Runbook: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
|
||||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||||
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
|
||||
- Runbook: [metrics-gap.md](../../docs/platform/runbooks/metrics-gap.md).
|
||||
- Runbook: [vault-unavailable.md](../../docs/platform/runbooks/vault-unavailable.md).
|
||||
- Boundaries: [../../docs/global/AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md), [../../docs/global/SANCTITY_RULES.md](../../docs/global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,174 @@
|
||||
# Skill — `diagnose-scheduling`
|
||||
|
||||
> **Layer:** mostly Layer 2 (advisory — produces diagnosis and recommended action). Layer 1 only when the recommendation is "open this PR with this diff."
|
||||
> **Scope:** infra workloads stuck `Pending`, scheduling onto wrong nodes, or with PVCs unbound.
|
||||
|
||||
This skill walks the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree and outputs a structured diagnosis + recommendation.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- "Why is `<release>` Pending on `<cluster>`?"
|
||||
- "Pods for `<app>` are scheduling on the wrong node pool."
|
||||
- "PVC for `<release>` is stuck Pending."
|
||||
|
||||
Do **not** use this skill to:
|
||||
|
||||
- Apply a fix without a separate explicit request (use `onboard-app` / a manual values PR).
|
||||
- Run `kubectl delete pod` / `kubectl drain` / any cluster mutation.
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
release: <release-name-or-app> # e.g. kube-state-metrics, contour-internal-0
|
||||
cluster: <cluster-folder> # e.g. k8s-supply-prd-ase1
|
||||
```
|
||||
|
||||
Optional but useful:
|
||||
|
||||
```yaml
|
||||
namespace: <ns> # if not standard
|
||||
symptom: <free-text> # what the user is seeing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps (deterministic walk)
|
||||
|
||||
### Step 1 — Identify the values file
|
||||
|
||||
```bash
|
||||
release=<release>
|
||||
cluster=<cluster>
|
||||
|
||||
ls helm-overrides/$cluster/$release/custom-values.yaml 2>/dev/null \
|
||||
|| find helm-overrides/$cluster -maxdepth 2 -name 'custom-values.yaml' -path "*${release}*"
|
||||
```
|
||||
|
||||
If nothing: surface "no override file found for `$release` on `$cluster` — is it onboarded? does the release name match the directory?" and stop.
|
||||
|
||||
### Step 2 — Read the values' scheduling block
|
||||
|
||||
```bash
|
||||
yq e '{nodeSelector: .nodeSelector, tolerations: .tolerations, affinity: .affinity}' \
|
||||
helm-overrides/$cluster/$release/custom-values.yaml
|
||||
```
|
||||
|
||||
### Step 3 — Determine the cluster's actual scheduling profile
|
||||
|
||||
```bash
|
||||
# Sample 3 sibling apps to see the cluster's key style
|
||||
for f in $(ls helm-overrides/$cluster/*/custom-values.yaml 2>/dev/null | head -3); do
|
||||
echo "--- $f ---"
|
||||
yq e '.nodeSelector' "$f"
|
||||
done
|
||||
```
|
||||
|
||||
Identify whether the cluster uses `dedicated:` keys or `cloud.google.com/compute-class:` keys.
|
||||
|
||||
### Step 4 — Run the live-cluster checks (if accessible)
|
||||
|
||||
```bash
|
||||
NS=${namespace:-$(yq e '.namespace // "default"' helm-overrides/$cluster/$release/custom-values.yaml)}
|
||||
CTX=<kubectl-context-for-$cluster>
|
||||
|
||||
kubectl --context=$CTX -n $NS get pods -o wide
|
||||
kubectl --context=$CTX -n $NS describe pod <pending-pod> | tail -30
|
||||
kubectl --context=$CTX -n $NS get events --sort-by=.lastTimestamp | tail -20
|
||||
kubectl --context=$CTX -n $NS get pvc
|
||||
```
|
||||
|
||||
If you can't reach the cluster: mark step 4 "unknown — needs operator with cluster access" and produce a partial diagnosis from steps 1–3.
|
||||
|
||||
### Step 5 — Walk the [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md) decision tree
|
||||
|
||||
For each node, record:
|
||||
|
||||
- The check performed.
|
||||
- The observed value.
|
||||
- Which branch you take.
|
||||
|
||||
### Step 6 — Produce the structured report
|
||||
|
||||
```markdown
|
||||
## Diagnosis: <release> on <cluster>
|
||||
|
||||
**Values file:** `helm-overrides/<cluster>/<release>/custom-values.yaml`
|
||||
**Namespace:** `<ns>`
|
||||
**Pod state:** `Pending` / `Running on wrong node` / `PVC Pending` / ...
|
||||
|
||||
### Values-side checks
|
||||
- [✅/⚠️/❌] `nodeSelector` key style matches cluster: `<dedicated|compute-class>`
|
||||
- [✅/⚠️/❌] `nodeSelector` value matches an existing pool / ComputeClass on this cluster
|
||||
- [✅/⚠️/❌] `tolerations` cover the node taints
|
||||
- [✅/⚠️/❌] `resources.requests` reasonable for cluster's pool sizes
|
||||
- [✅/⚠️/❌] `persistence.storageClass` (if set) exists in `manifests/storageclass/`
|
||||
- [✅/⚠️/❌] `existingSecret` (if set) exists per the cluster's `external-secrets/`
|
||||
|
||||
### Cluster-side checks (if accessible)
|
||||
- Pod events (top 5):
|
||||
```
|
||||
<events>
|
||||
```
|
||||
- Node availability summary:
|
||||
```
|
||||
<kubectl get nodes -o wide top 5>
|
||||
```
|
||||
|
||||
### Root cause hypothesis
|
||||
<one paragraph — e.g. "Values use `cloud.google.com/compute-class: contour-internal-0-cc`,
|
||||
but `k8s-supply-prd-ase1` is a standard GKE cluster (uses `dedicated:` keys). Pods cannot
|
||||
match any node. Almost certainly copied from `k8s-central-prd-ase1`'s values.">
|
||||
|
||||
### Recommended next step
|
||||
- **Layer 1 fix (PR):** "Open a PR rewriting `nodeSelector` and `tolerations` in `helm-overrides/<cluster>/<release>/custom-values.yaml` from scratch using sibling apps on `<cluster>` as reference. Specifically: replace `cloud.google.com/compute-class: contour-internal-0-cc` with `dedicated: <pool>`."
|
||||
- **Layer 2 advisory:** (if needed) "Recommend cluster owner increase node-pool max from N to M to relieve resource pressure."
|
||||
- **Layer 3 refusal:** (if applicable) "Cannot infer the right pool name from this repo alone — needs cluster owner to confirm."
|
||||
|
||||
### References
|
||||
- [pod-pending-scheduling.md §1](../../docs/platform/runbooks/pod-pending-scheduling.md)
|
||||
- [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) (if Contour)
|
||||
- [SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
A single markdown report (the structured shape above). The skill does **not**:
|
||||
|
||||
- Open a PR (use the matching procedure / `onboard-app` skill).
|
||||
- Mutate the cluster.
|
||||
- Click Sync.
|
||||
|
||||
If the recommendation is a Layer 1 PR, name the procedure (e.g. "follow [pod-pending-scheduling §1](../../docs/platform/runbooks/pod-pending-scheduling.md) — values rewrite") rather than blind-generating the diff.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
1. **A pod scheduled-but-on-wrong-node** is harder to diagnose than a Pending pod. Always check `kubectl get pod <pod> -o wide` to see the actual node.
|
||||
2. **PVC `Pending` is sometimes a chained failure** — the pod that would consume it is `Pending` because of scheduling, and the PVC's `WaitForFirstConsumer` mode means it won't bind until a pod is scheduled. Resolve scheduling first.
|
||||
3. **Some clusters have `taints` that look like keys but are values** — read the actual node label/taint, not the values file's interpretation.
|
||||
4. **`db-*` dataplane clusters** have minimal sibling apps to compare against. Be extra careful.
|
||||
5. **Cluster-level issues** (CNI broken, kubelet wedged, node-pool quota) look like scheduling failures from inside the values. Always note the cluster-side context if uncertain.
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Mostly Layer 2. Output a diagnosis; don't execute. If the diagnosis points at a Layer 1 fix, recommend the matching procedure or `onboard-app` skill; don't open the PR as a side effect.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||||
- Runbook: [ingress-down.md](../../docs/platform/runbooks/ingress-down.md).
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||||
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).
|
||||
@@ -0,0 +1,203 @@
|
||||
# Skill — `onboard-app`
|
||||
|
||||
> **Layer:** Layer 1 — agent generates diff(s) and opens PR(s); humans review and Sync.
|
||||
> **Scope:** the slice that lives in `devops-infra-helm-charts`. The matching Argo `Application` slice is in `github.com/Meesho/devops-infra-argo-config` and is paired but separate.
|
||||
|
||||
This skill is the agent-callable form of [docs/platform/procedures/onboard-app-to-cluster.md](../../docs/platform/procedures/onboard-app-to-cluster.md).
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
Triggers like:
|
||||
|
||||
- "Onboard `<app>` to `<cluster>`."
|
||||
- "Add a `kube-state-metrics` override for `k8s-foo-prd-ase1`."
|
||||
- "Bring up `external-dns` on the new cluster."
|
||||
|
||||
Do **not** use this skill for:
|
||||
|
||||
- Adding a brand-new cluster (use the `onboard-new-cluster` procedure).
|
||||
- Bumping a chart's version (use `bump-chart-version`).
|
||||
- Migrating a chart blue-green (use the `blue-green-chart-migration` procedure directly).
|
||||
|
||||
---
|
||||
|
||||
## Input
|
||||
|
||||
Required:
|
||||
|
||||
```yaml
|
||||
app: <chart-name> # must exist in helm-templates/
|
||||
target_cluster: <cluster-folder> # must exist in helm-overrides/
|
||||
release_name: <name> # often == app
|
||||
workload_namespace: <ns> # the K8s namespace pods run in
|
||||
image_tag: <tag> # NEVER 'latest'
|
||||
resources:
|
||||
cpu_request: <e.g. 250m>
|
||||
memory_request: <e.g. 512Mi>
|
||||
cpu_limit: <e.g. 1>
|
||||
memory_limit: <e.g. 2Gi>
|
||||
node_pool_key: <e.g. dedicated OR cloud.google.com/compute-class>
|
||||
node_pool_value: <e.g. monitoring OR monitoring-cc>
|
||||
```
|
||||
|
||||
Optional:
|
||||
|
||||
```yaml
|
||||
needs_external_dns: <bool> # if true, add a Service to external-dns-services/
|
||||
needs_external_secret: <bool> # if true, add an ExternalSecret to external-secrets/
|
||||
needs_compute_class: <bool> # if true (Autopilot), add a ComputeClass under computeclass/
|
||||
replicas: <N or autoscaling-block>
|
||||
persistence:
|
||||
enabled: <bool>
|
||||
storage_class: <name> # MUST exist in manifests/storageclass/
|
||||
size: <e.g. 100Gi>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps (deterministic)
|
||||
|
||||
1. **Verify pre-conditions.**
|
||||
```bash
|
||||
ls helm-templates/<app>/Chart.yaml # chart exists
|
||||
ls helm-overrides/<target_cluster>/ # cluster exists
|
||||
! ls helm-overrides/<target_cluster>/<app>/ # app not already onboarded here
|
||||
ls manifests/storageclass/<storage_class>.yaml # if persistence.enabled
|
||||
```
|
||||
|
||||
2. **Read the cluster's scheduling profile.** Sample a sibling app on the **same** cluster:
|
||||
```bash
|
||||
ls helm-overrides/<target_cluster>/ | grep -v '^contour' | head -3
|
||||
yq e '.nodeSelector' helm-overrides/<target_cluster>/<sibling>/custom-values.yaml
|
||||
```
|
||||
Confirm the agent's `node_pool_key` matches the cluster's actual style (`dedicated:` vs `cloud.google.com/compute-class:`). Mismatch → **fail fast** and ask.
|
||||
|
||||
3. **Generate `helm-overrides/<target_cluster>/<app>/custom-values.yaml`.** Use this template:
|
||||
|
||||
```yaml
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<app>
|
||||
tag: <image_tag>
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
replicaCount: <replicas-or-omit-if-autoscaling>
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: <cpu_request>
|
||||
memory: <memory_request>
|
||||
limits:
|
||||
cpu: <cpu_limit>
|
||||
memory: <memory_limit>
|
||||
|
||||
nodeSelector:
|
||||
<node_pool_key>: <node_pool_value>
|
||||
tolerations:
|
||||
- key: <node_pool_key>
|
||||
value: <node_pool_value>
|
||||
effect: NoSchedule
|
||||
|
||||
# if persistence.enabled
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: <storage_class>
|
||||
size: <size>
|
||||
accessModes: [ReadWriteOnce]
|
||||
```
|
||||
|
||||
4. **(If `needs_external_dns: true`)** Generate `helm-overrides/<target_cluster>/<app>/external-dns-services/<svc>.yaml` per [raw-manifest-sidecar-schema.md §Service for external-dns binding](../../docs/platform/schemas/raw-manifest-sidecar-schema.md). Sample a sibling cluster's pattern.
|
||||
|
||||
5. **(If `needs_external_secret: true`)** Generate an `ExternalSecret` under `helm-overrides/<target_cluster>/external-secrets/`. Verify the cluster has a `SecretStore` / `ClusterSecretStore` (sample sibling apps' `existingSecret:` references).
|
||||
|
||||
6. **(If `needs_compute_class: true`)** Generate `helm-overrides/<target_cluster>/<app>/computeclass/<node_pool_value>.yaml`. The `metadata.name` MUST equal `<node_pool_value>`.
|
||||
|
||||
7. **Validate.**
|
||||
```bash
|
||||
yamllint helm-overrides/<target_cluster>/<app>/custom-values.yaml
|
||||
helm template <release_name> helm-templates/<app> \
|
||||
-f helm-overrides/<target_cluster>/<app>/custom-values.yaml > /dev/null
|
||||
```
|
||||
Render must succeed. If it errors, fail and surface the error.
|
||||
|
||||
8. **Open the PR.**
|
||||
```bash
|
||||
git checkout -b onboard/<app>-on-<target_cluster>
|
||||
git add helm-overrides/<target_cluster>/<app>/
|
||||
git commit -m "Onboard <app> to <target_cluster>"
|
||||
git push origin onboard/<app>-on-<target_cluster>
|
||||
gh pr create --base main --title "Onboard <app> to <target_cluster>"
|
||||
```
|
||||
|
||||
PR body (use a heredoc):
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
Onboards `<app>` to `<target_cluster>`.
|
||||
|
||||
- Procedure: `docs/platform/procedures/onboard-app-to-cluster.md`
|
||||
- Skill: `skills/infra/onboard-app.md`
|
||||
- Sister-repo PR: <link to devops-infra-argo-config PR or "TBD">
|
||||
|
||||
## Validation
|
||||
- `yamllint` passed
|
||||
- `helm template` rendered cleanly
|
||||
- Scheduling profile verified against sibling apps on `<target_cluster>`
|
||||
|
||||
## Approvers
|
||||
- App owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
|
||||
## CMR
|
||||
<ticket or "n/a">
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
A single PR diff containing 1–3+ new files in this repo:
|
||||
|
||||
- `helm-overrides/<target_cluster>/<app>/custom-values.yaml`
|
||||
- (optional) `helm-overrides/<target_cluster>/<app>/external-dns-services/<svc>.yaml`
|
||||
- (optional) `helm-overrides/<target_cluster>/<app>/computeclass/<name>.yaml`
|
||||
- (optional) `helm-overrides/<target_cluster>/external-secrets/<name>.yaml`
|
||||
|
||||
The skill does **not**:
|
||||
|
||||
- Open the sister-repo PR (separate skill / manual). It must be drafted by the user or a follow-up step.
|
||||
- Run `argocd app sync` (Layer 1 boundary).
|
||||
- Modify `helm-templates/<chart>/`.
|
||||
- Modify `repository.yaml`.
|
||||
|
||||
---
|
||||
|
||||
## Pattern reference
|
||||
|
||||
A clean recent onboarding to compare against: pick any merged PR titled "Onboard ... to k8s-...". `git log --oneline --grep='Onboard' -i | head -10`.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
1. **Don't copy a sibling cluster's values verbatim** — per-cluster scheduling differs ([SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md)).
|
||||
2. **Don't set `automated.{prune,selfHeal}` in the sister-repo Application** ([ADR-A5](../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)).
|
||||
3. **Verify `image.tag`** is real before opening the PR — Argo CD only catches a missing tag at sync time.
|
||||
4. **The release name (`metadata.name` of the resulting `Application`) is set in the sister repo, not here.** Don't assume it; verify with the user.
|
||||
5. **For Contour**, always cross-reference [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) — the matrix is load-bearing.
|
||||
|
||||
---
|
||||
|
||||
## Layer constraint
|
||||
|
||||
Layer 1. Open the PR; do not merge it; do not Sync. Reviewer + sister-repo PR + Argo CD UI Sync click are the human gates.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [onboard-app-to-cluster.md](../../docs/platform/procedures/onboard-app-to-cluster.md).
|
||||
- Schema: [custom-values-schema.md](../../docs/platform/schemas/custom-values-schema.md).
|
||||
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md).
|
||||
Reference in New Issue
Block a user