Procedure: Fleet-Wide Tool Rollout

Step-by-step guide for adding a new tool to multiple clusters in a single PR.

Layer: 1-T (Tool-Mediated). Agent generates the diff and opens a PR.

Blast radius: ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.

Per AI Blitz Plan §5.2. Skill: skills/infra/fleet-wide-rollout.md.

When to use this procedure

Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
Rolling out a security policy tool (e.g., Kyverno) fleet-wide
Deploying a new mandatory platform component across all production clusters

Prerequisites

Before touching any values file:

Chart exists in devops-infra-helm-charts/helm-templates/<chartDir>/.
Override values exist for EACH target cluster: devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml. ValuesDir names may differ per cluster — verify each one.
Decide whether this is a full fleet rollout or subset (specific envs, BUs, or roles).

If any chart or override is missing, stop. Create them in devops-infra-helm-charts first.

Steps

Step 1: Identify target clusters

List all values files for the target env:

ls values/prd/
ls values/int/

Filter to the clusters you need. Common subsets:

Subset	Description
All prd	All files in `values/prd/`
Data clusters	`datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu`
App clusters	`demand`, `supply`, `farmiso`, `central`
Admin cluster	`admin-prd`, `devops-admin`, `sec-admin`

Step 2: Verify override exists for each cluster

for cluster in <cluster-list>; do
  echo -n "$cluster: "
  ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
    && echo "ok" || echo "MISSING"
done

Halt if any cluster is missing its override. The override must be added to devops-infra-helm-charts first. Only proceed when all clusters are covered.

Step 3: Compute Application names for all clusters

For each target cluster, compute <name>-<mungedCluster>-<env>. Verify no name collisions with existing entries.

Step 4: Add appSpec entries to all target values files

For each cluster values file, append:

  - name: <tool>
    namespace: <namespace>
    chartDir: <chartDir>
    valuesDir: <valuesDir>

The valuesDir value may differ per cluster if override directories are cluster-named.

Step 5: Validate each cluster

for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
  echo "=== $f ==="
  helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
  yamllint "$f"
done

All files must render cleanly.

Step 6: Open PR

Title: onboard <tool> to [all prd clusters | <subset description>]
PR body must include:
- Complete list of modified clusters
- The chartDir and valuesDir used
- Link to the devops-infra-helm-charts PR that added the chart/overrides (if applicable)
- Confirmation that all override directories exist
Required: Platform team review

Staged rollout alternative

If you want cluster-by-cluster rollout (to catch issues early):

Start with one low-risk cluster (e.g., k8s-datascience-prd-ase1).
Open PR 1, merge, verify in ArgoCD.
Open PR 2 with the remaining clusters.

Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.

Checklist

Chart exists in devops-infra-helm-charts/helm-templates/
Override exists for EVERY target cluster in devops-infra-helm-charts/helm-overrides/<cluster>/
All Application names computed and verified no collisions
helm template passes for every modified values file
yamllint passes for every modified values file
PR body lists all modified clusters
Pre-commit hooks pass

3.9 KiB Raw Blame History