# Procedure: Fleet-Wide Tool Rollout

> Step-by-step guide for adding a new tool to multiple clusters in a single PR.
>
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
>
> **Blast radius:** ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.
>
> Per AI Blitz Plan §5.2. Skill: [skills/infra/fleet-wide-rollout.md](../../../skills/infra/fleet-wide-rollout.md).

---

## When to use this procedure

- Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
- Rolling out a security policy tool (e.g., Kyverno) fleet-wide
- Deploying a new mandatory platform component across all production clusters

---

## Prerequisites

Before touching any values file:

1. **Chart exists** in `devops-infra-helm-charts/helm-templates/<chartDir>/`.
2. **Override values exist** for EACH target cluster: `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml`. ValuesDir names may differ per cluster — verify each one.
3. Decide whether this is a **full fleet** rollout or **subset** (specific envs, BUs, or roles).

If any chart or override is missing, stop. Create them in `devops-infra-helm-charts` first.

---

## Steps

### Step 1: Identify target clusters

List all values files for the target env:

```bash
ls values/prd/
ls values/int/
```

Filter to the clusters you need. Common subsets:

| Subset | Description |
| ------ | ----------- |
| All prd | All files in `values/prd/` |
| Data clusters | `datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu` |
| App clusters | `demand`, `supply`, `farmiso`, `central` |
| Admin cluster | `admin-prd`, `devops-admin`, `sec-admin` |

### Step 2: Verify override exists for each cluster

```bash
for cluster in <cluster-list>; do
  echo -n "$cluster: "
  ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
    && echo "ok" || echo "MISSING"
done
```

**Halt** if any cluster is missing its override. The override must be added to `devops-infra-helm-charts` first. Only proceed when all clusters are covered.

### Step 3: Compute Application names for all clusters

For each target cluster, compute `<name>-<mungedCluster>-<env>`. Verify no name collisions with existing entries.

### Step 4: Add appSpec entries to all target values files

For each cluster values file, append:

```yaml
  - name: <tool>
    namespace: <namespace>
    chartDir: <chartDir>
    valuesDir: <valuesDir>
```

The `valuesDir` value may differ per cluster if override directories are cluster-named.

### Step 5: Validate each cluster

```bash
for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
  echo "=== $f ==="
  helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
  yamllint "$f"
done
```

All files must render cleanly.

### Step 6: Open PR

- Title: `onboard <tool> to [all prd clusters | <subset description>]`
- PR body must include:
  - Complete list of modified clusters
  - The `chartDir` and `valuesDir` used
  - Link to the `devops-infra-helm-charts` PR that added the chart/overrides (if applicable)
  - Confirmation that all override directories exist
- Required: Platform team review

---

## Staged rollout alternative

If you want cluster-by-cluster rollout (to catch issues early):

1. Start with one low-risk cluster (e.g., `k8s-datascience-prd-ase1`).
2. Open PR 1, merge, verify in ArgoCD.
3. Open PR 2 with the remaining clusters.

Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.

---

## Checklist

- [ ] Chart exists in `devops-infra-helm-charts/helm-templates/`
- [ ] Override exists for EVERY target cluster in `devops-infra-helm-charts/helm-overrides/<cluster>/`
- [ ] All Application names computed and verified no collisions
- [ ] `helm template` passes for every modified values file
- [ ] `yamllint` passes for every modified values file
- [ ] PR body lists all modified clusters
- [ ] Pre-commit hooks pass