127 lines
3.9 KiB
Markdown
127 lines
3.9 KiB
Markdown
# Procedure: Fleet-Wide Tool Rollout
|
|
|
|
> Step-by-step guide for adding a new tool to multiple clusters in a single PR.
|
|
>
|
|
> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
|
|
>
|
|
> **Blast radius:** ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.
|
|
>
|
|
> Per AI Blitz Plan §5.2. Skill: [skills/infra/fleet-wide-rollout.md](../../../skills/infra/fleet-wide-rollout.md).
|
|
|
|
---
|
|
|
|
## When to use this procedure
|
|
|
|
- Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
|
|
- Rolling out a security policy tool (e.g., Kyverno) fleet-wide
|
|
- Deploying a new mandatory platform component across all production clusters
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
Before touching any values file:
|
|
|
|
1. **Chart exists** in `devops-infra-helm-charts/helm-templates/<chartDir>/`.
|
|
2. **Override values exist** for EACH target cluster: `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml`. ValuesDir names may differ per cluster — verify each one.
|
|
3. Decide whether this is a **full fleet** rollout or **subset** (specific envs, BUs, or roles).
|
|
|
|
If any chart or override is missing, stop. Create them in `devops-infra-helm-charts` first.
|
|
|
|
---
|
|
|
|
## Steps
|
|
|
|
### Step 1: Identify target clusters
|
|
|
|
List all values files for the target env:
|
|
|
|
```bash
|
|
ls values/prd/
|
|
ls values/int/
|
|
```
|
|
|
|
Filter to the clusters you need. Common subsets:
|
|
|
|
| Subset | Description |
|
|
| ------ | ----------- |
|
|
| All prd | All files in `values/prd/` |
|
|
| Data clusters | `datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu` |
|
|
| App clusters | `demand`, `supply`, `farmiso`, `central` |
|
|
| Admin cluster | `admin-prd`, `devops-admin`, `sec-admin` |
|
|
|
|
### Step 2: Verify override exists for each cluster
|
|
|
|
```bash
|
|
for cluster in <cluster-list>; do
|
|
echo -n "$cluster: "
|
|
ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
|
|
&& echo "ok" || echo "MISSING"
|
|
done
|
|
```
|
|
|
|
**Halt** if any cluster is missing its override. The override must be added to `devops-infra-helm-charts` first. Only proceed when all clusters are covered.
|
|
|
|
### Step 3: Compute Application names for all clusters
|
|
|
|
For each target cluster, compute `<name>-<mungedCluster>-<env>`. Verify no name collisions with existing entries.
|
|
|
|
### Step 4: Add appSpec entries to all target values files
|
|
|
|
For each cluster values file, append:
|
|
|
|
```yaml
|
|
- name: <tool>
|
|
namespace: <namespace>
|
|
chartDir: <chartDir>
|
|
valuesDir: <valuesDir>
|
|
```
|
|
|
|
The `valuesDir` value may differ per cluster if override directories are cluster-named.
|
|
|
|
### Step 5: Validate each cluster
|
|
|
|
```bash
|
|
for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
|
|
echo "=== $f ==="
|
|
helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
|
|
yamllint "$f"
|
|
done
|
|
```
|
|
|
|
All files must render cleanly.
|
|
|
|
### Step 6: Open PR
|
|
|
|
- Title: `onboard <tool> to [all prd clusters | <subset description>]`
|
|
- PR body must include:
|
|
- Complete list of modified clusters
|
|
- The `chartDir` and `valuesDir` used
|
|
- Link to the `devops-infra-helm-charts` PR that added the chart/overrides (if applicable)
|
|
- Confirmation that all override directories exist
|
|
- Required: Platform team review
|
|
|
|
---
|
|
|
|
## Staged rollout alternative
|
|
|
|
If you want cluster-by-cluster rollout (to catch issues early):
|
|
|
|
1. Start with one low-risk cluster (e.g., `k8s-datascience-prd-ase1`).
|
|
2. Open PR 1, merge, verify in ArgoCD.
|
|
3. Open PR 2 with the remaining clusters.
|
|
|
|
Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.
|
|
|
|
---
|
|
|
|
## Checklist
|
|
|
|
- [ ] Chart exists in `devops-infra-helm-charts/helm-templates/`
|
|
- [ ] Override exists for EVERY target cluster in `devops-infra-helm-charts/helm-overrides/<cluster>/`
|
|
- [ ] All Application names computed and verified no collisions
|
|
- [ ] `helm template` passes for every modified values file
|
|
- [ ] `yamllint` passes for every modified values file
|
|
- [ ] PR body lists all modified clusters
|
|
- [ ] Pre-commit hooks pass
|