added repo

2026-08-26 04:03:34 +05:30
parent 2389ec1fd6
commit 1055e1394f
150 changed files with 12395 additions and 0 deletions
@@ -0,0 +1,126 @@
+# Procedure: Fleet-Wide Tool Rollout
+
+> Step-by-step guide for adding a new tool to multiple clusters in a single PR.
+>
+> **Layer:** 1-T (Tool-Mediated). Agent generates the diff and opens a PR.
+>
+> **Blast radius:** ALL targeted clusters simultaneously. Auto-sync means all clusters deploy on merge.
+>
+> Per AI Blitz Plan §5.2. Skill: [skills/infra/fleet-wide-rollout.md](../../../skills/infra/fleet-wide-rollout.md).
+
+---
+
+## When to use this procedure
+
+- Adding an observability tool (e.g., Coroot, Pyroscope) to a set of clusters
+- Rolling out a security policy tool (e.g., Kyverno) fleet-wide
+- Deploying a new mandatory platform component across all production clusters
+
+---
+
+## Prerequisites
+
+Before touching any values file:
+
+1. **Chart exists** in `devops-infra-helm-charts/helm-templates/<chartDir>/`.
+2. **Override values exist** for EACH target cluster: `devops-infra-helm-charts/helm-overrides/<cluster>/<valuesDir>/custom-values.yaml`. ValuesDir names may differ per cluster — verify each one.
+3. Decide whether this is a **full fleet** rollout or **subset** (specific envs, BUs, or roles).
+
+If any chart or override is missing, stop. Create them in `devops-infra-helm-charts` first.
+
+---
+
+## Steps
+
+### Step 1: Identify target clusters
+
+List all values files for the target env:
+
+```bash
+ls values/prd/
+ls values/int/
+```
+
+Filter to the clusters you need. Common subsets:
+
+| Subset | Description |
+| ------ | ----------- |
+| All prd | All files in `values/prd/` |
+| Data clusters | `datascience`, `dataengg`, `dengspark`, `dscispark`, `dsgpu` |
+| App clusters | `demand`, `supply`, `farmiso`, `central` |
+| Admin cluster | `admin-prd`, `devops-admin`, `sec-admin` |
+
+### Step 2: Verify override exists for each cluster
+
+```bash
+for cluster in <cluster-list>; do
+  echo -n "$cluster: "
+  ls /path/to/devops-infra-helm-charts/helm-overrides/$cluster/<valuesDir>/custom-values.yaml \
+    && echo "ok" || echo "MISSING"
+done
+```
+
+**Halt** if any cluster is missing its override. The override must be added to `devops-infra-helm-charts` first. Only proceed when all clusters are covered.
+
+### Step 3: Compute Application names for all clusters
+
+For each target cluster, compute `<name>-<mungedCluster>-<env>`. Verify no name collisions with existing entries.
+
+### Step 4: Add appSpec entries to all target values files
+
+For each cluster values file, append:
+
+```yaml
+  - name: <tool>
+    namespace: <namespace>
+    chartDir: <chartDir>
+    valuesDir: <valuesDir>
+```
+
+The `valuesDir` value may differ per cluster if override directories are cluster-named.
+
+### Step 5: Validate each cluster
+
+```bash
+for f in values/prd/incubator-infra-k8s-<cluster>-prd-ase1-values.yaml; do
+  echo "=== $f ==="
+  helm template generic-argo-apps-chart/ -f "$f" | grep -c "kind: Application"
+  yamllint "$f"
+done
+```
+
+All files must render cleanly.
+
+### Step 6: Open PR
+
+- Title: `onboard <tool> to [all prd clusters | <subset description>]`
+- PR body must include:
+  - Complete list of modified clusters
+  - The `chartDir` and `valuesDir` used
+  - Link to the `devops-infra-helm-charts` PR that added the chart/overrides (if applicable)
+  - Confirmation that all override directories exist
+- Required: Platform team review
+
+---
+
+## Staged rollout alternative
+
+If you want cluster-by-cluster rollout (to catch issues early):
+
+1. Start with one low-risk cluster (e.g., `k8s-datascience-prd-ase1`).
+2. Open PR 1, merge, verify in ArgoCD.
+3. Open PR 2 with the remaining clusters.
+
+Staged rollout requires multiple PRs and more human attention but reduces blast radius per deploy.
+
+---
+
+## Checklist
+
+- [ ] Chart exists in `devops-infra-helm-charts/helm-templates/`
+- [ ] Override exists for EVERY target cluster in `devops-infra-helm-charts/helm-overrides/<cluster>/`
+- [ ] All Application names computed and verified no collisions
+- [ ] `helm template` passes for every modified values file
+- [ ] `yamllint` passes for every modified values file
+- [ ] PR body lists all modified clusters
+- [ ] Pre-commit hooks pass