devops-infra-helm-charts-gcp/docs/platform/procedures/blue-green-chart-migration.md

# Procedure — Blue-green chart migration (versioned siblings)

> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** the chart upgrade ships in a new sibling directory; no cluster cuts over until its `Application` is repointed. Each cluster's cutover is its own decision.
> **Approval:** platform team — multi-reviewer.

This procedure handles the case where you can't safely bump a chart's pinned `dependencies[].version` in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as **sibling chart directories** until every consuming cluster has migrated.

The repo already shows this pattern at work:

| Original | Migration target |
|----------|------------------|
| `argo-cd` | `argo-cd-green` |
| `contour` | `contour-v1.33.3` |
| `keda` | `keda-2.17.1` |
| `opentelemetry-collector` | `opentelemetry-collector-latest` |
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` |
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` |
| `sonarqube` | `sonarqube-old` *(reverse — `sonarqube` is the new one; `-old` retained for rollback)* |

See [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) for the rationale.

---

## When to use this procedure

- A major-version bump with breaking template changes.
- A bump with an immutable-field change (Deployment `spec.selector`, StatefulSet `volumeClaimTemplates`).
- A migration that needs cluster-by-cluster cutover with rollback windows.

This is **not** the right procedure for:

- A patch / minor bump that's a clean drop-in → use [update-chart-version.md](update-chart-version.md).
- Forking templates → use [fork-upstream-chart.md](fork-upstream-chart.md).

---

## Pre-conditions

- [ ] Platform team agrees a blue-green migration is required (not just a bump).
- [ ] You've identified the variant naming convention (`-green`, `-vX.Y.Z`, `-latest`, etc.).
- [ ] You've identified every consuming cluster: `grep -rl '<chart>' helm-overrides`.
- [ ] CMR open.

---

## Steps

### 1. Create the sibling chart directory

```bash
cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
cd helm-templates/<chart>-<variant>
```

Update `Chart.yaml`:

```diff
 apiVersion: v2
-name: <chart>
+name: <chart>-<variant>
 …
 dependencies:
   - name: <sub>
-    version: <old-version>
+    version: <new-version>
     repository: <upstream-repo>
```

```bash
helm dependency update
```

### 2. Render against a representative cluster's overrides — old variant

The point of blue-green is *no surprises*. Render the **old** chart with each consuming cluster's override and capture the output.

```bash
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
  cluster=$(echo "$f" | awk -F/ '{print $2}')
  helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
done
```

### 3. Render against the same overrides — new sibling variant

```bash
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
  cluster=$(echo "$f" | awk -F/ '{print $2}')
  helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
done
```

### 4. Diff old → new per cluster

```bash
for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
  echo "=== $cluster ==="
  diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
done
```

For each cluster, decide:

- Is the diff what you expected?
- Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
- Is the cutover safe to do without the workload owner present? (If no → schedule.)

### 5. Open PR-1: introduce the sibling chart

```bash
git checkout -b migrate/<chart>-to-<variant>-introduce
git add helm-templates/<chart>-<variant>/
git commit
git push origin migrate/<chart>-to-<variant>-introduce
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"
```

After merge, the new chart exists in the repo but **no cluster uses it yet** — the existing Argo `Application`s still point at `helm-templates/<chart>`.

### 6. Per-cluster cutover (one PR pair per cluster)

For each consuming cluster:

a. **Update the cluster's `custom-values.yaml`** if the new chart needs different values. Open as a values-side PR (this repo).

b. **Update the sister-repo `Application`** to repoint:

```diff
 spec:
   source:
     repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
     targetRevision: main
-    path: helm-templates/<chart>
+    path: helm-templates/<chart>-<variant>
```

c. **After both merge**, click Sync in the cluster's Argo CD UI.

d. **Soak** — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.

### 7. Open PR-N: retire the old sibling

After every cluster has cut over and soaked:

```bash
git checkout -b migrate/<chart>-retire-old
git rm -r helm-templates/<chart>
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
git commit
git push origin migrate/<chart>-retire-old
gh pr create --base main --title "migrate: retire old <chart> sibling"
```

PR description:

- Confirmation every cluster has cut over (`grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config` returns nothing).
- Confirmation soak period elapsed.
- Decision: delete vs rename to `-old` (kept for rollback).

---

## Why two PRs at the start, then per-cluster pairs, then a final retirement

| PR | Effect |
|----|--------|
| **PR-1: introduce sibling** | Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them. |
| **PR-2..N-1: per-cluster cutover (paired with sister repo)** | One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart. |
| **PR-N: retire old sibling** | Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible. |

Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.

---

## Anti-patterns

1. **Cutting over multiple clusters in one PR.** Bundle = no per-cluster rollback.
2. **Deleting the old sibling before every cluster has cut over.** Cluster N+1 has its `Application` pointing at a path that no longer exists; sync fails immediately.
3. **Cutting over without rendering first.** Surprises after merge.
4. **Skipping the soak period.** "It looked fine in the first 5 minutes" is not soak.
5. **Renaming the *new* sibling to drop the suffix** (e.g. `argo-cd-green` → `argo-cd`) before the old chart is retired. The path collision will break Argo CD's caching.

---

## Rollback (per cluster)

If a cluster's cutover fails:

1. Revert the sister-repo PR for that cluster.
2. Click Sync in the cluster's Argo CD — the `Application` re-renders against `<chart>` (the old sibling). The rollback is one cluster only.
3. Investigate; iterate.

If the new sibling has a fundamental problem affecting every cluster:

1. Open PR-X: revert PR-1 (delete the sibling).
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
3. Schedule a postmortem before re-attempting.

---

## Related

- Procedure: [update-chart-version.md](update-chart-version.md).
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md).
- ADR: [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
- [SANCTITY_RULES R8](../../global/SANCTITY_RULES.md).