devops-infra-helm-charts-gcp/docs/platform/procedures/blue-green-chart-migration.md

Procedure — Blue-green chart migration (versioned siblings)

Layer: Layer 1 — HIGH RISK. Blast radius: the chart upgrade ships in a new sibling directory; no cluster cuts over until its Application is repointed. Each cluster's cutover is its own decision. Approval: platform team — multi-reviewer.

This procedure handles the case where you can't safely bump a chart's pinned dependencies[].version in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as sibling chart directories until every consuming cluster has migrated.

The repo already shows this pattern at work:

Original	Migration target
argo-cd	argo-cd-green
contour	contour-v1.33.3
keda	keda-2.17.1
opentelemetry-collector	opentelemetry-collector-latest
victoria-metrics-cluster	victoria-metrics-cluster-latest
victoria-metrics-agent	victoria-metrics-agent-latest
sonarqube	sonarqube-old (reverse — sonarqube is the new one; -old retained for rollback)

See ADR-A2-blue-green-sibling-pattern.md for the rationale.

---

When to use this procedure

• A major-version bump with breaking template changes.
• A bump with an immutable-field change (Deployment spec.selector, StatefulSet volumeClaimTemplates).
• A migration that needs cluster-by-cluster cutover with rollback windows.

This is not the right procedure for:

• A patch / minor bump that's a clean drop-in → use update-chart-version.md.
• Forking templates → use fork-upstream-chart.md.

---

Pre-conditions

• Platform team agrees a blue-green migration is required (not just a bump).
• You've identified the variant naming convention (-green, -vX.Y.Z, -latest, etc.).
• You've identified every consuming cluster: grep -rl '<chart>' helm-overrides.
• CMR open.

---

Steps

1. Create the sibling chart directory

cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
cd helm-templates/<chart>-<variant>

Update Chart.yaml:

 apiVersion: v2
-name: <chart>
+name: <chart>-<variant>
 …
 dependencies:
   - name: <sub>
-    version: <old-version>
+    version: <new-version>
     repository: <upstream-repo>

helm dependency update

2. Render against a representative cluster's overrides — old variant

The point of blue-green is no surprises. Render the old chart with each consuming cluster's override and capture the output.

for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
  cluster=$(echo "$f" | awk -F/ '{print $2}')
  helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
done

3. Render against the same overrides — new sibling variant

for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
  cluster=$(echo "$f" | awk -F/ '{print $2}')
  helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
done

4. Diff old → new per cluster

for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
  echo "=== $cluster ==="
  diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
done

For each cluster, decide:

• Is the diff what you expected?
• Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
• Is the cutover safe to do without the workload owner present? (If no → schedule.)

5. Open PR-1: introduce the sibling chart

git checkout -b migrate/<chart>-to-<variant>-introduce
git add helm-templates/<chart>-<variant>/
git commit
git push origin migrate/<chart>-to-<variant>-introduce
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"

After merge, the new chart exists in the repo but no cluster uses it yet — the existing Argo Applications still point at helm-templates/<chart>.

6. Per-cluster cutover (one PR pair per cluster)

For each consuming cluster:

a. Update the cluster's custom-values.yaml if the new chart needs different values. Open as a values-side PR (this repo).

b. Update the sister-repo Application to repoint:

 spec:
   source:
     repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
     targetRevision: main
-    path: helm-templates/<chart>
+    path: helm-templates/<chart>-<variant>

c. After both merge, click Sync in the cluster's Argo CD UI.

d. Soak — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.

7. Open PR-N: retire the old sibling

After every cluster has cut over and soaked:

git checkout -b migrate/<chart>-retire-old
git rm -r helm-templates/<chart>
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
git commit
git push origin migrate/<chart>-retire-old
gh pr create --base main --title "migrate: retire old <chart> sibling"

PR description:

• Confirmation every cluster has cut over (grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config returns nothing).
• Confirmation soak period elapsed.
• Decision: delete vs rename to -old (kept for rollback).

---

Why two PRs at the start, then per-cluster pairs, then a final retirement

PR	Effect
PR-1: introduce sibling	Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them.
PR-2..N-1: per-cluster cutover (paired with sister repo)	One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart.
PR-N: retire old sibling	Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible.

Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.

---

Anti-patterns

1. Cutting over multiple clusters in one PR. Bundle = no per-cluster rollback.
2. Deleting the old sibling before every cluster has cut over. Cluster N+1 has its Application pointing at a path that no longer exists; sync fails immediately.
3. Cutting over without rendering first. Surprises after merge.
4. Skipping the soak period. "It looked fine in the first 5 minutes" is not soak.
5. Renaming the new sibling to drop the suffix (e.g. argo-cd-green → argo-cd) before the old chart is retired. The path collision will break Argo CD's caching.

---

Rollback (per cluster)

If a cluster's cutover fails:

1. Revert the sister-repo PR for that cluster.
2. Click Sync in the cluster's Argo CD — the Application re-renders against <chart> (the old sibling). The rollback is one cluster only.
3. Investigate; iterate.

If the new sibling has a fundamental problem affecting every cluster:

1. Open PR-X: revert PR-1 (delete the sibling).
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
3. Schedule a postmortem before re-attempting.

---

Related

• Procedure: update-chart-version.md.
• Procedure: fork-upstream-chart.md.
• ADR: ADR-A2-blue-green-sibling-pattern.md.
• SANCTITY_RULES R8.