added repo
This commit is contained in:
Your Name
@@ -0,0 +1,206 @@
|
||||
# Procedure — Blue-green chart migration (versioned siblings)
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** the chart upgrade ships in a new sibling directory; no cluster cuts over until its `Application` is repointed. Each cluster's cutover is its own decision.
|
||||
> **Approval:** platform team — multi-reviewer.
|
||||
|
||||
This procedure handles the case where you can't safely bump a chart's pinned `dependencies[].version` in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as **sibling chart directories** until every consuming cluster has migrated.
|
||||
|
||||
The repo already shows this pattern at work:
|
||||
|
||||
| Original | Migration target |
|
||||
|----------|------------------|
|
||||
| `argo-cd` | `argo-cd-green` |
|
||||
| `contour` | `contour-v1.33.3` |
|
||||
| `keda` | `keda-2.17.1` |
|
||||
| `opentelemetry-collector` | `opentelemetry-collector-latest` |
|
||||
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` |
|
||||
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` |
|
||||
| `sonarqube` | `sonarqube-old` *(reverse — `sonarqube` is the new one; `-old` retained for rollback)* |
|
||||
|
||||
See [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) for the rationale.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- A major-version bump with breaking template changes.
|
||||
- A bump with an immutable-field change (Deployment `spec.selector`, StatefulSet `volumeClaimTemplates`).
|
||||
- A migration that needs cluster-by-cluster cutover with rollback windows.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- A patch / minor bump that's a clean drop-in → use [update-chart-version.md](update-chart-version.md).
|
||||
- Forking templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] Platform team agrees a blue-green migration is required (not just a bump).
|
||||
- [ ] You've identified the variant naming convention (`-green`, `-vX.Y.Z`, `-latest`, etc.).
|
||||
- [ ] You've identified every consuming cluster: `grep -rl '<chart>' helm-overrides`.
|
||||
- [ ] CMR open.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Create the sibling chart directory
|
||||
|
||||
```bash
|
||||
cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
|
||||
cd helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
Update `Chart.yaml`:
|
||||
|
||||
```diff
|
||||
apiVersion: v2
|
||||
-name: <chart>
|
||||
+name: <chart>-<variant>
|
||||
…
|
||||
dependencies:
|
||||
- name: <sub>
|
||||
- version: <old-version>
|
||||
+ version: <new-version>
|
||||
repository: <upstream-repo>
|
||||
```
|
||||
|
||||
```bash
|
||||
helm dependency update
|
||||
```
|
||||
|
||||
### 2. Render against a representative cluster's overrides — old variant
|
||||
|
||||
The point of blue-green is *no surprises*. Render the **old** chart with each consuming cluster's override and capture the output.
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 3. Render against the same overrides — new sibling variant
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 4. Diff old → new per cluster
|
||||
|
||||
```bash
|
||||
for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
|
||||
echo "=== $cluster ==="
|
||||
diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
|
||||
done
|
||||
```
|
||||
|
||||
For each cluster, decide:
|
||||
|
||||
- Is the diff what you expected?
|
||||
- Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
|
||||
- Is the cutover safe to do without the workload owner present? (If no → schedule.)
|
||||
|
||||
### 5. Open PR-1: introduce the sibling chart
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-to-<variant>-introduce
|
||||
git add helm-templates/<chart>-<variant>/
|
||||
git commit
|
||||
git push origin migrate/<chart>-to-<variant>-introduce
|
||||
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"
|
||||
```
|
||||
|
||||
After merge, the new chart exists in the repo but **no cluster uses it yet** — the existing Argo `Application`s still point at `helm-templates/<chart>`.
|
||||
|
||||
### 6. Per-cluster cutover (one PR pair per cluster)
|
||||
|
||||
For each consuming cluster:
|
||||
|
||||
a. **Update the cluster's `custom-values.yaml`** if the new chart needs different values. Open as a values-side PR (this repo).
|
||||
|
||||
b. **Update the sister-repo `Application`** to repoint:
|
||||
|
||||
```diff
|
||||
spec:
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
- path: helm-templates/<chart>
|
||||
+ path: helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
c. **After both merge**, click Sync in the cluster's Argo CD UI.
|
||||
|
||||
d. **Soak** — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.
|
||||
|
||||
### 7. Open PR-N: retire the old sibling
|
||||
|
||||
After every cluster has cut over and soaked:
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-retire-old
|
||||
git rm -r helm-templates/<chart>
|
||||
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
|
||||
git commit
|
||||
git push origin migrate/<chart>-retire-old
|
||||
gh pr create --base main --title "migrate: retire old <chart> sibling"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Confirmation every cluster has cut over (`grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config` returns nothing).
|
||||
- Confirmation soak period elapsed.
|
||||
- Decision: delete vs rename to `-old` (kept for rollback).
|
||||
|
||||
---
|
||||
|
||||
## Why two PRs at the start, then per-cluster pairs, then a final retirement
|
||||
|
||||
| PR | Effect |
|
||||
|----|--------|
|
||||
| **PR-1: introduce sibling** | Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them. |
|
||||
| **PR-2..N-1: per-cluster cutover (paired with sister repo)** | One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart. |
|
||||
| **PR-N: retire old sibling** | Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible. |
|
||||
|
||||
Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Cutting over multiple clusters in one PR.** Bundle = no per-cluster rollback.
|
||||
2. **Deleting the old sibling before every cluster has cut over.** Cluster N+1 has its `Application` pointing at a path that no longer exists; sync fails immediately.
|
||||
3. **Cutting over without rendering first.** Surprises after merge.
|
||||
4. **Skipping the soak period.** "It looked fine in the first 5 minutes" is not soak.
|
||||
5. **Renaming the *new* sibling to drop the suffix** (e.g. `argo-cd-green` → `argo-cd`) before the old chart is retired. The path collision will break Argo CD's caching.
|
||||
|
||||
---
|
||||
|
||||
## Rollback (per cluster)
|
||||
|
||||
If a cluster's cutover fails:
|
||||
|
||||
1. Revert the sister-repo PR for that cluster.
|
||||
2. Click Sync in the cluster's Argo CD — the `Application` re-renders against `<chart>` (the old sibling). The rollback is one cluster only.
|
||||
3. Investigate; iterate.
|
||||
|
||||
If the new sibling has a fundamental problem affecting every cluster:
|
||||
|
||||
1. Open PR-X: revert PR-1 (delete the sibling).
|
||||
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
|
||||
3. Schedule a postmortem before re-attempting.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md).
|
||||
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
- ADR: [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- [SANCTITY_RULES R8](../../global/SANCTITY_RULES.md).
|
||||
Reference in New Issue
Block a user