120 lines
5.4 KiB
Markdown
120 lines
5.4 KiB
Markdown
# Procedure — Deboard a retired app from a cluster
|
||
|
||
> **Layer:** Layer 1 — HIGH RISK.
|
||
> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
|
||
> **Approval:** app owner + cluster owner. CMR mandatory.
|
||
|
||
This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.
|
||
|
||
---
|
||
|
||
## Pre-conditions (block deboarding if any are FALSE)
|
||
|
||
- [ ] App is genuinely retired on this cluster — confirmed by app owner.
|
||
- [ ] No traffic / scrape job / dependency is still hitting it.
|
||
- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
|
||
- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
|
||
- [ ] CMR approved.
|
||
|
||
```bash
|
||
# Where does this app exist today?
|
||
find helm-overrides -maxdepth 2 -type d -name '<app>'
|
||
```
|
||
|
||
---
|
||
|
||
## Steps
|
||
|
||
### 1. Decide the scope
|
||
|
||
| Scope | Then |
|
||
|-------|------|
|
||
| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
|
||
| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
|
||
| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |
|
||
|
||
### 2. Open the sister-repo `Application` removal PR FIRST
|
||
|
||
In `github.com/Meesho/devops-infra-argo-config`:
|
||
|
||
- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
|
||
- Merge.
|
||
- The cluster's Argo CD will mark the `Application` for removal on next reconcile.
|
||
|
||
This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.
|
||
|
||
### 3. Manually clean up the workload (if `automated.prune` was not set)
|
||
|
||
For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:
|
||
|
||
```bash
|
||
kubectl --context=<cluster> delete <kind>/<name> -n <ns>
|
||
# OR, if the whole namespace is dedicated to this release:
|
||
kubectl --context=<cluster> delete namespace <ns>
|
||
```
|
||
|
||
Coordinate with the app owner — wholesale namespace deletion is irreversible.
|
||
|
||
### 4. Open the values-side removal PR
|
||
|
||
```bash
|
||
git checkout -b deboard/<app>-from-<cluster>
|
||
git rm -r helm-overrides/<cluster>/<app>/
|
||
git commit
|
||
git push origin deboard/<app>-from-<cluster>
|
||
gh pr create --base main --title "deboard: <app> from <cluster>"
|
||
```
|
||
|
||
PR description:
|
||
|
||
- Procedure followed: this file.
|
||
- Confirmation pre-conditions are TRUE.
|
||
- Sister-repo PR (already merged).
|
||
- CMR ticket reference.
|
||
- Confirmation the workload was manually cleaned up (or scheduled).
|
||
|
||
### 5. (Optional) If this was the last cluster running the app
|
||
|
||
If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:
|
||
|
||
- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).
|
||
|
||
---
|
||
|
||
## "What if the app might come back?"
|
||
|
||
If retirement is provisional:
|
||
|
||
- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
|
||
- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.
|
||
|
||
---
|
||
|
||
## Anti-patterns
|
||
|
||
1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
|
||
2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
|
||
3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
|
||
4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
|
||
5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.
|
||
|
||
---
|
||
|
||
## Rollback
|
||
|
||
If you deboarded by mistake:
|
||
|
||
1. Revert the values-side PR (`git revert <merge-sha>`).
|
||
2. Revert the sister-repo PR.
|
||
3. Click Sync in the cluster's Argo CD UI.
|
||
4. The `Application` is recreated and renders the chart.
|
||
5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
|
||
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
|
||
- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.
|