# Procedure — Deboard a retired app from a cluster

> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
> **Approval:** app owner + cluster owner. CMR mandatory.

This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.

---

## Pre-conditions (block deboarding if any are FALSE)

- [ ] App is genuinely retired on this cluster — confirmed by app owner.
- [ ] No traffic / scrape job / dependency is still hitting it.
- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
- [ ] CMR approved.

```bash
# Where does this app exist today?
find helm-overrides -maxdepth 2 -type d -name '<app>'
```

---

## Steps

### 1. Decide the scope

| Scope | Then |
|-------|------|
| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |

### 2. Open the sister-repo `Application` removal PR FIRST

In `github.com/Meesho/devops-infra-argo-config`:

- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
- Merge.
- The cluster's Argo CD will mark the `Application` for removal on next reconcile.

This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.

### 3. Manually clean up the workload (if `automated.prune` was not set)

For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:

```bash
kubectl --context=<cluster> delete <kind>/<name> -n <ns>
# OR, if the whole namespace is dedicated to this release:
kubectl --context=<cluster> delete namespace <ns>
```

Coordinate with the app owner — wholesale namespace deletion is irreversible.

### 4. Open the values-side removal PR

```bash
git checkout -b deboard/<app>-from-<cluster>
git rm -r helm-overrides/<cluster>/<app>/
git commit
git push origin deboard/<app>-from-<cluster>
gh pr create --base main --title "deboard: <app> from <cluster>"
```

PR description:

- Procedure followed: this file.
- Confirmation pre-conditions are TRUE.
- Sister-repo PR (already merged).
- CMR ticket reference.
- Confirmation the workload was manually cleaned up (or scheduled).

### 5. (Optional) If this was the last cluster running the app

If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:

- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).

---

## "What if the app might come back?"

If retirement is provisional:

- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.

---

## Anti-patterns

1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.

---

## Rollback

If you deboarded by mistake:

1. Revert the values-side PR (`git revert <merge-sha>`).
2. Revert the sister-repo PR.
3. Click Sync in the cluster's Argo CD UI.
4. The `Application` is recreated and renders the chart.
5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).

---

## Related

- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.