Procedure — Deboard a retired app from a cluster

Layer: Layer 1 — HIGH RISK. Blast radius: the Helm release is removed from the cluster. If still in use, that's an outage. Approval: app owner + cluster owner. CMR mandatory.

This procedure removes an app's helm-overrides/<cluster>/<app>/ directory. It does not delete the workload directly — but the matching sister-repo Application PR (which must be paired) tells Argo CD to stop tracking the release.

Pre-conditions (block deboarding if any are FALSE)

App is genuinely retired on this cluster — confirmed by app owner.
No traffic / scrape job / dependency is still hitting it.
All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
CMR approved.

# Where does this app exist today?
find helm-overrides -maxdepth 2 -type d -name '<app>'

Steps

1. Decide the scope

Scope	Then
Single cluster	Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running.
Fleet-wide	Multiple PRs — one cluster per PR. Don't bundle.
App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`)	This is a migration, not a deboard — use blue-green-chart-migration.md.

2. Open the sister-repo `Application` removal PR FIRST

In github.com/Meesho/devops-infra-argo-config:

Remove (or scope out of) the Application / ApplicationSet entry that targets this cluster × app.
Merge.
The cluster's Argo CD will mark the Application for removal on next reconcile.

This step must precede the values-side removal. If you delete the values-side first, the Application will fail to render and the workload may go into an Errored state on the cluster.

3. Manually clean up the workload (if `automated.prune` was not set)

For most infra apps, the Application does not have automated.prune: true — so removing the Application does not delete the workload. Manually:

kubectl --context=<cluster> delete <kind>/<name> -n <ns>
# OR, if the whole namespace is dedicated to this release:
kubectl --context=<cluster> delete namespace <ns>

Coordinate with the app owner — wholesale namespace deletion is irreversible.

4. Open the values-side removal PR

git checkout -b deboard/<app>-from-<cluster>
git rm -r helm-overrides/<cluster>/<app>/
git commit
git push origin deboard/<app>-from-<cluster>
gh pr create --base main --title "deboard: <app> from <cluster>"

PR description:

Procedure followed: this file.
Confirmation pre-conditions are TRUE.
Sister-repo PR (already merged).
CMR ticket reference.
Confirmation the workload was manually cleaned up (or scheduled).

5. (Optional) If this was the last cluster running the app

If find helm-overrides -maxdepth 2 -type d -name '<app>' now returns nothing, consider:

Should helm-templates/<chart>/ also be removed? Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).

"What if the app might come back?"

If retirement is provisional:

Do not scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
Do keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.

Anti-patterns

Deleting the values-side first before the sister-repo Application is removed. The Application errors on next reconcile.
Bulk-deleting multiple clusters in one PR. One cluster per PR. Rollback granularity.
Forgetting to clean up the workload manually. The Helm release lingers on the cluster after the Application is removed (because most infra apps lack automated.prune).
Forgetting external-secrets cleanup. If the app referenced an ExternalSecret, the corresponding ExternalSecret resource (in the cluster's external-secrets/ directory) often outlives the app. Either repurpose it or delete it in the same PR.
Forgetting alert / dashboard cleanup. Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.

Rollback

If you deboarded by mistake:

Revert the values-side PR (git revert <merge-sha>).
Revert the sister-repo PR.
Click Sync in the cluster's Argo CD UI.
The Application is recreated and renders the chart.
However, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see manifests/storageclass/pd-standard-retain-dr.yaml).

Procedure: onboard-app-to-cluster.md — the inverse.
Procedure: blue-green-chart-migration.md — if "deboard" is actually a migration.
ADR: ADR-A5-manual-sync-default-for-infra.md — why the workload doesn't auto-delete.

5.4 KiB Raw Blame History Unescape Escape