added repo

2026-08-26 03:39:42 +05:30
parent 45c25a95af
commit b8575bb8b9
6889 changed files with 1217125 additions and 0 deletions
@@ -0,0 +1,119 @@
+# Procedure — Deboard a retired app from a cluster
+
+> **Layer:** Layer 1 — HIGH RISK.
+> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
+> **Approval:** app owner + cluster owner. CMR mandatory.
+
+This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.
+
+---
+
+## Pre-conditions (block deboarding if any are FALSE)
+
+- [ ] App is genuinely retired on this cluster — confirmed by app owner.
+- [ ] No traffic / scrape job / dependency is still hitting it.
+- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
+- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
+- [ ] CMR approved.
+
+```bash
+# Where does this app exist today?
+find helm-overrides -maxdepth 2 -type d -name '<app>'
+```
+
+---
+
+## Steps
+
+### 1. Decide the scope
+
+| Scope | Then |
+|-------|------|
+| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
+| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
+| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |
+
+### 2. Open the sister-repo `Application` removal PR FIRST
+
+In `github.com/Meesho/devops-infra-argo-config`:
+
+- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
+- Merge.
+- The cluster's Argo CD will mark the `Application` for removal on next reconcile.
+
+This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.
+
+### 3. Manually clean up the workload (if `automated.prune` was not set)
+
+For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:
+
+```bash
+kubectl --context=<cluster> delete <kind>/<name> -n <ns>
+# OR, if the whole namespace is dedicated to this release:
+kubectl --context=<cluster> delete namespace <ns>
+```
+
+Coordinate with the app owner — wholesale namespace deletion is irreversible.
+
+### 4. Open the values-side removal PR
+
+```bash
+git checkout -b deboard/<app>-from-<cluster>
+git rm -r helm-overrides/<cluster>/<app>/
+git commit
+git push origin deboard/<app>-from-<cluster>
+gh pr create --base main --title "deboard: <app> from <cluster>"
+```
+
+PR description:
+
+- Procedure followed: this file.
+- Confirmation pre-conditions are TRUE.
+- Sister-repo PR (already merged).
+- CMR ticket reference.
+- Confirmation the workload was manually cleaned up (or scheduled).
+
+### 5. (Optional) If this was the last cluster running the app
+
+If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:
+
+- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).
+
+---
+
+## "What if the app might come back?"
+
+If retirement is provisional:
+
+- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
+- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.
+
+---
+
+## Anti-patterns
+
+1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
+2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
+3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
+4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
+5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.
+
+---
+
+## Rollback
+
+If you deboarded by mistake:
+
+1. Revert the values-side PR (`git revert <merge-sha>`).
+2. Revert the sister-repo PR.
+3. Click Sync in the cluster's Argo CD UI.
+4. The `Application` is recreated and renders the chart.
+5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).
+
+---
+
+## Related
+
+- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
+- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
+- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.