Procedure: Deboard a Tool from a Cluster

Step-by-step guide for safely removing an appSpec entry from a cluster values file.

Layer: 1-T (Tool-Mediated). Agent generates the diff and opens a PR.

Blast radius: Single cluster. After merge, ArgoCD deletes the Application and its resources via finalizer.

Per AI Blitz Plan §5.2.

Warning: deletion is destructive

Removing an appSpec entry causes ArgoCD to delete the child Application, which triggers the resources-finalizer.argocd.argoproj.io to delete all Kubernetes resources the Application manages. This includes Deployments, Services, ConfigMaps, PVCs, and the namespace (if CreateNamespace=true and the namespace is not shared).

Confirm with the tool owner before proceeding.

Prerequisites

Confirm the tool is no longer needed on this cluster.
Confirm no other system depends on the tool's namespace (e.g., shared victoriametrics namespace — removing one app doesn't remove the namespace, but verify).
If the tool has persistent storage (PVC), confirm data can be discarded or has been backed up.

Steps

Step 1: Find the appSpec entry

grep -n -A 5 'name: <tool>' values/<env>/incubator-infra-<cluster>-values.yaml

Record the line numbers of the entire entry block.

# Does any other appSpec entry use the same namespace?
grep 'namespace: <namespace>' values/<env>/incubator-infra-<cluster>-values.yaml

If another entry shares the namespace, removing this entry will NOT delete the namespace — only the Application's resources. This is usually safe, but confirm.

Step 3: Compute the Application name that will be deleted

<name>-<mungedCluster>-<env>

Use this to verify the Application in ArgoCD before the PR merges.

Step 4: Remove the appSpec entry

Delete the entire entry block from the values file:

# Remove this block entirely:
  - name: <tool>
    namespace: <namespace>
    chartDir: <chartDir>
    valuesDir: <valuesDir>

Do not leave empty lines or dangling list markers.

Step 5: Validate

helm template generic-argo-apps-chart/ \
  -f values/<env>/incubator-infra-<cluster>-values.yaml

yamllint values/<env>/incubator-infra-<cluster>-values.yaml

Confirm the removed Application no longer appears in the rendered output.

Step 6: Open PR

Title: deboard <tool> from <cluster>
PR body must include: what the tool was doing, why it's being removed, who confirmed the removal, and namespace cleanup plan.
Required: Platform team review + tool owner acknowledgment

Step 7: Post-merge cleanup

After merge and ArgoCD sync completes:

Verify in ArgoCD that the Application has been deleted.
Verify the namespace is gone (or still exists if shared — expected).
If custom-values.yaml in devops-infra-helm-charts is no longer needed, clean it up in a separate PR.

Special case: decommissioning a cluster

If you are removing all tools from a cluster as part of cluster decommission:

Do not delete the incubator file in this PR — that is a separate step requiring confirmation.
First empty the appSpec list: appSpec: []
After all tools are confirmed deleted, open a second PR to delete the incubator and values files.

See escalation matrix: docs/global/escalation-matrix.md — cluster decommission requires coordinated sign-off.

Checklist

Tool owner confirmed removal
Namespace sharing checked — no unintended namespace deletion
Persistent storage impact assessed
Application name identified for post-merge verification
helm template renders without errors
yamllint passes
Pre-commit hooks pass
PR body includes removal rationale and namespace cleanup plan

3.8 KiB Raw Blame History