devops-infra-helm-charts-gcp/docs/platform/procedures/onboard-new-cluster.md

Procedure — Onboard a new cluster's overrides

Layer: Layer 1 — HIGH RISK. Blast radius: establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded. Approval: platform team + cluster owner. CMR mandatory.

When a new GKE cluster is provisioned (typically by terraform-gcp-infra or its successor), this repo gains a new helm-overrides/<cluster>/ directory and the sister repo gains the ApplicationSet (or per-cluster Applications) that route to it. This procedure covers the slice owned by devops-infra-helm-charts.

---

Pre-conditions

• The cluster exists in GCP — confirmed by the platform team.
• The cluster is registered in the relevant Argo CD instance(s).
• The cluster's nodeSelector / tolerations / computeClass topology is documented (node pool names + taints).
• The cluster has a SecretStore / ClusterSecretStore for external-secrets — or onboarding external-secrets is part of this PR.
• CMR open.

---

Steps

1. Confirm the naming convention

Pattern	Use
k8s-<bu>-prd-ase1[c]	Standard BU prod cluster (GCP zone-a or zone-c)
k8s-shared-int-ase1	Shared int
k8s-aurva-prd-ase1	Aurva
k8s-supply-dev-ase1	Dev/sandbox
db-<numeric-id>-...	Auto-named dataplane / data-tier

The directory name is a contract — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.

2. Determine the cluster's scheduling profile

Cluster type	Scheduling key
GKE Autopilot	cloud.google.com/compute-class
Standard GKE	dedicated:

Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of ComputeClass resources that need to land in helm-overrides/<cluster>/<app>/computeclass/.

3. Decide the minimal app set

Most clusters need at least:

• external-secrets — to materialise secrets from GCP Secret Manager
• kube-state-metrics — for fleet observability
• victoria-metrics-agent — to ship metrics to the central VM
• fluentd (or equivalent log shipper)

Plus per-cluster role:

• BU clusters: contour-internal-0, contour-internal-1, sometimes contour-external — see contour-nodeselector-tolerations-summary.md for the convention
• DSGPU / Spark clusters: GPU operators, keda for queue autoscaling
• Dataplane (db-*) clusters: minimal — kube-state-metrics + victoria-metrics-agent only

4. Create the directory and minimal apps

mkdir -p helm-overrides/<cluster>
cd helm-overrides/<cluster>

For each app in the minimal set, follow onboard-app-to-cluster.md (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).

5. Update the per-cluster Contour matrix

If the cluster runs Contour, add a section to contour-nodeselector-tolerations-summary.md with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.

6. (If GKE Autopilot) Add the ComputeClass resources

Each Autopilot Contour / pool needs a corresponding ComputeClass. Drop them in:

helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml

Schema: raw-manifest-sidecar-schema.md §ComputeClass. The metadata.name of each ComputeClass must equal the value the corresponding app's nodeSelector: {cloud.google.com/compute-class: <name>} references.

7. Open the base PR

git checkout -b cluster/<cluster>-base
git add helm-overrides/<cluster>/
git add contour-nodeselector-tolerations-summary.md   # if updated
git commit
git push origin cluster/<cluster>-base
gh pr create --base main --title "Onboard <cluster> — base override set"

PR description must include:

• Cluster owner approver tag.
• Platform team approver tag.
• CMR ticket.
• Sister-repo PR for the ApplicationSet / per-cluster Application set.
• The minimal app set being shipped, with per-app rationale.
• The cluster's scheduling profile (key style + sample nodeSelector).

8. Pair the sister-repo PR

In github.com/Meesho/devops-infra-argo-config, draft the ApplicationSet (or per-cluster Application set) that walks helm-overrides/<cluster>/. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).

9. Stage subsequent app additions

After the base PR merges:

• Each additional app (contour-internal-0, flagger, etc.) is its own PR via onboard-app-to-cluster.md.
• Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.

---

Anti-patterns

1. Naming the directory <cluster-name> slightly differently from how Argo CD registered it. Silent bind failure.
2. Cloning another cluster's directory verbatim. Per-cluster scheduling differs; external-secrets SecretStore references differ; image-tag pinning differs.
3. Onboarding 30 apps in the base PR. Stage them — base + per-app.
4. Forgetting the Contour matrix update. Future Contour edits then have no reference and silently mis-schedule.
5. Skipping the ComputeClass resources on a GKE Autopilot cluster. nodeSelector references a class that doesn't exist; pods Pending.

---

Rollback

If the cluster onboarding turns out to be premature:

• Revert the values-side PR.
• Revert the sister-repo ApplicationSet PR.
• The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't prune what's not in its scope after the Application is removed); for a clean wipe, manually kubectl delete ns the affected namespaces.

---

Related

• Procedure: onboard-app-to-cluster.md — for each app after the base PR.
• Schema: raw-manifest-sidecar-schema.md — for ComputeClass and ExternalSecret sidecars.
• Reference: contour-nodeselector-tolerations-summary.md.
• ADR: ADR-A3-per-cluster-scheduling.md.