# Procedure — Onboard a new cluster's overrides

> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded.
> **Approval:** platform team + cluster owner. CMR mandatory.

When a new GKE cluster is provisioned (typically by `terraform-gcp-infra` or its successor), this repo gains a new `helm-overrides/<cluster>/` directory and the sister repo gains the `ApplicationSet` (or per-cluster Applications) that route to it. This procedure covers the slice owned by `devops-infra-helm-charts`.

---

## Pre-conditions

- [ ] The cluster exists in GCP — confirmed by the platform team.
- [ ] The cluster is registered in the relevant Argo CD instance(s).
- [ ] The cluster's `nodeSelector` / `tolerations` / `computeClass` topology is documented (node pool names + taints).
- [ ] The cluster has a `SecretStore` / `ClusterSecretStore` for `external-secrets` — or onboarding `external-secrets` is part of this PR.
- [ ] CMR open.

---

## Steps

### 1. Confirm the naming convention

| Pattern | Use |
|---------|-----|
| `k8s-<bu>-prd-ase1[c]` | Standard BU prod cluster (GCP zone-a or zone-c) |
| `k8s-shared-int-ase1` | Shared int |
| `k8s-aurva-prd-ase1` | Aurva |
| `k8s-supply-dev-ase1` | Dev/sandbox |
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier |

The directory name is a **contract** — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.

### 2. Determine the cluster's scheduling profile

| Cluster type | Scheduling key |
|--------------|----------------|
| GKE Autopilot | `cloud.google.com/compute-class` |
| Standard GKE | `dedicated:` |

Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of `ComputeClass` resources that need to land in `helm-overrides/<cluster>/<app>/computeclass/`.

### 3. Decide the minimal app set

Most clusters need at least:

- `external-secrets` — to materialise secrets from GCP Secret Manager
- `kube-state-metrics` — for fleet observability
- `victoria-metrics-agent` — to ship metrics to the central VM
- `fluentd` (or equivalent log shipper)

Plus per-cluster role:

- BU clusters: `contour-internal-0`, `contour-internal-1`, sometimes `contour-external` — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) for the convention
- DSGPU / Spark clusters: GPU operators, `keda` for queue autoscaling
- Dataplane (`db-*`) clusters: minimal — `kube-state-metrics` + `victoria-metrics-agent` only

### 4. Create the directory and minimal apps

```bash
mkdir -p helm-overrides/<cluster>
cd helm-overrides/<cluster>
```

For each app in the minimal set, follow [onboard-app-to-cluster.md](onboard-app-to-cluster.md) (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).

### 5. Update the per-cluster Contour matrix

If the cluster runs Contour, add a section to [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.

### 6. (If GKE Autopilot) Add the `ComputeClass` resources

Each Autopilot Contour / pool needs a corresponding `ComputeClass`. Drop them in:

```
helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml
```

Schema: [raw-manifest-sidecar-schema.md §ComputeClass](../schemas/raw-manifest-sidecar-schema.md). The `metadata.name` of each `ComputeClass` must equal the value the corresponding app's `nodeSelector: {cloud.google.com/compute-class: <name>}` references.

### 7. Open the base PR

```bash
git checkout -b cluster/<cluster>-base
git add helm-overrides/<cluster>/
git add contour-nodeselector-tolerations-summary.md   # if updated
git commit
git push origin cluster/<cluster>-base
gh pr create --base main --title "Onboard <cluster> — base override set"
```

PR description must include:

- Cluster owner approver tag.
- Platform team approver tag.
- CMR ticket.
- Sister-repo PR for the `ApplicationSet` / per-cluster `Application` set.
- The minimal app set being shipped, with per-app rationale.
- The cluster's scheduling profile (key style + sample `nodeSelector`).

### 8. Pair the sister-repo PR

In `github.com/Meesho/devops-infra-argo-config`, draft the `ApplicationSet` (or per-cluster Application set) that walks `helm-overrides/<cluster>/`. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).

### 9. Stage subsequent app additions

After the base PR merges:

- Each additional app (`contour-internal-0`, `flagger`, etc.) is its own PR via [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
- Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.

---

## Anti-patterns

1. **Naming the directory `<cluster-name>` slightly differently** from how Argo CD registered it. Silent bind failure.
2. **Cloning another cluster's directory verbatim.** Per-cluster scheduling differs; `external-secrets` `SecretStore` references differ; image-tag pinning differs.
3. **Onboarding 30 apps in the base PR.** Stage them — base + per-app.
4. **Forgetting the Contour matrix update.** Future Contour edits then have no reference and silently mis-schedule.
5. **Skipping the `ComputeClass` resources** on a GKE Autopilot cluster. `nodeSelector` references a class that doesn't exist; pods Pending.

---

## Rollback

If the cluster onboarding turns out to be premature:

- Revert the values-side PR.
- Revert the sister-repo `ApplicationSet` PR.
- The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't `prune` what's not in its scope after the `Application` is removed); for a clean wipe, manually `kubectl delete ns` the affected namespaces.

---

## Related

- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — for each app after the base PR.
- Schema: [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md) — for `ComputeClass` and `ExternalSecret` sidecars.
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).