141 lines
6.3 KiB
Markdown
141 lines
6.3 KiB
Markdown
# Procedure — Onboard a new cluster's overrides
|
|
|
|
> **Layer:** Layer 1 — HIGH RISK.
|
|
> **Blast radius:** establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded.
|
|
> **Approval:** platform team + cluster owner. CMR mandatory.
|
|
|
|
When a new GKE cluster is provisioned (typically by `terraform-gcp-infra` or its successor), this repo gains a new `helm-overrides/<cluster>/` directory and the sister repo gains the `ApplicationSet` (or per-cluster Applications) that route to it. This procedure covers the slice owned by `devops-infra-helm-charts`.
|
|
|
|
---
|
|
|
|
## Pre-conditions
|
|
|
|
- [ ] The cluster exists in GCP — confirmed by the platform team.
|
|
- [ ] The cluster is registered in the relevant Argo CD instance(s).
|
|
- [ ] The cluster's `nodeSelector` / `tolerations` / `computeClass` topology is documented (node pool names + taints).
|
|
- [ ] The cluster has a `SecretStore` / `ClusterSecretStore` for `external-secrets` — or onboarding `external-secrets` is part of this PR.
|
|
- [ ] CMR open.
|
|
|
|
---
|
|
|
|
## Steps
|
|
|
|
### 1. Confirm the naming convention
|
|
|
|
| Pattern | Use |
|
|
|---------|-----|
|
|
| `k8s-<bu>-prd-ase1[c]` | Standard BU prod cluster (GCP zone-a or zone-c) |
|
|
| `k8s-shared-int-ase1` | Shared int |
|
|
| `k8s-aurva-prd-ase1` | Aurva |
|
|
| `k8s-supply-dev-ase1` | Dev/sandbox |
|
|
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier |
|
|
|
|
The directory name is a **contract** — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.
|
|
|
|
### 2. Determine the cluster's scheduling profile
|
|
|
|
| Cluster type | Scheduling key |
|
|
|--------------|----------------|
|
|
| GKE Autopilot | `cloud.google.com/compute-class` |
|
|
| Standard GKE | `dedicated:` |
|
|
|
|
Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of `ComputeClass` resources that need to land in `helm-overrides/<cluster>/<app>/computeclass/`.
|
|
|
|
### 3. Decide the minimal app set
|
|
|
|
Most clusters need at least:
|
|
|
|
- `external-secrets` — to materialise secrets from GCP Secret Manager
|
|
- `kube-state-metrics` — for fleet observability
|
|
- `victoria-metrics-agent` — to ship metrics to the central VM
|
|
- `fluentd` (or equivalent log shipper)
|
|
|
|
Plus per-cluster role:
|
|
|
|
- BU clusters: `contour-internal-0`, `contour-internal-1`, sometimes `contour-external` — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) for the convention
|
|
- DSGPU / Spark clusters: GPU operators, `keda` for queue autoscaling
|
|
- Dataplane (`db-*`) clusters: minimal — `kube-state-metrics` + `victoria-metrics-agent` only
|
|
|
|
### 4. Create the directory and minimal apps
|
|
|
|
```bash
|
|
mkdir -p helm-overrides/<cluster>
|
|
cd helm-overrides/<cluster>
|
|
```
|
|
|
|
For each app in the minimal set, follow [onboard-app-to-cluster.md](onboard-app-to-cluster.md) (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).
|
|
|
|
### 5. Update the per-cluster Contour matrix
|
|
|
|
If the cluster runs Contour, add a section to [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.
|
|
|
|
### 6. (If GKE Autopilot) Add the `ComputeClass` resources
|
|
|
|
Each Autopilot Contour / pool needs a corresponding `ComputeClass`. Drop them in:
|
|
|
|
```
|
|
helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml
|
|
```
|
|
|
|
Schema: [raw-manifest-sidecar-schema.md §ComputeClass](../schemas/raw-manifest-sidecar-schema.md). The `metadata.name` of each `ComputeClass` must equal the value the corresponding app's `nodeSelector: {cloud.google.com/compute-class: <name>}` references.
|
|
|
|
### 7. Open the base PR
|
|
|
|
```bash
|
|
git checkout -b cluster/<cluster>-base
|
|
git add helm-overrides/<cluster>/
|
|
git add contour-nodeselector-tolerations-summary.md # if updated
|
|
git commit
|
|
git push origin cluster/<cluster>-base
|
|
gh pr create --base main --title "Onboard <cluster> — base override set"
|
|
```
|
|
|
|
PR description must include:
|
|
|
|
- Cluster owner approver tag.
|
|
- Platform team approver tag.
|
|
- CMR ticket.
|
|
- Sister-repo PR for the `ApplicationSet` / per-cluster `Application` set.
|
|
- The minimal app set being shipped, with per-app rationale.
|
|
- The cluster's scheduling profile (key style + sample `nodeSelector`).
|
|
|
|
### 8. Pair the sister-repo PR
|
|
|
|
In `github.com/Meesho/devops-infra-argo-config`, draft the `ApplicationSet` (or per-cluster Application set) that walks `helm-overrides/<cluster>/`. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).
|
|
|
|
### 9. Stage subsequent app additions
|
|
|
|
After the base PR merges:
|
|
|
|
- Each additional app (`contour-internal-0`, `flagger`, etc.) is its own PR via [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
|
- Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.
|
|
|
|
---
|
|
|
|
## Anti-patterns
|
|
|
|
1. **Naming the directory `<cluster-name>` slightly differently** from how Argo CD registered it. Silent bind failure.
|
|
2. **Cloning another cluster's directory verbatim.** Per-cluster scheduling differs; `external-secrets` `SecretStore` references differ; image-tag pinning differs.
|
|
3. **Onboarding 30 apps in the base PR.** Stage them — base + per-app.
|
|
4. **Forgetting the Contour matrix update.** Future Contour edits then have no reference and silently mis-schedule.
|
|
5. **Skipping the `ComputeClass` resources** on a GKE Autopilot cluster. `nodeSelector` references a class that doesn't exist; pods Pending.
|
|
|
|
---
|
|
|
|
## Rollback
|
|
|
|
If the cluster onboarding turns out to be premature:
|
|
|
|
- Revert the values-side PR.
|
|
- Revert the sister-repo `ApplicationSet` PR.
|
|
- The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't `prune` what's not in its scope after the `Application` is removed); for a clean wipe, manually `kubectl delete ns` the affected namespaces.
|
|
|
|
---
|
|
|
|
## Related
|
|
|
|
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — for each app after the base PR.
|
|
- Schema: [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md) — for `ComputeClass` and `ExternalSecret` sidecars.
|
|
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
|
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|