79 lines
6.3 KiB
Markdown
79 lines
6.3 KiB
Markdown
# ADR-A3 — Per-cluster `nodeSelector` / `tolerations` / `computeClass`
|
||
|
||
> **Status:** Accepted (status quo — every cluster has bespoke scheduling).
|
||
> **Repo:** `devops-infra-helm-charts`.
|
||
> **Related:** [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md), [SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md), [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
Meesho's GKE fleet has two cluster types:
|
||
|
||
| Type | Scheduling primitives |
|
||
|------|------------------------|
|
||
| **Standard GKE** | Node pools with `dedicated:` taints and matching node labels |
|
||
| **GKE Autopilot** (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `ComputeClass` resources with `cloud.google.com/compute-class:` keys |
|
||
|
||
Within each type, individual clusters have their own node-pool / compute-class topology, designed for the workloads that cluster runs:
|
||
|
||
- `k8s-central-mqkafka-prd-ase1` has Kafka-optimised pools.
|
||
- `k8s-dsgpu-prd-ase1` has GPU-equipped Autopilot classes.
|
||
- `k8s-dengspark-prd-ase1` has Spark-executor pools.
|
||
- BU clusters (`k8s-supply-prd-ase1`, `k8s-demand-prd-ase1`, etc.) have per-app pools (`contour-external`, `contour-internal-0`, `monitoring`, …).
|
||
|
||
The `helm-overrides/<cluster>/<app>/custom-values.yaml` files reflect this — each cluster's values for the same app are different.
|
||
|
||
## Decision
|
||
|
||
`nodeSelector` / `tolerations` / `affinity` / `topologySpreadConstraints` / `cloud.google.com/compute-class` keys in this repo are **per-cluster, hand-authored, never copied**. The matrix of which Contour instance uses which key on which cluster is recorded in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md). For non-Contour apps, sample sibling apps on the same cluster.
|
||
|
||
## Rationale
|
||
|
||
1. **GKE Autopilot vs Standard isn't optional.** Autopilot's `ComputeClass` mechanism is mutually exclusive with standard `dedicated:` taints. A values block written for one type has no scheduling effect on the other — pods stay `Pending`.
|
||
|
||
2. **Per-cluster pool naming is intentional.** `contour-internal-0` on `k8s-supply-prd-ase1` is not the same node pool as `contour-internal-0` on `k8s-demand-prd-ase1` even if they share the name. The pool is sized differently, may have different machine types, may have different anti-affinity rules. Copying values across clusters works *by accident* sometimes; it fails *deliberately* the rest of the time.
|
||
|
||
3. **Multi-Contour-per-cluster pattern.** Most BU clusters run 5–6 Contour releases (`contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`). Each has its own pool. Cross-instance copying within the same cluster is also wrong.
|
||
|
||
4. **Operational reality.** When a cluster's node pool changes (a new pool added, an old one renamed), only that cluster's overrides need editing. Centralising scheduling values would mean every node-pool change becomes a fleet-wide PR.
|
||
|
||
5. **Reviewability.** A reviewer of a values diff can compare against the same file's git history (this cluster's previous state) without needing to know what other clusters look like. Cross-cluster consistency, when it exists, is incidental.
|
||
|
||
## Consequences
|
||
|
||
### Accepted
|
||
|
||
- **The most common silent bug** in this repo is `nodeSelector` / `tolerations` / `computeClass` copied from another cluster. ([SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md))
|
||
- **Cross-cluster cleanup is hard.** Renaming a pool (e.g. `monitoring` → `obs-shared`) is N PRs, one per cluster.
|
||
- **Onboarding a new cluster** is bespoke per app — every app needs its scheduling block authored from scratch ([onboard-new-cluster](../../docs/platform/procedures/onboard-new-cluster.md)).
|
||
- **Reasoning over the fleet** ("which apps are on which pool, fleet-wide?") requires `grep` across cluster directories.
|
||
|
||
### Mitigated
|
||
|
||
- **The Contour matrix file** ([contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md)) is the single source of truth for the Contour scheduling. **Read before editing any Contour values.**
|
||
- **The runbook** ([pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md)) walks the diagnosis when scheduling fails.
|
||
- **The skill** ([diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md)) gives an agent a deterministic diagnosis path.
|
||
|
||
### Open
|
||
|
||
- **A non-Contour scheduling matrix** has not been formalised. Sample-sibling-on-same-cluster is the working approach but isn't written down.
|
||
- **Auto-detection of "values copied from another cluster"** is plausible (compare a new file's `nodeSelector` against the cluster's own labels via kubectl). Not implemented.
|
||
- **Per-cluster topology drift over time** — when a cluster's underlying pools change in Terraform, the values here need a corresponding update. Today it's manual; ideally a Terraform-side hook would notify.
|
||
|
||
## Alternatives considered
|
||
|
||
| Alternative | Why not |
|
||
|-------------|---------|
|
||
| **A shared `_scheduling.yaml`** at the repo root or per-cluster, included via Helm subchart values. | Charts here mostly don't support arbitrary value-file inclusion (Argo CD's `valueFiles` does, but the structure would still need to map per-cluster). Would add a templating step that doesn't exist today. |
|
||
| **Centralised "platform values" subchart** that every release inherits. | Requires every chart to be a wrapper that depends on the platform subchart. Most upstream charts aren't structured for this. |
|
||
| **Programmatic generation** (a script that emits per-cluster overrides from a topology spec). | Plausible Phase-2 work — the topology spec would need to live somewhere (likely Terraform output), and the generator would need to handle every chart's idiosyncratic values shape. Not done today. |
|
||
| **Argo CD `ApplicationSet` with cluster generator + matrix.** | Would centralise routing but doesn't help author the scheduling values. The values still need to be cluster-specific somewhere. |
|
||
|
||
## References
|
||
|
||
- The matrix: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||
- Sanctity rule: [R5](../../docs/global/SANCTITY_RULES.md).
|
||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||
- Skill: [diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md).
|