devops-infra-helm-charts-gcp/wiki/analyses/ADR-A3-per-cluster-scheduling.md

ADR-A3 — Per-cluster nodeSelector / tolerations / computeClass

Status: Accepted (status quo — every cluster has bespoke scheduling). Repo: devops-infra-helm-charts. Related: contour-nodeselector-tolerations-summary.md, SANCTITY_RULES R5, pod-pending-scheduling.md.

---

Context

Meesho's GKE fleet has two cluster types:

Type	Scheduling primitives
Standard GKE	Node pools with dedicated: taints and matching node labels
GKE Autopilot (k8s-central-prd-ase1, k8s-dsgpu-prd-ase1, k8s-shared-int-ase1)	ComputeClass resources with cloud.google.com/compute-class: keys

Within each type, individual clusters have their own node-pool / compute-class topology, designed for the workloads that cluster runs:

• k8s-central-mqkafka-prd-ase1 has Kafka-optimised pools.
• k8s-dsgpu-prd-ase1 has GPU-equipped Autopilot classes.
• k8s-dengspark-prd-ase1 has Spark-executor pools.
• BU clusters (k8s-supply-prd-ase1, k8s-demand-prd-ase1, etc.) have per-app pools (contour-external, contour-internal-0, monitoring, …).

The helm-overrides/<cluster>/<app>/custom-values.yaml files reflect this — each cluster's values for the same app are different.

Decision

nodeSelector / tolerations / affinity / topologySpreadConstraints / cloud.google.com/compute-class keys in this repo are per-cluster, hand-authored, never copied. The matrix of which Contour instance uses which key on which cluster is recorded in contour-nodeselector-tolerations-summary.md. For non-Contour apps, sample sibling apps on the same cluster.

Rationale

1. GKE Autopilot vs Standard isn't optional. Autopilot's ComputeClass mechanism is mutually exclusive with standard dedicated: taints. A values block written for one type has no scheduling effect on the other — pods stay Pending.

2. Per-cluster pool naming is intentional. contour-internal-0 on k8s-supply-prd-ase1 is not the same node pool as contour-internal-0 on k8s-demand-prd-ase1 even if they share the name. The pool is sized differently, may have different machine types, may have different anti-affinity rules. Copying values across clusters works by accident sometimes; it fails deliberately the rest of the time.

3. Multi-Contour-per-cluster pattern. Most BU clusters run 5–6 Contour releases (contour-external, contour-external-1, contour-internal-0, contour-internal-1, contour-internal-intra-{0,1}). Each has its own pool. Cross-instance copying within the same cluster is also wrong.

4. Operational reality. When a cluster's node pool changes (a new pool added, an old one renamed), only that cluster's overrides need editing. Centralising scheduling values would mean every node-pool change becomes a fleet-wide PR.

5. Reviewability. A reviewer of a values diff can compare against the same file's git history (this cluster's previous state) without needing to know what other clusters look like. Cross-cluster consistency, when it exists, is incidental.

Consequences

Accepted

• The most common silent bug in this repo is nodeSelector / tolerations / computeClass copied from another cluster. (SANCTITY_RULES R5)
• Cross-cluster cleanup is hard. Renaming a pool (e.g. monitoring → obs-shared) is N PRs, one per cluster.
• Onboarding a new cluster is bespoke per app — every app needs its scheduling block authored from scratch (onboard-new-cluster).
• Reasoning over the fleet ("which apps are on which pool, fleet-wide?") requires grep across cluster directories.

Mitigated

• The Contour matrix file (contour-nodeselector-tolerations-summary.md) is the single source of truth for the Contour scheduling. Read before editing any Contour values.
• The runbook (pod-pending-scheduling.md) walks the diagnosis when scheduling fails.
• The skill (diagnose-scheduling.md) gives an agent a deterministic diagnosis path.

Open

• A non-Contour scheduling matrix has not been formalised. Sample-sibling-on-same-cluster is the working approach but isn't written down.
• Auto-detection of "values copied from another cluster" is plausible (compare a new file's nodeSelector against the cluster's own labels via kubectl). Not implemented.
• Per-cluster topology drift over time — when a cluster's underlying pools change in Terraform, the values here need a corresponding update. Today it's manual; ideally a Terraform-side hook would notify.

Alternatives considered

Alternative	Why not
A shared _scheduling.yaml at the repo root or per-cluster, included via Helm subchart values.	Charts here mostly don't support arbitrary value-file inclusion (Argo CD's valueFiles does, but the structure would still need to map per-cluster). Would add a templating step that doesn't exist today.
Centralised "platform values" subchart that every release inherits.	Requires every chart to be a wrapper that depends on the platform subchart. Most upstream charts aren't structured for this.
Programmatic generation (a script that emits per-cluster overrides from a topology spec).	Plausible Phase-2 work — the topology spec would need to live somewhere (likely Terraform output), and the generator would need to handle every chart's idiosyncratic values shape. Not done today.
Argo CD ApplicationSet with cluster generator + matrix.	Would centralise routing but doesn't help author the scheduling values. The values still need to be cluster-specific somewhere.

References

• The matrix: contour-nodeselector-tolerations-summary.md.
• Sanctity rule: R5.
• Runbook: pod-pending-scheduling.md.
• Skill: diagnose-scheduling.md.