gitadmin/devops-infra-helm-charts-gcp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added repo

Browse Source
This commit is contained in:
Your Name
2026-08-26 03:39:42 +05:30
parent 45c25a95af
commit b8575bb8b9
6889 changed files with 1217125 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+148
View File
@@ -0,0 +1,148 @@
# Architecture
This is a **GitOps Helm values repository**, not a service repo. There is no application code, no build, no tests — only declarative YAML (Helm charts, value overrides, Kubernetes manifests) and two git-hook shell scripts. Argo CD is the runtime; merging to `main` is the deployment.
## Section 1 — High-level design
### Repo purpose
Centralizes (1) cached/forked upstream Helm charts and (2) per-cluster value overrides for every infrastructure tool Meesho runs on its GKE fleet — observability (VictoriaMetrics, Mimir, Loki, Tempo, Grafana, OpenTelemetry, Pyroscope), ingress/edge (Contour/Envoy, ingress-nginx, cert-manager, external-dns, external-secrets), platform (Argo CD, Vault, Keda, Kyverno, Flagger, Jenkins, JFrog, Rancher, SonarQube), and data/AI (ClickHouse, Temporal, Superset, Deepgram, Aurva, Deepfence). It was carved out of `gcp-devops-admin` and is the source of truth for **what gets installed where, with what values**.
### System context
| Edge | Plays |
|------|-------|
| Sister repo `devops-infra-argo-config` | `github.com/Meesho/devops-infra-argo-config`. Holds the Argo CD `Application` / `ApplicationSet` manifests that point at this repo's `helm-overrides/<cluster>/<app>/` paths. Argo Application changes are PRs against that repo, not this one. |
| Argo CD instance(s) | Reconciles cluster state from this repo + the argo-config repo. One Argo CD per cluster (or per BU); each cluster has an `argocd/custom-values.yaml` here that configures *its own* Argo CD. |
| GKE cluster fleet | All consumers. Standard GKE clusters (`k8s-<bu>-prd-ase1`) plus auto-named dataplane clusters (`db-<numeric-id>-...`). Region: `asia-southeast1`. Project fleet: `meesho-admin-prd-0622`. |
| TruffleHog webhook | `https://observe.meeshogcp.in/api/webhook` — pre-commit hook reports verified secret findings. |
| CAC API | `https://observe.meeshogcp.in/api/cac/repos` — pre-commit allowlist; this repo isn't on the list, so the `cac validate` hook is a no-op. |
| Cursor metrics API | `https://cursor-server.meeshogcp.in/api/v1/...` — post-commit hook ships per-commit Cursor AI usage metrics. Local-only side effect. |
### Module boundaries
| Top-level dir | What it owns |
|---------------|--------------|
| `helm-templates/<chart>/` | Cached or forked upstream Helm chart. Touch only when a deliberate fork update is needed. Most are vanilla upstream — `Chart.yaml` + `templates/` + `values.yaml` (default upstream values). |
| `helm-templates/<chart>-<variant>/` | Versioned/blue-green sibling charts: `argo-cd` + `argo-cd-green`, `contour` + `contour-v1.33.3`, `keda` + `keda-2.17.1`, `opentelemetry-collector` + `-latest`, `victoria-metrics-cluster` + `-latest`, `victoria-metrics-agent` + `-latest`, `sonarqube` + `sonarqube-old`. The variant is the **target** of an in-flight chart upgrade — old version stays until the migration finishes. |
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | Cluster × application override values. Argo CD's `helm.valueFiles` points here; values merge over the chart's own `values.yaml` (or the upstream subchart's defaults when the local chart is a thin wrapper). |
| `helm-overrides/<cluster>/<app>/<extra>.yaml` | Non-`custom-values` files: extra Kubernetes resources rendered by an Argo Application's `path:` (e.g., `computeclass/*-cc.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`, `external-dns-services/*.yaml`). These are not Helm values — they are raw manifests applied alongside the Helm release. |
| `manifests/` | One-shot, cluster-scoped resources applied outside the Helm flow: `storageclass/`, `priorityclass/<cluster>/`, `jenkins-filestore-caching/{dev,prd}/`, `jenkins-gcs-caching/`, `jfrog-filestore-data/{dev,prd}/`. These are singletons — wrong values affect every workload in the cluster. |
| `pre-commit-scripts/` | `runner.sh` (parallel exec), `trufflehog-hook.sh` (verified-secret scan + webhook), `cac-validate.sh` (no-op here — gated on `configs/` path), `yaakhook.sh` (no-op here — gated on `api-collections/` path). |
| `post-commit-scripts/` | `runner.sh` + `commit-metric.sh` — Cursor AI commit metric collector (forks to background; never blocks). |
| `repository.yaml` | Owners (auto-managed by registry-bootstrap). |
### Architecture philosophy
**Reliability-first, surgical edits.** A bad values change can take down ingress, observability, or an entire cluster. Two rules govern every change:
1. **Reliability-first** — production blast radius is huge; mirror the existing pattern of neighbor cluster overrides; never delete keys without checking what depends on them; preserve explicit limits and HPA bounds.
2. **Surgical** — touch only what was asked. Don't refactor surrounding values, don't "normalize" across clusters in the same PR, don't drive-by-edit other charts in the same cluster's directory.
### Data flow (deployment lifecycle)
```
edit helm-overrides/<cluster>/<app>/custom-values.yaml
│
▼
git commit ──► pre-commit hooks (TruffleHog secrets scan)
│
▼
git push ──► PR → review → merge to main
│
▼
Argo CD on each cluster polls this repo + devops-infra-argo-config
│
▼
Application sync: helm template <chart> -f <override> → apply to cluster
│
▼
post-commit hook ships Cursor AI metrics (background, non-blocking)
```
Argo CD resolves `<chart>` from `helm-templates/` (when the Application uses `repoURL` of this repo with `path: helm-templates/<name>`) or pulls upstream by reading the local `Chart.yaml` dependencies (e.g., `argo-cd/Chart.yaml` declares `argo-cd 7.7.23` from `argoproj.github.io/argo-helm`).
### Cross-cutting concerns
- **Secret management** — TruffleHog pre-commit hook (`pre-commit-scripts/trufflehog-hook.sh`) blocks any verified secret. Reports go to the security webhook with content hash + commit + file/line metadata. NEVER bypass. Real secrets are externalised via `external-secrets` (per-cluster override exists in most clusters) backed by GCP Secret Manager / Vault.
- **Per-cluster scheduling** — every cluster has its own `nodeSelector` / `tolerations` / `computeclass` topology. The `contour-nodeselector-tolerations-summary.md` at the repo root documents the current matrix. GKE Autopilot clusters (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) use `cloud.google.com/compute-class` keys; standard clusters use `dedicated:` keys. Copying values between clusters without rewriting these is a reliable way to schedule pods on the wrong nodes.
- **Versioned chart migrations** — when upgrading a chart, the new version lives as a sibling dir (`argo-cd-green`, `contour-v1.33.3`, `keda-2.17.1`) until cutover. Both directories may be referenced by Argo Applications during the transition. Don't delete the old sibling without confirming no Application still points at it.
- **Image registry** — most overrides pin images to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho's internal Artifact Registry mirror), not upstream Docker Hub.
## Section 2 — Low-level details
### Cluster fleet (helm-overrides/)
Each top-level dir under `helm-overrides/` is one cluster. Two naming conventions:
| Convention | Example | Owner / type |
|------------|---------|--------------|
| `k8s-<bu>-prd-ase1` (and `-prd-ase1c`) | `k8s-central-prd-ase1`, `k8s-supply-prd-ase1`, `k8s-dataengg-prd-ase1` | Standard GKE cluster, BU-owned (central, supply, demand, dataengg, datascience, dengspark, dscispark, dsgpu, farmiso, ml-platform, admin, sec-admin, devops-admin, central-mqkafka). Project varies per BU (e.g., `meesho-supply-prd`, `meesho-datascience-prd`). All in `asia-southeast1`. |
| `k8s-shared-int-ase1` | (only) | Shared **integration** (pre-prod) cluster. The only non-prod cluster in this repo. |
| `k8s-aurva-prd-ase1` | (only) | Aurva integration. Limited override set (contour-internal, rancher only). |
| `db-<numeric-id>-...` | `db-2516183257845181-c-1204-195038-428` | Auto-named dataplane / data-tier clusters. Override sets are minimal (typically `kube-state-metrics` + `victoria-metrics-agent` only). `fullnameOverride` values use `dbc-<bu>-prd` form (e.g., `dbc-dsci-prd`). |
| `k8s-supply-dev-ase1` | (only) | Dev/sandbox supply cluster. |
Per-cluster READMEs (where present) say: "This folder contains the custom `values.yaml` files organized based on specific cluster names. Each subdirectory corresponds to a particular cluster and holds the configurations for the applications and tools deployed within that cluster."
### Application directory layout (per cluster)
Inside `helm-overrides/<cluster>/`, each subdirectory is an Argo CD Application. Common shapes:
| Layout | Meaning | Example |
|--------|---------|---------|
| `<app>/custom-values.yaml` | Single Helm release; values merged onto a chart from `helm-templates/<chart>` | `argocd/custom-values.yaml`, `etcd/custom-values.yaml`, `contour-internal-0/custom-values.yaml` |
| `<app>/<sub>.yaml` (no custom-values.yaml) | Argo Application's `path:` points here; raw manifests applied | `computeclass/contour-external-cc.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`, `external-dns-services/*.yaml` |
| Both | Helm release + sidecar raw manifests | rare; check the matching Application in `devops-infra-argo-config` |
Multiple Contour instances per cluster is the norm — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-0`, `contour-internal-intra-1`. Each maps to a different node pool / dedicated node taint or compute class. The mapping per cluster is recorded in `contour-nodeselector-tolerations-summary.md` at the repo root — read before adding/changing a Contour instance.
### Helm chart inventory
| Category | Charts in `helm-templates/` |
|----------|----------------------------|
| Argo / GitOps | `argo-cd`, `argo-cd-green` |
| Ingress / edge | `contour`, `contour-v1.33.3`, `contour-ca-issuer`, `contour-cert-checker`, `ingress-nginx`, `cert-manager`, `external-dns`, `external-secrets` |
| Observability — metrics | `prometheus-node-exporter`, `prometheus-stackdriver-exporter`, `kube-state-metrics`, `kube-events`, `victoria-metrics-{single,cluster,cluster-latest,agent,agent-latest,alert,alert-stateful,alerts-config,auth,mcp}`, `vm-alert-config`, `mimir-distributed`, `pmm`, `telegraf-operator` |
| Observability — logs/traces/profiles | `fluentd`, `loki-distributed`, `tempo-distributed`, `pyroscope`, `alloy`, `opentelemetry-collector`, `opentelemetry-collector-latest`, `opentelemetry-operator`, `elastalert2`, `coroot-node-agent`, `deepfence-console`, `deepfence-router` |
| UI / dashboards | `grafana`, `grafana-edge`, `grafana-mcp`, `kubernetes-dashboard`, `superset`, `uptime-kuma` |
| Workflow / CI/CD | `jenkins`, `jfrog`, `sonarqube`, `sonarqube-old`, `flagger`, `keda`, `keda-2.17.1`, `kyverno`, `loadtester`, `temporal`, `dind`, `canary-bot-gcp`, `paused-container` |
| Networking / DNS | `coredns`, `kube-dns`, `bifrost`, `conntrack-adjuster`, `node-thp-config` |
| Data / search / DB | `clickhouse`, `etcd`, `vault`, `elasticsearch-mcp`, `eck-operator`, `athens-proxy` |
| AI / 3rd-party | `aurva-dataplane`, `deepgram-onprem`, `rancher` |
74 charts total. Each has its own `Chart.yaml`. 19 also have a `Chart.lock` (charts with subchart dependencies that have been resolved with `helm dependency update`). Many of the local "charts" (e.g., `argo-cd/Chart.yaml`) are thin wrappers that declare the upstream chart as a dependency in `Chart.yaml` — the actual templates come from upstream. Others (e.g., `contour/`) carry a full vendored `templates/` tree.
### Manifests (singletons)
| Path | Scope | What it is |
|------|-------|------------|
| `manifests/storageclass/*.yaml` | Cluster-wide | StorageClasses: `pd-standard-retain-dr`, `sc-filestore-standard`, `sc-pd-ssd`, `sc-pd-standard`. Wrong change affects every PVC. |
| `manifests/priorityclass/<cluster>/*.yaml` | Per-cluster | `priorityclass-high.yaml`, `priorityclass-low.yaml` per BU cluster. Affects scheduling priority for every pod that references them. |
| `manifests/jenkins-filestore-caching/{dev,prd}/{pv,pvc}.yaml` | Per-env | Jenkins build cache PV/PVC backed by GCP Filestore. |
| `manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml` | Per-env | Jenkins GCS-backed cache. |
| `manifests/jfrog-filestore-data/{dev,prd}/` | Per-env | JFrog data PV/PVC. |
### Git hooks
| Hook | Path | Behavior |
|------|------|----------|
| pre-commit, pre-push | `pre-commit-scripts/runner.sh` | Forks every other `*.sh` in the same dir in parallel; fails the commit if any fails. |
| pre-commit, pre-push | `pre-commit-scripts/trufflehog-hook.sh` | Runs `trufflehog git file://. --since-commit HEAD --branch=$(git rev-parse --abbrev-ref HEAD) --json --results=verified`. On a verified hit: prints the finding and POSTs metadata (no raw secret) to `https://observe.meeshogcp.in/api/webhook`. Exit 1 blocks the commit. **Never bypass.** |
| pre-commit, pre-push | `pre-commit-scripts/cac-validate.sh` | Gated on `configs/` paths in the staged diff. This repo has no `configs/`, so it always early-exits. Documented for completeness. |
| pre-commit, pre-push | `pre-commit-scripts/yaakhook.sh` | Gated on `api-collections/` paths. This repo has none — early-exits. |
| post-commit | `post-commit-scripts/runner.sh` → `commit-metric.sh` | Two-phase: synchronous `start` writes a temp file with commit hash + repo info, spawns detached `continue` background process. `continue` polls the local Cursor SQLite DB up to 120 s for `aiCodeTracking.recentCommit.commitHash` to match HEAD, then POSTs the AI line-edit metrics to `https://cursor-server.meeshogcp.in/api/v1/add-commit-metrics`. Skips rebase/merge/cherry-pick commits. Always exits 0 — never blocks the hook. |
### Configuration touch points
There is no application config to touch. The only env-equivalent layer is `helm-overrides/<cluster>/<app>/custom-values.yaml` — every cluster × application pair is its own configuration unit. The CAC API at `https://observe.meeshogcp.in/api/cac/repos` would gate config schema validation, but this repo isn't on that allowlist.
### Critical invariants & gotchas
- **`helm-templates/` is mostly upstream code.** Most charts here are `helm pull`-ed copies of upstream charts (Bitnami, ArgoProj, VictoriaMetrics, Grafana). Editing a `templates/*.yaml` inside one of these is editing upstream — easy to forget on the next chart bump. Treat `helm-templates/<chart>/` as read-only unless you are explicitly forking; if you fork, document why in the chart's `README.md`.
- **Each cluster is unique on `nodeSelector` / `tolerations` / `computeClass`.** See `contour-nodeselector-tolerations-summary.md` at the repo root. GKE Autopilot clusters use `cloud.google.com/compute-class` keys; standard clusters use `dedicated:` keys. Copying a values block from one cluster to another without rewriting these schedules pods on the wrong nodes — or pending forever.
- **Versioned siblings are intentional, not duplicates.** `argo-cd` vs `argo-cd-green`, `contour` vs `contour-v1.33.3`, `keda` vs `keda-2.17.1`, `opentelemetry-collector` vs `-latest`, `sonarqube` vs `sonarqube-old`. Don't "consolidate" them. They support blue-green chart upgrades — both versions may be live during a migration.
- **`fullnameOverride` is load-bearing in dataplane overrides.** `db-*` cluster values use `fullnameOverride: <kind>-dbc-<bu>-prd` to keep release names stable across re-installs. Don't change these — Service DNS, PVC binding, and Argo Application names depend on them.
- **Argo CD reconciles from `main`.** A merge to `main` is a deploy. There is no staging branch — review the PR as if it ships to production, because it does.
- **`devops-infra-argo-config` is the routing layer.** A new chart in `helm-templates/` does nothing until an `Application` referencing it is added to the sister repo. A new cluster directory in `helm-overrides/` does nothing until an `ApplicationSet` covers it. Pair the two-repo change.
<!-- meesho-init: generated-at=2026-05-05T17:31:45Z base-sha=debd20fcc510e19bfe3123a08247dde39790643b+dirty -->
docs/global/AGENT_BOUNDARIES.md
+123
View File
@@ -0,0 +1,123 @@
# AGENT_BOUNDARIES.md
> **Scope:** operations agents may perform on the `devops-infra-helm-charts` repo.
> **Companion docs:** [SANCTITY_RULES.md](SANCTITY_RULES.md) (hard rules), [coding-guidelines/helm-values.md](coding-guidelines/helm-values.md) (style).
This document classifies every operation in this repo into Layer 1 / 2 / 3 with explicit blast radius and approval requirements. **An agent operating in this repo MUST consult this file before any write.**
---
## The 3-Layer Model (recap)
| Layer | Agent action | Safety gate |
|-------|--------------|-------------|
| **Layer 1 — Agent-Writable** | Generate diff, open PR. Do **not** apply directly to clusters. | PR review + Argo CD UI Sync click. |
| **Layer 2 — Agent-Readable (Advisory)** | Research, analyse, recommend. Human executes. | Human review + manual execution. |
| **Layer 3 — Agent-Blocked** | Refuse the write. Explain why. Cite this doc. | None applicable. |
---
## Per-operation classification
### Layer 1 — Agent-Writable (this repo's normal operating range)
All edits land via PR. A merge to `main` is then reconciled by Argo CD per cluster — most infra Applications use manual sync ([ADR-A5](../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)), so the workload deploy is a separate human Sync click.
| Operation | File(s) touched | Blast radius | Required approvers |
|-----------|-----------------|--------------|--------------------|
| Edit `custom-values.yaml` for one app on one cluster | `helm-overrides/<cluster>/<app>/custom-values.yaml` | One Helm release on one cluster | Service / app owner |
| Add a new app under an existing cluster | new dir + files in `helm-overrides/<cluster>/<app>/` | One new release | App owner + cluster owner; pair with Argo `Application` PR in sister repo |
| Add a sidecar raw manifest alongside an existing app | `helm-overrides/<cluster>/<app>/<extra>.yaml` | Adds Kubernetes resources rendered alongside the Helm release | App owner |
| Add a new override file under an existing app dir | new file in `helm-overrides/<cluster>/<app>/` | Layered into the same release if the Argo Application's `valueFiles` covers it | App owner |
| Append to an existing chart's `Chart.yaml` `dependencies[]` | `helm-templates/<chart>/Chart.yaml` + refresh `Chart.lock` via `helm dependency update` | Affects every consumer of that chart | Platform team |
| Bump a `dependencies[].version` in `Chart.yaml` | `helm-templates/<chart>/Chart.yaml` + `Chart.lock` | Same | Platform team — see [update-chart-version](../platform/procedures/update-chart-version.md) |
| Add a new chart sibling for blue-green migration | new dir `helm-templates/<chart>-<variant>/` | Migration target only; old sibling stays live | Platform team — see [blue-green-chart-migration](../platform/procedures/blue-green-chart-migration.md) |
| Edit the per-cluster Contour scheduling matrix | `contour-nodeselector-tolerations-summary.md` | Documentation; no runtime effect | Platform team |
### Layer 1 — HIGH RISK (write allowed, but require explicit approval and detailed rationale in PR)
| Operation | Why it's high risk |
|-----------|--------------------|
| Edit `helm-templates/<chart>/templates/` or `helm-templates/<chart>/values.yaml` | Most charts here are vanilla upstream pulled via `helm pull`. Edits silently fork the chart and get clobbered on the next upstream sync. **Only allowed if the fork is intentional and documented in that chart's `README.md` — see [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md).** |
| Delete a versioned sibling chart (`<chart>-green`, `<chart>-vX.Y.Z`, `<chart>-latest`, `<chart>-old`) | The variant is a blue-green migration target. Both versions may be live simultaneously. **Confirm zero references in `github.com/Meesho/devops-infra-argo-config` before deleting.** |
| Edit `manifests/storageclass/*.yaml` | Cluster-wide singleton; affects every PVC. **Platform-team review required.** |
| Edit `manifests/priorityclass/<cluster>/*.yaml` | Affects scheduling priority for every pod that references the class. Platform-team review. |
| Edit `manifests/{jenkins-filestore-caching,jenkins-gcs-caching,jfrog-filestore-data}/{dev,prd}/` | Per-env stateful PV/PVCs; wrong reclaim policy can drop CI build caches or JFrog binary data. Platform-team review. |
| Add a new cluster directory under `helm-overrides/` | Creates a new deployment target. **Pair with the cluster's `ApplicationSet` change in the sister repo. Per-cluster `nodeSelector` / `tolerations` / `computeClass` MUST be written from scratch, not copied from another cluster ([SANCTITY_RULES R5](SANCTITY_RULES.md)).** |
| Change `fullnameOverride` in any `custom-values.yaml` | Service DNS, PVC binding, ConfigMap/Secret references, and downstream Argo Application names depend on it being stable. **Almost always wrong to touch.** |
| Bulk find-replace across cluster directories ("normalise" labels, image tags, etc.) | The whole point of per-cluster overrides is divergence. Cross-cluster cleanup is its own PR with its own scope ([SANCTITY_RULES R6](SANCTITY_RULES.md)). |
| Pin an image tag to a registry outside `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/` | Production overrides go through Meesho's Artifact Registry mirror. Direct Docker Hub / Quay pulls are a supply-chain and rate-limit risk. |
### Layer 2 — Agent-Readable (advisory only)
Some operations a user might ask for touch systems an agent can analyse but not change.
| Operation | Why advisory |
|-----------|--------------|
| "Sync `<release>` now in Argo CD" | The Sync click is intentional human action. **Recommend the command (`argocd app sync <name>`) or the Argo CD UI path; do not execute.** |
| "Restart pods for `<release>`" | kubectl operation against a workload cluster. Recommend `kubectl rollout restart deploy/<name> -n <ns>`; do not execute. |
| "Why is this pod pending?" | Live cluster state. Recommend `kubectl describe pod` and walk [pod-pending-scheduling.md](../platform/runbooks/pod-pending-scheduling.md); do not infer. |
| "Why did the chart not render?" | Reproduce locally with `helm template`; recommend the fix. Do not push without PR. |
| "Add an alert rule for `<service>`" | Alert rules are in `victoria-metrics-alert*` chart values *here*, but routing/notifier config is elsewhere (Pulse / Slack webhook secrets). Recommend the right file; pair with the routing change. |
| "What's broken on the GKE cluster itself?" | Cluster-level GCP / GKE issues are out of scope for this repo. Recommend the Terraform repo or the platform team. |
| "Reconcile the Terraform drift for the cluster" | Out of scope — that's `terraform-gcp-infra`. Refuse + redirect. |
### Layer 3 — Agent-Blocked (refuse + explain)
| Operation | Why blocked | What would unblock |
|-----------|-------------|--------------------|
| Hand-edit `repository.yaml` | Owned by `registry-bootstrap` automation; manual edits are overwritten on the next bootstrap run. | Update the upstream registry that feeds registry-bootstrap. |
| Bypass pre-commit hooks (`--no-verify`, `git commit -n`, removing the hook) | TruffleHog is the last-line secret scan. ([SANCTITY_RULES R4](SANCTITY_RULES.md)) | If a hit is a known false positive, confirm with the platform/security team in writing first. |
| Push directly to `main` | Branch protection. ([SANCTITY_RULES R1](SANCTITY_RULES.md)) | (Never legitimate.) |
| Force-push to `main` | Same. | (Never legitimate.) |
| Run `helm install`, `helm upgrade`, or `kubectl apply` against any workload cluster | This is GitOps; in-cluster mutation creates drift Argo CD will reconcile away. | Use the Argo CD UI Sync flow, or the cluster's incident-response toolset. |
| Curl / probe / interact with `int.meesho.int`, `prd.meesho.int`, `int.mrouter.int`, `prd.mrouter.int`, or any workload-traffic `*.meeshogcp.in` host | Production traffic surfaces. ([SANCTITY_RULES R3](SANCTITY_RULES.md)) | (Pre-commit hook telemetry to `observe.meeshogcp.in` is automated; that is hook infrastructure, not agent action.) |
| Edit Argo `Application` / `ApplicationSet` manifests | They live in `github.com/Meesho/devops-infra-argo-config`. | Open a PR there. |
| Modify a chart whose `helm-templates/<chart>/` is vanilla upstream, without a documented fork rationale | An accidental fork is silently clobbered on the next sync. | Follow [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md) and document the fork in the chart's `README.md`. |
| Copy a `custom-values.yaml` from one cluster to another verbatim | Per-cluster `nodeSelector` / `tolerations` / `computeClass` differ. ([SANCTITY_RULES R5](SANCTITY_RULES.md)) | Author the override from scratch using the per-cluster matrix in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) and sibling files. |
---
## Cross-cuts: things to verify on **every** Layer 1 PR
Pre-commit hooks here are minimal — TruffleHog covers secrets; CAC and Yaak are no-ops because their gating paths don't exist in this repo. The agent is the next line of defence. On every PR, mentally run this checklist:
1. **Surgical scope.** The PR touches only the cluster × app the task asked for. No drive-by edits to neighbours.
2. **Per-cluster scheduling rewritten, not copied.** If the change involves `nodeSelector` / `tolerations` / `computeClass`, every cluster has its own topology. Copy-and-rename is the most common silent bug. ([contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md))
3. **`fullnameOverride` unchanged.** Unless the explicit headline of the PR is a release-name migration.
4. **Image tags pin to Meesho's GAR mirror** (`asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>`), not Docker Hub.
5. **Versioned siblings preserved.** A delete or rename of a `<chart>-green` / `-latest` / `-vX.Y.Z` directory only proceeds after grepping `github.com/Meesho/devops-infra-argo-config` for references.
6. **Chart.yaml + Chart.lock move together.** A `dependencies[].version` bump without a refreshed lockfile is incomplete.
7. **Sister repo paired (where required).** A new app under a cluster needs a matching `Application` in the sister repo. A new cluster directory needs a matching `ApplicationSet` (or per-cluster Application set) in the sister repo. Note the pairing in the PR description.
8. **No `*` or `latest` image tags introduced.** Argo CD's manual-sync default does not rescue you from a pulled-out-from-under-you image.
9. **`manifests/` left alone unless the PR's headline says so.** `storageclass/` and `priorityclass/<cluster>/` are cluster-wide singletons.
10. **TruffleHog passed.** Never bypass.
---
## Approval requirements summary
| Change category | Reviewer required | CMR required? |
|-----------------|-------------------|---------------|
| Single-cluster `custom-values.yaml` edit (no scheduling, no `fullnameOverride`) | App owner | Per BU policy |
| Add a new app override under an existing cluster | App owner + cluster owner | **Yes** |
| Add a new cluster directory | Platform team | **Yes** |
| `Chart.yaml` `dependencies[].version` bump | Platform team | **Yes** for prod-fleet charts (Argo CD, Contour, VictoriaMetrics, ingress) |
| Intentional fork of `helm-templates/<chart>/templates/` | Platform team — multi-reviewer | **Yes** |
| Delete a versioned chart sibling | Platform team — confirm zero sister-repo references | **Yes** |
| Edit `manifests/storageclass/` or `priorityclass/<cluster>/` | Platform team | **Yes** |
| Pre/post-commit hook script change | Platform team + security (if hook scope changes) | Per CMR matrix |
| `fullnameOverride` change | Platform team — multi-reviewer | **Yes** — emergency-only |
CMR = Change Management Request. Per Meesho process; not enforced in-repo, applied at org level.
---
## Escalation
If a request falls outside this matrix or you can't classify it cleanly:
1. Refuse the write.
2. Cite this document + the row that matches (or explain why no row matches).
3. Suggest the human asks the platform team or files a CMR.
4. Do not improvise around the boundary.
docs/global/SANCTITY_RULES.md
+122
View File
@@ -0,0 +1,122 @@
# SANCTITY_RULES.md
> Non-negotiable rules for the `devops-infra-helm-charts` repo.
> Read alongside [AGENT_BOUNDARIES.md](AGENT_BOUNDARIES.md) (per-operation Layer map) and [coding-guidelines/helm-values.md](coding-guidelines/helm-values.md) (style).
These are the rules whose violation is a process incident, not a clever shortcut. Each one is the result of a real failure mode (or proximity to one).
---
## R1 — `main` is production
Argo CD on every cluster reconciles from `main`. There is no staging branch. A merge is a deploy event.
- **No experimentation on `main`.** Always work on a feature/fix branch and open a PR.
- **No force-push to `main`** (enforced at GitHub org level).
- **No "I'll just amend that" after merge.** A new PR is the only path forward.
## R2 — Sister repo is the routing layer; this repo is the values layer
`devops-infra-helm-charts` holds *what gets installed and with what values*. `github.com/Meesho/devops-infra-argo-config` holds *where it gets routed* (the Argo `Application` / `ApplicationSet` manifests).
- **A new chart in `helm-templates/` does nothing** until an `Application` referencing it lands in the sister repo.
- **A new cluster directory in `helm-overrides/` does nothing** until an `ApplicationSet` (or per-cluster Application set) covers it.
- **Pair the two-repo change.** Cite the sister-repo PR in the description here, and vice versa.
## R3 — Production traffic surfaces are off-limits
Agent-initiated requests to `int.meesho.int`, `prd.meesho.int`, `int.mrouter.int`, `prd.mrouter.int`, and workload `*.meeshogcp.in` services are forbidden. Any accidental call can affect live traffic or data.
- **Never `curl`, `WebFetch`, query, or otherwise probe** these endpoints.
- The pre-commit hook telemetry endpoints (`observe.meeshogcp.in/api/webhook`, `cursor-server.meeshogcp.in/api/v1/...`) are *automated hook infrastructure*, not agent-initiated traffic. They are not a precedent for agent calls.
## R4 — Pre-commit hooks must pass
`pre-commit-scripts/runner.sh` invokes TruffleHog (verified-secret scan with webhook telemetry). CAC and Yaak hooks exist but are no-ops here (their gating paths — `configs/` and `api-collections/` — don't exist in this repo). TruffleHog is the last-line secret scan.
- **Never bypass** with `--no-verify`, `git commit -n`, or by removing the hook.
- **Never weaken a hook to "just get this through"** — fix the upstream cause.
- A verified TruffleHog hit means a secret is staged. Real secrets belong in `external-secrets` (per-cluster), backed by GCP Secret Manager / Vault — not in `custom-values.yaml`.
## R5 — Per-cluster scheduling is bespoke; never copy across clusters
Each cluster has its own node-pool topology. Some clusters use GKE Autopilot's `cloud.google.com/compute-class:` keys (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`); standard clusters use `dedicated:` keys. Per-Contour-instance, per-cluster mappings are recorded in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
- **Never copy `nodeSelector` / `tolerations` / `computeClass` blocks between clusters** without rewriting them from scratch against the destination cluster's topology.
- **Always read the matrix** before editing any Contour values; cross-reference sibling cluster files for non-Contour scheduling.
- Wrong values strand pods on wrong nodes or leave them `Pending` indefinitely.
## R6 — Surgical edits only
The repo has ~30 cluster directories × dozens of apps each. The whole point of per-cluster overrides is divergence — accumulated, deliberate, often for tiered traffic or specialised hardware.
- **Never "normalise" values across clusters in the same PR as a feature change.** Cross-cluster clean-ups are their own PRs with their own scope and CMRs.
- **Never auto-update labels, image tags, or comments** on apps you weren't asked to touch.
- **One change-type per PR.** Reviewer cognition matters; rollback granularity matters.
## R7 — `helm-templates/<chart>/` is mostly upstream
Most charts here are vanilla upstream pulled via `helm pull`. The local `Chart.yaml` is often a thin wrapper that declares the upstream chart as a dependency. Editing a `templates/*.yaml` or the chart's own `values.yaml` silently forks the chart, and the fork is clobbered on the next upstream sync.
- **Never edit `helm-templates/<chart>/templates/` or `values.yaml` casually.** If a fork is intentional, follow [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md) and document the reason in the chart's `README.md`.
- **Never bump `Chart.yaml` `dependencies[].version`** without (a) reading the upstream changelog, (b) running `helm dependency update` to refresh `Chart.lock`, and (c) calling out the bump in the PR description.
## R8 — Versioned chart siblings stay live during migrations
`<chart>` ↔ `<chart>-green` (blue-green), `<chart>` ↔ `<chart>-vX.Y.Z` (pinned upgrade target), `<chart>` ↔ `<chart>-latest` (work-in-progress), `<chart>` ↔ `<chart>-old` (retired but still referenced). Both directories may be referenced by Argo Applications during the migration window.
- **Never delete a versioned sibling** without confirming zero references in `github.com/Meesho/devops-infra-argo-config`.
- **Never "consolidate" siblings** into one chart in a maintenance PR. The split is intentional ([ADR-A2](../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md)).
## R9 — `fullnameOverride` is load-bearing
Helm's `fullnameOverride` controls the name of every released Service, Deployment, StatefulSet, ConfigMap, Secret, and PVC. Service DNS, PVC binding, ConfigMap references in other apps, and Argo Application names downstream all depend on it being stable.
- **Never change `fullnameOverride`** in any `custom-values.yaml`.
- A release-name migration is its own headline-of-the-PR change with platform-team approval, a documented before/after map, and an explicit reason.
## R10 — `manifests/` is cluster-wide singletons
`manifests/storageclass/*.yaml` is repo-global; a wrong StorageClass affects every PVC on every cluster that consumes it. `manifests/priorityclass/<cluster>/*.yaml` is per-cluster but cluster-wide; a wrong PriorityClass changes scheduling priority for every pod that references it.
- **Never edit `manifests/storageclass/`** or `manifests/priorityclass/<cluster>/` without platform-team review.
- **Never delete a StorageClass referenced by an existing PVC** — Kubernetes will not remove the StorageClass while bindings remain, but new PVCs will fail to provision.
## R11 — Image tags go through Meesho's Artifact Registry mirror
Production overrides pin images to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>`, not upstream Docker Hub or Quay. The mirror exists for supply-chain control and rate-limit isolation.
- **Never introduce a Docker Hub / Quay / GCR / ECR upstream tag** in a production override.
- **Never use `:latest` or unpinned tags** in production overrides; an Argo CD reconcile is not the same as a controlled rollout.
## R12 — `repository.yaml` is automation-owned
Generated by `registry-bootstrap`. Hand edits will be silently overwritten on the next bootstrap run.
- **Never hand-edit `repository.yaml`.** If owner data is wrong, fix it in the upstream registry that feeds registry-bootstrap.
## R13 — Branch protection trumps everything
Direct push to `main` is blocked at the GitHub org level. PRs go through code review.
- **Never propose workflows that bypass branch protection.** If a hotfix is genuinely urgent, the path is "open a PR with a `hotfix/*` branch and request emergency review," not "force-push to main."
## R14 — Pre-existing schema/label drift is not yours to fix
Some clusters have inconsistencies in label values, indentation, key ordering, or comment style — accumulated technical-debt items. Some apps lack values keys their newer siblings have. Some clusters use long BU names (`supply`) while others use short (`supl`).
- **Never normalise label or BU values in unrelated PRs.** Cross-cutting clean-ups are their own PRs with their own scope and CMRs.
- **Never reformat a YAML file in passing.** Diffs full of indentation churn drown out the real change.
---
## What "non-negotiable" means
Each rule above has a documented reason and is the result of a real failure mode (or proximity to one). If you find yourself wanting to break a rule, the path is:
1. Write up the case in a doc (or PR description) explaining what would change and why.
2. Loop in the platform team for review.
3. If the rule should change, change *the rule first* in this file (with a PR to update it), then act.
4. If the rule should not change, find another way.
A merge that violates a rule here is a process incident, not a clever shortcut.
docs/global/agent-operations-guide.md
+82
View File
@@ -0,0 +1,82 @@
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
# Agent Operations Guide
A meta-guide for any AI agent (or human contributor wearing the agent hat) doing work in this repo. Read this first; it tells you which other docs to load and in what order.
## Pre-flight (always, every task)
Before reading any task-specific file:
1. **Read the repo-root `CLAUDE.md`.** It is the authoritative source for the NEVER-DO list, cluster naming, versioned siblings, multi-Contour pattern, and the Layer constraint summary. Everything else in `docs/` derives from it.
2. **Read [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)** to confirm which Layer the operation lives in.
3. **Read [SANCTITY_RULES.md](./SANCTITY_RULES.md)** to confirm the operation is not on the don't-touch list.
4. **Read [`wiki/entities/DevOps Infra Helm Charts.md`](../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md)** for the conceptual model (this repo as values store; sister repo as routing layer).
5. **Skim [`docs/architecture.md`](../architecture.md)** if the task touches more than one chart or cluster.
If the task targets a specific chart family (Contour, Vault, Argo CD, observability stack), also load the relevant coding guideline before editing:
- [coding-guidelines/helm-values.md](./coding-guidelines/helm-values.md) — values-file conventions
- [coding-guidelines/argocd.md](./coding-guidelines/argocd.md) — Argo CD interaction model
- [coding-guidelines/observability.md](./coding-guidelines/observability.md) — VM / Mimir / Loki / Tempo / Grafana
## Authoring loop
The supported edit path for almost every task in this repo:
1. **Branch off `main`.** Never push to `main`. The branch name should describe the cluster × app being touched.
2. **Edit the single targeted file** under `helm-overrides/<cluster>/<app>/custom-values.yaml` (or sibling `<extra>.yaml`). No drive-by edits to other apps in the same directory. No cross-cluster "normalization" in the same PR.
3. **Dry-run with `helm template`** to confirm the values render. Use the command from the root `CLAUDE.md` Quick reference table:
```
helm template <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml
```
4. **Commit.** The pre-commit hooks run automatically:
- **TruffleHog** — secret scan, blocking. NEVER bypass. See [claude/08-pre-commit-and-hooks.md](../../claude/08-pre-commit-and-hooks.md).
- CAC and Yaak hooks are gated/no-op on this repo.
5. **Open a PR.** Pair with a sister-repo PR if a new Argo Application is being introduced.
6. **Reviewer + Argo CD UI Sync are the safety gates.** Once the PR is merged to `main`, Argo CD on the target cluster reconciles. Per-cluster `syncPolicy` (auto vs manual) is set in the sister repo `Meesho/devops-infra-argo-config`, not here.
See [platform/procedures/onboard-app-to-cluster.md](../platform/procedures/onboard-app-to-cluster.md) and [platform/procedures/update-chart-version.md](../platform/procedures/update-chart-version.md) for two of the most common authoring loops.
## When to refuse
Refuse the operation outright if it falls into one of these categories. Cite [SANCTITY_RULES.md](./SANCTITY_RULES.md) in the refusal:
- Direct push to `main`, or any `--force` push.
- Bypassing the pre-commit hook (`--no-verify`, `git commit -n`).
- Curl / probe / query against any production endpoint (`*.meesho.int`, `*.mrouter.int`, `*.meeshogcp.in`).
- Edit to `repository.yaml` (owned by `registry-bootstrap`).
- Edit to an Argo `Application` / `ApplicationSet` manifest (lives in sister repo).
- Cross-cluster copy-paste of `nodeSelector` / `tolerations` / `computeClass` without rewrite.
- Deletion of a versioned-sibling chart without confirming zero sister-repo references.
## When to escalate
If the operation is potentially valid but exceeds Layer-1 authority — chart fork, dep bump with breaking changes, StorageClass/PriorityClass edit, Vault HA work — stop and escalate per [escalation-matrix.md](./escalation-matrix.md). The escalation matrix maps each situation to an owner and a channel.
## Tooling commands (cheat sheet)
The repo-root `CLAUDE.md` Quick reference table is the source of truth. Reproduced here for convenience:
| Task | Command |
|------|---------|
| Install pre-commit hooks | `pre-commit install --hook-type pre-commit --hook-type pre-push --hook-type post-commit` |
| Re-run pre-commit on staged changes | `pre-commit run` |
| Render a chart locally | `helm template <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml` |
| Refresh subchart deps | `helm dependency update helm-templates/<chart>` |
| Diff against live release | `helm diff upgrade <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml` |
| Lint a chart | `helm lint helm-templates/<chart>` |
| Find which clusters override an app | `find helm-overrides -maxdepth 2 -type d -name '<app>'` |
## Output discipline
- Generate diffs and PRs; do not apply directly to clusters. In-cluster mutation is incident response, not authoring.
- Surgical edits only. Touch the cluster × application asked for; leave the rest.
- Cross-link reasoning to ADRs in [`wiki/analyses/`](../../wiki/analyses/) where helpful.
## See also
- [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)
- [SANCTITY_RULES.md](./SANCTITY_RULES.md)
- [escalation-matrix.md](./escalation-matrix.md)
- [coding-guidelines/helm-values.md](./coding-guidelines/helm-values.md)
- [../architecture.md](../architecture.md)
docs/global/coding-guidelines/argocd.md
+67
View File
@@ -0,0 +1,67 @@
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
# Coding Guideline — Argo CD interaction model
This repo holds **values** and **cached charts**. It does NOT own Argo CD `Application` or `ApplicationSet` manifests. Those live in the sister repo:
> [`github.com/Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config)
A merge to `main` here is a deploy event for every cluster whose Argo Application points at a path in this repo. The Argo `Application` defines the routing (which path on which cluster, sync policy, retry, prune); we define what that path renders to.
## Hard separation
| Concern | Owns it |
|---------|---------|
| `helm-templates/<chart>/` (cached/forked chart) | this repo |
| `helm-overrides/<cluster>/<app>/custom-values.yaml` (per-cluster values) | this repo |
| `manifests/storageclass/`, `manifests/priorityclass/<cluster>/` (singletons) | this repo |
| Argo `Application` (cluster, path, repoURL, targetRevision, destination namespace) | **sister repo** |
| Argo `ApplicationSet` (cluster generators, templating fan-out) | **sister repo** |
| `syncPolicy.automated.{prune,selfHeal}` decision | **sister repo** |
| `syncPolicy.syncOptions` (CreateNamespace, ServerSideApply) | **sister repo** |
| Sync waves / hooks via `argocd.argoproj.io/sync-wave` annotations | this repo (when expressed inside chart templates or raw sidecar manifests) |
## What this means for the agent
- **Never** add or edit a file matching `Application*.yaml` / `ApplicationSet*.yaml` here. If the task asks for one, redirect to the sister repo. See [escalation-matrix.md](../escalation-matrix.md) row 5.
- When introducing a **new app** to a cluster, the change is a **paired PR**: (a) a PR here adding `helm-overrides/<cluster>/<newapp>/custom-values.yaml`, and (b) a PR in the sister repo adding the matching `Application` manifest. Both must merge before the app deploys.
- When introducing a **new cluster**, the paired PR in the sister repo updates the `ApplicationSet` cluster generator. See [../platform/procedures/onboard-new-cluster.md](../../platform/procedures/onboard-new-cluster.md).
- When **removing** an app, deboard the Argo Application first (sister repo), let Argo prune, then remove the override directory here. See [../platform/procedures/deboard-app.md](../../platform/procedures/deboard-app.md).
## Sync policy: where the decision lives
`syncPolicy.automated.prune` and `syncPolicy.automated.selfHeal` live in the sister repo's `Application` spec. Convention on the platform:
- **Manual sync default** for infra components on prod clusters. Sync is a deliberate human click after a merge. Rationale documented in [`wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
- Auto-sync is reserved for low-risk leaf components (e.g., `kube-state-metrics`, monitoring agents) where reconciliation drift is benign.
Agents editing values here should assume **the sync click is the safety gate**. A merged PR is not yet deployed.
## Validating values before merge
The agent's responsibility is that the values **render** correctly. Argo CD will materialize the rendered output via `helm template`-equivalent server-side. Use the same command locally:
```
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml
```
If the chart has subchart dependencies (`Chart.yaml` `dependencies:`), run `helm dependency update helm-templates/<chart>` before templating, otherwise render will fail with `found in Chart.yaml, but missing in charts/ directory`.
For raw-manifest sidecars (`<extra>.yaml` files in the override dir), validate with `kubectl apply --dry-run=client -f <file>`. See [../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md).
## Common failure modes
| Symptom | First read |
|---------|-----------|
| `Sync` button click results in `OutOfSync` that won't resolve | [../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md) |
| Pods land but stay `Pending` | [../platform/runbooks/pod-pending-scheduling.md](../../platform/runbooks/pod-pending-scheduling.md) |
| Ingress 5xx after a Contour values change | [../platform/runbooks/ingress-down.md](../../platform/runbooks/ingress-down.md) |
## Cross-references
- Sister repo: [`Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config)
- [helm-values.md](./helm-values.md)
- [observability.md](./observability.md)
- [../escalation-matrix.md](../escalation-matrix.md)
- [`wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)
docs/global/coding-guidelines/helm-values.md
+167
View File
@@ -0,0 +1,167 @@
# Coding Guidelines — Helm values authoring
> The hard stops live in [../SANCTITY_RULES.md](../SANCTITY_RULES.md); the layer/scope rules in [../AGENT_BOUNDARIES.md](../AGENT_BOUNDARIES.md). This file is "how to write the values YAML well" — style, conventions, and the recurring footguns the linter doesn't catch.
---
## File-level conventions
### Where files go
| Path | Meaning |
|------|---------|
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | The primary Helm values file Argo's `valueFiles` references. **Default name. Do not invent alternatives.** |
| `helm-overrides/<cluster>/<app>/<extra>.yaml` | Sidecar raw manifests applied alongside the Helm release. Common shapes: `computeclass/<x>-cc.yaml`, `external-dns-services/<svc>.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`. |
| `helm-templates/<chart>/Chart.yaml` | Chart manifest; usually a thin wrapper declaring an upstream dep. |
| `helm-templates/<chart>/values.yaml` | The chart's own defaults — rarely edited; treat as upstream. |
| `helm-templates/<chart>/templates/` | Manifest templates — vanilla upstream unless intentionally forked. **Don't edit casually.** |
### Cluster directory naming
The cluster directory's name is a contract — it must match the Kubernetes cluster name registered in Argo CD. Conventions:
| Pattern | Use |
|---------|-----|
| `k8s-<bu>-prd-ase1` | Standard GKE prod cluster, BU-owned (AWS-style naming retained). |
| `k8s-<bu>-prd-ase1c` | GCP zone-c twin. |
| `k8s-shared-int-ase1` | Shared int (pre-prod) cluster. The only non-prod cluster. |
| `k8s-aurva-prd-ase1` | Aurva integration. |
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier clusters. Minimal override sets (typically `kube-state-metrics` + `victoria-metrics-agent`). |
| `k8s-supply-dev-ase1` | The lone dev/sandbox cluster. |
### App directory naming
Inside `helm-overrides/<cluster>/`, each subdirectory is one Argo Application = one Helm release.
- One Helm release per directory.
- Multiple Contour instances per cluster is the norm — `contour-external`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. **They are separate releases — don't merge them.**
- Versioned siblings (`argo-cd` ↔ `argo-cd-green`, `keda` ↔ `keda-2.17.1`) live as parallel directories under `helm-templates/`, but their per-cluster overrides typically live under one directory until the migration cuts over. See [blue-green-chart-migration](../../platform/procedures/blue-green-chart-migration.md).
---
## Field-level conventions (`custom-values.yaml`)
### Image references
- **Always pin images to Meesho's Artifact Registry mirror** for production:
```yaml
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/<image>
tag: <semver-or-sha>
```
- **Never use `:latest` or unpinned tags** in production overrides.
- **Never reference Docker Hub, Quay, GCR upstream, or ECR directly** in a production override. Mirror it via the platform team's image-pull workflow first.
### `fullnameOverride`
- **Set it once, never change it.** Service DNS, PVC binding, ConfigMap references, and downstream Argo Application names depend on stability. ([SANCTITY_RULES R9](../SANCTITY_RULES.md))
- For dataplane (`db-*`) clusters, the convention is `fullnameOverride: <kind>-dbc-<bu>-prd` (e.g. `kube-state-metrics-dbc-dsci-prd`).
- For BU clusters, omit `fullnameOverride` unless the chart's default name collides with another release in the same namespace.
### `nameOverride`
Almost never needed. Helm's `<release>-<chart>` naming is usually fine.
### Replica counts and HPA bounds
- Read the chart's defaults before specifying replicas. Some charts have HPA-managed replicas that conflict with `replicaCount`.
- For HPA-managed releases, set `minReplicas` and `maxReplicas` *and* leave `replicaCount` unset (or set to `null`).
- Don't lower `minReplicas` to zero unless the workload genuinely scales-from-zero.
### Resource requests and limits
- **Always set `resources.requests`** for production releases. Without requests, the scheduler treats the pod as best-effort.
- **Set `resources.limits`** unless the chart documentation explicitly recommends omitting them (some sidecars deliberately go limit-less).
- **Don't copy resources from another cluster.** Workload sizing is per-traffic-tier; the supply prd cluster's Contour requests are not the demand prd cluster's.
### `nodeSelector`, `tolerations`, `affinity`, `topologySpreadConstraints`
The single biggest source of silent mis-deploys. Per [SANCTITY_RULES R5](../SANCTITY_RULES.md):
| Cluster type | Key style | Example |
|--------------|-----------|---------|
| GKE Autopilot (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `cloud.google.com/compute-class` | `nodeSelector: {cloud.google.com/compute-class: contour-internal-0-cc}` |
| Standard GKE | `dedicated:` | `nodeSelector: {dedicated: contour-internal-0}` |
| Most others | `dedicated:` | same |
Cross-reference [`contour-nodeselector-tolerations-summary.md`](../../../contour-nodeselector-tolerations-summary.md) for the per-cluster Contour matrix. For non-Contour apps, copy from a sibling app on the *same* cluster, not from the same app on a *different* cluster.
### Probes
- Always set `livenessProbe` and `readinessProbe` for any long-running container.
- For workloads that take >30 s to warm (Jenkins, JFrog, ClickHouse), bump `initialDelaySeconds` accordingly — not the timeout, not the period.
- A `startupProbe` is the right tool for slow-warm containers; don't fight it with a 600s `initialDelaySeconds` on `livenessProbe`.
### Persistence
- StorageClass references go through the cluster-wide singletons in `manifests/storageclass/`: `pd-standard-retain-dr`, `sc-filestore-standard`, `sc-pd-ssd`, `sc-pd-standard`. **Never reference a StorageClass that doesn't exist in `manifests/storageclass/`.**
- For stateful releases, set `persistence.size` explicitly. Default sizes are rarely right.
- Never enable `persistence.enabled: true` without confirming the StorageClass and its retention/reclaim policy.
### Secrets
- **Never inline secret values** in `custom-values.yaml`. ([SANCTITY_RULES R4](../SANCTITY_RULES.md))
- Reference secrets by name: `existingSecret: <secret-name>`, where the secret is materialised by the per-cluster `external-secrets` app from GCP Secret Manager / Vault.
- Most clusters have an `external-secrets/` override directory; if your release needs a secret, the corresponding `ExternalSecret` lives there.
### Annotations and labels
- **Add labels conservatively.** Most charts already emit sensible label sets (`app.kubernetes.io/name`, etc.).
- For ingress (`Ingress`, `HTTPProxy`, Contour `Service`), `external-dns` annotations and AWS/GCP load-balancer annotations are normal — copy from a sibling on the same cluster.
- **Don't invent label keys.** If you find yourself adding `meesho.com/<something>`, double-check whether the project already has a convention for it.
---
## Field-level conventions (raw sidecar manifests)
For `helm-overrides/<cluster>/<app>/<extra>.yaml` files (no Helm templating, applied as-is):
- One Kubernetes resource per file unless they are tightly coupled.
- Use `apiVersion: v1` etc. — pin the API version explicitly.
- Set `metadata.namespace` (don't rely on the Argo Application's `destination.namespace` for these).
- For `ComputeClass` / `NodeClass` / `BackendConfig` / GKE-specific resources, sample a sibling cluster's existing file before authoring.
- For `external-dns-services/*.yaml`, the `Service` resource carries `external-dns.alpha.kubernetes.io/hostname` annotations — match the cluster's existing DNS pattern.
---
## YAML style
- **2-space indent. No tabs.**
- **Use single quotes for `'*'`** and other glob-like strings; bare strings elsewhere where unambiguous.
- **Trailing newline at EOF.**
- **No `---` document separators** unless you genuinely need multi-document YAML (rare in this repo).
- **Don't comment out fields; remove them.** The repo doesn't use commented-out scaffolding.
- **Preserve key order from siblings.** A reordered file is a noisy diff that drowns the real change.
- **Don't reformat unrelated YAML in passing.** ([SANCTITY_RULES R14](../SANCTITY_RULES.md))
---
## Diff hygiene
When opening a PR:
- **One change-type per PR.** Adding a service should not also "normalise labels on three other apps."
- **Keep diffs minimal.** Don't reformat surrounding YAML.
- **Cite the procedure followed** (link to one of `docs/platform/procedures/*.md`) in the PR description.
- **Show the validation you ran** — `helm template`, `yamllint`, sibling-file diff, the kubectl context you ran a `helm diff` against.
- **Pair the sister-repo PR** (`devops-infra-argo-config`) when adding a new app or cluster — link both.
---
## Common mistakes the hooks do **not** catch
These are the recurring footguns that pre-commit hooks won't flag:
1. **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster.** Pods stay `Pending`, or schedule on the wrong node pool.
2. **`fullnameOverride` modified.** Downstream Service DNS resolves to nothing.
3. **`spec.source.path` in the sister repo's `Application` not updated** to point at the new chart sibling after a blue-green migration.
4. **Image tag pinned to Docker Hub** or Quay instead of the GAR mirror.
5. **`replicaCount` set on an HPA-managed release.** HPA fights the static count.
6. **`persistence.storageClass` referencing a class that doesn't exist** on this cluster — PVC stays `Pending` forever.
7. **`existingSecret` referencing a secret the per-cluster `external-secrets` app doesn't create.** Pods crashloop on missing env.
8. **`Chart.yaml` `dependencies[].version` bumped without `helm dependency update`.** Argo CD will use the lockfile and silently render the old version.
9. **Edits inside `helm-templates/<chart>/templates/`** — silently fork the chart; clobbered on next upstream sync.
10. **Sidecar raw-manifest namespace mismatch** with the Helm release's namespace — orphaned resources.
The agent's job is to be the second pair of eyes on every one of these.
docs/global/coding-guidelines/observability.md
+78
View File
@@ -0,0 +1,78 @@
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
# Coding Guideline — Observability stack overrides
Conventions for editing values for the observability charts vendored in this repo:
- `victoria-metrics-cluster` and `victoria-metrics-cluster-latest`
- `victoria-metrics-agent` and `victoria-metrics-agent-latest`
- `vmalert`, `victoria-metrics-operator`
- `mimir-distributed`
- `loki`, `loki-distributed`
- `tempo`, `tempo-distributed`
- `grafana`
- `opentelemetry-collector` and `opentelemetry-collector-latest`
- `kube-state-metrics`, `node-exporter`, `metrics-server`
- `kube-prometheus-stack` (Prometheus alert rules)
The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as *data-pipeline* changes, not static config.
## Versioned siblings
`*-latest` siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See [`wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [../platform/procedures/blue-green-chart-migration.md](../../platform/procedures/blue-green-chart-migration.md).
## Cardinality discipline
Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:
- New `extraLabels` / `externalLabels` on `victoria-metrics-agent`.
- New `relabel_configs` that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
- New scrape targets in `additionalScrapeConfigs`.
If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.
## PromQL / alert-rule validation
Alert rules live in `kube-prometheus-stack`, `vmalert`, and Mimir ruler config. Before merging:
1. Render with `helm template`:
```
helm template prom helm-templates/kube-prometheus-stack \
-f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml
```
2. Extract the `PrometheusRule` objects and validate PromQL with `promtool check rules <file.yaml>` (Prometheus tooling) or `vmalert -dryRun -rule=<file.yaml>` for vmalert-specific rules.
3. Confirm the rule's `for:` window and `severity` label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.
## Retention and tenancy
- VictoriaMetrics `vmstorage.retentionPeriod` is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC `resources.requests.storage`.
- Loki and Mimir multi-tenancy is keyed on the `X-Scope-OrgID` header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
- Tempo trace retention is set via `compactor.compaction.block_retention`. Default is short (24h–48h); long retention is opt-in per cluster.
## Grafana dashboards & datasources
- Datasources are declared in the cluster's `grafana/custom-values.yaml` under `datasources.datasources.yaml`. Pin URLs to in-cluster Service DNS, not external endpoints.
- Dashboards bundled via `dashboardProviders` reference ConfigMaps; uniqueness of `uid` matters for panel-image links and alert dashboards.
## Validation checklist (always)
Before raising a PR that touches an observability chart:
- [ ] `helm template` renders without error.
- [ ] PromQL in any new alert rule passes `promtool check rules`.
- [ ] No new high-cardinality label is added without a bound.
- [ ] Retention / PVC sizing has not been changed silently.
- [ ] If touching a `*-latest` sibling, the matching Argo Application points at it. See [argocd.md](./argocd.md).
## Schema references
- Per-cluster scheduling fields (must be rewritten, never copy-pasted): [../../platform/schemas/custom-values-schema.md](../../platform/schemas/custom-values-schema.md)
- Raw manifest sidecars (e.g., `external-dns-services/*.yaml` for Grafana ingress): [../../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md)
- Storage backing for `vmstorage`, `loki` chunks, `tempo` blocks: [../../platform/schemas/storageclass-priorityclass-schema.md](../../platform/schemas/storageclass-priorityclass-schema.md)
## See also
- [helm-values.md](./helm-values.md) — values-file conventions
- [argocd.md](./argocd.md) — Argo CD interaction model
- [../escalation-matrix.md](../escalation-matrix.md)
- [../../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md)
docs/global/escalation-matrix.md
+39
View File
@@ -0,0 +1,39 @@
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
# Escalation Matrix
Use this table when a task crosses a Sanctity/Boundary line, when blast radius exceeds the agent's authority, or when a request belongs in a different repo. Always escalate **before** writing the change, not after.
Primary owner: **siddharth.pal@meesho.com**
Secondary owner: **samarth.nag@meesho.com**
Owner data is sourced from the repo-root [`repository.yaml`](../../repository.yaml). If owners change, update that file via the `registry-bootstrap` flow; do not edit `repository.yaml` by hand. See [SANCTITY_RULES.md](./SANCTITY_RULES.md) for the don't-touch list and [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md) for layer classification.
## Situation → owner → channel
| # | Situation | Who | Channel | Notes |
|---|-----------|-----|---------|-------|
| 1 | Edit suspected to silently fork an upstream chart (touching `helm-templates/<chart>/templates/` or `values.yaml` of a vanilla-pulled chart) | siddharth.pal | Slack `#devops-infra` + PR review | Document the intentional fork in the chart's `README.md`. See [procedures/fork-upstream-chart.md](../platform/procedures/fork-upstream-chart.md). |
| 2 | Cross-cluster cleanup or "normalization" requested in the same PR as a feature change | siddharth.pal | Slack `#devops-infra` | Refuse in-PR; request a separate cleanup PR. See [SANCTITY_RULES.md](./SANCTITY_RULES.md) §surgical-edits. |
| 3 | TruffleHog flagged a real secret in a staged commit | siddharth.pal + secret owner (BU on-call) | Slack `#sec-incidents` (private) + revoke pipeline | NEVER bypass with `--no-verify`. Rotate the credential, then move it to External Secrets Operator. |
| 4 | Edit to `repository.yaml` requested | siddharth.pal | Slack `#devops-infra` | This file is owned by `registry-bootstrap` automation. Refuse and redirect to that pipeline. |
| 5 | Edit to an Argo CD `Application` / `ApplicationSet` manifest requested | siddharth.pal | PR against sister repo `Meesho/devops-infra-argo-config` | This repo does not own routing manifests. See [coding-guidelines/argocd.md](./coding-guidelines/argocd.md). |
| 6 | Schedule fields (`nodeSelector`, `tolerations`, `computeClass`) copy-pasted from one cluster's override to another without rewriting | siddharth.pal | PR review block | Re-derive from per-cluster node-pool topology. See [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) and [runbooks/pod-pending-scheduling.md](../platform/runbooks/pod-pending-scheduling.md). |
| 7 | Chart dep version bump where upstream changelog flags breaking template changes | siddharth.pal | Slack `#devops-infra` + PR review | Run `helm dependency update`, refresh `Chart.lock`, dry-run `helm template` against every consumer cluster. See [procedures/update-chart-version.md](../platform/procedures/update-chart-version.md). |
| 8 | Vault HA write-path failure (seal status, Raft peer loss, unsealed standby) | siddharth.pal + platform on-call | Slack `#sec-incidents` + PagerDuty | Production secret-store outage. Do not modify Vault overrides without on-call ack. |
| 9 | Edit to `manifests/storageclass/*.yaml` or `manifests/priorityclass/<cluster>/*.yaml` | siddharth.pal + cluster BU owner | PR review (two reviewers) | Cluster-wide singleton; affects every PVC / scheduling priority. See [schemas/storageclass-priorityclass-schema.md](../platform/schemas/storageclass-priorityclass-schema.md). |
| 10 | Deletion of a versioned-sibling chart (`-green`, `-vX.Y.Z`, `-latest`, `-old`) | siddharth.pal | Slack `#devops-infra` + grep sister repo | Confirm zero references in `Meesho/devops-infra-argo-config` before deletion. See [analyses/ADR-A2-blue-green-sibling-pattern.md](../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md). |
## Refusal language
When refusing, state:
1. The Sanctity rule or Layer constraint that's tripped.
2. Which owner to ping (from the table above).
3. The repo or pipeline the request actually belongs in (sister repo, `registry-bootstrap`, Vault on-call, etc.).
## See also
- [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)
- [SANCTITY_RULES.md](./SANCTITY_RULES.md)
- [agent-operations-guide.md](./agent-operations-guide.md)
- Repo-root `CLAUDE.md` NEVER-DO list
docs/platform/procedures/add-contour-route.md
+231
View File
@@ -0,0 +1,231 @@
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
# Procedure — Add or modify a Contour HTTPProxy route
> **Layer:** Layer 1 — values diff + PR.
> **Blast radius:** one Contour release × one cluster (a misrouted host can break ingress for the whole BU).
> **Approval:** consuming-app owner + cluster owner. Platform team if the change touches `contour-external*`.
This procedure covers HTTPProxy route changes that flow through one of the cluster's Contour releases. Most clusters run **multiple Contour instances** — read the [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) (repo root) authoritative scheduling matrix before editing **any** Contour values.
---
## Multi-Contour topology
| Release name | Plane | Typical purpose |
|--------------|-------|-----------------|
| `contour-external` | North-south | Public / external traffic (terminates at GCP external LB) |
| `contour-external-1` | North-south | Second external Contour (blue-green or capacity split) |
| `contour-internal-0` | East-west | Internal traffic, plane 0 |
| `contour-internal-1` | East-west | Internal traffic, plane 1 |
| `contour-internal-intra-0` | Intra-cluster | Cluster-local east-west, plane 0 |
| `contour-internal-intra-1` | Intra-cluster | Cluster-local east-west, plane 1 |
Each is a separate Helm release, on a separate node pool, with its own `nodeSelector` / `tolerations` / `computeClass`. Picking the wrong release is the most common authoring mistake.
---
## When to use
- Adding a new HTTPProxy / route for a service.
- Changing TLS, retry, timeout, or rate-limit policy on an existing route.
- Re-pointing an HTTPProxy at a different upstream Service.
- Adding a new host to an existing route's `virtualhost.fqdn`.
Do **not** use this procedure for:
- Changing Contour itself (sizing, scheduling, image) — that's [modify-observability-config.md](modify-observability-config.md)-style infra editing on the Contour release.
- Bumping Contour's chart version — use [update-chart-version.md](update-chart-version.md) and consult the versioned sibling (`contour-v1.33.3`).
- Curling production hostnames to test — [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md) forbids it. Test from inside the cluster with a `curl` Pod.
---
## Inputs
| Input | Example |
|-------|---------|
| Target cluster | `k8s-supply-prd-ase1` |
| Target Contour release | `contour-internal-0` |
| HTTPProxy host | `api-foo.internal.meeshogcp.in` |
| Upstream Service | `foo-svc.foo-ns:8080` |
| TLS source | `cert-manager` / `external-secret` / `none` |
| Path prefixes | `/v1`, `/health` |
| Approval ticket | CMR-… (if BU policy) |
---
## Pre-conditions
- [ ] You know which Contour release is the right one for this host (north-south = `external*`; east-west between BUs = `internal-{0,1}`; cluster-local = `internal-intra-{0,1}`). When in doubt, sample existing HTTPProxies in the same namespace.
- [ ] The upstream Service exists or will exist by the time of Sync.
- [ ] The DNS name follows the cluster's `external-dns` pattern.
- [ ] The TLS source (Secret, Issuer, etc.) exists on the cluster.
---
## Steps
### 1. Identify the right Contour release
```bash
ls helm-overrides/<cluster>/ | grep '^contour'
```
Pick the release whose plane matches the new route's traffic class. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). If unsure, look at existing HTTPProxies on the cluster:
```bash
# Read existing routes already shipped via this repo
yq e '.. | select(has("httpproxies"))' \
helm-overrides/<cluster>/contour-internal-0/custom-values.yaml
```
### 2. Decide where the HTTPProxy lives
Two patterns exist:
| Pattern | When | Where to author |
|---------|------|-----------------|
| **Inline in Contour values** | Routes shared by the cluster's infra layer (e.g. Grafana, Argo CD). | `helm-overrides/<cluster>/<contour-release>/custom-values.yaml` under `httpproxies:` (if the chart supports it) or as a sidecar manifest in the same dir. |
| **In the consuming app's repo** | Routes for a specific service. | The service's own deployment artifacts. **Out of scope for this repo.** |
If the route is service-owned, **redirect** to the consuming team's repo and stop. This procedure only covers infra-layer HTTPProxies.
### 3. Author the HTTPProxy
Skeleton (sidecar manifest pattern):
```yaml
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
name: <name>
namespace: <ns>
spec:
virtualhost:
fqdn: <fqdn>
tls:
secretName: <tls-secret> # cert-manager-managed Secret
routes:
- conditions:
- prefix: /
services:
- name: <upstream-service>
port: <port>
timeoutPolicy:
response: 30s
retryPolicy:
count: 2
retryOn: 5xx
```
Drop into `helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml` (sidecar manifest) — see [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
### 4. Lint the HTTPProxy
```bash
# Schema check (kubectl --dry-run against the cluster's CRD)
kubectl --context=<ctx> --dry-run=server -f helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml apply
# Or local: validate against the projectcontour CRD schema
kubeconform -schema-location default -schema-location \
'https://raw.githubusercontent.com/projectcontour/contour/main/examples/contour/01-crds.yaml' \
helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml
```
### 5. Render the chart
```bash
helm template <contour-release> helm-templates/<contour-chart> \
-f helm-overrides/<cluster>/<contour-release>/custom-values.yaml > /tmp/contour.yaml
```
The render must succeed; the HTTPProxy sidecar is applied alongside, not through Helm — but rendering catches values-side errors that would block sync of the same Argo Application.
### 6. Test from inside the cluster (post-merge, not pre-merge)
**Do not** curl `<fqdn>.meeshogcp.in` from your laptop / a build agent. Run a curl Pod on-cluster:
```bash
kubectl --context=<ctx> run -it --rm curl-test \
--image=curlimages/curl --restart=Never -- \
curl -v -H "Host: <fqdn>" http://<contour-svc>.projectcontour.svc.cluster.local
```
### 7. Open the PR
```bash
git checkout -b contour/<cluster>-<route-name>
git add helm-overrides/<cluster>/<contour-release>/
git commit -m "contour(<cluster>/<contour-release>): add route <name>"
git push origin contour/<cluster>-<route-name>
gh pr create --base main
```
### PR description template
```markdown
## Summary
Adds (or modifies) HTTPProxy `<name>` on `<cluster>` via `<contour-release>` for FQDN `<fqdn>`.
## Why
<1-2 sentences>
## Topology
- Cluster: `<cluster>`
- Contour release: `<contour-release>` (plane: external / internal / intra)
- FQDN: `<fqdn>`
- Upstream Service: `<svc>:<port>` in namespace `<ns>`
- TLS: cert-manager Secret `<tls-secret>`
## Validation
- [ ] HTTPProxy CRD schema validation passed (kubeconform / kubectl --dry-run)
- [ ] `helm template` rendered cleanly
- [ ] On-cluster curl from a curl Pod returns expected status
- [ ] DNS / external-dns plumbed (existing wildcard or new external-dns Service)
## Approvers
- App owner: <handle>
- Cluster owner: <handle>
- Platform (if `contour-external*`): <handle>
```
### 8. After merge — Sync
Open the cluster's Argo CD UI, find the Contour release's Application, **Sync**. Verify:
```bash
kubectl --context=<ctx> -n projectcontour get httpproxy <name>
kubectl --context=<ctx> -n projectcontour describe httpproxy <name> | grep -A5 'Status:'
```
A `Valid: true` status means Contour accepted the route. `Valid: false` with a reason → fix the values and re-PR.
If the HTTPProxy is `Valid` but traffic still 5xx → [../runbooks/ingress-down.md](../runbooks/ingress-down.md) §4.
---
## Anti-patterns
1. **Wrong Contour release.** A route mounted on `contour-internal-intra-0` is unreachable from outside the cluster. Cross-check the matrix.
2. **Curling the production FQDN** from a developer machine to "test." Forbidden — see [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md).
3. **Hand-edited values for a single host across all Contour releases** — pick one release, justify it.
4. **`tls.passthrough` for plain HTTP services** — silent: TLS terminates upstream, doesn't.
5. **Wildcard hosts that overlap an existing HTTPProxy** — Contour status will mark one of them invalid; check before merging.
6. **Editing `helm-templates/contour*/`** to "tweak the chart." Layer 1 forbids casual chart edits — the chart is vanilla upstream.
---
## Rollback
- Revert the values PR.
- Sync the Contour Application — Argo will prune the HTTPProxy or restore the prior values.
---
## Related
- Reference (authoritative scheduling): [../../../contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- Runbook: [../runbooks/ingress-down.md](../runbooks/ingress-down.md).
- Schema: [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
- ADR: [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
docs/platform/procedures/blue-green-chart-migration.md
+206
View File
@@ -0,0 +1,206 @@
# Procedure — Blue-green chart migration (versioned siblings)
> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** the chart upgrade ships in a new sibling directory; no cluster cuts over until its `Application` is repointed. Each cluster's cutover is its own decision.
> **Approval:** platform team — multi-reviewer.
This procedure handles the case where you can't safely bump a chart's pinned `dependencies[].version` in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as **sibling chart directories** until every consuming cluster has migrated.
The repo already shows this pattern at work:
| Original | Migration target |
|----------|------------------|
| `argo-cd` | `argo-cd-green` |
| `contour` | `contour-v1.33.3` |
| `keda` | `keda-2.17.1` |
| `opentelemetry-collector` | `opentelemetry-collector-latest` |
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` |
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` |
| `sonarqube` | `sonarqube-old` *(reverse — `sonarqube` is the new one; `-old` retained for rollback)* |
See [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) for the rationale.
---
## When to use this procedure
- A major-version bump with breaking template changes.
- A bump with an immutable-field change (Deployment `spec.selector`, StatefulSet `volumeClaimTemplates`).
- A migration that needs cluster-by-cluster cutover with rollback windows.
This is **not** the right procedure for:
- A patch / minor bump that's a clean drop-in → use [update-chart-version.md](update-chart-version.md).
- Forking templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
---
## Pre-conditions
- [ ] Platform team agrees a blue-green migration is required (not just a bump).
- [ ] You've identified the variant naming convention (`-green`, `-vX.Y.Z`, `-latest`, etc.).
- [ ] You've identified every consuming cluster: `grep -rl '<chart>' helm-overrides`.
- [ ] CMR open.
---
## Steps
### 1. Create the sibling chart directory
```bash
cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
cd helm-templates/<chart>-<variant>
```
Update `Chart.yaml`:
```diff
apiVersion: v2
-name: <chart>
+name: <chart>-<variant>
…
dependencies:
- name: <sub>
- version: <old-version>
+ version: <new-version>
repository: <upstream-repo>
```
```bash
helm dependency update
```
### 2. Render against a representative cluster's overrides — old variant
The point of blue-green is *no surprises*. Render the **old** chart with each consuming cluster's override and capture the output.
```bash
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
cluster=$(echo "$f" | awk -F/ '{print $2}')
helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
done
```
### 3. Render against the same overrides — new sibling variant
```bash
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
cluster=$(echo "$f" | awk -F/ '{print $2}')
helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
done
```
### 4. Diff old → new per cluster
```bash
for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
echo "=== $cluster ==="
diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
done
```
For each cluster, decide:
- Is the diff what you expected?
- Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
- Is the cutover safe to do without the workload owner present? (If no → schedule.)
### 5. Open PR-1: introduce the sibling chart
```bash
git checkout -b migrate/<chart>-to-<variant>-introduce
git add helm-templates/<chart>-<variant>/
git commit
git push origin migrate/<chart>-to-<variant>-introduce
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"
```
After merge, the new chart exists in the repo but **no cluster uses it yet** — the existing Argo `Application`s still point at `helm-templates/<chart>`.
### 6. Per-cluster cutover (one PR pair per cluster)
For each consuming cluster:
a. **Update the cluster's `custom-values.yaml`** if the new chart needs different values. Open as a values-side PR (this repo).
b. **Update the sister-repo `Application`** to repoint:
```diff
spec:
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
targetRevision: main
- path: helm-templates/<chart>
+ path: helm-templates/<chart>-<variant>
```
c. **After both merge**, click Sync in the cluster's Argo CD UI.
d. **Soak** — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.
### 7. Open PR-N: retire the old sibling
After every cluster has cut over and soaked:
```bash
git checkout -b migrate/<chart>-retire-old
git rm -r helm-templates/<chart>
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
git commit
git push origin migrate/<chart>-retire-old
gh pr create --base main --title "migrate: retire old <chart> sibling"
```
PR description:
- Confirmation every cluster has cut over (`grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config` returns nothing).
- Confirmation soak period elapsed.
- Decision: delete vs rename to `-old` (kept for rollback).
---
## Why two PRs at the start, then per-cluster pairs, then a final retirement
| PR | Effect |
|----|--------|
| **PR-1: introduce sibling** | Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them. |
| **PR-2..N-1: per-cluster cutover (paired with sister repo)** | One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart. |
| **PR-N: retire old sibling** | Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible. |
Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.
---
## Anti-patterns
1. **Cutting over multiple clusters in one PR.** Bundle = no per-cluster rollback.
2. **Deleting the old sibling before every cluster has cut over.** Cluster N+1 has its `Application` pointing at a path that no longer exists; sync fails immediately.
3. **Cutting over without rendering first.** Surprises after merge.
4. **Skipping the soak period.** "It looked fine in the first 5 minutes" is not soak.
5. **Renaming the *new* sibling to drop the suffix** (e.g. `argo-cd-green` → `argo-cd`) before the old chart is retired. The path collision will break Argo CD's caching.
---
## Rollback (per cluster)
If a cluster's cutover fails:
1. Revert the sister-repo PR for that cluster.
2. Click Sync in the cluster's Argo CD — the `Application` re-renders against `<chart>` (the old sibling). The rollback is one cluster only.
3. Investigate; iterate.
If the new sibling has a fundamental problem affecting every cluster:
1. Open PR-X: revert PR-1 (delete the sibling).
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
3. Schedule a postmortem before re-attempting.
---
## Related
- Procedure: [update-chart-version.md](update-chart-version.md).
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md).
- ADR: [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
- [SANCTITY_RULES R8](../../global/SANCTITY_RULES.md).
docs/platform/procedures/deboard-app.md
+119
View File
@@ -0,0 +1,119 @@
# Procedure — Deboard a retired app from a cluster
> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
> **Approval:** app owner + cluster owner. CMR mandatory.
This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.
---
## Pre-conditions (block deboarding if any are FALSE)
- [ ] App is genuinely retired on this cluster — confirmed by app owner.
- [ ] No traffic / scrape job / dependency is still hitting it.
- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
- [ ] CMR approved.
```bash
# Where does this app exist today?
find helm-overrides -maxdepth 2 -type d -name '<app>'
```
---
## Steps
### 1. Decide the scope
| Scope | Then |
|-------|------|
| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |
### 2. Open the sister-repo `Application` removal PR FIRST
In `github.com/Meesho/devops-infra-argo-config`:
- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
- Merge.
- The cluster's Argo CD will mark the `Application` for removal on next reconcile.
This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.
### 3. Manually clean up the workload (if `automated.prune` was not set)
For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:
```bash
kubectl --context=<cluster> delete <kind>/<name> -n <ns>
# OR, if the whole namespace is dedicated to this release:
kubectl --context=<cluster> delete namespace <ns>
```
Coordinate with the app owner — wholesale namespace deletion is irreversible.
### 4. Open the values-side removal PR
```bash
git checkout -b deboard/<app>-from-<cluster>
git rm -r helm-overrides/<cluster>/<app>/
git commit
git push origin deboard/<app>-from-<cluster>
gh pr create --base main --title "deboard: <app> from <cluster>"
```
PR description:
- Procedure followed: this file.
- Confirmation pre-conditions are TRUE.
- Sister-repo PR (already merged).
- CMR ticket reference.
- Confirmation the workload was manually cleaned up (or scheduled).
### 5. (Optional) If this was the last cluster running the app
If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:
- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).
---
## "What if the app might come back?"
If retirement is provisional:
- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.
---
## Anti-patterns
1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.
---
## Rollback
If you deboarded by mistake:
1. Revert the values-side PR (`git revert <merge-sha>`).
2. Revert the sister-repo PR.
3. Click Sync in the cluster's Argo CD UI.
4. The `Application` is recreated and renders the chart.
5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).
---
## Related
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.
docs/platform/procedures/fork-upstream-chart.md
+158
View File
@@ -0,0 +1,158 @@
# Procedure — Intentionally fork an upstream chart
> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** every cluster that consumes this chart will render against the forked templates on next sync.
> **Approval:** platform team — multi-reviewer.
Most charts in `helm-templates/` are **vanilla upstream**, pulled via `helm pull`. The `Chart.yaml` is a thin wrapper declaring the upstream chart as a dependency; the actual templates come from the subchart in `charts/`. Editing a `templates/*.yaml` *under the wrapper* silently forks the chart, and the fork is clobbered the next time someone runs `helm dependency update`.
This procedure is the supported path when a fork is **intentional**.
---
## When to use this procedure
- Upstream chart lacks a feature you need (e.g. a values key that doesn't exist).
- Upstream behaviour conflicts with Meesho's environment (e.g. probe path doesn't work behind Contour).
- Upstream chart has a bug awaiting an upstream fix.
This is **not** the right procedure for:
- Adding a values knob — file a PR upstream, or override behaviour through existing knobs.
- Pinning to an old version — use [update-chart-version.md](update-chart-version.md) (or just don't bump).
- Style preferences — leave the upstream chart alone.
---
## Pre-conditions
- [ ] You've confirmed the desired behaviour cannot be achieved via the chart's existing values.
- [ ] You've checked whether an upstream PR / issue already covers this.
- [ ] Platform team agrees the fork is justified.
- [ ] The fork's rationale will be documented in the chart's `README.md`.
---
## Steps
### 1. Vendor the chart's templates
If the chart is currently a thin wrapper (templates come from a subchart in `charts/<sub>`), you must first promote the subchart's templates into the wrapper.
```bash
cd helm-templates/<chart>
ls charts/ # find the subchart .tgz
helm dependency update # ensure it's resolved
# Extract templates from the subchart .tgz
tar -xzf charts/<sub>-<ver>.tgz -C /tmp/
cp -R /tmp/<sub>/templates ./templates
cp -R /tmp/<sub>/values.yaml ./values.yaml.upstream
```
Now the wrapper has its own `templates/` — Helm will use those instead of the subchart's.
### 2. Edit `Chart.yaml` to reflect the fork
Remove the dependency (since the templates are now local), and bump `version:` (the chart's own version, not the subchart's):
```diff
apiVersion: v2
name: <chart>
-version: 0.1.0
+version: 0.1.0+fork.1
description: <description> — Meesho-forked from upstream <chart> <upstream-version>
-dependencies:
- - name: <sub>
- version: <upstream-version>
- repository: <upstream-repo>
```
### 3. Make the fork edits
Edit `templates/*.yaml` or `values.yaml` to apply the fix. Keep the diff minimal — every line away from upstream is technical debt.
### 4. Document the fork
Edit `helm-templates/<chart>/README.md` (create if missing). Use this template:
```markdown
# <chart>
**Forked from upstream <chart> <upstream-version>** at <date>.
## Why
<one-paragraph rationale — what behaviour the fork changes and why we couldn't achieve
it through values alone>
## What's changed
- `templates/<file>.yaml` — <one-line diff summary>
- `values.yaml` — <one-line diff summary>
## Upstream tracking
- Upstream PR: <link, if you've sent one>
- Upstream issue: <link>
- When upstream merges: revert this fork via [unfork procedure].
```
### 5. Render and diff
```bash
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
# Spot-check helm diff against a live cluster
helm diff upgrade <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml --kube-context=<context>
```
### 6. Commit and open the PR
```bash
git checkout -b fork/<chart>-<purpose>
git add helm-templates/<chart>/
git commit
git push origin fork/<chart>-<purpose>
gh pr create --base main --title "fork: <chart> — <one-line purpose>"
```
PR description must include:
- Procedure followed: this file.
- Why the fork is necessary (link to upstream issue/PR if any).
- What templates / values are changed and why.
- Platform team approver(s).
- Plan for unforking when upstream lands the fix.
---
## Anti-patterns
1. **Editing `templates/` without first vendoring** (templates from a subchart). Your edits live in `helm-templates/<chart>/templates/` but Helm renders from `charts/<sub>/templates/` — the edits do nothing, then get clobbered.
2. **Forking a chart and not documenting why.** Six months later nobody remembers; the fork looks like accidental drift.
3. **Forking instead of overriding via values.** Always check the chart's existing values surface first.
4. **Forking and then bumping the upstream version.** A bump runs `helm dependency update`, which can clobber the fork. The fork must be re-applied or the bump must explicitly re-vendor.
5. **Sweeping cleanup of upstream code** alongside the fork edit. The diff should be exactly the change you intended; everything else is upstream.
---
## Rollback / unforking
When upstream lands the fix and you want to return to vanilla:
1. Restore `Chart.yaml` to declare the upstream as a dependency (with the new upstream version that includes the fix).
2. Delete the local `templates/` and `values.yaml` (or rename them as `.upstream` for reference).
3. Run `helm dependency update`.
4. Update `helm-templates/<chart>/README.md` to remove the "forked" status.
5. Render against a sibling override and confirm the resulting manifests match what the fork was producing.
---
## Related
- Procedure: [update-chart-version.md](update-chart-version.md).
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
- [SANCTITY_RULES R7](../../global/SANCTITY_RULES.md).
docs/platform/procedures/modify-alert-rules.md
+227
View File
@@ -0,0 +1,227 @@
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
# Procedure — Modify Prometheus / VictoriaMetrics alert rules
> **Layer:** Layer 1 — values diff + PR.
> **Blast radius:** PagerDuty / on-call paging across the cluster's tenants. A wrong threshold pages everyone; a missing rule masks an outage.
> **Approval:** observability owner + on-call lead for the affected severity.
This procedure is the narrow alert-rule slice of [modify-observability-config.md](modify-observability-config.md). Use this when the entire intent of the PR is alert-rule editing — adding, removing, retuning thresholds, changing `for:` windows, or rewriting `expr:` PromQL.
---
## Where alert rules live
| File | Engine | Notes |
|------|--------|-------|
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert | Most clusters use vmalert against VM cluster as the primary alerting engine. Rules typically under `config.rules` or rendered as a `VMRule` CRD. |
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmalert (bundled) / VM ruler | Some clusters carry rules in the cluster chart's `vmalert.config` block. |
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` | Prometheus / `kube-prometheus-stack` | Rules under `additionalPrometheusRulesMap` or `serverFiles.alerting_rules.yml`. |
| `helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml` | Prometheus | Same as above; older clusters use this chart name. |
| `helm-overrides/<cluster>/mimir/custom-values.yaml` | Mimir ruler | Rules go through the ruler API; YAML in `runtimeConfig` or via a sidecar `ConfigMap`. |
Confirm which cluster runs which engine before editing — sample existing rules in the override.
---
## When to use
- Adding/removing one or more alert rules in the cluster's primary alerting engine.
- Tuning `expr:` PromQL or `for:` windows on existing rules.
- Re-routing alerts (changing `severity`, `team`, or other routing labels).
- Adding/removing recording rules for downstream alert efficiency.
Do **not** use this procedure for:
- Touching scrape configs, retention, datasources, or sizing in the same PR — split it. See [modify-observability-config.md](modify-observability-config.md).
- Bumping the alerting chart's version — see [update-chart-version.md](update-chart-version.md).
- Writing app-specific alert rules that belong in the consuming service's own values — those go in the service's repo.
---
## Inputs
| Input | Example |
|-------|---------|
| Target cluster | `k8s-supply-prd-ase1` |
| Target chart | `vmalert` / `prometheus` / `kube-prometheus-stack` / `mimir` |
| Rule name(s) added/changed | `HighErrorRate`, `KafkaConsumerLag` |
| `expr:` PromQL | `sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 0.05` |
| `for:` window | `5m` |
| Severity / routing labels | `severity: critical`, `team: supply-platform` |
| Approval ticket | CMR-… |
---
## Pre-conditions
- [ ] You have `promtool` and/or `vmalert` available locally.
- [ ] You know the cluster's Alertmanager / vmalert notifier routing (which `severity` / `team` label routes where).
- [ ] You have run the new PromQL expression against the cluster's VM/Prometheus to confirm it returns sensible values *before* writing the rule — open Grafana Explore.
- [ ] You have on-call sign-off if this rule pages on-call.
---
## Steps
### 1. Locate the rules block
```bash
yq e '.config.rules // .additionalPrometheusRulesMap // .serverFiles' \
helm-overrides/<cluster>/<chart>/custom-values.yaml
```
Identify the YAML path the engine expects:
- **vmalert**: `config.groups[].rules[]`.
- **kube-prometheus-stack**: `additionalPrometheusRulesMap.<group-name>.groups[].rules[]`.
- **prometheus**: `serverFiles.alerting_rules.yml.groups[].rules[]`.
- **mimir ruler**: per-tenant config — confirm the cluster's tenancy setup.
### 2. Author the rule
```yaml
- alert: <RuleName> # PascalCase, no spaces; surfaces in pages and dashboards
expr: |
<PromQL expression>
for: <duration> # e.g. 5m, 10m
labels:
severity: <warning|critical|info>
team: <owning-team> # routes to the right PagerDuty service
annotations:
summary: "<one-line>"
description: "<longer text, can reference $labels.* and $value>"
runbook_url: <link to runbook>
```
#### Threshold discipline
- Express thresholds as `rate(...) > 0.05` rather than `> 5%`. Ensure units in the expression match the units in `summary` / `description`.
- Avoid `for:` windows shorter than the scrape interval × 2. Sub-`1m` windows are flap factories.
- Avoid alerting on absolute counts (`sum(http_requests_total) > 1000`); prefer rates / ratios.
- For SLO-style alerts, use multi-window multi-burn-rate (Google SRE workbook). Single-window threshold alerts are a known anti-pattern.
#### Routing implications
Changing `severity` from `warning` to `critical` (or vice versa) **silently re-routes the page** — the rule may now wake on-call where it previously emailed (or vice versa). Same for `team:` — a typo here drops alerts into the wrong PagerDuty service. Cross-check the cluster's Alertmanager / vmalert routing config:
```bash
yq e '.config.route // .alertmanagerConfig' \
helm-overrides/<cluster>/<alertmanager-chart>/custom-values.yaml
```
### 3. Render the chart
```bash
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
```
### 4. Validate PromQL
For Prometheus / `kube-prometheus-stack`:
```bash
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
promtool check rules /tmp/rules.yaml
```
For vmalert:
```bash
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
vmalert -dryRun -rule=/tmp/vmrules.yaml
```
For Mimir ruler — `mimirtool rules check` against the rendered ruler config.
If validation fails: fix the PromQL. **Do not** merge a rule that fails parse.
### 5. Sanity-check expressions in Grafana Explore
In a non-production manner: open Grafana on the cluster, paste the `expr:` into Explore, run over the last 6h. Confirm:
- The series returned makes sense for the rule.
- The threshold isn't always-firing or never-firing in normal operation.
- Empty result is allowed (`absent_over_time(...)` style).
### 6. Open the PR
```bash
git checkout -b alerts/<cluster>-<rule-name>
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
git commit -m "alerts(<cluster>/<chart>): <add|tune|remove> <RuleName>"
git push origin alerts/<cluster>-<rule-name>
gh pr create --base main
```
### PR description template
```markdown
## Summary
<add|tune|remove> alert rule(s) on `<cluster>` × `<chart>`.
## Rules touched
| Rule | Change | New expr | New `for:` | Severity | Team |
|------|--------|----------|-----------|----------|------|
| `<RuleName>` | added | `<expr>` | `5m` | `critical` | `supply-platform` |
## Validation
- [ ] `promtool check rules` / `vmalert -dryRun` passed
- [ ] PromQL sanity-checked in Grafana Explore over last 6h
- [ ] Threshold appropriate (no flap risk, no always-firing)
- [ ] Routing labels (`severity`, `team`) verified against Alertmanager/vmalert routes
- [ ] Runbook URL added (`annotations.runbook_url`)
## Approvers
- Observability owner: <handle>
- On-call lead (severity=<x>): <handle>
```
### 7. After merge — Sync and observe
Sync the Argo Application. Watch:
```bash
# vmalert
kubectl --context=<ctx> -n monitoring logs deploy/vmalert | grep -i "<RuleName>\|error"
# kube-prometheus-stack
kubectl --context=<ctx> -n monitoring logs prometheus-k8s-0 | grep -i "<RuleName>\|error"
```
Open the alerting engine's UI (vmalert UI / Prometheus `/alerts`) to confirm the rule loaded.
### Watch for false positives
Stay on the alert for at least one full `for:` window after sync. If the rule pages immediately and was not expected to → revert.
---
## Anti-patterns
1. **Sub-1m `for:` windows** — flap-prone.
2. **Alerting on absolute counts** instead of rates/ratios.
3. **`severity` flips without on-call sign-off** — silent re-route.
4. **Missing `runbook_url`** — pages without remediation steps waste on-call cycles.
5. **PromQL that uses `{job=~"foo.*"}` regex without anchors** — slow query, sometimes false matches.
6. **Inlining the rule body in the chart `templates/`** to "make it sticky" — that's a chart fork; see NEVER-DO.
7. **Adding the rule on every cluster in one PR** — surgical only.
---
## Rollback
- Revert the PR. Sync.
- vmalert/Prometheus reload picks up the previous ruleset within seconds.
- Any in-flight alerts resolve on next evaluation.
---
## Related
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
- Procedure: [modify-observability-config.md](modify-observability-config.md) — broader observability edits.
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md) — when an alert is silent because the metric is gone.
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
docs/platform/procedures/modify-observability-config.md
+210
View File
@@ -0,0 +1,210 @@
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
# Procedure — Modify observability stack config
> **Layer:** Layer 1 — values diff + PR.
> **Blast radius:** one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU).
> **Approval:** observability owner (`siddharth.pal@meesho.com`) + cluster owner.
This procedure covers edits to the cluster-by-cluster observability override files:
| Override path | What it controls |
|---------------|------------------|
| `helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml` | Scrape targets, relabel rules, remote-write tenancy. |
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmstorage retention, vmselect/vminsert sizing, ruler config. |
| `helm-overrides/<cluster>/mimir/custom-values.yaml` (also `mimir-distributed`) | Mimir distributor/ingester/ruler config and tenancy. |
| `helm-overrides/<cluster>/loki/custom-values.yaml` (also `loki-distributed`) | Loki retention, ingestion limits, multi-tenant config. |
| `helm-overrides/<cluster>/tempo/custom-values.yaml` (also `tempo-distributed`) | Tempo block retention, distributor sizing. |
| `helm-overrides/<cluster>/grafana/custom-values.yaml` | Datasources, dashboard providers, plugins. |
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert alerting rules and notifier config. |
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` (or `kube-prometheus-stack`) | Prometheus alert rules, scrape config, retention. |
All of these are **versioned-sibling-aware** — confirm whether the cluster's Argo Application points at `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`, etc., before editing. See [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md) for stack conventions.
---
## When to use
- Adding/removing scrape targets, relabel rules, recording rules.
- Changing retention (`vmstorage.retentionPeriod`, Loki `retention_period`, Tempo `compaction.block_retention`).
- Adding/removing/tuning alert rules in `vmalert` or `kube-prometheus-stack`.
- Adding/removing Grafana datasources, dashboards, plugins.
- Tuning ingester/distributor sizing for Mimir/Loki/Tempo.
Do **not** use this procedure for:
- **Bumping the chart version** of an observability tool — use [update-chart-version.md](update-chart-version.md).
- **Onboarding a brand-new observability tool** to a cluster — use [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
- **A blue-green migration** between sibling charts (`-latest`) — use [blue-green-chart-migration.md](blue-green-chart-migration.md).
---
## Inputs
| Input | Example |
|-------|---------|
| Target cluster | `k8s-supply-prd-ase1` |
| Target chart | `victoria-metrics-cluster` |
| Versioned-sibling target (if applicable) | `victoria-metrics-cluster-latest` |
| Change kind | `retention bump`, `new scrape target`, `new alert rule`, `dashboard add` |
| Promql expression(s) touched (if any) | `sum(rate(...)) by (job) > 0.05` |
| Approval ticket | CMR-1234 (if BU policy requires) |
---
## Pre-conditions
- [ ] The cluster directory and chart override exist.
- [ ] The cluster's Argo `Application` points at the chart you're editing (and not its sibling). Check `github.com/Meesho/devops-infra-argo-config`.
- [ ] You have the previous PR diff for context (most observability edits are touching a known knob).
- [ ] You have access to PromQL/promtool (or vmalert binary) locally for rule validation.
---
## Steps
### 1. Confirm which sibling the cluster runs
```bash
# In the sister repo
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*
```
Find the `Application` whose `spec.source.path` points at this repo. Note whether it's `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`. **Editing the wrong one is silent** — the file diff merges, but no cluster picks it up.
### 2. Read the current values
```bash
yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less
```
Note current retention, scrape targets, and any inline alert rules.
### 3. Author the change
Edit `helm-overrides/<cluster>/<chart>/custom-values.yaml`. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.
#### Cardinality discipline
If the change adds a label, scrape target, or `relabel_configs` rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See [observability.md §Cardinality discipline](../../global/coding-guidelines/observability.md).
#### Retention changes
Increasing `vmstorage.retentionPeriod`, Loki `retention_period`, or Tempo `block_retention` grows the backing PVC. Confirm `persistence.size` (or `vmstorage.persistentVolume.size`) has been bumped to match — otherwise vmstorage runs out of disk silently.
#### Alert-rule edits
If the rule's `for:` window or `severity` label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.
### 4. Render the chart locally
```bash
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
```
The render must succeed. If it errors, fix the values before continuing.
### 5. Validate any PromQL touched
For Prometheus/`kube-prometheus-stack`:
```bash
# Extract PrometheusRule objects from rendered output
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
# Validate
promtool check rules /tmp/rules.yaml
```
For vmalert:
```bash
# Extract VMRule (or ConfigMap with rules)
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
# vmalert dry-run
vmalert -dryRun -rule=/tmp/vmrules.yaml
```
If `promtool` / `vmalert` is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.
### 6. (Grafana) Validate datasource URL is in-cluster
Datasources should point at in-cluster Service DNS (e.g. `http://victoria-metrics-cluster-vmselect:8481`), not external endpoints. **Never** pin a Grafana datasource at `*.meeshogcp.in`, `prd.meesho.int`, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.
### 7. Open the PR
```bash
git checkout -b obs/<cluster>-<chart>-<short-desc>
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
git commit -m "obs(<cluster>/<chart>): <short desc>"
git push origin obs/<cluster>-<chart>-<short-desc>
gh pr create --base main
```
### PR description template
```markdown
## Summary
<one-line: what changed and why>
## Cluster × chart
- Cluster: `<cluster>`
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`
## Validation
- [ ] `helm template` renders cleanly
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
- [ ] Cardinality bounded (no unbounded label introduced)
- [ ] Retention/PVC headroom confirmed (if retention changed)
- [ ] PagerDuty routing unchanged (if alert rule changed)
## Approvers
- Observability owner: <handle>
- Cluster owner: <handle>
## Sister repo
- N/A (no Application change required)
```
### 8. After merge
Argo CD on the target cluster reconciles. Most observability charts use **manual sync** (see [observability.md](../../global/coding-guidelines/observability.md)) — open the cluster's Argo CD UI, find the Application, click **Sync**. Watch:
```bash
kubectl --context=<ctx> -n <observability-ns> get pods -w
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50
```
If a metric goes missing post-sync → [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
---
## Anti-patterns
1. **Editing the wrong sibling.** The Argo Application points at `*-latest`; you edit the stable chart's values. Diff merges, nothing applies.
2. **Adding `pod_name` / `request_id` / `user_id` as a label** without aggregation — explodes cardinality.
3. **Bumping retention without bumping PVC.** vmstorage runs out of disk; ingest fails silently.
4. **Silent PagerDuty re-route.** Changing `severity:` from `warning` to `critical` (or vice versa) without coordinating on-call.
5. **Inlining production hostnames** as Grafana datasource URLs — see [SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3.
6. **Cross-cluster normalising** — touching every cluster's `custom-values.yaml` in one PR. Surgical only; one cluster per PR.
---
## Rollback
- Revert the values PR. Argo CD will re-render with the previous values; click Sync.
- For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the *config* but not the *data*.
---
## Related
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
- Procedure: [modify-alert-rules.md](modify-alert-rules.md) — alert-only edits.
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
- ADR: [../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
- Wiki: [../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md](../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md).
docs/platform/procedures/onboard-app-to-cluster.md
+204
View File
@@ -0,0 +1,204 @@
# Procedure — Onboard a new app to an existing cluster
> **Layer:** Layer 1 — Agent-Writable.
> **Blast radius:** one new Helm release on one cluster.
> **Approval:** app owner + cluster owner.
This procedure adds a new infrastructure-tooling Helm release to a cluster that already exists in `helm-overrides/`. It covers the slice owned by `devops-infra-helm-charts`. The matching Argo `Application` lives in `github.com/Meesho/devops-infra-argo-config` and must be paired.
---
## Inputs
| Input | Example |
|-------|---------|
| Chart name (must exist in `helm-templates/`) | `kube-state-metrics` |
| Target cluster directory | `k8s-supply-prd-ase1` |
| Release name | `kube-state-metrics` (matches chart name; may be different for variants) |
| Workload namespace | `monitoring` |
| Image tag | `v2.10.1` |
| Sized resources (CPU/memory requests + limits) | `250m / 512Mi` |
| Node-pool key (per cluster) | `dedicated: monitoring` |
| Whether this needs a sidecar `external-dns` Service | yes / no |
---
## Pre-conditions
- [ ] The chart exists in `helm-templates/<chart>/` with a current `Chart.yaml`.
- [ ] The cluster directory exists in `helm-overrides/<cluster>/`.
- [ ] The chart is appropriate for an *infra* release (services don't go here — they live in `devops-argo-config`).
- [ ] The matching sister-repo `Application` PR is drafted (or will be drafted in parallel).
- [ ] CMR ticket open if required by BU policy.
---
## Steps
### 1. Verify the chart renders with a sibling cluster's values
Pick a cluster that already runs this chart and use its values as a starting point:
```bash
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -60
```
If the render errors → the chart's dependencies may be unresolved. Run `helm dependency update helm-templates/<chart>` first.
### 2. Identify the cluster's scheduling profile
```bash
# Standard GKE: uses 'dedicated:' keys
grep -rh 'dedicated:' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
# GKE Autopilot: uses 'cloud.google.com/compute-class' keys
grep -rh 'cloud.google.com/compute-class' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
```
For Contour, cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). For others, copy from a sibling app on the **same** cluster — never from the same app on a different cluster ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md)).
### 3. Create the directory and `custom-values.yaml`
```bash
mkdir -p helm-overrides/<cluster>/<app>
$EDITOR helm-overrides/<cluster>/<app>/custom-values.yaml
```
Author from scratch using [custom-values-schema.md](../schemas/custom-values-schema.md) and the cluster's scheduling profile from step 2. Do **not** copy a sibling cluster's values verbatim.
Skeleton:
```yaml
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/<image>
tag: <tag>
replicaCount: <N>
resources:
requests: {cpu: <c>, memory: <m>}
limits: {cpu: <c>, memory: <m>}
nodeSelector:
<pool-key>: <pool-value>
tolerations:
- {key: <pool-key>, value: <pool-value>, effect: NoSchedule}
```
### 4. (If needed) Add sidecar raw manifests
If the app needs sidecar resources (e.g. `external-dns` `Service`, `ComputeClass`, `ExternalSecret`), drop them in the same directory under a subfolder:
```
helm-overrides/<cluster>/<app>/
custom-values.yaml
external-dns-services/<svc>.yaml
computeclass/<name>-cc.yaml
```
See [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
### 5. Validate locally
```bash
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
# Render
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -60
# Optional: dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
helm diff upgrade <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
--kube-context=<context>
```
### 6. Open the sister-repo PR
In `github.com/Meesho/devops-infra-argo-config`, draft an `Application` (or add to an existing `ApplicationSet`):
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: <release>-<cluster>
spec:
destination:
name: <cluster>
namespace: <workload-namespace>
source:
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
targetRevision: main
path: helm-overrides/<cluster>/<app>
helm:
valueFiles: [custom-values.yaml]
syncPolicy:
syncOptions: [CreateNamespace=true]
# Most infra apps DO NOT use automated sync — see ADR-A5
```
### 7. Open the values-side PR (this repo)
```bash
git checkout -b onboard/<chart>-on-<cluster>
git add helm-overrides/<cluster>/<app>/
git commit
# pre-commit hook runs TruffleHog
git push origin onboard/<chart>-on-<cluster>
gh pr create --base main --title "Onboard <chart> to <cluster>"
```
PR description:
- Procedure followed: this file.
- App owner approver tag.
- Cluster owner approver tag.
- Sister-repo PR link (`devops-infra-argo-config#<n>`).
- CMR ticket reference (if applicable).
- Confirmation that the chart renders cleanly and `helm diff` (if run) showed only additions.
### 8. After both merge — sync in Argo CD
`devops-infra-argo-config`'s reconciler will create the `Application` resource on the cluster's Argo CD. Most infra apps are **manual sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)), so the workload deploy is a separate step:
1. Open the cluster's Argo CD UI.
2. Search for the new `Application`.
3. Verify the manifest renders (Diff view shows the chart's resources).
4. Click **Sync**.
5. Watch the rollout: `kubectl --context=<context> get pods -n <ns> -w`.
---
## Anti-patterns
1. **Copying the entire `helm-overrides/<sibling-cluster>/<app>/` directory** verbatim. Per-cluster scheduling differs.
2. **Bundling onboarding with a chart-version bump.** Two separate PRs.
3. **Skipping the sister-repo PR.** Without an `Application`, the values do nothing.
4. **Setting `automated.{prune,selfHeal}: true`** in the sister-repo `Application` "to make life easier." Manual sync is the default safety property.
5. **Inlining secrets** in `custom-values.yaml`. Use `ExternalSecret`.
---
## Rollback
If the merge causes a problem before Sync:
- Revert the values-side PR (and the sister-repo PR).
- Argo CD will prune the `Application` resource on next reconcile of the sister repo.
If Sync was clicked and the workload broke:
- Click **Rollback** in Argo CD UI to the previous synced revision (if there is one).
- Or revert both PRs and re-Sync — the previous state had no `Application`, so the workload is removed.
---
## Related
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
- Procedure: [update-chart-version.md](update-chart-version.md) for bumping after onboarding.
- Procedure: [deboard-app.md](deboard-app.md) for retirement.
- Skill: [skills/infra/onboard-app.md](../../../skills/infra/onboard-app.md) — agent-callable wrapper.
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
docs/platform/procedures/onboard-new-cluster.md
+140
View File
@@ -0,0 +1,140 @@
# Procedure — Onboard a new cluster's overrides
> **Layer:** Layer 1 — HIGH RISK.
> **Blast radius:** establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded.
> **Approval:** platform team + cluster owner. CMR mandatory.
When a new GKE cluster is provisioned (typically by `terraform-gcp-infra` or its successor), this repo gains a new `helm-overrides/<cluster>/` directory and the sister repo gains the `ApplicationSet` (or per-cluster Applications) that route to it. This procedure covers the slice owned by `devops-infra-helm-charts`.
---
## Pre-conditions
- [ ] The cluster exists in GCP — confirmed by the platform team.
- [ ] The cluster is registered in the relevant Argo CD instance(s).
- [ ] The cluster's `nodeSelector` / `tolerations` / `computeClass` topology is documented (node pool names + taints).
- [ ] The cluster has a `SecretStore` / `ClusterSecretStore` for `external-secrets` — or onboarding `external-secrets` is part of this PR.
- [ ] CMR open.
---
## Steps
### 1. Confirm the naming convention
| Pattern | Use |
|---------|-----|
| `k8s-<bu>-prd-ase1[c]` | Standard BU prod cluster (GCP zone-a or zone-c) |
| `k8s-shared-int-ase1` | Shared int |
| `k8s-aurva-prd-ase1` | Aurva |
| `k8s-supply-dev-ase1` | Dev/sandbox |
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier |
The directory name is a **contract** — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.
### 2. Determine the cluster's scheduling profile
| Cluster type | Scheduling key |
|--------------|----------------|
| GKE Autopilot | `cloud.google.com/compute-class` |
| Standard GKE | `dedicated:` |
Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of `ComputeClass` resources that need to land in `helm-overrides/<cluster>/<app>/computeclass/`.
### 3. Decide the minimal app set
Most clusters need at least:
- `external-secrets` — to materialise secrets from GCP Secret Manager
- `kube-state-metrics` — for fleet observability
- `victoria-metrics-agent` — to ship metrics to the central VM
- `fluentd` (or equivalent log shipper)
Plus per-cluster role:
- BU clusters: `contour-internal-0`, `contour-internal-1`, sometimes `contour-external` — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) for the convention
- DSGPU / Spark clusters: GPU operators, `keda` for queue autoscaling
- Dataplane (`db-*`) clusters: minimal — `kube-state-metrics` + `victoria-metrics-agent` only
### 4. Create the directory and minimal apps
```bash
mkdir -p helm-overrides/<cluster>
cd helm-overrides/<cluster>
```
For each app in the minimal set, follow [onboard-app-to-cluster.md](onboard-app-to-cluster.md) (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).
### 5. Update the per-cluster Contour matrix
If the cluster runs Contour, add a section to [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.
### 6. (If GKE Autopilot) Add the `ComputeClass` resources
Each Autopilot Contour / pool needs a corresponding `ComputeClass`. Drop them in:
```
helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml
```
Schema: [raw-manifest-sidecar-schema.md §ComputeClass](../schemas/raw-manifest-sidecar-schema.md). The `metadata.name` of each `ComputeClass` must equal the value the corresponding app's `nodeSelector: {cloud.google.com/compute-class: <name>}` references.
### 7. Open the base PR
```bash
git checkout -b cluster/<cluster>-base
git add helm-overrides/<cluster>/
git add contour-nodeselector-tolerations-summary.md # if updated
git commit
git push origin cluster/<cluster>-base
gh pr create --base main --title "Onboard <cluster> — base override set"
```
PR description must include:
- Cluster owner approver tag.
- Platform team approver tag.
- CMR ticket.
- Sister-repo PR for the `ApplicationSet` / per-cluster `Application` set.
- The minimal app set being shipped, with per-app rationale.
- The cluster's scheduling profile (key style + sample `nodeSelector`).
### 8. Pair the sister-repo PR
In `github.com/Meesho/devops-infra-argo-config`, draft the `ApplicationSet` (or per-cluster Application set) that walks `helm-overrides/<cluster>/`. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).
### 9. Stage subsequent app additions
After the base PR merges:
- Each additional app (`contour-internal-0`, `flagger`, etc.) is its own PR via [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
- Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.
---
## Anti-patterns
1. **Naming the directory `<cluster-name>` slightly differently** from how Argo CD registered it. Silent bind failure.
2. **Cloning another cluster's directory verbatim.** Per-cluster scheduling differs; `external-secrets` `SecretStore` references differ; image-tag pinning differs.
3. **Onboarding 30 apps in the base PR.** Stage them — base + per-app.
4. **Forgetting the Contour matrix update.** Future Contour edits then have no reference and silently mis-schedule.
5. **Skipping the `ComputeClass` resources** on a GKE Autopilot cluster. `nodeSelector` references a class that doesn't exist; pods Pending.
---
## Rollback
If the cluster onboarding turns out to be premature:
- Revert the values-side PR.
- Revert the sister-repo `ApplicationSet` PR.
- The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't `prune` what's not in its scope after the `Application` is removed); for a clean wipe, manually `kubectl delete ns` the affected namespaces.
---
## Related
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — for each app after the base PR.
- Schema: [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md) — for `ComputeClass` and `ExternalSecret` sidecars.
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
docs/platform/procedures/update-chart-version.md
+176
View File
@@ -0,0 +1,176 @@
# Procedure — Update a chart's pinned dependency version
> **Layer:** Layer 1 — HIGH RISK (charts in `helm-templates/` are consumed by every cluster that runs them).
> **Blast radius:** every cluster that has an Argo `Application` referencing this chart will pick up the new version on next sync.
> **Approval:** platform team. CMR mandatory for prod-fleet charts.
In this repo, charts in `helm-templates/<chart>/` are typically thin wrappers — `Chart.yaml` declares an upstream chart as a dependency, and `Chart.lock` pins the resolved subchart. "Bumping the chart version" means:
1. Update `dependencies[].version` in `Chart.yaml`.
2. Run `helm dependency update` to refresh `Chart.lock` (and re-pull the subchart).
3. Test render with representative cluster overrides.
---
## When to use this procedure
- Upgrading a wrapper chart's pinned subchart version (e.g. `argo-cd 7.7.23` → `7.8.0`).
- Following a security advisory in an upstream chart.
This is **not** the right procedure for:
- A blue-green migration to a new major version → use [blue-green-chart-migration.md](blue-green-chart-migration.md).
- Editing chart templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
- Changing values without a chart bump → that's per-cluster `custom-values.yaml` work, not this.
---
## Pre-conditions
- [ ] The new upstream version exists and has a published changelog.
- [ ] You have read the changelog for breaking template / values changes.
- [ ] The bump is the **explicit headline** of the PR (not a side-effect of another change).
- [ ] CMR open if this is a prod-fleet chart (Argo CD, Contour, VictoriaMetrics, ingress, cert-manager).
---
## Steps
### 1. Read the changelog
```bash
# For most upstream charts:
helm repo update
helm search repo <repo>/<chart> --versions | head -20
# Read the chart's CHANGELOG.md or release notes on GitHub.
```
Look specifically for:
- **Removed values keys** — would silently no-op overrides.
- **Renamed values keys** — same.
- **CRD changes** — might require manual `kubectl apply` of new CRDs.
- **Breaking template changes** (e.g. label selector immutability on Deployments).
- **Required Kubernetes version bumps**.
If any of these apply, this is not a simple version bump — escalate to a blue-green migration ([blue-green-chart-migration.md](blue-green-chart-migration.md)).
### 2. Update `Chart.yaml`
```bash
$EDITOR helm-templates/<chart>/Chart.yaml
```
Find the `dependencies[]` entry and bump `version:`. Example:
```diff
dependencies:
- name: argo-cd
- version: 7.7.23
+ version: 7.8.0
repository: https://argoproj.github.io/argo-helm
```
### 3. Refresh `Chart.lock`
```bash
helm dependency update helm-templates/<chart>
```
This:
- Verifies the new version resolves.
- Updates `Chart.lock` with the new digest.
- Re-pulls the subchart `.tgz` into `helm-templates/<chart>/charts/`.
The lockfile must be committed alongside `Chart.yaml`. A bump without a refreshed lockfile is incomplete.
### 4. Render against a representative cluster's overrides
Pick a cluster that runs this chart with a non-trivial override set:
```bash
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' \
| grep -v '^helm-overrides/db-' | head -1)
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
```
If the render errors → the chart bump introduced a values incompatibility. **Stop.** Either:
- Find the new values shape and update affected `custom-values.yaml` files in this PR, OR
- Defer the bump until those updates are scoped.
### 5. Spot-check `helm diff` against a live cluster (recommended)
```bash
helm diff upgrade <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
--kube-context=<cluster-context>
```
What you want to see:
- Image tag bumps.
- Label updates (often `helm.sh/chart`).
- Possibly new CRDs or RBAC.
What's a red flag:
- `Deployment` selector changes (immutable; will fail to apply).
- `StatefulSet` `volumeClaimTemplates` changes.
- Resource removals you didn't expect.
### 6. Commit and open the PR
```bash
git checkout -b chart-bump/<chart>-<new-version>
git add helm-templates/<chart>/Chart.yaml helm-templates/<chart>/Chart.lock
git add helm-templates/<chart>/charts/ # if the .tgz was re-pulled
git commit
git push origin chart-bump/<chart>-<new-version>
gh pr create --base main --title "chart bump: <chart> <old> → <new>"
```
PR description:
- Procedure followed: this file.
- Old → new with a link to the upstream changelog.
- The list of clusters that consume this chart (`grep -rl '<chart>' helm-overrides | head`).
- The render(s) and diff(s) you ran.
- Whether any breaking-change fallout is bundled (preferably not — if so, scope to a separate PR).
### 7. After merge — Argo CD picks up the new chart on next sync
For each cluster running this chart:
- The cluster's Argo CD detects the chart hash change.
- The `Application` becomes `OutOfSync`.
- A human clicks **Sync** per [ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
- Watch rollout per cluster — don't sync 20 clusters simultaneously.
---
## Anti-patterns
1. **Bumping `Chart.yaml` without refreshing `Chart.lock`.** Argo CD uses the lockfile; the bump silently no-ops.
2. **Bundling a chart bump with values changes.** Two separate PRs — one for the bump, one for any values that need to change because of the bump.
3. **Bumping a major version (e.g. 7.x → 8.x) in place.** Use [blue-green-chart-migration.md](blue-green-chart-migration.md) — give yourself a sibling chart and migrate cluster-by-cluster.
4. **Syncing every cluster simultaneously after merge.** Stagger; one cluster at a time, with a soak between.
5. **Skipping the changelog read.** Surprises you'll regret.
---
## Rollback
Open a follow-up PR that reverts `Chart.yaml` and `Chart.lock` to the previous versions. After merge, each cluster's Argo CD shows `OutOfSync` against the old chart; click **Sync** to roll back per cluster.
If the chart bump landed CRDs that are now incompatible with the old version, the rollback may require manual CRD cleanup — coordinate with the platform team.
---
## Related
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — for major-version bumps.
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md) — when the bump requires template edits.
- Skill: [skills/infra/bump-chart-version.md](../../../skills/infra/bump-chart-version.md).
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
docs/platform/runbooks/argocd-sync-failure.md
+231
View File
@@ -0,0 +1,231 @@
# Runbook — Argo CD Sync Failure (infra release)
> **Type:** Decision tree.
> **Entry symptom:** an Argo CD `Application` for an infra release is `OutOfSync`, errored, or stuck `Progressing`.
> **Layer:** mostly Layer 1 (read state, propose YAML diff). Some branches are Layer 2 (advisory).
This runbook handles sync failures for infra Applications routed by `github.com/Meesho/devops-infra-argo-config` and rendering against this repo's `helm-overrides/<cluster>/<app>/`.
---
## Entry — gather context
```bash
APP=<release> # e.g. argocd, contour-internal-0
CLUSTER=<cluster-name> # e.g. k8s-supply-prd-ase1
NS=$(argocd app get $APP -o json | jq -r '.spec.destination.namespace')
PROJECT=$(argocd app get $APP -o json | jq -r '.spec.project')
argocd app get $APP # headline
argocd app get $APP -o json | jq -r '.status.conditions[]?'
argocd app get $APP -o json | jq -r '.status.operationState.message // empty'
```
Note which Argo CD instance you're hitting — most infra Applications live in a per-cluster Argo CD install.
---
## Decision tree
```text
START
│
└── Is `argocd app get $APP` known to this Argo at all?
│
├── NO → §1 — Application not found
│
└── YES → What's the symptom?
│
├── Sync failed with an error message → §2 — Errored sync
├── Sync stuck `Progressing` for >5 min → §3 — Stuck progressing
├── App is `OutOfSync` but Sync hasn't run → §4 — OutOfSync only
└── `Synced`+`Healthy` but workload bad → §5 — Wrong workload (leave runbook)
```
---
## §1 — Application not found
| Sub-check | Action |
|-----------|--------|
| Are you on the right Argo CD instance? | Most v2 infra apps live in per-cluster Argo CDs. |
| Was the app deboarded recently? | `git -C <argo-config> log --diff-filter=D -- 'apps/<cluster>/<app>*'` and `git log --diff-filter=D -- 'helm-overrides/<cluster>/<app>/'`. |
| Did the values directory land on `main`? | `git log --all -- 'helm-overrides/<cluster>/<app>/'`. |
If the file *should* exist on `main` but the `Application` resource isn't created → **Layer 2** — escalate to the platform team. The cluster's Argo CD bootstrap (`ApplicationSet`) may not be picking up the path.
---
## §2 — Errored sync (read the error message)
### §2a — `repository not accessible / authentication required`
| Action |
|--------|
| Check `spec.source.repoURL` is `github.com/Meesho/devops-infra-helm-charts.git`. |
| If yes, the credentials in Argo CD's repo-list need refreshing. **Layer 2** — recommend platform team rotates credentials. |
### §2b — `path 'X' does not exist in repo Y`
```text
path 'helm-overrides/k8s-supply-prd-ase1/argocd' does not exist
```
| Action |
|--------|
| `ls helm-overrides/<cluster>/<app>/` on `main`. |
| If absent → either the values-side PR wasn't merged, or the path was typo'd in the sister-repo `Application`. **Layer 1** — open a fix PR (sister repo). |
| If a blue-green migration just landed: the `Application` may be pointing at the **old** chart path that was retired. **Layer 1** — repoint the `Application` to the new sibling path. |
### §2c — `Helm template error` / `values file not found`
```text
open helm-overrides/<cluster>/<app>/custom-values.yaml: no such file or directory
```
| Action |
|--------|
| Verify the file exists on `main`: `git ls-tree origin/main -- helm-overrides/<cluster>/<app>/custom-values.yaml`. |
| If absent → onboarding is incomplete. **Layer 1** — open the missing values-side PR. |
### §2d — `unable to render manifests` / `template error`
Helm template error inside the chart (missing required value, type mismatch).
| Action |
|--------|
| Reproduce locally: `helm template <app> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml`. |
| Determine whether the fix is in this repo (rare — usually values shape changed) or `helm-templates/<chart>` (more common after a chart bump). **Layer 1**. |
| If the error is `Cannot use existing release: ...` — see §2h. |
### §2e — `cluster not found / dial tcp ... no route to host`
| Action |
|--------|
| **Layer 2** — escalate to platform team. Cluster API server unreachable, or cluster-secret stale in Argo CD. |
| Do **not** edit `spec.destination.{server,name}` to redirect; that masks the underlying cluster issue. |
### §2f — `forbidden: ...` / admission webhook deny
```text
admission webhook "validate.kyverno.svc-fail" denied the request
```
| Action |
|--------|
| Look at the rule that denied (Kyverno? PSP? OPA? GKE Autopilot policy?). |
| Often the chart's manifest violates a cluster policy (e.g. `runAsUser: 0`, missing `securityContext`). |
| **Layer 1** — fix in chart values; pair with the policy team if the policy is wrong. |
| GKE Autopilot specifically denies many privileged settings — read the deny message carefully. |
### §2g — `webhook errored: ... cert-manager / external-secrets / kyverno`
A webhook that should validate the new resource is itself unhealthy.
| Action |
|--------|
| `kubectl get pods -n cert-manager` (or the relevant operator's namespace) — is it running? |
| **Layer 2** — recommend recovering the webhook before re-syncing this app. |
### §2h — `cannot patch ... immutable field`
Most often: `Deployment.spec.selector` or `StatefulSet.volumeClaimTemplates`. A chart bump that changes labels.
| Action |
|--------|
| Read the upstream changelog to confirm. |
| **Layer 2** — recommend deleting the old `Deployment` / `StatefulSet` (with the workload owner) so the chart can recreate it. **Do not delete blindly** — for `StatefulSet`, the PVCs survive but the rollout is disruptive. |
| For systemic immutable-field changes across a chart bump, this is a sign the bump should have been a [blue-green migration](../procedures/blue-green-chart-migration.md). Roll back, plan the migration. |
### §2i — `dependent CRD ... not installed`
The chart needs a CRD that doesn't exist yet on the cluster.
| Action |
|--------|
| Check whether the chart includes the CRD in `templates/crds/` (most upstream charts ship CRDs). |
| If yes: the chart's `helm template` may not include CRDs by default — Argo CD has `IncludeCRDs` semantics; check the `Application`'s `helm.skipCrds` setting. |
| If the CRD is supposed to come from a different chart (`cert-manager`, `kube-prometheus-stack`): **Layer 2** — sync that chart first. |
---
## §3 — Stuck `Progressing` for > 5 minutes
The sync started but resources aren't reconciling.
| Sub-check | Action |
|-----------|--------|
| `argocd app get $APP --refresh` shows resource-level status. | Look for `Progressing` resources. |
| Is a Deployment failing to roll out? | `kubectl rollout status deploy/<name> -n $NS`. If yes → see [pod-pending-scheduling.md](pod-pending-scheduling.md) or [ingress-down.md](ingress-down.md). |
| Is a Job hung? | `kubectl describe job/<name> -n $NS`. Old `Job`s sometimes block syncs (Helm pre-/post-install hooks). |
| Is a `PreSync`/`PostSync` hook hanging? | `kubectl get pods -n $NS -l argocd.argoproj.io/hook=PostSync`. |
If the workload itself is the problem, leave this runbook.
---
## §4 — `OutOfSync` only (no error, sync hasn't run)
Argo CD sees a diff between git and the cluster. **Most infra apps are intentionally manual-sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)).
| Sub-check | Action |
|-----------|--------|
| Is this expected? (e.g. you just merged a PR.) | Click Sync. |
| Diff suspicious? (e.g. someone `kubectl edit`-ed.) | `argocd app diff $APP`. If out-of-band edit happened, the GitOps contract was violated; recommend reverting the manual change or capturing it in a PR. |
| Diff has sat for > 1 day? | Notify the app owner — manual-sync apps rot if no one clicks. |
---
## §5 — `Synced` and `Healthy` but workload misbehaving
Argo thinks all is fine; the workload is broken. Not a sync failure. **Leave this runbook.**
| Symptom | Where to go |
|---------|-------------|
| Pods crashlooping | [pod-pending-scheduling.md](pod-pending-scheduling.md) §3 |
| Ingress 5xx | [ingress-down.md](ingress-down.md) |
| Specific feature broken | App-team playbook |
---
## §6 — Special: blue-green migration in flight
If this app is in a `<chart>` ↔ `<chart>-<variant>` migration:
- Confirm which variant the `Application` points at (check `spec.source.path`).
- The chart name may have changed in the new variant; release-name pinning via `fullnameOverride` may be required to keep the same Service DNS during cutover.
- A failed sync mid-migration is the trigger to roll back (`spec.source.path` ← old) and Sync, not to push forward.
- Read [blue-green-chart-migration.md](../procedures/blue-green-chart-migration.md) before deciding.
---
## Escalation matrix
| Symptom | Action | Escalate to |
|---------|--------|-------------|
| §1 + bootstrap looks healthy | Investigate further | App owner |
| §2a (repo auth) | Confirm allowed repoURL; rotate creds | Platform team |
| §2b (path missing) | Fix in this repo or sister repo | App owner |
| §2c, §2d (helm render) | Reproduce; fix values or chart | App owner / platform team |
| §2e (cluster unreachable) | Don't edit destination | Platform team |
| §2f (admission webhook) | Fix in chart values | App + policy team |
| §2g (webhook unhealthy) | Recover the webhook first | Platform team |
| §2h (immutable field) | Probably needs blue-green | Platform team |
| §3 (stuck > 30 min) | Check pod events; consider workload rollback | App owner |
---
## Done conditions
- `argocd app get $APP` shows `Synced` + `Healthy`.
- The PR or manual fix that resolved it is on `main`.
- If the failure was caused by a regression, a postmortem / RCA is scheduled.
---
## Related
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
- Runbook: [ingress-down.md](ingress-down.md).
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md).
- Boundaries: [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md).
docs/platform/runbooks/ingress-down.md
+197
View File
@@ -0,0 +1,197 @@
# Runbook — Ingress (Contour) is down on a cluster
> **Type:** Decision tree.
> **Entry symptom:** services on a cluster are returning 5xx, not reachable, or DNS doesn't resolve to working endpoints.
> **Layer:** mostly Layer 2 (advisory — recommend kubectl actions). Layer 1 only when the fix is a values diff in this repo.
Contour is the ingress for almost every BU cluster, and most clusters run **multiple Contour releases** — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. The matrix is in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Read it before editing any Contour values.**
---
## Entry — gather context
```bash
CLUSTER=<cluster>
CTX=<kubectl-context>
# Which Contour instances does this cluster run?
ls helm-overrides/$CLUSTER | grep '^contour'
# Pod state for each Contour
for c in $(ls helm-overrides/$CLUSTER | grep '^contour'); do
echo "=== $c ==="
kubectl --context=$CTX get pods -n projectcontour -l app.kubernetes.io/instance=$c -o wide 2>/dev/null \
|| kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c -o wide
done
```
---
## Decision tree
```text
START
│
└── Which Contour instance is affected?
│
├── External (`contour-external*`) → §A — North-south traffic
├── Internal (`contour-internal-*`) → §B — East-west traffic
└── Both / unsure → §C — Cluster-wide (worst case)
```
For each branch:
```text
└── What's the failure shape?
│
├── Pods Pending → §1 — Scheduling failure
├── Pods CrashLooping → §2 — Contour boot failure
├── Pods Running but no endpoints → §3 — Service / load-balancer unhealthy
├── HTTPS responses are 5xx → §4 — Backend / cert / config error
└── DNS doesn't resolve → §5 — external-dns / DNS plumbing
```
---
## §1 — Contour pods Pending
This is almost always a scheduling-key mismatch. **The single most common Contour incident.**
```bash
kubectl --context=$CTX describe pod <contour-pod>
```
Read the `Events:` section. Common causes:
| Reason | Fix in |
|--------|--------|
| `0/N nodes available: 1 node(s) had untolerated taint <key>=<value>` | Tolerations in `helm-overrides/<cluster>/<contour-instance>/custom-values.yaml`. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Layer 1.** |
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | `nodeSelector` in the values. Same fix. **Layer 1.** |
| `0/N nodes available: 1 node(s) had no available compute class` (GKE Autopilot) | Either the `ComputeClass` resource is missing, or `nodeSelector: cloud.google.com/compute-class: <X>` references a class that doesn't exist. Check `helm-overrides/<cluster>/<contour-instance>/computeclass/*-cc.yaml`. **Layer 1.** |
| `0/N nodes available: insufficient cpu/memory` | Node-pool autoscaler is at max. **Layer 2** — escalate to cluster owner / platform team. |
The fix is always to bring the values' `nodeSelector` / `tolerations` into agreement with the cluster's actual node-pool topology. Never guess — read the matrix and the cluster's other apps.
---
## §2 — Contour pods CrashLooping
```bash
kubectl --context=$CTX logs <contour-pod> -c contour --previous
kubectl --context=$CTX logs <contour-pod> -c envoy --previous
```
| Pattern | Cause | Fix in |
|---------|-------|--------|
| `failed to load TLS certificates` | Missing `Secret`, expired cert, wrong key | `cert-manager` / `external-secrets`. **Layer 2.** |
| `error parsing config: invalid HTTPProxy` | A user `HTTPProxy` is malformed and Contour refuses to load | The user's namespace. **Layer 2** — recommend `kubectl get httpproxy -A` to find the offender, then fix in the consuming team's repo. |
| `bind: address already in use` | Two Contour pods on the same node fighting for the host port | Pod anti-affinity in values. **Layer 1.** |
| `error: ratelimit service ... unavailable` | `contour-rate-limit` sidecar / external service is down | Operator action. **Layer 2.** |
---
## §3 — Contour Running but no endpoints / LB unhealthy
The Contour pods are healthy, but downstream LB / DNS / Service routing is broken.
```bash
kubectl --context=$CTX get svc -n projectcontour
kubectl --context=$CTX get endpoints -n projectcontour
kubectl --context=$CTX describe svc <contour-svc> -n projectcontour
```
| Sub-check | Action |
|-----------|--------|
| `Service` of type `LoadBalancer` has no `EXTERNAL-IP` | GCP LB provisioning failed. Check the `Service` annotations match the cluster's LB pattern. **Layer 2** — escalate. |
| `Endpoints` empty | The Contour pods aren't matching the `Service` selector. Often a label drift between values and the chart's defaults. **Layer 1** — fix selectors. |
| `Service` annotations mention `cloud.google.com/load-balancer-type: Internal` but external traffic is expected | Wrong Contour instance / Service annotation. **Layer 1.** |
| `Health checks failing` on the GCP LB | Backend pods aren't ready. See §4. |
---
## §4 — HTTPS responses are 5xx
Don't curl the production endpoint yourself — that violates [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md). Instead, recommend the user check from a controlled vantage:
```bash
# From inside the cluster
kubectl --context=$CTX run -it --rm curl-test --image=curlimages/curl --restart=Never -- \
curl -v https://<service>.<ns>.svc.cluster.local
# Contour access logs
kubectl --context=$CTX logs <envoy-pod> -c envoy | tail -50
```
| Pattern | Cause | Fix in |
|---------|-------|--------|
| `503 no_healthy_upstream` | Backend pods all unhealthy | App's pod readiness — see [pod-pending-scheduling.md](pod-pending-scheduling.md). **Layer 2.** |
| `502 upstream connect error` | Backend connection refused / TLS mismatch | App config / mTLS. **Layer 2.** |
| `404 route not found` | No matching `HTTPProxy`/`Ingress` | The user's `HTTPProxy` is missing or has the wrong host/path. **Layer 2.** |
| `500` from the app | Application error | App-team playbook. **Out of scope for this runbook.** |
---
## §5 — DNS doesn't resolve
| Sub-check | Action |
|-----------|--------|
| Is `external-dns` running on this cluster? | `kubectl --context=$CTX get pods -n external-dns`. |
| Are there `Service` resources with `external-dns.alpha.kubernetes.io/hostname` annotations? | `kubectl --context=$CTX get svc -A -o json \| jq '.items[] \| select(.metadata.annotations."external-dns.alpha.kubernetes.io/hostname")'`. |
| Did `external-dns` reconcile recently? | `kubectl --context=$CTX logs deploy/external-dns -n external-dns \| tail -30`. |
| Is the Cloud DNS zone wired up? | **Layer 2** — out of scope; escalate to platform team. |
---
## §C — Cluster-wide ingress outage
If both internal and external Contour are degraded simultaneously:
1. **Stop. Don't iterate values fixes.** This is incident-grade.
2. **Layer 2 — escalate to platform team immediately.**
3. Check whether a recent merge in this repo or the sister repo correlates: `git log --since='2 hours ago' -- helm-overrides/$CLUSTER/contour*` and the same in `devops-infra-argo-config`.
4. If a recent merge is implicated: revert it, click Sync to roll back to the previous state.
5. If no recent merge: cluster-level issue (node pool, network policy, GCP LB) — out of scope for this repo.
---
## When this repo *is* the right place to fix
A Contour outage traces back to `devops-infra-helm-charts` only when:
1. **`nodeSelector` / `tolerations` / `computeClass`** were copied from the wrong cluster.
2. **A chart bump** introduced an immutable-selector change or removed a values key.
3. **A blue-green migration** was cutover prematurely (Application points at `<contour>-vX.Y.Z` but values are still on the old shape).
4. **`fullnameOverride`** was changed (very rare, but catastrophic).
For 80%+ of ingress incidents, the fix is **outside** this repo (cluster issues, app-side issues, GCP LB, DNS).
---
## Escalation matrix
| Symptom | First responder | Escalate to |
|---------|-----------------|-------------|
| §1 (pending — scheduling) | Yourself with the values fix | Cluster owner if node pool full |
| §2 (CrashLoop — TLS) | cert-manager team | Security if cert source unknown |
| §2 (CrashLoop — invalid HTTPProxy) | Consuming team | Platform if Contour itself is broken |
| §3 (no endpoints / LB) | Cluster owner | Platform team |
| §4 (5xx) | App team | Platform team if Contour-side |
| §5 (DNS) | Platform team | — |
| §C (cluster-wide outage) | Platform team — pager | — |
---
## Done conditions
- `kubectl get pods -n projectcontour` shows N/N Ready for the affected instances.
- Synthetic probes from the app team return expected status codes.
- If the root cause was in this repo, the fix is on `main` and synced.
---
## Related
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
docs/platform/runbooks/metrics-gap.md
+233
View File
@@ -0,0 +1,233 @@
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
# Runbook — Metrics gap on a cluster / namespace
> **Type:** Decision tree.
> **Entry symptom:** "Grafana panels are blank for `<cluster>` / `<namespace>` / `<service>`," or an alert that should be firing isn't, or VictoriaMetrics shows `no data`.
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
The observability path on Meesho's GKE fleet is:
```
workload Pod (exposes /metrics)
└→ victoria-metrics-agent (vmagent) scrapes
└→ remote_write to victoria-metrics-cluster (vmstorage)
└→ vmselect ← Grafana / vmalert query
```
A metrics gap can be at any hop. Walk this tree top-down — the most common root cause is hop 1 (scrape config).
---
## Entry — gather context
```bash
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<workload-namespace-where-metric-is-missing>
METRIC=<metric-name-or-job-label>
# Confirm the cluster runs VM agent + cluster
ls helm-overrides/$CLUSTER | grep -E '^victoria-metrics-(agent|cluster)'
# Confirm pods are healthy
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
```
If pods are missing/CrashLooping → that's the gap. Skip to §5.
---
## Decision tree
```text
START
│
├── Is the metric known to be emitted by the workload?
│ ├── No → §0 — Workload not emitting; out of scope (app team)
│ └── Yes →
│
├── §1 — Is vmagent scraping the workload?
│ ├── No → fix scrape config (Layer 1)
│ └── Yes →
│
├── §2 — Are scrape targets healthy (status=up)?
│ ├── Down → fix endpoint reachability (Layer 2 / Layer 1)
│ └── Up →
│
├── §3 — Are relabel rules dropping the metric?
│ ├── Yes → adjust relabel_configs (Layer 1)
│ └── No →
│
├── §4 — Is remote_write succeeding?
│ ├── No → vmagent → vmstorage path broken (Layer 2 / Layer 1)
│ └── Yes →
│
├── §5 — Is vmstorage healthy and ingesting?
│ ├── No → vmstorage outage (Layer 2)
│ └── Yes →
│
└── §6 — Is the Grafana datasource / tenant correct?
└── Misrouted query → fix datasource URL or tenant header (Layer 1)
```
---
## §0 — Workload not emitting
Out of scope for this repo. Confirm with:
```bash
# Port-forward and curl /metrics directly
kubectl --context=$CTX -n $NS port-forward <pod> 9090:<metrics-port> &
curl -s localhost:9090/metrics | grep -i "$METRIC"
```
If `/metrics` is empty or doesn't contain `$METRIC` → app team. Stop.
---
## §1 — Is vmagent scraping the workload?
```bash
# vmagent UI exposes /api/v1/targets
kubectl --context=$CTX -n monitoring port-forward svc/victoria-metrics-agent 8429:8429 &
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.labels.namespace=="'$NS'")'
```
If no targets for `$NS`:
- Look at `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml`.
- Check the `additionalScrapeConfigs` (or `config.scrape_configs`) for a job that matches the workload's labels / namespace selector.
- Common cause: a `kubernetes_sd_configs` `namespaces.names` filter excludes `$NS`.
- Common cause: a missing `Pod`/`Service`/`PodMonitor` annotation `prometheus.io/scrape: "true"` on the workload.
**Layer 1 fix** (if the gap is in the scrape config): edit `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml` per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
**Layer 2 fix** (if the gap is on the workload — missing annotation): hand off to app team.
---
## §2 — Targets healthy?
```bash
# In the same vmagent /targets output
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up") | {labels, lastError}'
```
If targets are `down`:
| `lastError` | Cause | Fix |
|-------------|-------|-----|
| `connection refused` | Workload not listening on declared port | App team — Layer 2. |
| `i/o timeout` | NetworkPolicy / firewall blocking vmagent → workload | Check `NetworkPolicy` in `$NS`. Often a NetworkPolicy allowing only intra-namespace traffic and not vmagent's namespace. **Layer 2** — recommend the workload team allow vmagent. |
| `x509: certificate signed by unknown authority` | mTLS misconfigured | App team — Layer 2. |
| `404 Not Found` | Wrong path (default `/metrics` vs custom) | Add `metrics_path:` in scrape config. **Layer 1.** |
---
## §3 — Relabel rules dropping the metric?
```bash
yq e '.config.scrape_configs[].metric_relabel_configs, .config.scrape_configs[].relabel_configs' \
helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml
```
Look for:
- `action: drop` rules with regexes matching `$METRIC`.
- `action: keep` rules whose regex *excludes* `$METRIC`.
- `action: labeldrop` removing a label the query uses.
Test in vmagent's UI under the *Targets* tab — it shows the labels post-relabel.
**Layer 1 fix:** loosen the relabel rule. Re-PR per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
---
## §4 — Remote_write succeeding?
```bash
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent | grep -i 'remote_write\|error\|failed' | tail -20
```
| Symptom | Cause | Fix |
|---------|-------|-----|
| `429 Too Many Requests` from vmstorage | vmstorage ingest saturated | Scale vmstorage / vminsert (Layer 1 — see [observability.md](../../global/coding-guidelines/observability.md)). |
| `connection refused` to vmstorage URL | vmstorage Service down or wrong URL | Verify `remoteWrite.url` in vmagent values matches the live `vmstorage` Service DNS. Layer 1. |
| `out of bounds timestamp` | Clock skew on vmagent's node | Layer 2 — node time sync. |
| `series limit exceeded` | Cardinality bomb on vmstorage tenant | Layer 1 — drop the offending label. See [observability.md §Cardinality](../../global/coding-guidelines/observability.md). |
---
## §5 — vmstorage healthy?
```bash
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
kubectl --context=$CTX -n monitoring describe pod vmstorage-0 | tail -30
kubectl --context=$CTX -n monitoring exec vmstorage-0 -- df -h /storage
```
Failure modes:
| Symptom | Layer | Fix |
|---------|-------|-----|
| Pod CrashLooping with `out of disk` | Layer 1 | Bump `persistence.size`. Note: PVC growth requires the StorageClass to support `allowVolumeExpansion: true`. See [../schemas/storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md). |
| Pod CrashLooping with retention/index errors | Layer 2 | Escalate — may need data-side intervention. |
| Pod Pending | Layer 1 | Scheduling issue. See [pod-pending-scheduling.md](pod-pending-scheduling.md) and [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md). |
| Pod Running but vmselect can't reach it | Layer 1 / 2 | Verify the headless Service and StatefulSet pod-DNS records. |
---
## §6 — Grafana datasource / tenant correct?
```bash
yq e '.datasources.datasources.yaml.datasources[] | select(.name == "*VictoriaMetrics*" or .type == "prometheus")' \
helm-overrides/$CLUSTER/grafana/custom-values.yaml
```
| Sub-check | Action |
|-----------|--------|
| `url:` points at the correct in-cluster vmselect Service DNS | If wrong, **Layer 1** — fix the values per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md). |
| `httpHeaderName1: X-Scope-OrgID` (multi-tenant clusters only) | If the cluster runs multi-tenant VM, the tenant header must be set. Layer 1 fix. |
| Datasource `url:` points at an external `*.meeshogcp.in` host | Forbidden — see [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3. Repoint at in-cluster Service DNS. |
---
## Remediation summary
| Hop | Likely fix | Layer | Procedure |
|-----|-----------|-------|-----------|
| §1 scrape | Add scrape config / fix selector | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §2 endpoint | Workload-side / NetworkPolicy | 2 | Hand off to app team |
| §3 relabel | Loosen drop rule | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §4 remote_write | Scale vmstorage / fix URL | 1/2 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
| §5 vmstorage | PVC grow / scheduling fix | 1/2 | [pod-pending-scheduling.md](pod-pending-scheduling.md) |
| §6 datasource | Fix Grafana datasource | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
---
## Escalation triggers
- §5 vmstorage outage with no obvious values-side fix → platform team (observability owner).
- Multi-cluster simultaneous metrics gap → platform team — cluster-level / control-plane issue.
- Cardinality explosion impacting vmstorage stability → platform team + workload owner — joint fix.
---
## Done conditions
- The query that was returning `no data` returns the expected series in Grafana Explore.
- No remote_write errors in vmagent logs for at least 5 minutes post-fix.
- Alerts that depend on the metric have transitioned from `pending` / silent back to expected state.
---
## Related
- Procedure: [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
- Procedure: [../procedures/modify-alert-rules.md](../procedures/modify-alert-rules.md) — if the gap is "alert silent" not "metric missing."
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — for §5 vmstorage scheduling failures.
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if a sync didn't take.
docs/platform/runbooks/pod-pending-scheduling.md
+192
View File
@@ -0,0 +1,192 @@
# Runbook — Pod Pending / wrong-node scheduling
> **Type:** Decision tree.
> **Entry symptom:** an infra workload's pods are `Pending` indefinitely, or scheduling onto the wrong node pool.
> **Layer:** mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.
The single most common values-side bug in this repo is **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster**. ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md))
---
## Entry — gather context
```bash
APP=<release>
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace>
kubectl --context=$CTX get pods -n $NS -o wide
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40 # Events: section
```
---
## Decision tree
```text
START
│
└── What state are the pods in?
│
├── Pending — never scheduled → §1 — Pending pods
├── ContainerCreating long → §1 — Pending pods
├── Running but on the WRONG node pool → §2 — Wrong-pool scheduling
├── ImagePullBackOff / ErrImagePull → §3 — Image pull
├── CrashLoopBackOff → §4 — Crash loop
├── Running but PVC unbound → §5 — Storage
└── Running fine → leave runbook
```
---
## §1 — Pending pods
```bash
kubectl --context=$CTX describe pod <pod> -n $NS
```
Read the `Events:` section. Common patterns:
| Reason | Diagnosis | Fix in |
|--------|-----------|--------|
| `0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}` | Pod has wrong toleration or no toleration. | `helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. **Layer 1.** |
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | Pod's `nodeSelector` doesn't match any node label. | Values `nodeSelector:`. **Layer 1.** |
| `0/N nodes available: ... had no available compute class` (Autopilot) | `cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster. | (a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. **Layer 1.** |
| `0/N nodes available: insufficient cpu` / `insufficient memory` | Node-pool autoscaler at max, or pod's `requests` too high. | Either reduce `resources.requests`, or **Layer 2** — escalate to cluster owner to raise node-pool max. |
| `0/N nodes available: pod has unbound immediate PersistentVolumeClaims` | PVC is `Pending`. | See §5. |
| `0/N nodes available: didn't tolerate node-pressure taint` | Node has `node.kubernetes.io/disk-pressure` etc. | **Layer 2** — cluster-level issue. |
| `volume "X" not found` | PVC bound to a non-existent PV. | See §5. |
**The diagnostic for "wrong cluster's scheduling values":**
```bash
# What does the pod's nodeSelector say?
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
| yq e '.spec.nodeSelector'
# What labels do the cluster's nodes actually have?
kubectl --context=$CTX get nodes --show-labels | head -3
# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
grep -h 'dedicated:\|cloud.google.com/compute-class' \
helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head
```
If the values use `dedicated:` but the cluster only has `cloud.google.com/compute-class:` keys (or vice versa), the values were copied from a sibling cluster. **Author the values from scratch** using the cluster's own key style.
---
## §2 — Running but on the wrong node pool
The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).
| Sub-check | Action |
|-----------|--------|
| What `nodeSelector` does the pod actually have? | `kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`. |
| Where is it running? | `kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels. |
| Is the values-side `nodeSelector` too permissive? | If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect. |
The fix is to make the `nodeSelector` selective enough that only the intended pool matches. **Layer 1.**
---
## §3 — Image pull failing
```text
ErrImagePull / ImagePullBackOff
```
| Sub-check | Action |
|-----------|--------|
| Is the image pinned to Meesho's GAR mirror? | `kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's [SANCTITY_RULES R11](../../global/SANCTITY_RULES.md) violation. **Layer 1** — fix the image reference. |
| Does the tag exist in the registry? | Out-of-band check (registry UI). |
| Is the registry-pull credential present on the cluster? | `kubectl get secret -n $NS \| grep gcr-pull`. **Layer 2** if missing. |
The image tag is set in `image.repository` / `image.tag` of `custom-values.yaml`. **Layer 1.**
---
## §4 — CrashLoopBackOff
```bash
kubectl --context=$CTX logs <pod> -n $NS --previous
kubectl --context=$CTX describe pod <pod> -n $NS
```
| Pattern | Likely cause | Fix in |
|---------|--------------|--------|
| Application stack trace, missing config | App expects an env var / file that isn't there | `custom-values.yaml` config section. **Layer 1.** |
| `connection refused` to a dependency | Dependency not up; or wrong DNS | Dependency team. **Layer 2.** |
| `exit code 137` | OOMKilled — `kubectl describe` confirms | Bump `resources.limits.memory` in values. **Layer 1.** |
| `permission denied` on a file | Volume mount / `securityContext` | Values. **Layer 1.** |
| `existing Secret <X> not found` | `existingSecret:` references a secret that doesn't exist | Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. **Layer 1.** |
| Init container failed | Init logs explain | `kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`. |
---
## §5 — Running but PVC unbound
```bash
kubectl --context=$CTX get pvc -n $NS
kubectl --context=$CTX describe pvc <pvc> -n $NS
```
| Sub-check | Action |
|-----------|--------|
| `Events: Failed to provision volume with StorageClass "<X>"` | The StorageClass doesn't exist on this cluster. | `ls manifests/storageclass/<X>.yaml`. If absent, fix `persistence.storageClass:` in values to a real class. **Layer 1.** |
| `Events: ProvisioningFailed: googleapi: Error 403` | CSI driver lacks IAM permission. **Layer 2.** | Platform / IAM team. |
| PVC `Pending` with no events | StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled. | Schedule the pod (resolve §1 first). |
| PVC bound but pod can't mount | Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`). | Either set replicas to 1, or use a `StatefulSet` chart variant, or use a Filestore-backed `ReadWriteMany` class. **Layer 1.** |
---
## §6 — When this repo *is* the right place to fix
For these cases, the fix is a values diff in `helm-overrides/<cluster>/<app>/custom-values.yaml`:
1. Wrong-cluster `nodeSelector` / `tolerations` / `computeClass`.
2. Image tag pointing outside the GAR mirror.
3. `resources.requests` / `limits` mis-sized (OOMKill, throttling).
4. `persistence.storageClass` referencing a non-existent class.
5. `existingSecret:` referencing a non-existent secret.
6. `replicaCount` set on an HPA-managed release.
For these cases, the fix is **outside** this repo:
- Node pool full → cluster-owner / Terraform.
- Image not in GAR → image-mirror automation / build pipeline.
- CSI provisioning failures → platform / IAM team.
- App-internal crashes (config, dependencies) → app team.
---
## Escalation matrix
| Symptom | First responder | Escalate to |
|---------|-----------------|-------------|
| §1 — wrong scheduling values | Yourself with values fix | Cluster owner if topology is unclear |
| §1 — node pool full | Cluster owner | Platform team if quota raise needed |
| §3 — image pull (mirror miss) | Yourself with values fix | Platform if mirror push is missing |
| §4 — OOMKilled | Yourself with `resources.limits` bump | App team if root cause is leak |
| §4 — secret missing | Yourself with `ExternalSecret` add | Security if cluster-level `SecretStore` is missing |
| §5 — CSI provision failure | Platform team | — |
| §C — cluster-wide scheduling failure | Platform team — pager | — |
---
## Done conditions
- `kubectl rollout status deploy/<name> -n $NS` returns "successfully rolled out".
- `kubectl get pods -n $NS` shows N/N Ready for the expected replica count.
- Pods scheduled onto the **intended** node pool (verify `kubectl get pods -o wide` shows the right node names).
- If root cause was in this repo, the fix is on `main` and synced.
---
## Related
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
- Runbook: [ingress-down.md](ingress-down.md).
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md).
docs/platform/runbooks/vault-unavailable.md
+251
View File
@@ -0,0 +1,251 @@
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
# Runbook — Vault unavailable / External Secrets failing to render
> **Type:** Decision tree.
> **Entry symptom:** Pods are CrashLooping referencing a missing `Secret`, or `kubectl describe externalsecret` shows `SecretSyncedError`, or `kubectl get secrets <name>` returns NotFound for a name an `ExternalSecret` should be creating.
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
>
> **Important Layer 3 boundary:** **Vault HA itself is Layer 3.** Vault runs in production-only with no lower environment, so the agent must NOT attempt server-side fixes (unsealing, leader-election toggling, raft config changes, restoring from snapshot). Those are platform-team / security-team operations. The agent's role here is *diagnose, narrow root cause, and escalate*.
---
## Architecture refresher
```
workload Pod (mounts Secret <name>)
↑ created by
External Secrets Operator (ESO) controller
│
└─ reads SecretStore / ClusterSecretStore
│
├─ kind: gcpsm → GCP Secret Manager
│ └─ auth via Workload Identity (KSA → GSA binding)
└─ kind: vault → Vault HA cluster
└─ auth via Kubernetes auth (KSA token review)
```
Most clusters in the fleet use **GCP Secret Manager** as the primary backend (via Workload Identity), with Vault HA as the secondary for legacy services. Some clusters use Vault as the primary. Confirm before troubleshooting:
```bash
ls helm-overrides/<cluster>/external-secrets/
yq e '.spec.provider' helm-overrides/<cluster>/external-secrets/*.yaml 2>/dev/null
```
---
## Entry — gather context
```bash
CLUSTER=<cluster>
CTX=<kubectl-context>
NS=<namespace-of-the-failing-workload>
ES=<external-secret-name>
# ESO controller status
kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets | tail -50
# The failing ExternalSecret
kubectl --context=$CTX -n $NS describe externalsecret $ES
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.status'
# The SecretStore / ClusterSecretStore it references
STORE=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.name}')
KIND=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.kind}')
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status, .spec.provider'
```
---
## Decision tree
```text
START
│
├── §1 — Is ESO controller running and healthy?
│ ├── No → §1a — ESO outage
│ └── Yes →
│
├── §2 — Is the SecretStore / ClusterSecretStore Ready?
│ ├── No → §2a — Store config / auth broken
│ └── Yes →
│
├── §3 — Does the ExternalSecret reference a real remote key?
│ ├── No → §3a — Bad spec.data[].remoteRef.key (Layer 1 likely)
│ └── Yes →
│
├── §4 — Is the auth path working? (WI binding or Vault K8s auth)
│ ├── No → §4a — Identity binding (Layer 2/3)
│ └── Yes →
│
└── §5 — Is the upstream backend healthy?
├── GCP Secret Manager 5xx → §5a — escalate to platform
└── Vault unsealed / leader OK? → §5b — Vault outage (Layer 3)
```
---
## §1 — ESO controller health
```bash
kubectl --context=$CTX -n external-secrets get pods
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=100 | grep -iE 'error|failed|panic'
```
| Symptom | Cause | Layer | Action |
|---------|-------|-------|--------|
| Pods CrashLooping | Bad chart upgrade or RBAC misconfig | 1 | Check `helm-overrides/<cluster>/external-secrets/custom-values.yaml`. Last bump? Revert. |
| Pods Pending | Scheduling — wrong nodeSelector | 1 | See [pod-pending-scheduling.md](pod-pending-scheduling.md). |
| Pods Running, no logs about reconcile | Cluster-watch RBAC missing | 2 | Escalate. |
| `Forbidden` errors on CRD list | RBAC on the CRDs | 1 / 2 | Verify chart values' `rbac.create: true`. |
---
## §2 — SecretStore / ClusterSecretStore status
```bash
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status'
```
Look for `conditions[].status` and `conditions[].message`.
| Status / message | Cause | Layer | Action |
|------------------|-------|-------|--------|
| `Ready: False, ValidationFailed` | Store spec invalid | 1 | Fix the `SecretStore` YAML in `helm-overrides/<cluster>/external-secrets/`. |
| `Ready: False, InvalidProviderConfig` | Provider block malformed | 1 | Validate `spec.provider.gcpsm.projectID` / `spec.provider.vault.server`. |
| `Ready: False, AuthFailed` (gcpsm) | Workload Identity binding broken | 2/3 | §4 below. |
| `Ready: False, AuthFailed` (vault) | KSA token review fails | 2/3 | §4 below. |
| `Ready: True` | Store OK; problem is elsewhere | — | Continue to §3. |
---
## §3 — ExternalSecret spec validity
```bash
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.spec.data, .spec.dataFrom'
```
For each `remoteRef.key`:
- **GCP SM:** the key is the secret name in the project. Verify it exists:
```bash
gcloud secrets list --project=<gcp-project> --filter="name:<key>"
```
(Read-only — no write to GCP SM from the agent.)
- **Vault:** the key is the path under the engine. Cannot directly verify without Vault access; rely on the ESO controller's reconcile error message.
If the controller logs say `secret not found in backend` → the `remoteRef.key` is wrong. **Layer 1 fix:** correct the key in the `ExternalSecret` YAML.
---
## §4 — Auth path
### §4a — GCP Secret Manager (Workload Identity)
```bash
# The ServiceAccount the ESO controller (or this ExternalSecret's pod) runs as
kubectl --context=$CTX -n external-secrets get sa external-secrets -o yaml | yq e '.metadata.annotations'
# Should have:
# iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
```
| Sub-check | Layer | Action |
|-----------|-------|--------|
| KSA missing the `iam.gke.io/gcp-service-account` annotation | 1 | Add via `helm-overrides/<cluster>/external-secrets/custom-values.yaml` `serviceAccount.annotations`. |
| GSA exists but no IAM binding to KSA | 2 | Escalate to platform — IAM is out of repo. |
| GSA lacks `roles/secretmanager.secretAccessor` on the project | 2 | Escalate to platform. |
### §4b — Vault Kubernetes auth
| Sub-check | Layer | Action |
|-----------|-------|--------|
| `SecretStore.spec.provider.vault.auth.kubernetes.role` references a Vault role | — | Read-only; verify against existing working stores on the same cluster. |
| ESO logs say `permission denied` from Vault | 3 | The role/policy on the Vault server is wrong. **Cannot fix from this repo.** Escalate to security / Vault platform team. |
| ESO logs say `Vault is sealed` | 3 | **Vault HA outage. Do not attempt to unseal.** Escalate immediately. |
---
## §5 — Backend health
### §5a — GCP Secret Manager
GCP SM is a managed service. 5xx from it is rare and is a GCP-side incident. Action: escalate to platform; check GCP status dashboard. **No fix in this repo.**
### §5b — Vault HA
Vault HA on Meesho's fleet runs in production-only, no lower env. It is **Layer 3** — agent does not write to or operate Vault. Diagnostic *read* of pod state is OK; mutation is not.
```bash
# Diagnostic only
kubectl --context=$CTX -n vault get pods
kubectl --context=$CTX -n vault logs <vault-pod> | tail -50
# Look for: "core: Vault is sealed", "leader election", "raft", panic stacks
```
| Observed | Action |
|----------|--------|
| Some Vault pods sealed (HA quorum still up) | Escalate to security team. **Do not unseal.** |
| All Vault pods sealed (full outage) | Page security team. ESO will surface stale data only as long as the in-memory cache holds. |
| Leader-election thrashing | Escalate; could be a network/raft issue. |
| Vault pod Pending | Scheduling — see [pod-pending-scheduling.md](pod-pending-scheduling.md). Even here, restart of a Vault pod requires a security-team-led unseal afterwards. |
---
## Mitigations while Vault is down
If a workload's `ExternalSecret` is failing because Vault is unavailable, options are limited:
1. **Wait** — ESO caches the last-rendered Secret value. Pods that already mounted continue. New pod scheduling fails until Vault returns.
2. **Switch the `ExternalSecret` to GCP SM** if the same secret exists there (most do, with Vault as legacy). Layer 1 PR — change `secretStoreRef.name` to the GCP SM store. **Coordinate with security** before doing this in an outage.
3. **Hand-create the Secret as a temporary `kubectl apply`** — out of scope for the agent. This is incident response by a human operator.
The agent should **not** auto-cut option 2 without explicit human approval — switching the source of truth for a secret has security implications.
---
## Escalation matrix
| Symptom | First responder | Escalate to |
|---------|-----------------|-------------|
| §1 (ESO down) | DevOps on-call | Platform team |
| §2 (Store config) | DevOps on-call | Layer 1 PR + reviewer |
| §3 (Bad remoteRef) | DevOps on-call | Layer 1 PR + reviewer |
| §4a (WI binding) | Platform team | — |
| §4b (Vault role/policy) | Security team | — |
| §5a (GCP SM 5xx) | Platform team | GCP support |
| §5b (Vault outage) | **Security team — page** | — |
---
## Done conditions
- `ExternalSecret` `status.conditions[type=Ready].status == True`.
- The downstream `Secret` exists with expected keys.
- Workload pods consuming the Secret are Running.
- ESO logs are quiet for at least 5 minutes after the fix.
---
## What this repo can and cannot fix
| Fix kind | Layer | This repo? |
|----------|-------|-----------|
| `ExternalSecret` / `SecretStore` YAML edits | 1 | Yes — values PR. |
| ESO chart values (sizing, RBAC, metrics) | 1 | Yes. |
| Workload-Identity KSA annotation | 1 | Yes (in chart values' `serviceAccount.annotations`). |
| GSA IAM bindings on GCP | 2 | No — platform team. |
| Vault server-side role/policy | 3 | **Never** — security team. |
| Vault unseal / raft / leader | 3 | **Never** — security team. |
---
## Related
- Schema: [../schemas/raw-manifest-sidecar-schema.md §ExternalSecret](../schemas/raw-manifest-sidecar-schema.md).
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — if any of the above pods are Pending.
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if the `ExternalSecret` itself didn't sync.
- Boundaries: [../../global/AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md), [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md).
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
docs/platform/schemas/custom-values-schema.md
+379
View File
@@ -0,0 +1,379 @@
# Schema — `custom-values.yaml` (Helm override)
> Field-by-field annotation of the values file Argo CD's `valueFiles` references.
> Authoritative for `helm-overrides/<cluster>/<app>/custom-values.yaml`.
>
> Cross-reference: [coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md), [../../architecture.md](../../architecture.md).
---
## What this file is — and isn't
A `custom-values.yaml` is **not** a Helm chart. It is the *override* layer Argo CD merges over the chart's own `values.yaml` (or, for thin-wrapper charts, the upstream subchart's defaults).
- **Schema is owned by the chart, not by us.** Every key here must be a key the chart understands. Adding a key the chart doesn't read does nothing.
- **No multi-doc YAML.** One document per file.
- **No `apiVersion` / `kind`.** This file is values, not a Kubernetes manifest.
---
## Top-level shape (typical)
```yaml
# image registry / repository / tag
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/<image>
tag: <semver-or-sha>
pullPolicy: IfNotPresent
# release-name pinning (rarely changed once set)
fullnameOverride: <release>-<bu>-prd # e.g. kube-state-metrics-dbc-dsci-prd
# scaling
replicaCount: 3 # static, OR
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# scheduling — bespoke per cluster
nodeSelector:
dedicated: <pool-key> # standard GKE
# cloud.google.com/compute-class: <cc> # GKE Autopilot
tolerations:
- key: dedicated
value: <pool-key>
effect: NoSchedule
# resources
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1
memory: 2Gi
# persistence (stateful charts only)
persistence:
enabled: true
storageClass: sc-pd-ssd # MUST exist in manifests/storageclass/
size: 100Gi
accessModes: [ReadWriteOnce]
# secrets — by reference, never inline
existingSecret: <name-managed-by-external-secrets>
# probes
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
initialDelaySeconds: 5
periodSeconds: 5
# ingress / Contour HTTPProxy (if applicable)
ingress:
enabled: false # most internal services
# chart-specific top-level keys
# (varies — read the chart's values.yaml for the full schema)
```
---
## §image
| Field | Required | Convention | Notes |
|-------|----------|------------|-------|
| `image.registry` | yes (production) | `asia-southeast1-docker.pkg.dev` | Meesho's Artifact Registry mirror. ([SANCTITY_RULES R11](../../global/SANCTITY_RULES.md)) |
| `image.repository` | yes | `meesho-devops-admin-0622/admin/sre/<image>` | Mirror path. |
| `image.tag` | yes (production) | semver, datestamp, or git SHA | Never `latest`. Never unpinned. |
| `image.pullPolicy` | no | `IfNotPresent` | `Always` only for development; pulls slow rollout. |
| `image.pullSecrets` | no | omit | Mirror is public to fleet; pull secrets are a lock-in trap. |
Note: chart authors disagree on the shape — some use `image: {repository, tag}` (no `registry`), some use a single `image: <fully-qualified>`, some split into `image.repository: <registry>/<repo>`. **Read the chart's own `values.yaml`** before structuring this block.
---
## §release identity
| Field | Required | Convention | Notes |
|-------|----------|------------|-------|
| `fullnameOverride` | varies | Set once at release creation; stable forever | Service DNS, PVC binding, ConfigMap refs depend on it. ([SANCTITY_RULES R9](../../global/SANCTITY_RULES.md)) |
| `nameOverride` | rare | Omit unless explicitly needed | |
| `commonLabels` / `commonAnnotations` | rare | Omit unless the chart documents the pattern | |
For dataplane (`db-*`) clusters: `fullnameOverride: <kind>-dbc-<bu>-prd` is the convention (e.g. `kube-state-metrics-dbc-dsci-prd`). For BU clusters, omit unless the chart's default name collides.
---
## §scaling
Charts diverge sharply here. Common shapes:
```yaml
# Pattern A — static replicas
replicaCount: 3
# Pattern B — chart-level autoscaling block
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# Pattern C — separate HPA resource (KEDA / VictoriaMetrics)
hpa:
enabled: true
min: 3
max: 20
metrics:
- type: Resource
resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}
```
| Rule | Why |
|------|-----|
| **Don't set `replicaCount`** when `autoscaling.enabled: true`. | HPA and the chart's static deployment fight; replica count flaps. |
| **Set both `minReplicas` and `maxReplicas`** when autoscaling. | Without `min`, the HPA can scale to zero on a quiet hour. |
| **`minReplicas` ≥ 2** for any production-traffic-path workload. | Single-replica releases die on node drain. |
---
## §scheduling
The single biggest source of silent mis-deploys. Per [SANCTITY_RULES R5](../../global/SANCTITY_RULES.md):
| Cluster | Key style |
|---------|-----------|
| GKE Autopilot (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `cloud.google.com/compute-class:` |
| Standard GKE | `dedicated:` |
```yaml
# Standard GKE — Contour internal-0 on most clusters
nodeSelector:
dedicated: contour-internal-0
tolerations:
- key: dedicated
value: contour-internal-0
effect: NoSchedule
# often a second toleration to allow scheduling onto shared nodes
- key: dedicated
value: contour-shared
effect: NoSchedule
# GKE Autopilot — same Contour internal-0 on k8s-central-prd-ase1
nodeSelector:
cloud.google.com/compute-class: contour-internal-0-cc
tolerations:
- key: cloud.google.com/compute-class
value: contour-internal-0-cc
effect: NoSchedule
- key: cloud.google.com/compute-class
value: contour-shared-cc
effect: NoSchedule
```
The full per-cluster Contour matrix lives in [`contour-nodeselector-tolerations-summary.md`](../../../contour-nodeselector-tolerations-summary.md). For non-Contour apps, copy from a sibling app on the **same** cluster.
| Field | Notes |
|-------|-------|
| `nodeSelector` | Hard requirement — no scheduling onto non-matching nodes. Use exactly one key (the cluster's pool key). |
| `tolerations` | Allow scheduling onto tainted nodes. May list multiple to allow shared nodes alongside dedicated. |
| `affinity.nodeAffinity` | Use sparingly; `nodeSelector` is sufficient for most cases here. |
| `topologySpreadConstraints` | Use for multi-zone resilience on chatty workloads (Contour, ingress-nginx). |
---
## §resources
```yaml
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1
memory: 2Gi
```
| Rule | Why |
|------|-----|
| **Always set `requests`** in production. | Without requests, pods are best-effort; first to evict on memory pressure. |
| **Set `limits` unless the chart explicitly recommends omitting them.** | Some sidecars (cert-manager, external-dns) deliberately skip `limits`. |
| **Don't copy resources from another cluster.** | Workload sizing is per-traffic-tier. Supply prd is not demand prd. |
| **CPU limits cause throttling**, not OOMKill. Memory limits cause OOMKill. Tune accordingly. |
---
## §persistence
```yaml
persistence:
enabled: true
storageClass: sc-pd-ssd
size: 100Gi
accessModes: [ReadWriteOnce]
```
Allowed `storageClass` values (must match a file under `manifests/storageclass/`):
| Class | Backend | Use case |
|-------|---------|----------|
| `pd-standard-retain-dr` | GCP Persistent Disk standard, retention-on-delete | DR-critical state |
| `sc-pd-ssd` | GCP PD SSD | Default for write-heavy workloads (etcd, ClickHouse) |
| `sc-pd-standard` | GCP PD standard | Default for read-heavy / archival |
| `sc-filestore-standard` | GCP Filestore | Shared volumes (Jenkins build cache, JFrog filestore) |
| Rule | Why |
|------|-----|
| **Never reference a `storageClass`** that doesn't exist in `manifests/storageclass/`. | PVC stays Pending forever. |
| **`size` must be set explicitly.** | Chart defaults are often wrong (8Gi for everything). |
| **`accessModes`** for `ReadWriteMany` requires Filestore. PD-backed classes are RWO. |
---
## §secrets
```yaml
# Chart-specific — read the chart's values.yaml
existingSecret: <name>
# OR
auth:
existingSecret: <name>
# OR
envFrom:
- secretRef: {name: <name>}
```
| Rule | Why |
|------|-----|
| **Never inline `password:`, `apiKey:`, etc.** | Pre-commit TruffleHog will catch many; it won't catch all. ([SANCTITY_RULES R4](../../global/SANCTITY_RULES.md)) |
| **The `Secret` resource is materialised** by the cluster's `external-secrets` app. If the secret name doesn't appear in `helm-overrides/<cluster>/external-secrets/`, it doesn't exist. |
| **Reference, don't author.** Adding the secret material to a Helm-rendered `Secret` template defeats the External Secrets pattern. |
---
## §probes
```yaml
livenessProbe:
httpGet: {path: /healthz, port: http}
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
readinessProbe:
httpGet: {path: /ready, port: http}
initialDelaySeconds: 5
periodSeconds: 5
# For slow-warm workloads (Jenkins, JFrog, ClickHouse)
startupProbe:
httpGet: {path: /healthz, port: http}
initialDelaySeconds: 60
periodSeconds: 10
failureThreshold: 60 # 10 minutes total runway
```
| Rule | Why |
|------|-----|
| **Always set `liveness` + `readiness`** for any long-running container. |
| **Use `startupProbe` for slow-warm workloads** instead of inflating `livenessProbe.initialDelaySeconds` to 600 s. |
| **Don't set the same probe to both `liveness` and `readiness`** — they have different purposes (kill vs gate). |
---
## §ingress (Contour `HTTPProxy` / `Ingress`)
For external-facing apps, ingress is via Contour `HTTPProxy`. Specifics depend on the cluster's Contour topology (multiple Contour instances per cluster — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md)).
- The `external-dns` annotation is the DNS-binding step — match the cluster's DNS pattern.
- TLS terminates at Contour with `cert-manager`-issued certs; reference the issuer by name.
- Internal services use `contour-internal-0` / `contour-internal-1`; external services use `contour-external` / `contour-external-1`. The class is set via the `Service` annotation, not the values file directly.
---
## End-to-end example — a typical observability-side override
`helm-overrides/k8s-supply-prd-ase1/kube-state-metrics/custom-values.yaml`:
```yaml
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/kube-state-metrics
tag: v2.10.1
replicaCount: 2
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi
nodeSelector:
dedicated: monitoring
tolerations:
- key: dedicated
value: monitoring
effect: NoSchedule
prometheus:
monitor:
enabled: true
honorLabels: true
```
---
## End-to-end example — a dataplane override
`helm-overrides/db-2516183257845181-c-1204-195038-428/victoria-metrics-agent/custom-values.yaml`:
```yaml
fullnameOverride: vmagent-dbc-supply-prd
image:
registry: asia-southeast1-docker.pkg.dev
repository: meesho-devops-admin-0622/admin/sre/victoria-metrics-agent
tag: v1.96.0
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 256Mi
```
(Dataplane clusters intentionally have minimal overrides.)
---
## Validation
```bash
# Render the chart locally with this values file
helm template <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -80
# Dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
helm diff upgrade <release> helm-templates/<chart> \
-f helm-overrides/<cluster>/<app>/custom-values.yaml
# Lint
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
helm lint helm-templates/<chart>
# Sanity: storageClass referenced exists
yq e '.persistence.storageClass' helm-overrides/<cluster>/<app>/custom-values.yaml \
| xargs -I{} ls manifests/storageclass/{}.yaml 2>/dev/null \
|| echo "WARN: storageClass not in manifests/storageclass/"
```
docs/platform/schemas/incubator-values-schema.md
+195
View File
@@ -0,0 +1,195 @@
> Per AI Blitz Plan §platform.schemas. Layer: 1. Repo: devops-infra-helm-charts.
# Schema — Incubator-tool values + sidecar contract
> **Scope:** the values-side contract for incubator (newly-vendored) infrastructure tools that live under `helm-overrides/<cluster>/<incubator-tool>/`.
>
> **Out of scope (explicit):** the Argo CD `Application` / `ApplicationSet` manifest that registers the incubator tool with a cluster's Argo. Those manifests live in the **sister repo** `github.com/Meesho/devops-infra-argo-config`, NOT here. This file documents only what `devops-infra-helm-charts` is responsible for: the values + raw sidecar manifests Argo CD reads from this repo.
This schema complements [custom-values-schema.md](custom-values-schema.md) (general values shape) and [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) (sidecar manifest shapes). Read both first.
---
## What "incubator" means here
A tool is **incubator** while it is being trialled on one or two clusters before fleet-wide rollout. In this repo that maps to:
- A new chart directory under `helm-templates/<incubator-tool>/` (often a thin wrapper `Chart.yaml` with an upstream dep).
- An override under `helm-overrides/<cluster>/<incubator-tool>/` for the trial cluster(s) only — typically `k8s-shared-int-ase1` (integration / pre-prod) first, sometimes one BU prod cluster as a canary.
- Possibly raw sidecar manifests alongside the values file.
Once the tool graduates from incubator status, the override pattern is identical to other infra tools — the "incubator" label is operational, not structural. This schema codifies the conventions that keep early-stage adoption sane.
---
## Directory layout
```
helm-overrides/<cluster>/<incubator-tool>/
├── custom-values.yaml # Helm values (required if chart is Helm-based)
├── computeclass/ # Optional — GKE Autopilot ComputeClass
│ └── <name>-cc.yaml
├── external-dns-services/ # Optional — Service for external-dns annotation
│ └── <fqdn>.yaml
├── external-secrets/ # Optional — ExternalSecret resources
│ └── <name>.yaml
└── <incubator-specific>.yaml # Tool-specific raw manifest (CRD, ConfigMap)
```
The Argo `Application` for this directory (sister repo) determines whether files are Helm-rendered, raw-applied, or layered (see [raw-manifest-sidecar-schema.md §Layered-with-Helm vs standalone](raw-manifest-sidecar-schema.md)).
---
## `custom-values.yaml` keys an incubator chart should expose
A chart promoted to incubator status MUST surface (i.e. let the override file set without forking templates) at minimum:
| Key | Why required |
|-----|--------------|
| `image.registry`, `image.repository`, `image.tag` | Production overrides pin to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho Artifact Registry mirror, not Docker Hub). |
| `image.pullPolicy` | Default `IfNotPresent`. |
| `resources.requests.cpu`, `.memory` | No defaults assumed; Autopilot scheduling requires explicit requests. |
| `resources.limits.cpu`, `.memory` | Same. |
| `nodeSelector` | Per-cluster scheduling (see [custom-values-schema.md](custom-values-schema.md)). |
| `tolerations` | Per-cluster taints. |
| `affinity` (optional) | Pod anti-affinity for replicated tools. |
| `serviceAccount.create`, `.name`, `.annotations` | Workload Identity — `iam.gke.io/gcp-service-account` annotation goes here. |
| `replicaCount` (or `autoscaling.{enabled,minReplicas,maxReplicas}`) | Sizing. |
| `persistence.enabled`, `.storageClass`, `.size` | If stateful. `storageClass` MUST exist in `manifests/storageclass/`. See [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md). |
| `priorityClassName` | If the tool needs preemption priority — must reference a class in `manifests/priorityclass/<cluster>/`. |
| `podLabels`, `podAnnotations` | For prometheus.io scrape annotations and team-attribution labels. |
| `extraEnv` (or `env`) | For environment-specific knobs the chart's templates don't already accept. |
If the upstream chart doesn't expose these — that's a chart-bug. **Do not** fork the chart in `helm-templates/` to add them ([NEVER-DO list](../../../CLAUDE.md)). Either upstream-PR the chart or wrap with a small Meesho-owned chart that re-exposes the keys.
---
## Values-file conventions for incubator tools
1. **Pin every image tag.** Never `latest`, never an unpinned SHA. This is the same rule as production overrides; incubator status does not relax it.
2. **Set explicit `nodeSelector` and `tolerations`** authored from scratch using sibling apps on the same cluster. Cross-cluster copying is forbidden — see [custom-values-schema.md](custom-values-schema.md).
3. **Use `fullnameOverride` deliberately or not at all.** Once set, do not change ([NEVER-DO list](../../../CLAUDE.md)). Incubator graduations to other clusters should reuse the same `fullnameOverride` string for portability.
4. **Don't enable `autoscaling`** in the first incubator deploy. Pin `replicaCount: 1` (or 2 for HA-mandatory) until you have a load profile.
5. **Don't expose the tool externally** in the incubator phase. No `external-dns-services/` until it's promoted to a cluster's stable inventory.
6. **Annotate the values file** with a top-of-file YAML comment: `# Incubator: cluster=<...>, owner=<...>, graduation-target=<date>`. Comment is plain text; not parsed; serves as reviewer signal.
7. **Confine secrets to `ExternalSecret`** under `external-secrets/`. No inline `existingSecret:` referencing a hand-applied Secret.
---
## Raw sidecar manifests in incubator directories
Concrete shapes already documented elsewhere; this section calls out which ones routinely appear with incubators.
### `computeclass/*-cc.yaml` (GKE Autopilot only)
Required when the override's `nodeSelector` uses `cloud.google.com/compute-class: <name>` and that class doesn't already exist on the cluster. The `metadata.name` MUST equal the value referenced. See [raw-manifest-sidecar-schema.md §ComputeClass](raw-manifest-sidecar-schema.md).
Example incubator pattern:
```yaml
apiVersion: autoscaling.gke.io/v1
kind: ComputeClass
metadata:
name: <incubator-tool>-cc
namespace: <ns>
spec:
priorities:
- machineFamily: c4d
minCores: 2
minMemory: 8Gi
nodePoolAutoCreation:
enabled: true
```
Per-cluster only — never copy between clusters.
### `external-dns-services/*.yaml`
Skip in the first incubator deploy. Only add once the tool has a stable internal-only DNS need. See [raw-manifest-sidecar-schema.md §Service for external-dns binding](raw-manifest-sidecar-schema.md).
### `external-secrets/*.yaml`
Always present if the tool needs secrets. Use the existing per-cluster `SecretStore` / `ClusterSecretStore`; do not author new stores in an incubator PR. See [raw-manifest-sidecar-schema.md §ExternalSecret](raw-manifest-sidecar-schema.md).
### `elastic-cluster/argo-launch.yaml`-style operator-managed CRD launches
If the incubator tool is an operator (e.g. ECK, Pyroscope), a sidecar CRD instance often lives alongside the operator's chart values:
```
helm-overrides/<cluster>/<incubator-operator>/
├── custom-values.yaml # operator chart values
└── <crd-instance>/
└── argo-launch.yaml # the actual workload CRD instance
```
Argo CD applies both in one Application (directory loader). The CRD instance must:
- Reference an `apiVersion` whose CRD the operator has already installed.
- Pin its own image / version explicitly.
- Avoid hand-rolling values that the operator templates would otherwise compute.
---
## Argo CD — sister repo contract (for cross-reference only)
The Application that points at this directory lives in `github.com/Meesho/devops-infra-argo-config`. For incubator tools the Application typically:
- Has `syncPolicy.automated` **disabled** (manual sync — incubator-grade safety; see [argocd.md](../../global/coding-guidelines/argocd.md)).
- Is named `<incubator-tool>-<cluster>` to be unambiguous.
- Lives in the cluster's namespace under `apps/<cluster>/`.
This file does NOT instruct on authoring the Application — that PR is in the sister repo. Cross-link the sister-repo PR in the values-side PR description.
---
## Validation
Same as any Helm override. Before opening a PR:
```bash
yamllint helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml
helm template <release> helm-templates/<incubator-tool> \
-f helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml > /tmp/render.yaml
# Validate any sidecar manifests against the cluster's CRDs
kubectl --context=<ctx> --dry-run=server \
-f helm-overrides/<cluster>/<incubator-tool>/<sidecar>.yaml apply
```
For tool-specific CRDs that aren't in vanilla Kubernetes, prefer `--dry-run=server` (the cluster validates against the registered CRD) over `--dry-run=client` (only client-side schema, often misses required fields).
---
## Graduation: incubator → stable
When the tool has been stable for some period (usually 2–4 weeks) and is ready to fleet-roll:
1. Remove the top-of-file `# Incubator:` comment.
2. Open new override directories in target clusters — author from scratch each time, do not copy.
3. Open the matching sister-repo `Application` PRs (one per new cluster) or extend the `ApplicationSet`.
4. If the chart was a thin wrapper, audit `Chart.yaml` `dependencies[].version` is current.
5. Promote the chart's documentation in `helm-templates/<incubator-tool>/README.md` (if a fork was needed) or note in PR description that no fork was needed.
---
## Anti-patterns
1. **Forking the chart in `helm-templates/<incubator-tool>/templates/`** to add a missing values key. Wrap or upstream-PR; don't fork. See NEVER-DO.
2. **Copying the entire incubator directory across clusters** for graduation. Per-cluster scheduling is bespoke.
3. **Enabling autoscaling on day one.** No load profile = thrashy autoscaler.
4. **Promoting from `k8s-shared-int-ase1` straight to a critical BU cluster.** Insert a low-traffic prod canary first.
5. **Inlining secrets** "just for the trial." TruffleHog will block; even if it didn't, the leak is real.
6. **Setting `fullnameOverride`** without a graduation plan. The string travels — bad ones travel forever.
---
## Related
- Schema: [custom-values-schema.md](custom-values-schema.md) — general values shape.
- Schema: [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) — sidecar shapes.
- Schema: [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md) — cluster-singleton resources.
- Procedure: [../procedures/onboard-app-to-cluster.md](../procedures/onboard-app-to-cluster.md) — the procedure used to land an incubator override.
- Procedure: [../procedures/fork-upstream-chart.md](../procedures/fork-upstream-chart.md) — only when a fork is genuinely required.
- Coding guideline: [../../global/coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md).
- Coding guideline: [../../global/coding-guidelines/argocd.md](../../global/coding-guidelines/argocd.md) — sister-repo Application conventions.
docs/platform/schemas/raw-manifest-sidecar-schema.md
+191
View File
@@ -0,0 +1,191 @@
# Schema — Raw-manifest sidecars in `helm-overrides/`
> Field-by-field guidance for `helm-overrides/<cluster>/<app>/<extra>.yaml` — Kubernetes manifests applied **alongside** a Helm release, not through it.
---
## What this is
Some Argo CD Applications point at a directory containing both a Helm `custom-values.yaml` *and* one or more raw Kubernetes manifests. The raw manifests are not Helm-rendered; Argo CD applies them as-is.
Common shapes seen in this repo:
| Path pattern | What it is |
|--------------|------------|
| `helm-overrides/<cluster>/<app>/computeclass/*-cc.yaml` | GKE Autopilot `ComputeClass` resource — declares a node-pool/compute-class profile referenced by `nodeSelector` in the Helm values. |
| `helm-overrides/<cluster>/<app>/external-dns-services/<svc>.yaml` | A `Service` that exists purely to carry an `external-dns.alpha.kubernetes.io/hostname` annotation, binding a DNS name to a workload. |
| `helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml` | An `ElasticCluster` (ECK CRD) launched alongside the operator. |
| `helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml` | Inlined Alertmanager config materialised as a `ConfigMap`. |
If the sister-repo `Application` for this directory has `path: helm-overrides/<cluster>/<app>/`, then **every YAML file in the directory is applied** — Argo CD's directory loader treats them as a single deployment unit.
---
## File-level conventions
| Rule | Why |
|------|-----|
| **One Kubernetes resource per file** unless they're tightly coupled (e.g. a `Service` + a `ServiceAccount` referenced by it). | Reviewer cognition; rollback granularity. |
| **Pin `apiVersion` explicitly.** | `extensions/v1beta1` and `networking.k8s.io/v1beta1` are real footguns — both are gone in current Kubernetes. |
| **Set `metadata.namespace` explicitly.** | Don't rely on the Argo `Application.spec.destination.namespace` carrying through — different cluster Argos behave differently here. |
| **Trailing newline at EOF.** |
| **No multi-doc (`---`) within one file** unless genuinely required. |
---
## Top-level shape
```yaml
apiVersion: <api-version> # e.g. v1, networking.k8s.io/v1, autoscaling.gke.io/v1
kind: <kind> # e.g. Service, ComputeClass, ConfigMap, ExternalSecret
metadata:
name: <name>
namespace: <namespace>
labels: {...} # optional
annotations: {...} # often the load-bearing field (DNS, LB)
spec:
... # kind-specific
```
---
## Common kinds in this repo
### `ComputeClass` (`autoscaling.gke.io/v1`)
GKE Autopilot uses `ComputeClass` resources to declare named node profiles. The Helm values reference them via `cloud.google.com/compute-class:` keys.
```yaml
apiVersion: autoscaling.gke.io/v1
kind: ComputeClass
metadata:
name: contour-internal-0-cc
namespace: projectcontour
spec:
priorities:
- machineFamily: c4d
minCores: 4
minMemory: 16Gi
nodePoolAutoCreation:
enabled: true
```
| Rule | Why |
|------|-----|
| **`metadata.name` MUST match the value referenced by `nodeSelector` in the Helm values.** | Otherwise pods stay `Pending`. |
| **Per-cluster.** A `ComputeClass` for `k8s-central-prd-ase1` does not exist on `k8s-supply-prd-ase1`. Don't share files. |
| **The matrix of which `ComputeClass` exists where** is recorded in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). |
### `Service` for `external-dns` binding
```yaml
apiVersion: v1
kind: Service
metadata:
name: <hostname>-dns
namespace: <ns>
annotations:
external-dns.alpha.kubernetes.io/hostname: <fqdn>.meeshogcp.in
cloud.google.com/load-balancer-type: Internal
spec:
type: ClusterIP
selector: {app.kubernetes.io/name: <app>}
ports:
- port: 80
targetPort: 8080
```
| Rule | Why |
|------|-----|
| **`metadata.annotations.external-dns.alpha.kubernetes.io/hostname`** is the contract. Check the cluster's existing DNS pattern before authoring. |
| **`selector` must match labels of an actual workload Pod** in the same namespace. |
| **Cloud-LB annotations** (`cloud.google.com/load-balancer-type`, etc.) — copy from a sibling on the same cluster. |
### `ExternalSecret` (per-cluster `external-secrets/`)
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: <secret-name>
namespace: <ns>
spec:
refreshInterval: 1h
secretStoreRef:
name: gcp-sm-store # cluster-level SecretStore, defined elsewhere
kind: ClusterSecretStore
target:
name: <secret-name>
creationPolicy: Owner
data:
- secretKey: <env-key>
remoteRef:
key: <gcp-secret-name>
version: latest
```
| Rule | Why |
|------|-----|
| **`spec.target.name` is the `Secret` name** the workload's `existingSecret:` references. |
| **`spec.secretStoreRef.name`** must reference a `SecretStore` / `ClusterSecretStore` that already exists on the cluster. |
| **`creationPolicy: Owner`** is the default; `creationPolicy: Merge` is for the rare case where another tool also manages the same `Secret`. |
### `ConfigMap` for inlined config
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: <name>
namespace: <ns>
data:
<key>: |
<multi-line content>
```
| Rule | Why |
|------|-----|
| **Use `|` (literal block scalar)** for multi-line content with significant whitespace. |
| **Don't inline secrets** (TruffleHog will catch obvious ones; subtle ones slip through). |
| **Reference from the workload's chart values** via the `ConfigMap` name; don't duplicate config across files. |
### `ElasticCluster` / other CRDs
For ECK, Pyroscope, etc. — the schema is the operator's, not Kubernetes's. **Read the operator's CRD docs** before authoring; copy a sibling cluster's existing `argo-launch.yaml` first.
---
## Layered-with-Helm vs standalone
| Pattern | Argo Application points at |
|---------|----------------------------|
| Pure Helm | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` |
| Pure raw manifests | `path: helm-overrides/<cluster>/<app>/` with `directory.recurse: true`, no `helm:` block |
| Layered | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` AND extra files alongside |
Argo CD's behaviour for layered directories depends on its `directory.include` / `directory.exclude` settings — when in doubt, read the sister-repo `Application` to see exactly what gets picked up.
---
## Validation
```bash
# Validate the manifest against its API
kubectl --dry-run=client -f helm-overrides/<cluster>/<app>/<extra>.yaml apply
# Lint
yamllint helm-overrides/<cluster>/<app>/<extra>.yaml
# Confirm the Argo Application loads this path
# (read the matching file in github.com/Meesho/devops-infra-argo-config)
```
---
## Anti-patterns
1. **Inlining secrets** in a `ConfigMap` "to ship a hotfix." It's still a secret. Use `ExternalSecret`.
2. **Using `apiVersion: v1beta1`** of a CRD whose stable version exists. Pin to the highest stable.
3. **A `Service` whose `selector` doesn't match any Pod** — silent: the Service exists, DNS resolves, no endpoints.
4. **Cross-cluster cloning** of a `ComputeClass` or DNS `Service` without rewriting cluster-specific fields.
5. **Hand-rendered Helm output** dropped into a sidecar file (a release "freeze"). The chart bumps and your snapshot rots.
docs/platform/schemas/storageclass-priorityclass-schema.md
+158
View File
@@ -0,0 +1,158 @@
# Schema — `manifests/storageclass/` and `manifests/priorityclass/`
> Cluster-wide singletons. **High blast radius — every PVC / scheduling decision in the cluster is affected.**
>
> Cross-reference: [SANCTITY_RULES R10](../../global/SANCTITY_RULES.md), [coding-guidelines/helm-values.md §persistence](../../global/coding-guidelines/helm-values.md).
---
## §StorageClass (`manifests/storageclass/*.yaml`)
These are **repo-global** — one StorageClass file is applied to every cluster that consumes it. A wrong reclaim policy or volume-binding mode breaks every new PVC.
### Inventory
| File | Backend | Reclaim | Binding | Use case |
|------|---------|---------|---------|----------|
| `pd-standard-retain-dr.yaml` | GCP PD standard | `Retain` | `WaitForFirstConsumer` | DR-critical state — do not lose data on PVC delete |
| `sc-pd-ssd.yaml` | GCP PD SSD | `Delete` | `WaitForFirstConsumer` | Default for write-heavy (etcd, ClickHouse) |
| `sc-pd-standard.yaml` | GCP PD standard | `Delete` | `WaitForFirstConsumer` | Default for read-heavy / archival |
| `sc-filestore-standard.yaml` | GCP Filestore | varies | varies | Shared `ReadWriteMany` volumes (Jenkins build cache, JFrog filestore) |
### Schema
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sc-pd-ssd # MUST match filename
annotations:
storageclass.kubernetes.io/is-default-class: "false" # exactly one default per cluster
provisioner: pd.csi.storage.gke.io
parameters:
type: pd-ssd
reclaimPolicy: Delete # Delete | Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```
| Field | Convention | Notes |
|-------|------------|-------|
| `metadata.name` | matches filename (without `.yaml`) | Helm values reference by name. |
| `metadata.annotations."storageclass.kubernetes.io/is-default-class"` | `"false"` for all of these | Exactly one StorageClass should be `"true"` per cluster (declared elsewhere, often by Terraform/cluster-bootstrap). |
| `provisioner` | `pd.csi.storage.gke.io` (PD) or `filestore.csi.storage.gke.io` (Filestore) | GCP CSI drivers. |
| `parameters.type` | `pd-ssd`, `pd-standard`, `pd-balanced` | Cost/perf trade-off. |
| `reclaimPolicy` | `Delete` (default) or `Retain` (DR) | **Changing this on an existing StorageClass does not retroactively change PVCs.** |
| `volumeBindingMode` | `WaitForFirstConsumer` | Avoids zone-mismatch on multi-zone clusters. `Immediate` only for `ReadWriteMany`. |
| `allowVolumeExpansion` | `true` | Required for online resizing. Default false on some classes — set explicitly. |
### Hard rules
1. **Never delete a StorageClass referenced by an existing PVC.** New PVCs against the deleted class fail; existing bound PVCs survive but lose the ability to expand.
2. **Never change `reclaimPolicy` from `Delete` → `Retain`** as a "safety improvement" without auditing every PVC. Existing bound PVs keep their old policy; new PVs get the new policy. Drift.
3. **Never change `provisioner`** — that's a delete-and-recreate, not an edit. Existing PVs become orphaned.
4. **Never make a new StorageClass the cluster default** without coordinating with the platform team. A wrong default class hijacks every PVC that doesn't pin a class explicitly.
5. **Validation:** every StorageClass referenced by any `helm-overrides/*/persistence.storageClass:` must have a corresponding file here.
```bash
# Find every PVC reference
grep -rE 'storageClass(Name)?:' helm-overrides | sort -u
# Find every StorageClass file
ls manifests/storageclass/*.yaml
```
---
## §PriorityClass (`manifests/priorityclass/<cluster>/*.yaml`)
**Per-cluster** but **cluster-wide** — affects scheduling priority for every pod that references the class.
### Inventory pattern
```
manifests/priorityclass/
k8s-central-prd-ase1/
priorityclass-high.yaml
priorityclass-low.yaml
k8s-supply-prd-ase1/
priorityclass-high.yaml
priorityclass-low.yaml
…
```
Most BU clusters get a `high` and a `low` class. Specialty clusters (`mqkafka`, `dsgpu`, dataplane `db-*`) sometimes have additional tiers.
### Schema
```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000 # higher = more important
globalDefault: false # NEVER true on these — would clash with system defaults
description: "High-priority workloads — promoted ahead of general pods on resource pressure"
preemptionPolicy: PreemptLowerPriority # default; alternative is Never
```
| Field | Convention | Notes |
|-------|------------|-------|
| `metadata.name` | `high-priority`, `low-priority`, etc. | Workload pods reference by name. |
| `value` | High: 1,000,000 / Low: 100 / Critical (rare): >1,000,000,000 | Kubernetes system pods reserve `> 2,000,000,000`; don't conflict. |
| `globalDefault` | **always `false`** | A `true` here applies to every pod that doesn't pin a class — chaos. |
| `description` | one-line | For audit log. |
| `preemptionPolicy` | `PreemptLowerPriority` (default) or `Never` | `Never` for batch workloads that shouldn't kick others out. |
### Hard rules
1. **Never set `globalDefault: true`** on any PriorityClass here. The cluster's implicit default is what we want.
2. **Never raise `value`** of an existing class without auditing what gets preempted. A bump from 1,000,000 → 10,000,000 changes which pods get evicted under pressure.
3. **Never delete a PriorityClass** referenced by any workload — pods that referenced it become invalid.
4. **The same `name` must mean the same thing across clusters.** `high-priority` on supply ≠ `high-priority` on demand at the *value* level is an audit nightmare. Keep values consistent.
5. **Validation:** find every pod-spec reference:
```bash
grep -rE 'priorityClassName:' helm-overrides
```
---
## §The "PV/PVC singletons" — Jenkins / JFrog filestore
Beyond StorageClass and PriorityClass, `manifests/` also holds **per-env, one-shot** PV/PVC pairs:
```
manifests/jenkins-filestore-caching/
dev/{pv,pvc}.yaml
prd/{pv,pvc}.yaml
manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml
manifests/jfrog-filestore-data/
dev/...
prd/...
```
These are **already-created PVs being adopted** — typically because the underlying disk (Filestore mount, GCS bucket) was provisioned by Terraform or by hand. The PV file binds the existing disk to the cluster; the PVC file binds a workload to the PV.
| Rule | Why |
|------|-----|
| **Don't change `spec.csi.volumeHandle` / `spec.gcePersistentDisk.pdName`** without coordinating with Terraform. The handle is the disk identity. |
| **Don't change `spec.persistentVolumeReclaimPolicy`** from `Retain` to `Delete` on these. The disk has data on it. |
| **Don't move dev→prd or vice versa** — they reference different physical disks. |
| **Treat as platform-team review.** Any change here is a 1:1 disk operation. |
---
## Validation script
```bash
# StorageClass: every name in helm-overrides has a file here
for sc in $(grep -rhE 'storageClass(Name)?:' helm-overrides \
| sed -E 's/.*storageClass(Name)?:\s*//' \
| tr -d '"' | sort -u); do
[ -f "manifests/storageclass/${sc}.yaml" ] || echo "MISSING: $sc"
done
# PriorityClass: every name referenced has a file in the matching cluster
grep -rE 'priorityClassName:' helm-overrides
# (cross-reference manually against manifests/priorityclass/<cluster>/)
```