added repo
This commit is contained in:
@@ -0,0 +1,123 @@
|
||||
# AGENT_BOUNDARIES.md
|
||||
|
||||
> **Scope:** operations agents may perform on the `devops-infra-helm-charts` repo.
|
||||
> **Companion docs:** [SANCTITY_RULES.md](SANCTITY_RULES.md) (hard rules), [coding-guidelines/helm-values.md](coding-guidelines/helm-values.md) (style).
|
||||
|
||||
This document classifies every operation in this repo into Layer 1 / 2 / 3 with explicit blast radius and approval requirements. **An agent operating in this repo MUST consult this file before any write.**
|
||||
|
||||
---
|
||||
|
||||
## The 3-Layer Model (recap)
|
||||
|
||||
| Layer | Agent action | Safety gate |
|
||||
|-------|--------------|-------------|
|
||||
| **Layer 1 — Agent-Writable** | Generate diff, open PR. Do **not** apply directly to clusters. | PR review + Argo CD UI Sync click. |
|
||||
| **Layer 2 — Agent-Readable (Advisory)** | Research, analyse, recommend. Human executes. | Human review + manual execution. |
|
||||
| **Layer 3 — Agent-Blocked** | Refuse the write. Explain why. Cite this doc. | None applicable. |
|
||||
|
||||
---
|
||||
|
||||
## Per-operation classification
|
||||
|
||||
### Layer 1 — Agent-Writable (this repo's normal operating range)
|
||||
|
||||
All edits land via PR. A merge to `main` is then reconciled by Argo CD per cluster — most infra Applications use manual sync ([ADR-A5](../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)), so the workload deploy is a separate human Sync click.
|
||||
|
||||
| Operation | File(s) touched | Blast radius | Required approvers |
|
||||
|-----------|-----------------|--------------|--------------------|
|
||||
| Edit `custom-values.yaml` for one app on one cluster | `helm-overrides/<cluster>/<app>/custom-values.yaml` | One Helm release on one cluster | Service / app owner |
|
||||
| Add a new app under an existing cluster | new dir + files in `helm-overrides/<cluster>/<app>/` | One new release | App owner + cluster owner; pair with Argo `Application` PR in sister repo |
|
||||
| Add a sidecar raw manifest alongside an existing app | `helm-overrides/<cluster>/<app>/<extra>.yaml` | Adds Kubernetes resources rendered alongside the Helm release | App owner |
|
||||
| Add a new override file under an existing app dir | new file in `helm-overrides/<cluster>/<app>/` | Layered into the same release if the Argo Application's `valueFiles` covers it | App owner |
|
||||
| Append to an existing chart's `Chart.yaml` `dependencies[]` | `helm-templates/<chart>/Chart.yaml` + refresh `Chart.lock` via `helm dependency update` | Affects every consumer of that chart | Platform team |
|
||||
| Bump a `dependencies[].version` in `Chart.yaml` | `helm-templates/<chart>/Chart.yaml` + `Chart.lock` | Same | Platform team — see [update-chart-version](../platform/procedures/update-chart-version.md) |
|
||||
| Add a new chart sibling for blue-green migration | new dir `helm-templates/<chart>-<variant>/` | Migration target only; old sibling stays live | Platform team — see [blue-green-chart-migration](../platform/procedures/blue-green-chart-migration.md) |
|
||||
| Edit the per-cluster Contour scheduling matrix | `contour-nodeselector-tolerations-summary.md` | Documentation; no runtime effect | Platform team |
|
||||
|
||||
### Layer 1 — HIGH RISK (write allowed, but require explicit approval and detailed rationale in PR)
|
||||
|
||||
| Operation | Why it's high risk |
|
||||
|-----------|--------------------|
|
||||
| Edit `helm-templates/<chart>/templates/` or `helm-templates/<chart>/values.yaml` | Most charts here are vanilla upstream pulled via `helm pull`. Edits silently fork the chart and get clobbered on the next upstream sync. **Only allowed if the fork is intentional and documented in that chart's `README.md` — see [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md).** |
|
||||
| Delete a versioned sibling chart (`<chart>-green`, `<chart>-vX.Y.Z`, `<chart>-latest`, `<chart>-old`) | The variant is a blue-green migration target. Both versions may be live simultaneously. **Confirm zero references in `github.com/Meesho/devops-infra-argo-config` before deleting.** |
|
||||
| Edit `manifests/storageclass/*.yaml` | Cluster-wide singleton; affects every PVC. **Platform-team review required.** |
|
||||
| Edit `manifests/priorityclass/<cluster>/*.yaml` | Affects scheduling priority for every pod that references the class. Platform-team review. |
|
||||
| Edit `manifests/{jenkins-filestore-caching,jenkins-gcs-caching,jfrog-filestore-data}/{dev,prd}/` | Per-env stateful PV/PVCs; wrong reclaim policy can drop CI build caches or JFrog binary data. Platform-team review. |
|
||||
| Add a new cluster directory under `helm-overrides/` | Creates a new deployment target. **Pair with the cluster's `ApplicationSet` change in the sister repo. Per-cluster `nodeSelector` / `tolerations` / `computeClass` MUST be written from scratch, not copied from another cluster ([SANCTITY_RULES R5](SANCTITY_RULES.md)).** |
|
||||
| Change `fullnameOverride` in any `custom-values.yaml` | Service DNS, PVC binding, ConfigMap/Secret references, and downstream Argo Application names depend on it being stable. **Almost always wrong to touch.** |
|
||||
| Bulk find-replace across cluster directories ("normalise" labels, image tags, etc.) | The whole point of per-cluster overrides is divergence. Cross-cluster cleanup is its own PR with its own scope ([SANCTITY_RULES R6](SANCTITY_RULES.md)). |
|
||||
| Pin an image tag to a registry outside `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/` | Production overrides go through Meesho's Artifact Registry mirror. Direct Docker Hub / Quay pulls are a supply-chain and rate-limit risk. |
|
||||
|
||||
### Layer 2 — Agent-Readable (advisory only)
|
||||
|
||||
Some operations a user might ask for touch systems an agent can analyse but not change.
|
||||
|
||||
| Operation | Why advisory |
|
||||
|-----------|--------------|
|
||||
| "Sync `<release>` now in Argo CD" | The Sync click is intentional human action. **Recommend the command (`argocd app sync <name>`) or the Argo CD UI path; do not execute.** |
|
||||
| "Restart pods for `<release>`" | kubectl operation against a workload cluster. Recommend `kubectl rollout restart deploy/<name> -n <ns>`; do not execute. |
|
||||
| "Why is this pod pending?" | Live cluster state. Recommend `kubectl describe pod` and walk [pod-pending-scheduling.md](../platform/runbooks/pod-pending-scheduling.md); do not infer. |
|
||||
| "Why did the chart not render?" | Reproduce locally with `helm template`; recommend the fix. Do not push without PR. |
|
||||
| "Add an alert rule for `<service>`" | Alert rules are in `victoria-metrics-alert*` chart values *here*, but routing/notifier config is elsewhere (Pulse / Slack webhook secrets). Recommend the right file; pair with the routing change. |
|
||||
| "What's broken on the GKE cluster itself?" | Cluster-level GCP / GKE issues are out of scope for this repo. Recommend the Terraform repo or the platform team. |
|
||||
| "Reconcile the Terraform drift for the cluster" | Out of scope — that's `terraform-gcp-infra`. Refuse + redirect. |
|
||||
|
||||
### Layer 3 — Agent-Blocked (refuse + explain)
|
||||
|
||||
| Operation | Why blocked | What would unblock |
|
||||
|-----------|-------------|--------------------|
|
||||
| Hand-edit `repository.yaml` | Owned by `registry-bootstrap` automation; manual edits are overwritten on the next bootstrap run. | Update the upstream registry that feeds registry-bootstrap. |
|
||||
| Bypass pre-commit hooks (`--no-verify`, `git commit -n`, removing the hook) | TruffleHog is the last-line secret scan. ([SANCTITY_RULES R4](SANCTITY_RULES.md)) | If a hit is a known false positive, confirm with the platform/security team in writing first. |
|
||||
| Push directly to `main` | Branch protection. ([SANCTITY_RULES R1](SANCTITY_RULES.md)) | (Never legitimate.) |
|
||||
| Force-push to `main` | Same. | (Never legitimate.) |
|
||||
| Run `helm install`, `helm upgrade`, or `kubectl apply` against any workload cluster | This is GitOps; in-cluster mutation creates drift Argo CD will reconcile away. | Use the Argo CD UI Sync flow, or the cluster's incident-response toolset. |
|
||||
| Curl / probe / interact with `int.meesho.int`, `prd.meesho.int`, `int.mrouter.int`, `prd.mrouter.int`, or any workload-traffic `*.meeshogcp.in` host | Production traffic surfaces. ([SANCTITY_RULES R3](SANCTITY_RULES.md)) | (Pre-commit hook telemetry to `observe.meeshogcp.in` is automated; that is hook infrastructure, not agent action.) |
|
||||
| Edit Argo `Application` / `ApplicationSet` manifests | They live in `github.com/Meesho/devops-infra-argo-config`. | Open a PR there. |
|
||||
| Modify a chart whose `helm-templates/<chart>/` is vanilla upstream, without a documented fork rationale | An accidental fork is silently clobbered on the next sync. | Follow [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md) and document the fork in the chart's `README.md`. |
|
||||
| Copy a `custom-values.yaml` from one cluster to another verbatim | Per-cluster `nodeSelector` / `tolerations` / `computeClass` differ. ([SANCTITY_RULES R5](SANCTITY_RULES.md)) | Author the override from scratch using the per-cluster matrix in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) and sibling files. |
|
||||
|
||||
---
|
||||
|
||||
## Cross-cuts: things to verify on **every** Layer 1 PR
|
||||
|
||||
Pre-commit hooks here are minimal — TruffleHog covers secrets; CAC and Yaak are no-ops because their gating paths don't exist in this repo. The agent is the next line of defence. On every PR, mentally run this checklist:
|
||||
|
||||
1. **Surgical scope.** The PR touches only the cluster × app the task asked for. No drive-by edits to neighbours.
|
||||
2. **Per-cluster scheduling rewritten, not copied.** If the change involves `nodeSelector` / `tolerations` / `computeClass`, every cluster has its own topology. Copy-and-rename is the most common silent bug. ([contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md))
|
||||
3. **`fullnameOverride` unchanged.** Unless the explicit headline of the PR is a release-name migration.
|
||||
4. **Image tags pin to Meesho's GAR mirror** (`asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>`), not Docker Hub.
|
||||
5. **Versioned siblings preserved.** A delete or rename of a `<chart>-green` / `-latest` / `-vX.Y.Z` directory only proceeds after grepping `github.com/Meesho/devops-infra-argo-config` for references.
|
||||
6. **Chart.yaml + Chart.lock move together.** A `dependencies[].version` bump without a refreshed lockfile is incomplete.
|
||||
7. **Sister repo paired (where required).** A new app under a cluster needs a matching `Application` in the sister repo. A new cluster directory needs a matching `ApplicationSet` (or per-cluster Application set) in the sister repo. Note the pairing in the PR description.
|
||||
8. **No `*` or `latest` image tags introduced.** Argo CD's manual-sync default does not rescue you from a pulled-out-from-under-you image.
|
||||
9. **`manifests/` left alone unless the PR's headline says so.** `storageclass/` and `priorityclass/<cluster>/` are cluster-wide singletons.
|
||||
10. **TruffleHog passed.** Never bypass.
|
||||
|
||||
---
|
||||
|
||||
## Approval requirements summary
|
||||
|
||||
| Change category | Reviewer required | CMR required? |
|
||||
|-----------------|-------------------|---------------|
|
||||
| Single-cluster `custom-values.yaml` edit (no scheduling, no `fullnameOverride`) | App owner | Per BU policy |
|
||||
| Add a new app override under an existing cluster | App owner + cluster owner | **Yes** |
|
||||
| Add a new cluster directory | Platform team | **Yes** |
|
||||
| `Chart.yaml` `dependencies[].version` bump | Platform team | **Yes** for prod-fleet charts (Argo CD, Contour, VictoriaMetrics, ingress) |
|
||||
| Intentional fork of `helm-templates/<chart>/templates/` | Platform team — multi-reviewer | **Yes** |
|
||||
| Delete a versioned chart sibling | Platform team — confirm zero sister-repo references | **Yes** |
|
||||
| Edit `manifests/storageclass/` or `priorityclass/<cluster>/` | Platform team | **Yes** |
|
||||
| Pre/post-commit hook script change | Platform team + security (if hook scope changes) | Per CMR matrix |
|
||||
| `fullnameOverride` change | Platform team — multi-reviewer | **Yes** — emergency-only |
|
||||
|
||||
CMR = Change Management Request. Per Meesho process; not enforced in-repo, applied at org level.
|
||||
|
||||
---
|
||||
|
||||
## Escalation
|
||||
|
||||
If a request falls outside this matrix or you can't classify it cleanly:
|
||||
|
||||
1. Refuse the write.
|
||||
2. Cite this document + the row that matches (or explain why no row matches).
|
||||
3. Suggest the human asks the platform team or files a CMR.
|
||||
4. Do not improvise around the boundary.
|
||||
@@ -0,0 +1,122 @@
|
||||
# SANCTITY_RULES.md
|
||||
|
||||
> Non-negotiable rules for the `devops-infra-helm-charts` repo.
|
||||
> Read alongside [AGENT_BOUNDARIES.md](AGENT_BOUNDARIES.md) (per-operation Layer map) and [coding-guidelines/helm-values.md](coding-guidelines/helm-values.md) (style).
|
||||
|
||||
These are the rules whose violation is a process incident, not a clever shortcut. Each one is the result of a real failure mode (or proximity to one).
|
||||
|
||||
---
|
||||
|
||||
## R1 — `main` is production
|
||||
|
||||
Argo CD on every cluster reconciles from `main`. There is no staging branch. A merge is a deploy event.
|
||||
|
||||
- **No experimentation on `main`.** Always work on a feature/fix branch and open a PR.
|
||||
- **No force-push to `main`** (enforced at GitHub org level).
|
||||
- **No "I'll just amend that" after merge.** A new PR is the only path forward.
|
||||
|
||||
## R2 — Sister repo is the routing layer; this repo is the values layer
|
||||
|
||||
`devops-infra-helm-charts` holds *what gets installed and with what values*. `github.com/Meesho/devops-infra-argo-config` holds *where it gets routed* (the Argo `Application` / `ApplicationSet` manifests).
|
||||
|
||||
- **A new chart in `helm-templates/` does nothing** until an `Application` referencing it lands in the sister repo.
|
||||
- **A new cluster directory in `helm-overrides/` does nothing** until an `ApplicationSet` (or per-cluster Application set) covers it.
|
||||
- **Pair the two-repo change.** Cite the sister-repo PR in the description here, and vice versa.
|
||||
|
||||
## R3 — Production traffic surfaces are off-limits
|
||||
|
||||
Agent-initiated requests to `int.meesho.int`, `prd.meesho.int`, `int.mrouter.int`, `prd.mrouter.int`, and workload `*.meeshogcp.in` services are forbidden. Any accidental call can affect live traffic or data.
|
||||
|
||||
- **Never `curl`, `WebFetch`, query, or otherwise probe** these endpoints.
|
||||
- The pre-commit hook telemetry endpoints (`observe.meeshogcp.in/api/webhook`, `cursor-server.meeshogcp.in/api/v1/...`) are *automated hook infrastructure*, not agent-initiated traffic. They are not a precedent for agent calls.
|
||||
|
||||
## R4 — Pre-commit hooks must pass
|
||||
|
||||
`pre-commit-scripts/runner.sh` invokes TruffleHog (verified-secret scan with webhook telemetry). CAC and Yaak hooks exist but are no-ops here (their gating paths — `configs/` and `api-collections/` — don't exist in this repo). TruffleHog is the last-line secret scan.
|
||||
|
||||
- **Never bypass** with `--no-verify`, `git commit -n`, or by removing the hook.
|
||||
- **Never weaken a hook to "just get this through"** — fix the upstream cause.
|
||||
- A verified TruffleHog hit means a secret is staged. Real secrets belong in `external-secrets` (per-cluster), backed by GCP Secret Manager / Vault — not in `custom-values.yaml`.
|
||||
|
||||
## R5 — Per-cluster scheduling is bespoke; never copy across clusters
|
||||
|
||||
Each cluster has its own node-pool topology. Some clusters use GKE Autopilot's `cloud.google.com/compute-class:` keys (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`); standard clusters use `dedicated:` keys. Per-Contour-instance, per-cluster mappings are recorded in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||||
|
||||
- **Never copy `nodeSelector` / `tolerations` / `computeClass` blocks between clusters** without rewriting them from scratch against the destination cluster's topology.
|
||||
- **Always read the matrix** before editing any Contour values; cross-reference sibling cluster files for non-Contour scheduling.
|
||||
- Wrong values strand pods on wrong nodes or leave them `Pending` indefinitely.
|
||||
|
||||
## R6 — Surgical edits only
|
||||
|
||||
The repo has ~30 cluster directories × dozens of apps each. The whole point of per-cluster overrides is divergence — accumulated, deliberate, often for tiered traffic or specialised hardware.
|
||||
|
||||
- **Never "normalise" values across clusters in the same PR as a feature change.** Cross-cluster clean-ups are their own PRs with their own scope and CMRs.
|
||||
- **Never auto-update labels, image tags, or comments** on apps you weren't asked to touch.
|
||||
- **One change-type per PR.** Reviewer cognition matters; rollback granularity matters.
|
||||
|
||||
## R7 — `helm-templates/<chart>/` is mostly upstream
|
||||
|
||||
Most charts here are vanilla upstream pulled via `helm pull`. The local `Chart.yaml` is often a thin wrapper that declares the upstream chart as a dependency. Editing a `templates/*.yaml` or the chart's own `values.yaml` silently forks the chart, and the fork is clobbered on the next upstream sync.
|
||||
|
||||
- **Never edit `helm-templates/<chart>/templates/` or `values.yaml` casually.** If a fork is intentional, follow [fork-upstream-chart](../platform/procedures/fork-upstream-chart.md) and document the reason in the chart's `README.md`.
|
||||
- **Never bump `Chart.yaml` `dependencies[].version`** without (a) reading the upstream changelog, (b) running `helm dependency update` to refresh `Chart.lock`, and (c) calling out the bump in the PR description.
|
||||
|
||||
## R8 — Versioned chart siblings stay live during migrations
|
||||
|
||||
`<chart>` ↔ `<chart>-green` (blue-green), `<chart>` ↔ `<chart>-vX.Y.Z` (pinned upgrade target), `<chart>` ↔ `<chart>-latest` (work-in-progress), `<chart>` ↔ `<chart>-old` (retired but still referenced). Both directories may be referenced by Argo Applications during the migration window.
|
||||
|
||||
- **Never delete a versioned sibling** without confirming zero references in `github.com/Meesho/devops-infra-argo-config`.
|
||||
- **Never "consolidate" siblings** into one chart in a maintenance PR. The split is intentional ([ADR-A2](../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md)).
|
||||
|
||||
## R9 — `fullnameOverride` is load-bearing
|
||||
|
||||
Helm's `fullnameOverride` controls the name of every released Service, Deployment, StatefulSet, ConfigMap, Secret, and PVC. Service DNS, PVC binding, ConfigMap references in other apps, and Argo Application names downstream all depend on it being stable.
|
||||
|
||||
- **Never change `fullnameOverride`** in any `custom-values.yaml`.
|
||||
- A release-name migration is its own headline-of-the-PR change with platform-team approval, a documented before/after map, and an explicit reason.
|
||||
|
||||
## R10 — `manifests/` is cluster-wide singletons
|
||||
|
||||
`manifests/storageclass/*.yaml` is repo-global; a wrong StorageClass affects every PVC on every cluster that consumes it. `manifests/priorityclass/<cluster>/*.yaml` is per-cluster but cluster-wide; a wrong PriorityClass changes scheduling priority for every pod that references it.
|
||||
|
||||
- **Never edit `manifests/storageclass/`** or `manifests/priorityclass/<cluster>/` without platform-team review.
|
||||
- **Never delete a StorageClass referenced by an existing PVC** — Kubernetes will not remove the StorageClass while bindings remain, but new PVCs will fail to provision.
|
||||
|
||||
## R11 — Image tags go through Meesho's Artifact Registry mirror
|
||||
|
||||
Production overrides pin images to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>`, not upstream Docker Hub or Quay. The mirror exists for supply-chain control and rate-limit isolation.
|
||||
|
||||
- **Never introduce a Docker Hub / Quay / GCR / ECR upstream tag** in a production override.
|
||||
- **Never use `:latest` or unpinned tags** in production overrides; an Argo CD reconcile is not the same as a controlled rollout.
|
||||
|
||||
## R12 — `repository.yaml` is automation-owned
|
||||
|
||||
Generated by `registry-bootstrap`. Hand edits will be silently overwritten on the next bootstrap run.
|
||||
|
||||
- **Never hand-edit `repository.yaml`.** If owner data is wrong, fix it in the upstream registry that feeds registry-bootstrap.
|
||||
|
||||
## R13 — Branch protection trumps everything
|
||||
|
||||
Direct push to `main` is blocked at the GitHub org level. PRs go through code review.
|
||||
|
||||
- **Never propose workflows that bypass branch protection.** If a hotfix is genuinely urgent, the path is "open a PR with a `hotfix/*` branch and request emergency review," not "force-push to main."
|
||||
|
||||
## R14 — Pre-existing schema/label drift is not yours to fix
|
||||
|
||||
Some clusters have inconsistencies in label values, indentation, key ordering, or comment style — accumulated technical-debt items. Some apps lack values keys their newer siblings have. Some clusters use long BU names (`supply`) while others use short (`supl`).
|
||||
|
||||
- **Never normalise label or BU values in unrelated PRs.** Cross-cutting clean-ups are their own PRs with their own scope and CMRs.
|
||||
- **Never reformat a YAML file in passing.** Diffs full of indentation churn drown out the real change.
|
||||
|
||||
---
|
||||
|
||||
## What "non-negotiable" means
|
||||
|
||||
Each rule above has a documented reason and is the result of a real failure mode (or proximity to one). If you find yourself wanting to break a rule, the path is:
|
||||
|
||||
1. Write up the case in a doc (or PR description) explaining what would change and why.
|
||||
2. Loop in the platform team for review.
|
||||
3. If the rule should change, change *the rule first* in this file (with a PR to update it), then act.
|
||||
4. If the rule should not change, find another way.
|
||||
|
||||
A merge that violates a rule here is a process incident, not a clever shortcut.
|
||||
@@ -0,0 +1,82 @@
|
||||
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Agent Operations Guide
|
||||
|
||||
A meta-guide for any AI agent (or human contributor wearing the agent hat) doing work in this repo. Read this first; it tells you which other docs to load and in what order.
|
||||
|
||||
## Pre-flight (always, every task)
|
||||
|
||||
Before reading any task-specific file:
|
||||
|
||||
1. **Read the repo-root `CLAUDE.md`.** It is the authoritative source for the NEVER-DO list, cluster naming, versioned siblings, multi-Contour pattern, and the Layer constraint summary. Everything else in `docs/` derives from it.
|
||||
2. **Read [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)** to confirm which Layer the operation lives in.
|
||||
3. **Read [SANCTITY_RULES.md](./SANCTITY_RULES.md)** to confirm the operation is not on the don't-touch list.
|
||||
4. **Read [`wiki/entities/DevOps Infra Helm Charts.md`](../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md)** for the conceptual model (this repo as values store; sister repo as routing layer).
|
||||
5. **Skim [`docs/architecture.md`](../architecture.md)** if the task touches more than one chart or cluster.
|
||||
|
||||
If the task targets a specific chart family (Contour, Vault, Argo CD, observability stack), also load the relevant coding guideline before editing:
|
||||
- [coding-guidelines/helm-values.md](./coding-guidelines/helm-values.md) — values-file conventions
|
||||
- [coding-guidelines/argocd.md](./coding-guidelines/argocd.md) — Argo CD interaction model
|
||||
- [coding-guidelines/observability.md](./coding-guidelines/observability.md) — VM / Mimir / Loki / Tempo / Grafana
|
||||
|
||||
## Authoring loop
|
||||
|
||||
The supported edit path for almost every task in this repo:
|
||||
|
||||
1. **Branch off `main`.** Never push to `main`. The branch name should describe the cluster × app being touched.
|
||||
2. **Edit the single targeted file** under `helm-overrides/<cluster>/<app>/custom-values.yaml` (or sibling `<extra>.yaml`). No drive-by edits to other apps in the same directory. No cross-cluster "normalization" in the same PR.
|
||||
3. **Dry-run with `helm template`** to confirm the values render. Use the command from the root `CLAUDE.md` Quick reference table:
|
||||
```
|
||||
helm template <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
```
|
||||
4. **Commit.** The pre-commit hooks run automatically:
|
||||
- **TruffleHog** — secret scan, blocking. NEVER bypass. See [claude/08-pre-commit-and-hooks.md](../../claude/08-pre-commit-and-hooks.md).
|
||||
- CAC and Yaak hooks are gated/no-op on this repo.
|
||||
5. **Open a PR.** Pair with a sister-repo PR if a new Argo Application is being introduced.
|
||||
6. **Reviewer + Argo CD UI Sync are the safety gates.** Once the PR is merged to `main`, Argo CD on the target cluster reconciles. Per-cluster `syncPolicy` (auto vs manual) is set in the sister repo `Meesho/devops-infra-argo-config`, not here.
|
||||
|
||||
See [platform/procedures/onboard-app-to-cluster.md](../platform/procedures/onboard-app-to-cluster.md) and [platform/procedures/update-chart-version.md](../platform/procedures/update-chart-version.md) for two of the most common authoring loops.
|
||||
|
||||
## When to refuse
|
||||
|
||||
Refuse the operation outright if it falls into one of these categories. Cite [SANCTITY_RULES.md](./SANCTITY_RULES.md) in the refusal:
|
||||
|
||||
- Direct push to `main`, or any `--force` push.
|
||||
- Bypassing the pre-commit hook (`--no-verify`, `git commit -n`).
|
||||
- Curl / probe / query against any production endpoint (`*.meesho.int`, `*.mrouter.int`, `*.meeshogcp.in`).
|
||||
- Edit to `repository.yaml` (owned by `registry-bootstrap`).
|
||||
- Edit to an Argo `Application` / `ApplicationSet` manifest (lives in sister repo).
|
||||
- Cross-cluster copy-paste of `nodeSelector` / `tolerations` / `computeClass` without rewrite.
|
||||
- Deletion of a versioned-sibling chart without confirming zero sister-repo references.
|
||||
|
||||
## When to escalate
|
||||
|
||||
If the operation is potentially valid but exceeds Layer-1 authority — chart fork, dep bump with breaking changes, StorageClass/PriorityClass edit, Vault HA work — stop and escalate per [escalation-matrix.md](./escalation-matrix.md). The escalation matrix maps each situation to an owner and a channel.
|
||||
|
||||
## Tooling commands (cheat sheet)
|
||||
|
||||
The repo-root `CLAUDE.md` Quick reference table is the source of truth. Reproduced here for convenience:
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Install pre-commit hooks | `pre-commit install --hook-type pre-commit --hook-type pre-push --hook-type post-commit` |
|
||||
| Re-run pre-commit on staged changes | `pre-commit run` |
|
||||
| Render a chart locally | `helm template <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml` |
|
||||
| Refresh subchart deps | `helm dependency update helm-templates/<chart>` |
|
||||
| Diff against live release | `helm diff upgrade <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml` |
|
||||
| Lint a chart | `helm lint helm-templates/<chart>` |
|
||||
| Find which clusters override an app | `find helm-overrides -maxdepth 2 -type d -name '<app>'` |
|
||||
|
||||
## Output discipline
|
||||
|
||||
- Generate diffs and PRs; do not apply directly to clusters. In-cluster mutation is incident response, not authoring.
|
||||
- Surgical edits only. Touch the cluster × application asked for; leave the rest.
|
||||
- Cross-link reasoning to ADRs in [`wiki/analyses/`](../../wiki/analyses/) where helpful.
|
||||
|
||||
## See also
|
||||
|
||||
- [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)
|
||||
- [SANCTITY_RULES.md](./SANCTITY_RULES.md)
|
||||
- [escalation-matrix.md](./escalation-matrix.md)
|
||||
- [coding-guidelines/helm-values.md](./coding-guidelines/helm-values.md)
|
||||
- [../architecture.md](../architecture.md)
|
||||
@@ -0,0 +1,67 @@
|
||||
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Coding Guideline — Argo CD interaction model
|
||||
|
||||
This repo holds **values** and **cached charts**. It does NOT own Argo CD `Application` or `ApplicationSet` manifests. Those live in the sister repo:
|
||||
|
||||
> [`github.com/Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config)
|
||||
|
||||
A merge to `main` here is a deploy event for every cluster whose Argo Application points at a path in this repo. The Argo `Application` defines the routing (which path on which cluster, sync policy, retry, prune); we define what that path renders to.
|
||||
|
||||
## Hard separation
|
||||
|
||||
| Concern | Owns it |
|
||||
|---------|---------|
|
||||
| `helm-templates/<chart>/` (cached/forked chart) | this repo |
|
||||
| `helm-overrides/<cluster>/<app>/custom-values.yaml` (per-cluster values) | this repo |
|
||||
| `manifests/storageclass/`, `manifests/priorityclass/<cluster>/` (singletons) | this repo |
|
||||
| Argo `Application` (cluster, path, repoURL, targetRevision, destination namespace) | **sister repo** |
|
||||
| Argo `ApplicationSet` (cluster generators, templating fan-out) | **sister repo** |
|
||||
| `syncPolicy.automated.{prune,selfHeal}` decision | **sister repo** |
|
||||
| `syncPolicy.syncOptions` (CreateNamespace, ServerSideApply) | **sister repo** |
|
||||
| Sync waves / hooks via `argocd.argoproj.io/sync-wave` annotations | this repo (when expressed inside chart templates or raw sidecar manifests) |
|
||||
|
||||
## What this means for the agent
|
||||
|
||||
- **Never** add or edit a file matching `Application*.yaml` / `ApplicationSet*.yaml` here. If the task asks for one, redirect to the sister repo. See [escalation-matrix.md](../escalation-matrix.md) row 5.
|
||||
- When introducing a **new app** to a cluster, the change is a **paired PR**: (a) a PR here adding `helm-overrides/<cluster>/<newapp>/custom-values.yaml`, and (b) a PR in the sister repo adding the matching `Application` manifest. Both must merge before the app deploys.
|
||||
- When introducing a **new cluster**, the paired PR in the sister repo updates the `ApplicationSet` cluster generator. See [../platform/procedures/onboard-new-cluster.md](../../platform/procedures/onboard-new-cluster.md).
|
||||
- When **removing** an app, deboard the Argo Application first (sister repo), let Argo prune, then remove the override directory here. See [../platform/procedures/deboard-app.md](../../platform/procedures/deboard-app.md).
|
||||
|
||||
## Sync policy: where the decision lives
|
||||
|
||||
`syncPolicy.automated.prune` and `syncPolicy.automated.selfHeal` live in the sister repo's `Application` spec. Convention on the platform:
|
||||
|
||||
- **Manual sync default** for infra components on prod clusters. Sync is a deliberate human click after a merge. Rationale documented in [`wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||||
- Auto-sync is reserved for low-risk leaf components (e.g., `kube-state-metrics`, monitoring agents) where reconciliation drift is benign.
|
||||
|
||||
Agents editing values here should assume **the sync click is the safety gate**. A merged PR is not yet deployed.
|
||||
|
||||
## Validating values before merge
|
||||
|
||||
The agent's responsibility is that the values **render** correctly. Argo CD will materialize the rendered output via `helm template`-equivalent server-side. Use the same command locally:
|
||||
|
||||
```
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
```
|
||||
|
||||
If the chart has subchart dependencies (`Chart.yaml` `dependencies:`), run `helm dependency update helm-templates/<chart>` before templating, otherwise render will fail with `found in Chart.yaml, but missing in charts/ directory`.
|
||||
|
||||
For raw-manifest sidecars (`<extra>.yaml` files in the override dir), validate with `kubectl apply --dry-run=client -f <file>`. See [../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
## Common failure modes
|
||||
|
||||
| Symptom | First read |
|
||||
|---------|-----------|
|
||||
| `Sync` button click results in `OutOfSync` that won't resolve | [../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md) |
|
||||
| Pods land but stay `Pending` | [../platform/runbooks/pod-pending-scheduling.md](../../platform/runbooks/pod-pending-scheduling.md) |
|
||||
| Ingress 5xx after a Contour values change | [../platform/runbooks/ingress-down.md](../../platform/runbooks/ingress-down.md) |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Sister repo: [`Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config)
|
||||
- [helm-values.md](./helm-values.md)
|
||||
- [observability.md](./observability.md)
|
||||
- [../escalation-matrix.md](../escalation-matrix.md)
|
||||
- [`wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)
|
||||
@@ -0,0 +1,167 @@
|
||||
# Coding Guidelines — Helm values authoring
|
||||
|
||||
> The hard stops live in [../SANCTITY_RULES.md](../SANCTITY_RULES.md); the layer/scope rules in [../AGENT_BOUNDARIES.md](../AGENT_BOUNDARIES.md). This file is "how to write the values YAML well" — style, conventions, and the recurring footguns the linter doesn't catch.
|
||||
|
||||
---
|
||||
|
||||
## File-level conventions
|
||||
|
||||
### Where files go
|
||||
|
||||
| Path | Meaning |
|
||||
|------|---------|
|
||||
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | The primary Helm values file Argo's `valueFiles` references. **Default name. Do not invent alternatives.** |
|
||||
| `helm-overrides/<cluster>/<app>/<extra>.yaml` | Sidecar raw manifests applied alongside the Helm release. Common shapes: `computeclass/<x>-cc.yaml`, `external-dns-services/<svc>.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`. |
|
||||
| `helm-templates/<chart>/Chart.yaml` | Chart manifest; usually a thin wrapper declaring an upstream dep. |
|
||||
| `helm-templates/<chart>/values.yaml` | The chart's own defaults — rarely edited; treat as upstream. |
|
||||
| `helm-templates/<chart>/templates/` | Manifest templates — vanilla upstream unless intentionally forked. **Don't edit casually.** |
|
||||
|
||||
### Cluster directory naming
|
||||
|
||||
The cluster directory's name is a contract — it must match the Kubernetes cluster name registered in Argo CD. Conventions:
|
||||
|
||||
| Pattern | Use |
|
||||
|---------|-----|
|
||||
| `k8s-<bu>-prd-ase1` | Standard GKE prod cluster, BU-owned (AWS-style naming retained). |
|
||||
| `k8s-<bu>-prd-ase1c` | GCP zone-c twin. |
|
||||
| `k8s-shared-int-ase1` | Shared int (pre-prod) cluster. The only non-prod cluster. |
|
||||
| `k8s-aurva-prd-ase1` | Aurva integration. |
|
||||
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier clusters. Minimal override sets (typically `kube-state-metrics` + `victoria-metrics-agent`). |
|
||||
| `k8s-supply-dev-ase1` | The lone dev/sandbox cluster. |
|
||||
|
||||
### App directory naming
|
||||
|
||||
Inside `helm-overrides/<cluster>/`, each subdirectory is one Argo Application = one Helm release.
|
||||
|
||||
- One Helm release per directory.
|
||||
- Multiple Contour instances per cluster is the norm — `contour-external`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. **They are separate releases — don't merge them.**
|
||||
- Versioned siblings (`argo-cd` ↔ `argo-cd-green`, `keda` ↔ `keda-2.17.1`) live as parallel directories under `helm-templates/`, but their per-cluster overrides typically live under one directory until the migration cuts over. See [blue-green-chart-migration](../../platform/procedures/blue-green-chart-migration.md).
|
||||
|
||||
---
|
||||
|
||||
## Field-level conventions (`custom-values.yaml`)
|
||||
|
||||
### Image references
|
||||
|
||||
- **Always pin images to Meesho's Artifact Registry mirror** for production:
|
||||
```yaml
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<image>
|
||||
tag: <semver-or-sha>
|
||||
```
|
||||
- **Never use `:latest` or unpinned tags** in production overrides.
|
||||
- **Never reference Docker Hub, Quay, GCR upstream, or ECR directly** in a production override. Mirror it via the platform team's image-pull workflow first.
|
||||
|
||||
### `fullnameOverride`
|
||||
|
||||
- **Set it once, never change it.** Service DNS, PVC binding, ConfigMap references, and downstream Argo Application names depend on stability. ([SANCTITY_RULES R9](../SANCTITY_RULES.md))
|
||||
- For dataplane (`db-*`) clusters, the convention is `fullnameOverride: <kind>-dbc-<bu>-prd` (e.g. `kube-state-metrics-dbc-dsci-prd`).
|
||||
- For BU clusters, omit `fullnameOverride` unless the chart's default name collides with another release in the same namespace.
|
||||
|
||||
### `nameOverride`
|
||||
|
||||
Almost never needed. Helm's `<release>-<chart>` naming is usually fine.
|
||||
|
||||
### Replica counts and HPA bounds
|
||||
|
||||
- Read the chart's defaults before specifying replicas. Some charts have HPA-managed replicas that conflict with `replicaCount`.
|
||||
- For HPA-managed releases, set `minReplicas` and `maxReplicas` *and* leave `replicaCount` unset (or set to `null`).
|
||||
- Don't lower `minReplicas` to zero unless the workload genuinely scales-from-zero.
|
||||
|
||||
### Resource requests and limits
|
||||
|
||||
- **Always set `resources.requests`** for production releases. Without requests, the scheduler treats the pod as best-effort.
|
||||
- **Set `resources.limits`** unless the chart documentation explicitly recommends omitting them (some sidecars deliberately go limit-less).
|
||||
- **Don't copy resources from another cluster.** Workload sizing is per-traffic-tier; the supply prd cluster's Contour requests are not the demand prd cluster's.
|
||||
|
||||
### `nodeSelector`, `tolerations`, `affinity`, `topologySpreadConstraints`
|
||||
|
||||
The single biggest source of silent mis-deploys. Per [SANCTITY_RULES R5](../SANCTITY_RULES.md):
|
||||
|
||||
| Cluster type | Key style | Example |
|
||||
|--------------|-----------|---------|
|
||||
| GKE Autopilot (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `cloud.google.com/compute-class` | `nodeSelector: {cloud.google.com/compute-class: contour-internal-0-cc}` |
|
||||
| Standard GKE | `dedicated:` | `nodeSelector: {dedicated: contour-internal-0}` |
|
||||
| Most others | `dedicated:` | same |
|
||||
|
||||
Cross-reference [`contour-nodeselector-tolerations-summary.md`](../../../contour-nodeselector-tolerations-summary.md) for the per-cluster Contour matrix. For non-Contour apps, copy from a sibling app on the *same* cluster, not from the same app on a *different* cluster.
|
||||
|
||||
### Probes
|
||||
|
||||
- Always set `livenessProbe` and `readinessProbe` for any long-running container.
|
||||
- For workloads that take >30 s to warm (Jenkins, JFrog, ClickHouse), bump `initialDelaySeconds` accordingly — not the timeout, not the period.
|
||||
- A `startupProbe` is the right tool for slow-warm containers; don't fight it with a 600s `initialDelaySeconds` on `livenessProbe`.
|
||||
|
||||
### Persistence
|
||||
|
||||
- StorageClass references go through the cluster-wide singletons in `manifests/storageclass/`: `pd-standard-retain-dr`, `sc-filestore-standard`, `sc-pd-ssd`, `sc-pd-standard`. **Never reference a StorageClass that doesn't exist in `manifests/storageclass/`.**
|
||||
- For stateful releases, set `persistence.size` explicitly. Default sizes are rarely right.
|
||||
- Never enable `persistence.enabled: true` without confirming the StorageClass and its retention/reclaim policy.
|
||||
|
||||
### Secrets
|
||||
|
||||
- **Never inline secret values** in `custom-values.yaml`. ([SANCTITY_RULES R4](../SANCTITY_RULES.md))
|
||||
- Reference secrets by name: `existingSecret: <secret-name>`, where the secret is materialised by the per-cluster `external-secrets` app from GCP Secret Manager / Vault.
|
||||
- Most clusters have an `external-secrets/` override directory; if your release needs a secret, the corresponding `ExternalSecret` lives there.
|
||||
|
||||
### Annotations and labels
|
||||
|
||||
- **Add labels conservatively.** Most charts already emit sensible label sets (`app.kubernetes.io/name`, etc.).
|
||||
- For ingress (`Ingress`, `HTTPProxy`, Contour `Service`), `external-dns` annotations and AWS/GCP load-balancer annotations are normal — copy from a sibling on the same cluster.
|
||||
- **Don't invent label keys.** If you find yourself adding `meesho.com/<something>`, double-check whether the project already has a convention for it.
|
||||
|
||||
---
|
||||
|
||||
## Field-level conventions (raw sidecar manifests)
|
||||
|
||||
For `helm-overrides/<cluster>/<app>/<extra>.yaml` files (no Helm templating, applied as-is):
|
||||
|
||||
- One Kubernetes resource per file unless they are tightly coupled.
|
||||
- Use `apiVersion: v1` etc. — pin the API version explicitly.
|
||||
- Set `metadata.namespace` (don't rely on the Argo Application's `destination.namespace` for these).
|
||||
- For `ComputeClass` / `NodeClass` / `BackendConfig` / GKE-specific resources, sample a sibling cluster's existing file before authoring.
|
||||
- For `external-dns-services/*.yaml`, the `Service` resource carries `external-dns.alpha.kubernetes.io/hostname` annotations — match the cluster's existing DNS pattern.
|
||||
|
||||
---
|
||||
|
||||
## YAML style
|
||||
|
||||
- **2-space indent. No tabs.**
|
||||
- **Use single quotes for `'*'`** and other glob-like strings; bare strings elsewhere where unambiguous.
|
||||
- **Trailing newline at EOF.**
|
||||
- **No `---` document separators** unless you genuinely need multi-document YAML (rare in this repo).
|
||||
- **Don't comment out fields; remove them.** The repo doesn't use commented-out scaffolding.
|
||||
- **Preserve key order from siblings.** A reordered file is a noisy diff that drowns the real change.
|
||||
- **Don't reformat unrelated YAML in passing.** ([SANCTITY_RULES R14](../SANCTITY_RULES.md))
|
||||
|
||||
---
|
||||
|
||||
## Diff hygiene
|
||||
|
||||
When opening a PR:
|
||||
|
||||
- **One change-type per PR.** Adding a service should not also "normalise labels on three other apps."
|
||||
- **Keep diffs minimal.** Don't reformat surrounding YAML.
|
||||
- **Cite the procedure followed** (link to one of `docs/platform/procedures/*.md`) in the PR description.
|
||||
- **Show the validation you ran** — `helm template`, `yamllint`, sibling-file diff, the kubectl context you ran a `helm diff` against.
|
||||
- **Pair the sister-repo PR** (`devops-infra-argo-config`) when adding a new app or cluster — link both.
|
||||
|
||||
---
|
||||
|
||||
## Common mistakes the hooks do **not** catch
|
||||
|
||||
These are the recurring footguns that pre-commit hooks won't flag:
|
||||
|
||||
1. **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster.** Pods stay `Pending`, or schedule on the wrong node pool.
|
||||
2. **`fullnameOverride` modified.** Downstream Service DNS resolves to nothing.
|
||||
3. **`spec.source.path` in the sister repo's `Application` not updated** to point at the new chart sibling after a blue-green migration.
|
||||
4. **Image tag pinned to Docker Hub** or Quay instead of the GAR mirror.
|
||||
5. **`replicaCount` set on an HPA-managed release.** HPA fights the static count.
|
||||
6. **`persistence.storageClass` referencing a class that doesn't exist** on this cluster — PVC stays `Pending` forever.
|
||||
7. **`existingSecret` referencing a secret the per-cluster `external-secrets` app doesn't create.** Pods crashloop on missing env.
|
||||
8. **`Chart.yaml` `dependencies[].version` bumped without `helm dependency update`.** Argo CD will use the lockfile and silently render the old version.
|
||||
9. **Edits inside `helm-templates/<chart>/templates/`** — silently fork the chart; clobbered on next upstream sync.
|
||||
10. **Sidecar raw-manifest namespace mismatch** with the Helm release's namespace — orphaned resources.
|
||||
|
||||
The agent's job is to be the second pair of eyes on every one of these.
|
||||
@@ -0,0 +1,78 @@
|
||||
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Coding Guideline — Observability stack overrides
|
||||
|
||||
Conventions for editing values for the observability charts vendored in this repo:
|
||||
|
||||
- `victoria-metrics-cluster` and `victoria-metrics-cluster-latest`
|
||||
- `victoria-metrics-agent` and `victoria-metrics-agent-latest`
|
||||
- `vmalert`, `victoria-metrics-operator`
|
||||
- `mimir-distributed`
|
||||
- `loki`, `loki-distributed`
|
||||
- `tempo`, `tempo-distributed`
|
||||
- `grafana`
|
||||
- `opentelemetry-collector` and `opentelemetry-collector-latest`
|
||||
- `kube-state-metrics`, `node-exporter`, `metrics-server`
|
||||
- `kube-prometheus-stack` (Prometheus alert rules)
|
||||
|
||||
The observability stack is the most rule-heavy domain in the repo: alert rules drive paging, retention drives storage cost, and label cardinality drives both. Treat values changes as *data-pipeline* changes, not static config.
|
||||
|
||||
## Versioned siblings
|
||||
|
||||
`*-latest` siblings exist alongside the stable chart for each VictoriaMetrics and OTel collector chart. Both can be live simultaneously during a blue-green migration. When editing, confirm which sibling the target Argo Application points at (sister repo). See [`wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [../platform/procedures/blue-green-chart-migration.md](../../platform/procedures/blue-green-chart-migration.md).
|
||||
|
||||
## Cardinality discipline
|
||||
|
||||
Series cardinality on VictoriaMetrics / Mimir is the dominant cost driver. Before adding any of the following, ensure the new label is bounded:
|
||||
|
||||
- New `extraLabels` / `externalLabels` on `victoria-metrics-agent`.
|
||||
- New `relabel_configs` that emit a label sourced from a high-cardinality metric source (pod name, request path, user id, request id).
|
||||
- New scrape targets in `additionalScrapeConfigs`.
|
||||
|
||||
If a label can take more than ~few-hundred distinct values, drop it or aggregate before storage.
|
||||
|
||||
## PromQL / alert-rule validation
|
||||
|
||||
Alert rules live in `kube-prometheus-stack`, `vmalert`, and Mimir ruler config. Before merging:
|
||||
|
||||
1. Render with `helm template`:
|
||||
```
|
||||
helm template prom helm-templates/kube-prometheus-stack \
|
||||
-f helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml
|
||||
```
|
||||
2. Extract the `PrometheusRule` objects and validate PromQL with `promtool check rules <file.yaml>` (Prometheus tooling) or `vmalert -dryRun -rule=<file.yaml>` for vmalert-specific rules.
|
||||
3. Confirm the rule's `for:` window and `severity` label match the cluster's PagerDuty routing — getting this wrong silently swaps which on-call gets paged.
|
||||
|
||||
## Retention and tenancy
|
||||
|
||||
- VictoriaMetrics `vmstorage.retentionPeriod` is per-cluster. Increasing it grows the PVC; never increase without confirming PVC headroom and matching PVC `resources.requests.storage`.
|
||||
- Loki and Mimir multi-tenancy is keyed on the `X-Scope-OrgID` header. Tenant lists live in cluster-specific overrides; do not assume tenants match across clusters.
|
||||
- Tempo trace retention is set via `compactor.compaction.block_retention`. Default is short (24h–48h); long retention is opt-in per cluster.
|
||||
|
||||
## Grafana dashboards & datasources
|
||||
|
||||
- Datasources are declared in the cluster's `grafana/custom-values.yaml` under `datasources.datasources.yaml`. Pin URLs to in-cluster Service DNS, not external endpoints.
|
||||
- Dashboards bundled via `dashboardProviders` reference ConfigMaps; uniqueness of `uid` matters for panel-image links and alert dashboards.
|
||||
|
||||
## Validation checklist (always)
|
||||
|
||||
Before raising a PR that touches an observability chart:
|
||||
|
||||
- [ ] `helm template` renders without error.
|
||||
- [ ] PromQL in any new alert rule passes `promtool check rules`.
|
||||
- [ ] No new high-cardinality label is added without a bound.
|
||||
- [ ] Retention / PVC sizing has not been changed silently.
|
||||
- [ ] If touching a `*-latest` sibling, the matching Argo Application points at it. See [argocd.md](./argocd.md).
|
||||
|
||||
## Schema references
|
||||
|
||||
- Per-cluster scheduling fields (must be rewritten, never copy-pasted): [../../platform/schemas/custom-values-schema.md](../../platform/schemas/custom-values-schema.md)
|
||||
- Raw manifest sidecars (e.g., `external-dns-services/*.yaml` for Grafana ingress): [../../platform/schemas/raw-manifest-sidecar-schema.md](../../platform/schemas/raw-manifest-sidecar-schema.md)
|
||||
- Storage backing for `vmstorage`, `loki` chunks, `tempo` blocks: [../../platform/schemas/storageclass-priorityclass-schema.md](../../platform/schemas/storageclass-priorityclass-schema.md)
|
||||
|
||||
## See also
|
||||
|
||||
- [helm-values.md](./helm-values.md) — values-file conventions
|
||||
- [argocd.md](./argocd.md) — Argo CD interaction model
|
||||
- [../escalation-matrix.md](../escalation-matrix.md)
|
||||
- [../../platform/runbooks/argocd-sync-failure.md](../../platform/runbooks/argocd-sync-failure.md)
|
||||
@@ -0,0 +1,39 @@
|
||||
> Per AI Blitz Plan §global. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Escalation Matrix
|
||||
|
||||
Use this table when a task crosses a Sanctity/Boundary line, when blast radius exceeds the agent's authority, or when a request belongs in a different repo. Always escalate **before** writing the change, not after.
|
||||
|
||||
Primary owner: **siddharth.pal@meesho.com**
|
||||
Secondary owner: **samarth.nag@meesho.com**
|
||||
|
||||
Owner data is sourced from the repo-root [`repository.yaml`](../../repository.yaml). If owners change, update that file via the `registry-bootstrap` flow; do not edit `repository.yaml` by hand. See [SANCTITY_RULES.md](./SANCTITY_RULES.md) for the don't-touch list and [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md) for layer classification.
|
||||
|
||||
## Situation → owner → channel
|
||||
|
||||
| # | Situation | Who | Channel | Notes |
|
||||
|---|-----------|-----|---------|-------|
|
||||
| 1 | Edit suspected to silently fork an upstream chart (touching `helm-templates/<chart>/templates/` or `values.yaml` of a vanilla-pulled chart) | siddharth.pal | Slack `#devops-infra` + PR review | Document the intentional fork in the chart's `README.md`. See [procedures/fork-upstream-chart.md](../platform/procedures/fork-upstream-chart.md). |
|
||||
| 2 | Cross-cluster cleanup or "normalization" requested in the same PR as a feature change | siddharth.pal | Slack `#devops-infra` | Refuse in-PR; request a separate cleanup PR. See [SANCTITY_RULES.md](./SANCTITY_RULES.md) §surgical-edits. |
|
||||
| 3 | TruffleHog flagged a real secret in a staged commit | siddharth.pal + secret owner (BU on-call) | Slack `#sec-incidents` (private) + revoke pipeline | NEVER bypass with `--no-verify`. Rotate the credential, then move it to External Secrets Operator. |
|
||||
| 4 | Edit to `repository.yaml` requested | siddharth.pal | Slack `#devops-infra` | This file is owned by `registry-bootstrap` automation. Refuse and redirect to that pipeline. |
|
||||
| 5 | Edit to an Argo CD `Application` / `ApplicationSet` manifest requested | siddharth.pal | PR against sister repo `Meesho/devops-infra-argo-config` | This repo does not own routing manifests. See [coding-guidelines/argocd.md](./coding-guidelines/argocd.md). |
|
||||
| 6 | Schedule fields (`nodeSelector`, `tolerations`, `computeClass`) copy-pasted from one cluster's override to another without rewriting | siddharth.pal | PR review block | Re-derive from per-cluster node-pool topology. See [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md) and [runbooks/pod-pending-scheduling.md](../platform/runbooks/pod-pending-scheduling.md). |
|
||||
| 7 | Chart dep version bump where upstream changelog flags breaking template changes | siddharth.pal | Slack `#devops-infra` + PR review | Run `helm dependency update`, refresh `Chart.lock`, dry-run `helm template` against every consumer cluster. See [procedures/update-chart-version.md](../platform/procedures/update-chart-version.md). |
|
||||
| 8 | Vault HA write-path failure (seal status, Raft peer loss, unsealed standby) | siddharth.pal + platform on-call | Slack `#sec-incidents` + PagerDuty | Production secret-store outage. Do not modify Vault overrides without on-call ack. |
|
||||
| 9 | Edit to `manifests/storageclass/*.yaml` or `manifests/priorityclass/<cluster>/*.yaml` | siddharth.pal + cluster BU owner | PR review (two reviewers) | Cluster-wide singleton; affects every PVC / scheduling priority. See [schemas/storageclass-priorityclass-schema.md](../platform/schemas/storageclass-priorityclass-schema.md). |
|
||||
| 10 | Deletion of a versioned-sibling chart (`-green`, `-vX.Y.Z`, `-latest`, `-old`) | siddharth.pal | Slack `#devops-infra` + grep sister repo | Confirm zero references in `Meesho/devops-infra-argo-config` before deletion. See [analyses/ADR-A2-blue-green-sibling-pattern.md](../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md). |
|
||||
|
||||
## Refusal language
|
||||
|
||||
When refusing, state:
|
||||
1. The Sanctity rule or Layer constraint that's tripped.
|
||||
2. Which owner to ping (from the table above).
|
||||
3. The repo or pipeline the request actually belongs in (sister repo, `registry-bootstrap`, Vault on-call, etc.).
|
||||
|
||||
## See also
|
||||
|
||||
- [AGENT_BOUNDARIES.md](./AGENT_BOUNDARIES.md)
|
||||
- [SANCTITY_RULES.md](./SANCTITY_RULES.md)
|
||||
- [agent-operations-guide.md](./agent-operations-guide.md)
|
||||
- Repo-root `CLAUDE.md` NEVER-DO list
|
||||
Reference in New Issue
Block a user