added repo
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 00 — Overview
|
||||
|
||||
## Why this repo exists
|
||||
|
||||
`devops-infra-helm-charts` is the GitOps source-of-truth for **what infrastructure tooling runs on Meesho's GKE fleet, where, and with what values**. It is one of two repos that together compose the platform's deploy plane:
|
||||
|
||||
- **This repo** — values + cached/forked charts. Answers "what does cluster X's Argo CD agent stack look like?"
|
||||
- **Sister repo** — [`Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config) — Argo `Application` / `ApplicationSet` manifests. Answers "which cluster pulls which path from the values repo, with what sync policy?"
|
||||
|
||||
A merge to `main` here is a **deploy event**: every Argo CD instance whose Application points at a touched path will reconcile, on its own cadence (auto-sync) or on a human Sync click (manual-sync — the prod default).
|
||||
|
||||
See [`../wiki/entities/DevOps Infra Helm Charts.md`](../wiki/entities/DevOps%20Infra%20Helm%20Charts.md) for the conceptual model and [`../docs/architecture.md`](../docs/architecture.md) for the full deploy lifecycle.
|
||||
|
||||
## What's in it
|
||||
|
||||
| Top-level | Role |
|
||||
|-----------|------|
|
||||
| `helm-templates/<chart>/` | 74 cached or forked upstream charts (Argo CD, Contour, VictoriaMetrics, Mimir, Loki, Tempo, Vault, Keda, Kyverno, Jenkins, JFrog, Grafana…). |
|
||||
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | Per-cluster × per-app Helm values. Edited daily. |
|
||||
| `helm-overrides/<cluster>/<app>/<extra>.yaml` | Raw manifests applied alongside the Helm release (compute-class definitions, external-DNS records, etc.). |
|
||||
| `manifests/storageclass/`, `manifests/priorityclass/<cluster>/` | Cluster-wide singletons. High blast radius. |
|
||||
| `manifests/{jenkins,jfrog}-…/{dev,prd}/` | Per-env one-shot PV/PVC manifests. |
|
||||
| `pre-commit-scripts/` | TruffleHog (active, blocking); CAC + Yaak (no-op here). |
|
||||
| `post-commit-scripts/` | Cursor AI commit metric collector (background). |
|
||||
| `repository.yaml` | Owners, auto-managed by `registry-bootstrap`. |
|
||||
| `contour-nodeselector-tolerations-summary.md` | Per-cluster Contour scheduling matrix. |
|
||||
|
||||
Detailed walkthrough in [`./01-repo-structure.md`](./01-repo-structure.md).
|
||||
|
||||
## What's NOT in it
|
||||
|
||||
- Argo CD `Application` / `ApplicationSet` manifests — those live in the **sister repo**. See [`../docs/global/coding-guidelines/argocd.md`](../docs/global/coding-guidelines/argocd.md).
|
||||
- Application / service code — lives in service repos.
|
||||
- Workload-cluster `kubectl apply` operations — that's incident response, not authoring.
|
||||
- Production endpoint probes (`*.meesho.int`, `*.mrouter.int`, `*.meeshogcp.in`) — never call from an agent. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
## How a change reaches a cluster
|
||||
|
||||
1. Branch off `main`.
|
||||
2. Edit a single `helm-overrides/<cluster>/<app>/custom-values.yaml`.
|
||||
3. Local dry-run: `helm template <release> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml`.
|
||||
4. Commit — TruffleHog runs (blocking). Never `--no-verify`.
|
||||
5. Open a PR. Reviewer is the first safety gate.
|
||||
6. Merge to `main`.
|
||||
7. Argo CD on the target cluster either auto-reconciles (low-risk leaves) or waits for a human Sync click (prod infra default).
|
||||
|
||||
That click is the **second** safety gate. The combined property (reviewer + Sync) is the system's safety floor while a tool-mediated edit path (`helm-values-tool`) is still being built.
|
||||
|
||||
See [`./05-deploy-lifecycle.md`](./05-deploy-lifecycle.md) for the full flow with failure modes, and [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md) for the hook details.
|
||||
|
||||
## Layer classification
|
||||
|
||||
This repo is **Layer 1 — Agent-Writable** (config repo). Most edits are agent-eligible via PR. Several operations are Layer 1 *high-risk* or Layer 3 (refuse). The full mapping is in the root `CLAUDE.md` *Layer constraint summary* table; the agent-facing summary is [`../docs/global/AGENT_BOUNDARIES.md`](../docs/global/AGENT_BOUNDARIES.md) and [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
## Where to go next
|
||||
|
||||
- New to the repo: read [`./01-repo-structure.md`](./01-repo-structure.md) → [`./02-cluster-fleet.md`](./02-cluster-fleet.md) → [`./05-deploy-lifecycle.md`](./05-deploy-lifecycle.md).
|
||||
- About to edit values: [`./04-override-hierarchy.md`](./04-override-hierarchy.md) and [`../docs/global/coding-guidelines/helm-values.md`](../docs/global/coding-guidelines/helm-values.md).
|
||||
- About to bump a chart: [`../docs/platform/procedures/update-chart-version.md`](../docs/platform/procedures/update-chart-version.md).
|
||||
- About to onboard an app: [`../skills/infra/onboard-app.md`](../skills/infra/onboard-app.md) and [`../docs/platform/procedures/onboard-app-to-cluster.md`](../docs/platform/procedures/onboard-app-to-cluster.md).
|
||||
- Glossary: [`./10-glossary-and-references.md`](./10-glossary-and-references.md).
|
||||
@@ -0,0 +1,70 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 01 — Repository structure
|
||||
|
||||
Directory walkthrough with the *why* attached. The root `CLAUDE.md` *Repository layout* table is the canonical short version; this page extends it with rationale and links into the rest of the tree.
|
||||
|
||||
## `helm-templates/<chart>/` — cached / forked upstream charts
|
||||
|
||||
About 74 chart directories. Three flavours:
|
||||
|
||||
1. **Vanilla upstream cache** — pulled via `helm pull <repo>/<chart>` and committed verbatim. Edits to `templates/` here silently fork the chart and get clobbered on the next refresh. Most charts are this flavour. See [`../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md`](../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
2. **Thin wrapper** — `Chart.yaml` declares `dependencies:`, `templates/` is small or empty, and the real content lives in the subchart. Used to bind multiple sub-charts as one Argo Application.
|
||||
3. **Intentional fork** — `templates/` is meaningfully edited. Each fork should explain itself in that chart's `README.md`. Forks are rare and need explicit owner approval. See [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md).
|
||||
|
||||
Why cache at all? Network/ingress robustness for Argo CD on every cluster, and a stable target for the values to bind against. Trade-off and alternatives in [`../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md`](../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
|
||||
## `helm-overrides/<cluster>/<app>/custom-values.yaml`
|
||||
|
||||
The day-to-day editing surface. Path encodes the destination:
|
||||
|
||||
- `<cluster>` → directory name matches the GKE cluster name (`k8s-central-prd-ase1`, `k8s-shared-int-ase1`, `k8s-aurva-prd-ase1`, `db-<id>-…`).
|
||||
- `<app>` → directory name matches the Argo Application name in the sister repo (and usually matches the chart name in `helm-templates/`, but doesn't have to — many apps target a versioned-sibling chart).
|
||||
|
||||
The content is a Helm values overlay merged onto `helm-templates/<chart>/values.yaml` at render time. Schema: [`../docs/platform/schemas/custom-values-schema.md`](../docs/platform/schemas/custom-values-schema.md). Composition rules: [`./04-override-hierarchy.md`](./04-override-hierarchy.md).
|
||||
|
||||
## `helm-overrides/<cluster>/<app>/<extra>.yaml`
|
||||
|
||||
Raw Kubernetes manifests dropped alongside the Helm release. They are NOT consumed by Helm — Argo applies them directly. Common patterns:
|
||||
|
||||
- `computeclass/*-cc.yaml` — GKE Autopilot `ComputeClass` objects.
|
||||
- `elastic-cluster/argo-launch.yaml` — Elasticsearch Operator CR.
|
||||
- `external-dns-services/*.yaml` — `Service` objects with `external-dns` annotations to publish DNS records.
|
||||
|
||||
Schema and conventions: [`../docs/platform/schemas/raw-manifest-sidecar-schema.md`](../docs/platform/schemas/raw-manifest-sidecar-schema.md). Why they live here rather than in dedicated manifest dirs: [`../wiki/analyses/ADR-A4-raw-manifest-sidecars-in-helm-overrides.md`](../wiki/analyses/ADR-A4-raw-manifest-sidecars-in-helm-overrides.md).
|
||||
|
||||
## `manifests/storageclass/` and `manifests/priorityclass/<cluster>/`
|
||||
|
||||
Cluster-wide singletons. A wrong StorageClass affects every PVC; a wrong PriorityClass changes scheduling priority for every pod that references it. Two-reviewer policy. Schema: [`../docs/platform/schemas/storageclass-priorityclass-schema.md`](../docs/platform/schemas/storageclass-priorityclass-schema.md). Blast-radius detail: [`./07-singletons-and-blast-radius.md`](./07-singletons-and-blast-radius.md).
|
||||
|
||||
## `manifests/{jenkins-filestore-caching,jenkins-gcs-caching,jfrog-filestore-data}/{dev,prd}/`
|
||||
|
||||
Per-env one-shot PV / PVC manifests for stateful systems that pre-date a Helm-managed model. Treated as immutable once bound; resize via PVC `resources.requests.storage` rather than re-creating.
|
||||
|
||||
## `pre-commit-scripts/`
|
||||
|
||||
- **TruffleHog secret scan** — active, blocking. NEVER bypass. See [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md) and [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
- **CAC, Yaak hooks** — gated on file paths this repo doesn't have, so they no-op here. Same scripts run for real in service repos.
|
||||
|
||||
## `post-commit-scripts/`
|
||||
|
||||
- **Cursor AI commit metric collector** — background, non-blocking. Posts metric pings to `observe.meeshogcp.in`. This is platform-managed infrastructure, not agent-initiated.
|
||||
|
||||
## `repository.yaml`
|
||||
|
||||
Owners + secondary owners. Managed by the `registry-bootstrap` automation. Editing by hand is on the don't-touch list — see [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 4.
|
||||
|
||||
## `contour-nodeselector-tolerations-summary.md`
|
||||
|
||||
Per-cluster Contour scheduling matrix at the repo root. **Read this before any Contour values edit.** Multi-Contour pattern (`contour-external`, `contour-external-1`, `contour-internal-{0,1}`, `contour-internal-intra-{0,1}`) is detailed in [`./03-chart-inventory.md`](./03-chart-inventory.md).
|
||||
|
||||
## `docs/`, `claude/`, `skills/`, `wiki/`
|
||||
|
||||
The Blitz documentation tree. Entry points:
|
||||
|
||||
- [`../docs/architecture.md`](../docs/architecture.md) — full deploy lifecycle and gotchas.
|
||||
- [`../docs/global/`](../docs/global/) — agent boundaries, sanctity rules, escalation, coding guidelines.
|
||||
- [`../docs/platform/`](../docs/platform/) — procedures, runbooks, schemas.
|
||||
- [`../skills/infra/`](../skills/infra/) — task playbooks.
|
||||
- [`../wiki/`](../wiki/) — entity model and ADRs.
|
||||
- [`./00-overview.md`](./00-overview.md) — top of this `claude/` index.
|
||||
@@ -0,0 +1,80 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 02 — Cluster fleet
|
||||
|
||||
The repo's cluster directories under `helm-overrides/<cluster>/` are the canonical list of clusters this platform serves. Naming follows three patterns; each implies a different scheduling primitive set, which is why **schedule fields are never copy-pasted between clusters** (see [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md)).
|
||||
|
||||
## Naming taxonomy
|
||||
|
||||
### `k8s-<bu>-prd-ase1[c]`
|
||||
|
||||
Standard GKE prod cluster, BU-owned. All in `asia-southeast1`, fleet `meesho-admin-prd-0622`. Trailing `c` denotes a secondary cluster for the same BU.
|
||||
|
||||
Known BUs in this repo:
|
||||
|
||||
- `central`, `central-mqkafka`
|
||||
- `supply`, `supply-dev`
|
||||
- `demand`
|
||||
- `dataengg`, `datascience`, `dengspark`, `dengspark-di`, `dengspark-notebook`, `dscispark`
|
||||
- `dsgpu`
|
||||
- `farmiso`
|
||||
- `ml-platform`
|
||||
- `admin`, `sec-admin`, `devops-admin`
|
||||
|
||||
A few of these are GKE **Autopilot** clusters (different scheduling primitives — see below):
|
||||
- `k8s-central-prd-ase1`
|
||||
- `k8s-dsgpu-prd-ase1`
|
||||
- `k8s-shared-int-ase1`
|
||||
|
||||
The rest are **standard** GKE.
|
||||
|
||||
### `k8s-shared-int-ase1`
|
||||
|
||||
Shared **integration** (pre-prod) cluster — the only non-prod cluster in the repo. Used for integration testing of platform changes before they hit any prod cluster. Autopilot.
|
||||
|
||||
### `k8s-aurva-prd-ase1`
|
||||
|
||||
Aurva integration (third-party security tooling). Minimal override set.
|
||||
|
||||
### `db-<numeric-id>-...`
|
||||
|
||||
Auto-named dataplane / data-tier clusters. Each carries a minimal override set, typically `kube-state-metrics` and `victoria-metrics-agent` only. The `fullnameOverride` convention here is `<kind>-dbc-<bu>-prd` so that metrics labels stay legible across the data-tier fleet.
|
||||
|
||||
## Autopilot vs standard scheduling
|
||||
|
||||
This is the dominant reason scheduling fields cannot be copy-pasted across clusters.
|
||||
|
||||
### Standard GKE clusters
|
||||
|
||||
Use:
|
||||
- `nodeSelector.dedicated: <pool-tag>`
|
||||
- `tolerations[].key: dedicated`
|
||||
- Node pools are explicitly provisioned per workload class.
|
||||
|
||||
### Autopilot clusters
|
||||
|
||||
Use:
|
||||
- `nodeSelector."cloud.google.com/compute-class": <ComputeClass-name>`
|
||||
- `tolerations[].key: cloud.google.com/compute-class` (when applicable)
|
||||
- A `ComputeClass` raw manifest is dropped at `helm-overrides/<cluster>/<app>/computeclass/*-cc.yaml` to declare the class. See [`../docs/platform/schemas/raw-manifest-sidecar-schema.md`](../docs/platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
Mismatched fields → pods strand on the wrong nodes or stay `Pending`. Diagnosis flow: [`../docs/platform/runbooks/pod-pending-scheduling.md`](../docs/platform/runbooks/pod-pending-scheduling.md). Background: [`../wiki/analyses/ADR-A3-per-cluster-scheduling.md`](../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
|
||||
## Multi-Contour scheduling
|
||||
|
||||
Multi-Contour clusters run `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-0`, `contour-internal-intra-1` — each a separate Helm release pinned to its own node pool / dedicated taint or compute class. The per-cluster matrix (which Contour goes where) is documented in the repo-root `contour-nodeselector-tolerations-summary.md`. **Always cross-reference that file before touching a Contour values override.**
|
||||
|
||||
## Cluster onboarding
|
||||
|
||||
Adding a new cluster directory is Layer-1 *high-risk*. The procedure (paired PR with the sister repo's `ApplicationSet`) is in [`../docs/platform/procedures/onboard-new-cluster.md`](../docs/platform/procedures/onboard-new-cluster.md).
|
||||
|
||||
## Cluster deboarding
|
||||
|
||||
Removing a cluster is rare and requires draining Argo Applications first. There is no in-repo procedure today; escalate to the primary owner — see [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md).
|
||||
|
||||
## See also
|
||||
|
||||
- [`./03-chart-inventory.md`](./03-chart-inventory.md) — versioned siblings + multi-Contour pattern
|
||||
- [`./04-override-hierarchy.md`](./04-override-hierarchy.md) — how cluster + chart compose
|
||||
- [`./07-singletons-and-blast-radius.md`](./07-singletons-and-blast-radius.md) — `manifests/priorityclass/<cluster>/`
|
||||
- [`../contour-nodeselector-tolerations-summary.md`](../contour-nodeselector-tolerations-summary.md)
|
||||
@@ -0,0 +1,82 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 03 — Chart inventory
|
||||
|
||||
`helm-templates/` carries about 74 chart directories. They fall into three categories by structure and one cross-cutting category by lifecycle.
|
||||
|
||||
## By structure
|
||||
|
||||
### 1. Vanilla upstream cache
|
||||
|
||||
Most charts are pulled verbatim via `helm pull <repo>/<chart>` and committed. The `templates/` are *not* edited. Editing them silently forks the chart and the edits get clobbered the next time someone refreshes the cache. The pre-commit hooks do not catch this — only PR review does.
|
||||
|
||||
If a fork is intentional, it must be documented in the chart's `README.md` and called out in the PR. See [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md).
|
||||
|
||||
Background: [`../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md`](../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
|
||||
### 2. Thin wrapper
|
||||
|
||||
`Chart.yaml` declares `dependencies:` pointing at one or more upstream charts. The local `templates/` is empty or carries only a thin glue manifest. Used so a single Argo Application can install a stack (e.g., kube-prometheus-stack carries Prometheus + Alertmanager + Grafana + node-exporter + kube-state-metrics together).
|
||||
|
||||
When bumping a wrapper, refresh `Chart.lock` with `helm dependency update helm-templates/<chart>`. See [`../docs/platform/procedures/update-chart-version.md`](../docs/platform/procedures/update-chart-version.md).
|
||||
|
||||
### 3. Intentional fork
|
||||
|
||||
A small number of charts have deliberate `templates/` edits — local CRD patches, label injection, removed sub-resources we don't want, etc. Each fork should be self-documenting in its `README.md`. If the rationale is missing, treat the fork as suspect and escalate per [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md).
|
||||
|
||||
## By lifecycle — versioned siblings
|
||||
|
||||
A chart family often has two siblings live simultaneously to support blue-green migrations:
|
||||
|
||||
| Stable | Migration target | Used for |
|
||||
|--------|------------------|----------|
|
||||
| `argo-cd` | `argo-cd-green` | Green-deploy of the Argo CD control plane itself |
|
||||
| `contour` | `contour-v1.33.3` | Pinned-version migration of the ingress data plane |
|
||||
| `keda` | `keda-2.17.1` | Autoscaler version cutover |
|
||||
| `opentelemetry-collector` | `opentelemetry-collector-latest` | OTel collector cutover |
|
||||
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` | VM cluster cutover |
|
||||
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` | VM agent cutover |
|
||||
| `sonarqube` | (was forward; `sonarqube-old` retained) | SonarQube major cutover |
|
||||
|
||||
The `-green` / `-vX.Y.Z` / `-latest` / `-old` suffix names the **migration target** (or in `-old`'s case, the kept-around predecessor). Both can be live at once on different clusters or even on the same cluster (different Argo Applications). Deletion of a sibling requires confirming zero references in the sister repo. See [`../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [`../docs/platform/procedures/blue-green-chart-migration.md`](../docs/platform/procedures/blue-green-chart-migration.md).
|
||||
|
||||
## Multi-Contour pattern
|
||||
|
||||
Contour is unique in that a single cluster runs **multiple separate Contour Helm releases**, each pinned to its own node pool / compute class. The standard set:
|
||||
|
||||
- `contour-external` — public-facing ingress, primary
|
||||
- `contour-external-1` — public-facing ingress, secondary (capacity / blue-green)
|
||||
- `contour-internal-0`, `contour-internal-1` — internal mesh ingress, redundant pair
|
||||
- `contour-internal-intra-0`, `contour-internal-intra-1` — intra-VPC ingress, redundant pair
|
||||
|
||||
Per-cluster matrix of which release goes on which node pool: repo-root `contour-nodeselector-tolerations-summary.md`. Each release has its own `helm-overrides/<cluster>/<contour-release>/custom-values.yaml`.
|
||||
|
||||
## Notable individual charts
|
||||
|
||||
| Chart | Notes |
|
||||
|-------|-------|
|
||||
| `argo-cd` / `argo-cd-green` | Self-managing — Argo CD installs itself. Sync policy must be careful. |
|
||||
| `vault` | Stateful HA on Raft. Edits to seal config or HA storage require platform-team review. |
|
||||
| `external-secrets` | Source of truth for secret materialization on each cluster. See [`./06-secrets-and-identity.md`](./06-secrets-and-identity.md). |
|
||||
| `kyverno` | Cluster-policy enforcement. Edits change admission behaviour for all workloads. |
|
||||
| `kube-prometheus-stack` | Bundles Prometheus + Alertmanager. Alert rules pages on-call — validate PromQL. See [`../docs/global/coding-guidelines/observability.md`](../docs/global/coding-guidelines/observability.md). |
|
||||
| `external-dns` | Publishes DNS records to Cloud DNS. Often paired with sidecar `external-dns-services/*.yaml` raw manifests. |
|
||||
| `cert-manager` | Issues TLS certs (Let's Encrypt + Vault). |
|
||||
|
||||
## Image registry convention
|
||||
|
||||
Production overrides pin images to Meesho's Artifact Registry mirror:
|
||||
|
||||
```
|
||||
asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>
|
||||
```
|
||||
|
||||
…rather than Docker Hub directly. Mirroring decouples deploys from upstream registry availability and rate limits.
|
||||
|
||||
## See also
|
||||
|
||||
- [`./01-repo-structure.md`](./01-repo-structure.md)
|
||||
- [`./02-cluster-fleet.md`](./02-cluster-fleet.md)
|
||||
- [`./04-override-hierarchy.md`](./04-override-hierarchy.md)
|
||||
- [`../docs/platform/procedures/update-chart-version.md`](../docs/platform/procedures/update-chart-version.md)
|
||||
- [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md)
|
||||
@@ -0,0 +1,72 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 04 — Override hierarchy
|
||||
|
||||
How the per-cluster override file, the chart's default values, and any raw-manifest sidecars compose at deploy time.
|
||||
|
||||
## The three layers
|
||||
|
||||
For a given Argo Application targeting `<cluster>` × `<app>`, the rendered manifests come from three sources:
|
||||
|
||||
1. **Chart defaults** — `helm-templates/<chart>/values.yaml`. The upstream values, possibly customized in an intentional fork. This is the lowest precedence.
|
||||
2. **Cluster override** — `helm-overrides/<cluster>/<app>/custom-values.yaml`. Helm-merged on top of (1). This is what the agent edits day-to-day.
|
||||
3. **Raw manifest sidecars** — `helm-overrides/<cluster>/<app>/<extra>.yaml` (and optionally subdirectories like `computeclass/`, `external-dns-services/`). These are NOT consumed by Helm. Argo applies them directly to the cluster, in the same Application.
|
||||
|
||||
The Argo Application in the sister repo declares which `path:` (the override directory) and which `helm.valueFiles:` to use. Conventionally the Application points at the override directory and lists `custom-values.yaml`; raw sidecars in the same directory are picked up by Argo's manifest discovery.
|
||||
|
||||
## Helm merge semantics
|
||||
|
||||
Helm performs a **deep merge** of (2) over (1):
|
||||
|
||||
- Maps merge key-by-key.
|
||||
- Lists are **replaced wholesale**, not merged. This is the most common surprise — to extend an upstream list (`tolerations`, `extraArgs`, `extraEnv`), copy the upstream list into the override and edit there. Don't write a list expecting it to append.
|
||||
- `null` in the override deletes the key set in defaults.
|
||||
|
||||
If you need surgical list editing rather than wholesale replacement, you must fork the chart and rewrite the template — almost never the right call. See [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md).
|
||||
|
||||
## Dry-running the merge
|
||||
|
||||
Always render before pushing. The command from the root `CLAUDE.md` Quick reference table:
|
||||
|
||||
```
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
```
|
||||
|
||||
For wrapper charts (`Chart.yaml` `dependencies:`), refresh subcharts first:
|
||||
|
||||
```
|
||||
helm dependency update helm-templates/<chart>
|
||||
```
|
||||
|
||||
For raw sidecars, validate separately:
|
||||
|
||||
```
|
||||
kubectl apply --dry-run=client -f helm-overrides/<cluster>/<app>/<extra>.yaml
|
||||
```
|
||||
|
||||
## Where each piece of config belongs
|
||||
|
||||
| Config | Goes in | Why |
|
||||
|--------|---------|-----|
|
||||
| Image tag pin (production registry) | `custom-values.yaml` | Per-cluster pinning is the whole point of the override layer. |
|
||||
| `replicaCount`, resource requests | `custom-values.yaml` | Per-cluster capacity tuning. |
|
||||
| `nodeSelector`, `tolerations`, `computeClass` | `custom-values.yaml` | Per-cluster node-pool topology. **Never copy-paste across clusters.** |
|
||||
| `fullnameOverride` | `custom-values.yaml` | Pinned to keep Service DNS / PVC binding stable. **Never change** an existing one. |
|
||||
| Helm-managed Service / Deployment / ConfigMap | chart's `templates/` (don't touch) | Owned by upstream chart. |
|
||||
| External-DNS record bound to a Service the chart doesn't manage | `external-dns-services/*.yaml` raw sidecar | Not part of the chart's surface. |
|
||||
| `ComputeClass` definition (Autopilot) | `computeclass/*-cc.yaml` raw sidecar | Cluster-scoped object the chart can't render. |
|
||||
| StorageClass / PriorityClass | `manifests/storageclass/`, `manifests/priorityclass/<cluster>/` | Cluster-wide singleton, separate from any one Application. |
|
||||
| Secret values | **External Secrets Operator** + GCP Secret Manager / Vault | Never in `custom-values.yaml`. See [`./06-secrets-and-identity.md`](./06-secrets-and-identity.md). |
|
||||
|
||||
## Schema details
|
||||
|
||||
- Override schema: [`../docs/platform/schemas/custom-values-schema.md`](../docs/platform/schemas/custom-values-schema.md)
|
||||
- Raw-sidecar schema: [`../docs/platform/schemas/raw-manifest-sidecar-schema.md`](../docs/platform/schemas/raw-manifest-sidecar-schema.md)
|
||||
- Singleton schema: [`../docs/platform/schemas/storageclass-priorityclass-schema.md`](../docs/platform/schemas/storageclass-priorityclass-schema.md)
|
||||
|
||||
## See also
|
||||
|
||||
- [`./00-overview.md`](./00-overview.md)
|
||||
- [`./05-deploy-lifecycle.md`](./05-deploy-lifecycle.md)
|
||||
- [`../docs/global/coding-guidelines/helm-values.md`](../docs/global/coding-guidelines/helm-values.md)
|
||||
@@ -0,0 +1,103 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 05 — Deploy lifecycle
|
||||
|
||||
End-to-end story of how a values change reaches a live cluster, where the safety gates are, and what happens when they fail.
|
||||
|
||||
## The path
|
||||
|
||||
```
|
||||
edit override (branch) → commit (TruffleHog runs) → push → PR
|
||||
→ reviewer approves → merge to main
|
||||
→ Argo CD on cluster reconciles (auto-sync OR human Sync click)
|
||||
→ manifests applied → workload changes
|
||||
```
|
||||
|
||||
## Stage 1 — Branch and edit
|
||||
|
||||
- Branch off `main`. Never push to `main` directly.
|
||||
- Edit one `helm-overrides/<cluster>/<app>/custom-values.yaml` (or its raw sidecars).
|
||||
- No drive-by edits, no cross-cluster normalization in the same PR. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
## Stage 2 — Local validation
|
||||
|
||||
- `helm template` against the override — confirms render succeeds.
|
||||
- For wrappers: `helm dependency update` first.
|
||||
- For raw sidecars: `kubectl apply --dry-run=client`.
|
||||
|
||||
If render fails locally, it will fail in Argo CD's `OutOfSync → SyncFailed`. Fix before pushing.
|
||||
|
||||
## Stage 3 — Commit
|
||||
|
||||
`git commit` triggers pre-commit hooks:
|
||||
|
||||
- **TruffleHog** — blocking. Real secrets bounce. Never `--no-verify`.
|
||||
- **CAC, Yaak** — gated on paths this repo doesn't have, no-op.
|
||||
|
||||
Post-commit:
|
||||
|
||||
- **Cursor metric collector** — background, non-blocking. Pings `observe.meeshogcp.in` with commit telemetry. Failure here does not block.
|
||||
|
||||
Detail: [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md).
|
||||
|
||||
## Stage 4 — PR + review (safety gate 1)
|
||||
|
||||
The reviewer's job:
|
||||
|
||||
1. Confirm the change touches only the cluster × app named in the PR.
|
||||
2. Confirm any per-cluster scheduling fields were rewritten, not copy-pasted.
|
||||
3. Confirm `fullnameOverride` is unchanged.
|
||||
4. Confirm no secret materializes in the file.
|
||||
5. Confirm chart `Chart.yaml` dep bumps came with `Chart.lock` refresh and a changelog reference.
|
||||
6. Confirm versioned-sibling deletes have no sister-repo references.
|
||||
|
||||
If a chart fork is suspected, escalate per [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 1.
|
||||
|
||||
## Stage 5 — Merge to `main`
|
||||
|
||||
Merging is the deploy event. Argo CD on every cluster whose Application points at the changed path will move to `OutOfSync`.
|
||||
|
||||
## Stage 6 — Argo CD reconcile (safety gate 2)
|
||||
|
||||
Two reconciliation modes, set per Application in the sister repo:
|
||||
|
||||
- **Manual sync** (prod default for infra) — Argo waits for a human Sync click. Engineer reviews the diff in the Argo UI before applying.
|
||||
- **Auto-sync** — Argo applies on its own. Reserved for low-risk leaves (`kube-state-metrics`, monitoring agents).
|
||||
|
||||
Background on the manual-sync default: [`../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||||
|
||||
The Argo Application also defines:
|
||||
|
||||
- `syncPolicy.automated.prune` — whether Argo deletes objects no longer in Git.
|
||||
- `syncPolicy.automated.selfHeal` — whether Argo reverts manual cluster edits.
|
||||
- `syncOptions` — `CreateNamespace`, `ServerSideApply`, `RespectIgnoreDifferences`, retry/backoff.
|
||||
- Sync waves via annotations (in chart templates or sidecars).
|
||||
|
||||
These all live in the **sister repo**, not here. See [`../docs/global/coding-guidelines/argocd.md`](../docs/global/coding-guidelines/argocd.md).
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Failure | Where it surfaces | Read |
|
||||
|---------|------------------|------|
|
||||
| Render error in `helm template` | Argo Application status `ComparisonError` | Re-render locally; fix values |
|
||||
| `OutOfSync → SyncFailed` after Sync click | Argo UI events | [`../docs/platform/runbooks/argocd-sync-failure.md`](../docs/platform/runbooks/argocd-sync-failure.md) |
|
||||
| Pods land but stay `Pending` | `kubectl get pods` on target cluster | [`../docs/platform/runbooks/pod-pending-scheduling.md`](../docs/platform/runbooks/pod-pending-scheduling.md) |
|
||||
| Ingress 5xx after Contour change | `contour-external` envoy logs / synthetic probes | [`../docs/platform/runbooks/ingress-down.md`](../docs/platform/runbooks/ingress-down.md) |
|
||||
| Drift reappears after `kubectl edit` | `selfHeal: true` doing its job | Edit Git, not the cluster |
|
||||
|
||||
## Sister-repo coupling
|
||||
|
||||
Almost every non-trivial change is a **paired PR**:
|
||||
|
||||
- New app on cluster: PR here (override) + PR in sister repo (Application).
|
||||
- New cluster: PR here (cluster directory) + PR in sister repo (`ApplicationSet` cluster generator).
|
||||
- Blue-green sibling cutover: PR here (sibling values) + PR in sister repo (Application `targetRevision` / chart path).
|
||||
|
||||
Procedures: [`../docs/platform/procedures/onboard-app-to-cluster.md`](../docs/platform/procedures/onboard-app-to-cluster.md), [`../docs/platform/procedures/onboard-new-cluster.md`](../docs/platform/procedures/onboard-new-cluster.md), [`../docs/platform/procedures/blue-green-chart-migration.md`](../docs/platform/procedures/blue-green-chart-migration.md), [`../docs/platform/procedures/deboard-app.md`](../docs/platform/procedures/deboard-app.md).
|
||||
|
||||
## See also
|
||||
|
||||
- [`./00-overview.md`](./00-overview.md)
|
||||
- [`./04-override-hierarchy.md`](./04-override-hierarchy.md)
|
||||
- [`../docs/architecture.md`](../docs/architecture.md)
|
||||
- [`../docs/global/agent-operations-guide.md`](../docs/global/agent-operations-guide.md)
|
||||
@@ -0,0 +1,70 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 06 — Secrets and identity
|
||||
|
||||
This repo is **values + cached charts**, all of it world-readable from Git. No secret value should ever exist in a file here. The platform pattern is to express secrets as *references* and let the in-cluster machinery materialize them.
|
||||
|
||||
## The components
|
||||
|
||||
### External Secrets Operator (ESO)
|
||||
|
||||
Each cluster carries an `external-secrets/` override directory. ESO runs in the cluster, reads `ExternalSecret` CRs, fetches the secret value from a backend (GCP Secret Manager or Vault), and materializes a Kubernetes `Secret` for workloads to mount.
|
||||
|
||||
- The `ExternalSecret` CR refers to a backend by **name only** (e.g., `gcpsm/prod/<service>/api-token`). The CR is checked into Git; the value is not.
|
||||
- The backend is configured per-cluster in `external-secrets/custom-values.yaml`.
|
||||
|
||||
### GCP Secret Manager
|
||||
|
||||
The default backend for most prod clusters. Secrets are project-scoped under the cluster's GCP project. Workload Identity binds the ESO service account to a Google service account that has `secretmanager.secretAccessor` on the relevant secrets.
|
||||
|
||||
### Vault
|
||||
|
||||
Used where additional capabilities are required — dynamic credentials, transit encryption, PKI. Vault runs in-cluster on Raft HA. Edits to Vault overrides (`helm-overrides/<cluster>/vault/custom-values.yaml`) — especially seal config, HA storage, autounseal — require platform-team review. Vault HA write-path failure is a paging incident. See [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 8.
|
||||
|
||||
### Workload Identity
|
||||
|
||||
GKE Workload Identity binds Kubernetes service accounts to Google service accounts via the `iam.gke.io/gcp-service-account` annotation. This is how pods authenticate to GCP APIs (Secret Manager, Cloud Storage, Pub/Sub) without long-lived keys.
|
||||
|
||||
The annotation is set in the chart's values (`serviceAccount.annotations`) — agent-editable. The IAM binding itself is set up out-of-band via Terraform in the platform IaC repo, not here.
|
||||
|
||||
## What never goes in this repo
|
||||
|
||||
- Plain-text passwords, API tokens, certificates, private keys.
|
||||
- Base64-encoded secrets in `Secret` manifests.
|
||||
- TLS keys / certs (use `cert-manager` issuers + ESO references instead).
|
||||
- GCP service-account JSON keys (Workload Identity replaces them).
|
||||
- OAuth client secrets (Secret Manager → ESO).
|
||||
- Webhook URLs that contain a credential token in the path.
|
||||
|
||||
If the value would be useful to an attacker who clones this repo, it does not belong here.
|
||||
|
||||
## TruffleHog — last line of defense
|
||||
|
||||
The pre-commit hook scans staged content for high-entropy strings and known secret formats. **NEVER bypass.**
|
||||
|
||||
- `git commit --no-verify` is blocked by Sanctity rule.
|
||||
- If the hook flags a real secret, rotate the credential first (the moment it touched a Git working tree it is already half-burned), then move it to ESO.
|
||||
- If the hook flags a false positive, fix the regex in `pre-commit-scripts/` rather than excluding the file.
|
||||
|
||||
Detail: [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md).
|
||||
|
||||
## The pre-commit hook telemetry exception
|
||||
|
||||
The post-commit Cursor metric collector POSTs to `observe.meeshogcp.in`. This is platform-managed automation, not agent-initiated, and is the only outbound call to a `*.meeshogcp.in` host the agent will ever observe in this repo. The agent must still refuse any *new* call to such hosts. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
## Common patterns
|
||||
|
||||
| Pattern | Looks like |
|
||||
|---------|-----------|
|
||||
| Pod reads a Secret Manager value | `ExternalSecret` CR → ESO materializes `Secret` → pod mounts via `envFrom.secretRef` or `volumes.secret` |
|
||||
| Pod calls a GCP API | KSA annotated with `iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com` (Workload Identity) |
|
||||
| TLS for an Ingress | `cert-manager` `Certificate` CR + Vault PKI or Let's Encrypt issuer |
|
||||
| Vault dynamic DB credential | Vault DB secrets engine + ESO `VaultDynamicSecret` (or app-side Vault Agent sidecar) |
|
||||
|
||||
## See also
|
||||
|
||||
- [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md)
|
||||
- [`./07-singletons-and-blast-radius.md`](./07-singletons-and-blast-radius.md)
|
||||
- [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md)
|
||||
- [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md)
|
||||
- [`../docs/global/coding-guidelines/helm-values.md`](../docs/global/coding-guidelines/helm-values.md)
|
||||
@@ -0,0 +1,85 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 07 — Singletons and blast radius
|
||||
|
||||
Most files in this repo affect one cluster × one application. A small number of files affect **everything** on a cluster (or every cluster). These are the singletons. They get a different review bar.
|
||||
|
||||
## `manifests/storageclass/*.yaml`
|
||||
|
||||
Cluster-wide `StorageClass` objects. Every PVC on the cluster either references one of these by name or relies on the default annotation (`storageclass.kubernetes.io/is-default-class: "true"`).
|
||||
|
||||
Wrong here means:
|
||||
|
||||
- New PVCs bind to a different disk type (cost, latency, IOPS change).
|
||||
- `volumeBindingMode` change (Immediate ↔ WaitForFirstConsumer) changes scheduling semantics for every stateful workload.
|
||||
- Default-class flip changes behaviour of every chart that doesn't pin a class explicitly.
|
||||
|
||||
**Layer-1 high risk.** Two reviewers, one of whom must be a cluster BU owner. Schema: [`../docs/platform/schemas/storageclass-priorityclass-schema.md`](../docs/platform/schemas/storageclass-priorityclass-schema.md).
|
||||
|
||||
## `manifests/priorityclass/<cluster>/*.yaml`
|
||||
|
||||
Cluster-wide `PriorityClass` objects, partitioned by cluster directory. Every pod that sets `spec.priorityClassName: <name>` resolves against this set.
|
||||
|
||||
Wrong here means:
|
||||
|
||||
- A `value:` change can swap which workloads preempt others under capacity pressure.
|
||||
- A `globalDefault: true` flip changes behaviour of every pod that omits `priorityClassName`.
|
||||
- Removing a `PriorityClass` referenced by a live workload causes admission failure on next pod create.
|
||||
|
||||
**Layer-1 high risk.** Same review policy as StorageClass. Schema: same file as above.
|
||||
|
||||
## `repository.yaml`
|
||||
|
||||
Owners, secondary owners, repo metadata. Owned by the **`registry-bootstrap` automation**, not by humans. Hand edits will be reverted on the next `registry-bootstrap` run.
|
||||
|
||||
**Layer-3 — refuse.** If asked to edit, redirect to `registry-bootstrap`. See [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 4.
|
||||
|
||||
## `manifests/{jenkins-filestore-caching,jenkins-gcs-caching,jfrog-filestore-data}/{dev,prd}/`
|
||||
|
||||
Per-env, one-shot PV / PVC manifests for stateful systems (Jenkins build cache, JFrog binary store). Bound to GCP Filestore or GCS. Once a PVC is bound to a PV with a real backend, you cannot move it without data migration.
|
||||
|
||||
**Layer-1 high risk.** Resize via `resources.requests.storage` only; do not recreate.
|
||||
|
||||
## Versioned-sibling chart deletion
|
||||
|
||||
Deleting a chart directory under `helm-templates/` (e.g., removing `argo-cd-green` after a successful migration) is irreversible from Argo CD's point of view — any cluster whose Application still points at it will fail to render.
|
||||
|
||||
**Layer-1 high risk.** Confirm zero references in [`Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config) first. See [`../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) and [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 10.
|
||||
|
||||
## Chart `templates/` edits
|
||||
|
||||
Editing `helm-templates/<chart>/templates/` or `values.yaml` of a vanilla-pulled chart silently forks it. The next refresh clobbers the edit, but until then it ships to every cluster that consumes the chart.
|
||||
|
||||
**Layer-1 high risk.** Allowed only if the fork is intentional and documented in the chart's `README.md`. See [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md).
|
||||
|
||||
## Kyverno cluster policies
|
||||
|
||||
`helm-overrides/<cluster>/kyverno/custom-values.yaml` configures admission policies. A new `enforce`-mode `ClusterPolicy` can block every pod admission on a cluster.
|
||||
|
||||
**Layer-1 high risk.** Roll out in `audit` mode first, observe `PolicyReport`s, then flip to `enforce`.
|
||||
|
||||
## Argo CD itself (`argo-cd` / `argo-cd-green`)
|
||||
|
||||
Argo CD self-manages — it deploys itself from this repo. A bad values change can break the control plane that would otherwise heal it. Recovery requires `kubectl` access to apply a hand-rendered manifest.
|
||||
|
||||
**Layer-1 high risk.** Always cut a green sibling first; never edit the live release directly.
|
||||
|
||||
## Decision summary
|
||||
|
||||
| Singleton | Layer | Review policy |
|
||||
|-----------|-------|---------------|
|
||||
| `manifests/storageclass/*.yaml` | 1 high-risk | Two reviewers, one cluster BU owner |
|
||||
| `manifests/priorityclass/<cluster>/*.yaml` | 1 high-risk | Two reviewers, one cluster BU owner |
|
||||
| `repository.yaml` | 3 | Refuse; redirect to `registry-bootstrap` |
|
||||
| `manifests/{jenkins,jfrog}-…/{dev,prd}/` | 1 high-risk | Two reviewers; resize-only edits |
|
||||
| Versioned-sibling chart deletion | 1 high-risk | Confirm sister-repo zero references |
|
||||
| `helm-templates/<chart>/templates/` edits | 1 high-risk | README must document the fork |
|
||||
| Kyverno enforce-mode policy | 1 high-risk | Audit-mode rollout first |
|
||||
| Argo CD self-managed values | 1 high-risk | Cut a green sibling first |
|
||||
|
||||
## See also
|
||||
|
||||
- [`./01-repo-structure.md`](./01-repo-structure.md)
|
||||
- [`../docs/platform/schemas/storageclass-priorityclass-schema.md`](../docs/platform/schemas/storageclass-priorityclass-schema.md)
|
||||
- [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md)
|
||||
- [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md)
|
||||
@@ -0,0 +1,69 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 08 — Pre-commit and hooks
|
||||
|
||||
What runs when you `git commit` here, what blocks, what doesn't, and why nothing should be bypassed.
|
||||
|
||||
## Hook installation
|
||||
|
||||
One-time, per clone:
|
||||
|
||||
```
|
||||
pre-commit install \
|
||||
--hook-type pre-commit \
|
||||
--hook-type pre-push \
|
||||
--hook-type post-commit
|
||||
```
|
||||
|
||||
If the hooks aren't installed, the local commit will skip them — but PR review is the catch-net, and a missed scan in a feature branch can still catch the secret before merge.
|
||||
|
||||
## Active hooks
|
||||
|
||||
### TruffleHog (pre-commit, blocking)
|
||||
|
||||
Scans the staged content for high-entropy strings and known secret patterns (AWS keys, GCP service-account JSON, GitHub tokens, generic JWTs, etc.).
|
||||
|
||||
- **Blocks the commit** on any positive match.
|
||||
- **NEVER bypass** with `git commit --no-verify` or `git commit -n`. This is on the don't-touch list — see [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
- If the hook fires on a **real secret**: stop, rotate the credential immediately (any value that touched a Git working tree is half-burned), then move to External Secrets Operator. See [`./06-secrets-and-identity.md`](./06-secrets-and-identity.md).
|
||||
- If the hook fires on a **false positive**: fix the regex in `pre-commit-scripts/` rather than skip-listing the file. The fix is reusable across the org.
|
||||
|
||||
### CAC and Yaak (pre-commit / pre-push, gated)
|
||||
|
||||
These hooks exist in the platform's standard `.pre-commit-config.yaml`, but they are gated on file paths this repo doesn't carry (CAC config files, Yaak collections). They no-op here. The same scripts run for real in service repos.
|
||||
|
||||
If a future change ever introduces matching paths, the hooks will start firing — read their messages and fix forward. Do not disable.
|
||||
|
||||
## Background hooks
|
||||
|
||||
### Cursor AI commit metric collector (post-commit, non-blocking)
|
||||
|
||||
Posts a metric ping to `observe.meeshogcp.in` describing the commit (author, files touched, AI tool used). Runs in the background, does not block, and silently drops on failure.
|
||||
|
||||
- This is **the only sanctioned outbound call to a `*.meeshogcp.in` host** the agent should ever observe in this repo. Agents must still refuse to *initiate* any such call themselves. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
- If the post-commit script is failing, that's a platform issue — escalate per [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md). Do not remove the script.
|
||||
|
||||
## Why never `--no-verify`
|
||||
|
||||
A bypassed pre-commit hook is invisible to the PR reviewer. Real secrets ship through merged PRs are very expensive to recover from:
|
||||
|
||||
- The credential itself must be rotated everywhere it's used.
|
||||
- The Git history must be force-rewritten (and even then, the Git push may be cached on a mirror).
|
||||
- Any system that ingested the secret value (CI logs, Slack quotes, downstream forks) is now compromised.
|
||||
|
||||
The 5 seconds saved bypassing the hook is a 5-day-or-more incident later.
|
||||
|
||||
## When the hook is wrong
|
||||
|
||||
Two kinds of false-positive:
|
||||
|
||||
1. **Pattern over-matches** — TruffleHog regex matches a non-secret high-entropy string (a hash, a UUID, a build label). Fix: tighten the regex in `pre-commit-scripts/`.
|
||||
2. **Genuine fixture / test data** — a fake-looking string in a chart's example values or test fixture. Fix: same — tighten the pattern, or move the fixture to a path TruffleHog already excludes (chart `templates/` test fixtures usually qualify).
|
||||
|
||||
Either way, the fix is in the hook, not in the bypass.
|
||||
|
||||
## See also
|
||||
|
||||
- [`./06-secrets-and-identity.md`](./06-secrets-and-identity.md)
|
||||
- [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md)
|
||||
- [`../docs/global/agent-operations-guide.md`](../docs/global/agent-operations-guide.md)
|
||||
@@ -0,0 +1,67 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 09 — Common tasks
|
||||
|
||||
Index of the procedures and skills checked into the repo. Pick the task, follow the link, run the playbook.
|
||||
|
||||
## Procedures (deeper, multi-step, often paired with sister-repo PR)
|
||||
|
||||
| Task | Read |
|
||||
|------|------|
|
||||
| Onboard a new app to an existing cluster | [`../docs/platform/procedures/onboard-app-to-cluster.md`](../docs/platform/procedures/onboard-app-to-cluster.md) |
|
||||
| Onboard a new cluster | [`../docs/platform/procedures/onboard-new-cluster.md`](../docs/platform/procedures/onboard-new-cluster.md) |
|
||||
| Deboard / remove an app from a cluster | [`../docs/platform/procedures/deboard-app.md`](../docs/platform/procedures/deboard-app.md) |
|
||||
| Bump a chart version (Chart.yaml deps) | [`../docs/platform/procedures/update-chart-version.md`](../docs/platform/procedures/update-chart-version.md) |
|
||||
| Cut a blue-green sibling and migrate to it | [`../docs/platform/procedures/blue-green-chart-migration.md`](../docs/platform/procedures/blue-green-chart-migration.md) |
|
||||
| Intentionally fork an upstream chart | [`../docs/platform/procedures/fork-upstream-chart.md`](../docs/platform/procedures/fork-upstream-chart.md) |
|
||||
|
||||
## Skills (focused, single-task playbooks)
|
||||
|
||||
| Skill | Read |
|
||||
|-------|------|
|
||||
| Bump a chart version (concise checklist) | [`../skills/infra/bump-chart-version.md`](../skills/infra/bump-chart-version.md) |
|
||||
| Diagnose pods stuck `Pending` (scheduling) | [`../skills/infra/diagnose-scheduling.md`](../skills/infra/diagnose-scheduling.md) |
|
||||
| Onboard an app (concise checklist) | [`../skills/infra/onboard-app.md`](../skills/infra/onboard-app.md) |
|
||||
|
||||
## Runbooks (failure response)
|
||||
|
||||
| Symptom | Read |
|
||||
|---------|------|
|
||||
| Argo CD `OutOfSync → SyncFailed` | [`../docs/platform/runbooks/argocd-sync-failure.md`](../docs/platform/runbooks/argocd-sync-failure.md) |
|
||||
| Pods stuck `Pending` | [`../docs/platform/runbooks/pod-pending-scheduling.md`](../docs/platform/runbooks/pod-pending-scheduling.md) |
|
||||
| Ingress 5xx after a Contour change | [`../docs/platform/runbooks/ingress-down.md`](../docs/platform/runbooks/ingress-down.md) |
|
||||
|
||||
## Schemas (reference while editing)
|
||||
|
||||
| Surface | Read |
|
||||
|---------|------|
|
||||
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | [`../docs/platform/schemas/custom-values-schema.md`](../docs/platform/schemas/custom-values-schema.md) |
|
||||
| Raw `<extra>.yaml` sidecars in override dirs | [`../docs/platform/schemas/raw-manifest-sidecar-schema.md`](../docs/platform/schemas/raw-manifest-sidecar-schema.md) |
|
||||
| Cluster-wide `StorageClass` / `PriorityClass` | [`../docs/platform/schemas/storageclass-priorityclass-schema.md`](../docs/platform/schemas/storageclass-priorityclass-schema.md) |
|
||||
|
||||
## Coding guidelines
|
||||
|
||||
| Domain | Read |
|
||||
|--------|------|
|
||||
| Helm values conventions | [`../docs/global/coding-guidelines/helm-values.md`](../docs/global/coding-guidelines/helm-values.md) |
|
||||
| Argo CD interaction model | [`../docs/global/coding-guidelines/argocd.md`](../docs/global/coding-guidelines/argocd.md) |
|
||||
| Observability stack | [`../docs/global/coding-guidelines/observability.md`](../docs/global/coding-guidelines/observability.md) |
|
||||
|
||||
## Operating discipline
|
||||
|
||||
| Topic | Read |
|
||||
|-------|------|
|
||||
| Pre-flight + authoring loop | [`../docs/global/agent-operations-guide.md`](../docs/global/agent-operations-guide.md) |
|
||||
| Don't-touch list | [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md) |
|
||||
| Layer classification | [`../docs/global/AGENT_BOUNDARIES.md`](../docs/global/AGENT_BOUNDARIES.md) |
|
||||
| Escalation table | [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) |
|
||||
|
||||
## ADRs (rationale, when you want to know *why*)
|
||||
|
||||
| ADR | Read |
|
||||
|-----|------|
|
||||
| Why cache charts here vs pull from upstream at deploy | [`../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md`](../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md) |
|
||||
| Why versioned-sibling charts (`-green`, `-vX.Y.Z`, `-latest`) | [`../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) |
|
||||
| Why per-cluster scheduling fields cannot be shared | [`../wiki/analyses/ADR-A3-per-cluster-scheduling.md`](../wiki/analyses/ADR-A3-per-cluster-scheduling.md) |
|
||||
| Why raw manifest sidecars live next to overrides | [`../wiki/analyses/ADR-A4-raw-manifest-sidecars-in-helm-overrides.md`](../wiki/analyses/ADR-A4-raw-manifest-sidecars-in-helm-overrides.md) |
|
||||
| Why manual Argo sync is the prod default | [`../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) |
|
||||
@@ -0,0 +1,61 @@
|
||||
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# 10 — Glossary and references
|
||||
|
||||
Short definitions for the terms that recur in this repo's docs, and outbound links for deep dives.
|
||||
|
||||
## Glossary
|
||||
|
||||
**Argo CD** — GitOps continuous-delivery controller. Reconciles a cluster's actual state to a Git-declared desired state. Each cluster runs its own Argo CD instance; each Argo CD instance hosts a set of `Application` objects.
|
||||
|
||||
**Application (Argo)** — A single deployable unit. Points at a Git repo + path + revision + chart-and-values config, and a destination (cluster + namespace). In our setup, the source path is in **this** repo; the Application manifest itself is in the **sister repo**.
|
||||
|
||||
**ApplicationSet** — A controller-side template that fans out one Application per cluster (or per cluster × app). Used in the sister repo to express "deploy `victoria-metrics-agent` to every prod cluster" once instead of N times.
|
||||
|
||||
**BU (Business Unit)** — Meesho-internal grouping that owns a cluster. Encoded in the cluster name: `k8s-<bu>-prd-ase1[c]`. Examples: `central`, `supply`, `demand`, `dataengg`, `ml-platform`.
|
||||
|
||||
**Autopilot (GKE)** — Google's managed-node-pool flavour of GKE. Scheduling primitives are different from standard GKE — uses `cloud.google.com/compute-class` instead of `dedicated:` taints. The repo has three Autopilot clusters: `k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`. See [`./02-cluster-fleet.md`](./02-cluster-fleet.md).
|
||||
|
||||
**ESO (External Secrets Operator)** — In-cluster operator that reads `ExternalSecret` CRs and materializes Kubernetes `Secret` objects from a remote backend (GCP Secret Manager, Vault). The mechanism that keeps secret values out of this repo. See [`./06-secrets-and-identity.md`](./06-secrets-and-identity.md).
|
||||
|
||||
**ComputeClass** — A GKE Autopilot CR that describes a node-pool selection policy (machine family, accelerators, spot eligibility). Workloads target a ComputeClass via `nodeSelector."cloud.google.com/compute-class": <name>`. CRs live as raw sidecars in `helm-overrides/<cluster>/<app>/computeclass/`.
|
||||
|
||||
**fullnameOverride** — A Helm values key consumed by most charts to fix the resource name prefix. **Load-bearing** — Service DNS names, PVC bindings, ConfigMap references all key off it. Never change for a live release. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
**Sister repo** — [`Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config). Owns the Argo `Application` / `ApplicationSet` manifests that point at paths in this repo. See [`../docs/global/coding-guidelines/argocd.md`](../docs/global/coding-guidelines/argocd.md).
|
||||
|
||||
**External Secrets** — short for the External Secrets Operator (above), or the `ExternalSecret` CR it consumes.
|
||||
|
||||
**Blue-green sibling** — A second chart directory under `helm-templates/` (`-green`, `-vX.Y.Z`, `-latest`, `-old`) that exists alongside the stable chart to support a phased migration. Both can be live simultaneously. See [`../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md`](../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
|
||||
**helm-overrides** — Top-level dir holding per-cluster × per-app values overlays (`<cluster>/<app>/custom-values.yaml`) and raw-manifest sidecars. The agent-edited surface.
|
||||
|
||||
**helm-templates** — Top-level dir holding cached / forked upstream charts. Mostly read-only. Edits silently fork unless intentional.
|
||||
|
||||
**Manual sync** — Argo `syncPolicy.automated` is unset; reconciliation requires a human Sync click in the Argo UI. The default for prod infra. See [`../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||||
|
||||
**Workload Identity** — GKE feature that binds a Kubernetes service account to a Google service account via the `iam.gke.io/gcp-service-account` annotation. Replaces long-lived JSON service-account keys.
|
||||
|
||||
**TruffleHog** — Pre-commit secret scanner. Active and blocking on this repo. Never bypass.
|
||||
|
||||
**Layer 1 / Layer 3** — Agent authority classification from the AI Blitz Plan. Layer 1 = agent-writable (this repo, mostly). Layer 3 = refuse and redirect (`repository.yaml` edits, production endpoint probes). See [`../docs/global/AGENT_BOUNDARIES.md`](../docs/global/AGENT_BOUNDARIES.md).
|
||||
|
||||
## References — internal
|
||||
|
||||
- Repo-root `CLAUDE.md` — authoritative facts list.
|
||||
- [`../docs/architecture.md`](../docs/architecture.md) — full deploy lifecycle and gotchas.
|
||||
- [`../wiki/entities/DevOps Infra Helm Charts.md`](../wiki/entities/DevOps%20Infra%20Helm%20Charts.md) — entity model.
|
||||
- [`./09-common-tasks.md`](./09-common-tasks.md) — task index.
|
||||
|
||||
## References — external
|
||||
|
||||
- **Sister repo** (Argo Application manifests): [`github.com/Meesho/devops-infra-argo-config`](https://github.com/Meesho/devops-infra-argo-config)
|
||||
- **AI Blitz Plan** — internal Confluence; ask the primary owner for the current link.
|
||||
- **Argo CD** — [argo-cd.readthedocs.io](https://argo-cd.readthedocs.io/), chart at [github.com/argoproj/argo-helm](https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd)
|
||||
- **Contour** — [projectcontour.io](https://projectcontour.io/), chart at [github.com/bitnami/charts/tree/main/bitnami/contour](https://github.com/bitnami/charts/tree/main/bitnami/contour)
|
||||
- **VictoriaMetrics** — [docs.victoriametrics.com](https://docs.victoriametrics.com/), charts at [github.com/VictoriaMetrics/helm-charts](https://github.com/VictoriaMetrics/helm-charts)
|
||||
- **HashiCorp Vault** — [developer.hashicorp.com/vault](https://developer.hashicorp.com/vault), chart at [github.com/hashicorp/vault-helm](https://github.com/hashicorp/vault-helm)
|
||||
- **KEDA** — [keda.sh](https://keda.sh/), chart at [github.com/kedacore/charts](https://github.com/kedacore/charts)
|
||||
- **Kyverno** — [kyverno.io](https://kyverno.io/), chart at [github.com/kyverno/kyverno/tree/main/charts](https://github.com/kyverno/kyverno/tree/main/charts)
|
||||
- **External Secrets Operator** — [external-secrets.io](https://external-secrets.io/)
|
||||
- **GKE Autopilot ComputeClass** — [cloud.google.com/kubernetes-engine/docs/concepts/autopilot-compute-classes](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-compute-classes)
|
||||
Reference in New Issue
Block a user