149 lines
16 KiB
Markdown
149 lines
16 KiB
Markdown
# Architecture
|
||
|
||
This is a **GitOps Helm values repository**, not a service repo. There is no application code, no build, no tests — only declarative YAML (Helm charts, value overrides, Kubernetes manifests) and two git-hook shell scripts. Argo CD is the runtime; merging to `main` is the deployment.
|
||
|
||
## Section 1 — High-level design
|
||
|
||
### Repo purpose
|
||
|
||
Centralizes (1) cached/forked upstream Helm charts and (2) per-cluster value overrides for every infrastructure tool Meesho runs on its GKE fleet — observability (VictoriaMetrics, Mimir, Loki, Tempo, Grafana, OpenTelemetry, Pyroscope), ingress/edge (Contour/Envoy, ingress-nginx, cert-manager, external-dns, external-secrets), platform (Argo CD, Vault, Keda, Kyverno, Flagger, Jenkins, JFrog, Rancher, SonarQube), and data/AI (ClickHouse, Temporal, Superset, Deepgram, Aurva, Deepfence). It was carved out of `gcp-devops-admin` and is the source of truth for **what gets installed where, with what values**.
|
||
|
||
### System context
|
||
|
||
| Edge | Plays |
|
||
|------|-------|
|
||
| Sister repo `devops-infra-argo-config` | `github.com/Meesho/devops-infra-argo-config`. Holds the Argo CD `Application` / `ApplicationSet` manifests that point at this repo's `helm-overrides/<cluster>/<app>/` paths. Argo Application changes are PRs against that repo, not this one. |
|
||
| Argo CD instance(s) | Reconciles cluster state from this repo + the argo-config repo. One Argo CD per cluster (or per BU); each cluster has an `argocd/custom-values.yaml` here that configures *its own* Argo CD. |
|
||
| GKE cluster fleet | All consumers. Standard GKE clusters (`k8s-<bu>-prd-ase1`) plus auto-named dataplane clusters (`db-<numeric-id>-...`). Region: `asia-southeast1`. Project fleet: `meesho-admin-prd-0622`. |
|
||
| TruffleHog webhook | `https://observe.meeshogcp.in/api/webhook` — pre-commit hook reports verified secret findings. |
|
||
| CAC API | `https://observe.meeshogcp.in/api/cac/repos` — pre-commit allowlist; this repo isn't on the list, so the `cac validate` hook is a no-op. |
|
||
| Cursor metrics API | `https://cursor-server.meeshogcp.in/api/v1/...` — post-commit hook ships per-commit Cursor AI usage metrics. Local-only side effect. |
|
||
|
||
### Module boundaries
|
||
|
||
| Top-level dir | What it owns |
|
||
|---------------|--------------|
|
||
| `helm-templates/<chart>/` | Cached or forked upstream Helm chart. Touch only when a deliberate fork update is needed. Most are vanilla upstream — `Chart.yaml` + `templates/` + `values.yaml` (default upstream values). |
|
||
| `helm-templates/<chart>-<variant>/` | Versioned/blue-green sibling charts: `argo-cd` + `argo-cd-green`, `contour` + `contour-v1.33.3`, `keda` + `keda-2.17.1`, `opentelemetry-collector` + `-latest`, `victoria-metrics-cluster` + `-latest`, `victoria-metrics-agent` + `-latest`, `sonarqube` + `sonarqube-old`. The variant is the **target** of an in-flight chart upgrade — old version stays until the migration finishes. |
|
||
| `helm-overrides/<cluster>/<app>/custom-values.yaml` | Cluster × application override values. Argo CD's `helm.valueFiles` points here; values merge over the chart's own `values.yaml` (or the upstream subchart's defaults when the local chart is a thin wrapper). |
|
||
| `helm-overrides/<cluster>/<app>/<extra>.yaml` | Non-`custom-values` files: extra Kubernetes resources rendered by an Argo Application's `path:` (e.g., `computeclass/*-cc.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`, `external-dns-services/*.yaml`). These are not Helm values — they are raw manifests applied alongside the Helm release. |
|
||
| `manifests/` | One-shot, cluster-scoped resources applied outside the Helm flow: `storageclass/`, `priorityclass/<cluster>/`, `jenkins-filestore-caching/{dev,prd}/`, `jenkins-gcs-caching/`, `jfrog-filestore-data/{dev,prd}/`. These are singletons — wrong values affect every workload in the cluster. |
|
||
| `pre-commit-scripts/` | `runner.sh` (parallel exec), `trufflehog-hook.sh` (verified-secret scan + webhook), `cac-validate.sh` (no-op here — gated on `configs/` path), `yaakhook.sh` (no-op here — gated on `api-collections/` path). |
|
||
| `post-commit-scripts/` | `runner.sh` + `commit-metric.sh` — Cursor AI commit metric collector (forks to background; never blocks). |
|
||
| `repository.yaml` | Owners (auto-managed by registry-bootstrap). |
|
||
|
||
### Architecture philosophy
|
||
|
||
**Reliability-first, surgical edits.** A bad values change can take down ingress, observability, or an entire cluster. Two rules govern every change:
|
||
1. **Reliability-first** — production blast radius is huge; mirror the existing pattern of neighbor cluster overrides; never delete keys without checking what depends on them; preserve explicit limits and HPA bounds.
|
||
2. **Surgical** — touch only what was asked. Don't refactor surrounding values, don't "normalize" across clusters in the same PR, don't drive-by-edit other charts in the same cluster's directory.
|
||
|
||
### Data flow (deployment lifecycle)
|
||
|
||
```
|
||
edit helm-overrides/<cluster>/<app>/custom-values.yaml
|
||
│
|
||
▼
|
||
git commit ──► pre-commit hooks (TruffleHog secrets scan)
|
||
│
|
||
▼
|
||
git push ──► PR → review → merge to main
|
||
│
|
||
▼
|
||
Argo CD on each cluster polls this repo + devops-infra-argo-config
|
||
│
|
||
▼
|
||
Application sync: helm template <chart> -f <override> → apply to cluster
|
||
│
|
||
▼
|
||
post-commit hook ships Cursor AI metrics (background, non-blocking)
|
||
```
|
||
|
||
Argo CD resolves `<chart>` from `helm-templates/` (when the Application uses `repoURL` of this repo with `path: helm-templates/<name>`) or pulls upstream by reading the local `Chart.yaml` dependencies (e.g., `argo-cd/Chart.yaml` declares `argo-cd 7.7.23` from `argoproj.github.io/argo-helm`).
|
||
|
||
### Cross-cutting concerns
|
||
|
||
- **Secret management** — TruffleHog pre-commit hook (`pre-commit-scripts/trufflehog-hook.sh`) blocks any verified secret. Reports go to the security webhook with content hash + commit + file/line metadata. NEVER bypass. Real secrets are externalised via `external-secrets` (per-cluster override exists in most clusters) backed by GCP Secret Manager / Vault.
|
||
- **Per-cluster scheduling** — every cluster has its own `nodeSelector` / `tolerations` / `computeclass` topology. The `contour-nodeselector-tolerations-summary.md` at the repo root documents the current matrix. GKE Autopilot clusters (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) use `cloud.google.com/compute-class` keys; standard clusters use `dedicated:` keys. Copying values between clusters without rewriting these is a reliable way to schedule pods on the wrong nodes.
|
||
- **Versioned chart migrations** — when upgrading a chart, the new version lives as a sibling dir (`argo-cd-green`, `contour-v1.33.3`, `keda-2.17.1`) until cutover. Both directories may be referenced by Argo Applications during the transition. Don't delete the old sibling without confirming no Application still points at it.
|
||
- **Image registry** — most overrides pin images to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho's internal Artifact Registry mirror), not upstream Docker Hub.
|
||
|
||
## Section 2 — Low-level details
|
||
|
||
### Cluster fleet (helm-overrides/)
|
||
|
||
Each top-level dir under `helm-overrides/` is one cluster. Two naming conventions:
|
||
|
||
| Convention | Example | Owner / type |
|
||
|------------|---------|--------------|
|
||
| `k8s-<bu>-prd-ase1` (and `-prd-ase1c`) | `k8s-central-prd-ase1`, `k8s-supply-prd-ase1`, `k8s-dataengg-prd-ase1` | Standard GKE cluster, BU-owned (central, supply, demand, dataengg, datascience, dengspark, dscispark, dsgpu, farmiso, ml-platform, admin, sec-admin, devops-admin, central-mqkafka). Project varies per BU (e.g., `meesho-supply-prd`, `meesho-datascience-prd`). All in `asia-southeast1`. |
|
||
| `k8s-shared-int-ase1` | (only) | Shared **integration** (pre-prod) cluster. The only non-prod cluster in this repo. |
|
||
| `k8s-aurva-prd-ase1` | (only) | Aurva integration. Limited override set (contour-internal, rancher only). |
|
||
| `db-<numeric-id>-...` | `db-2516183257845181-c-1204-195038-428` | Auto-named dataplane / data-tier clusters. Override sets are minimal (typically `kube-state-metrics` + `victoria-metrics-agent` only). `fullnameOverride` values use `dbc-<bu>-prd` form (e.g., `dbc-dsci-prd`). |
|
||
| `k8s-supply-dev-ase1` | (only) | Dev/sandbox supply cluster. |
|
||
|
||
Per-cluster READMEs (where present) say: "This folder contains the custom `values.yaml` files organized based on specific cluster names. Each subdirectory corresponds to a particular cluster and holds the configurations for the applications and tools deployed within that cluster."
|
||
|
||
### Application directory layout (per cluster)
|
||
|
||
Inside `helm-overrides/<cluster>/`, each subdirectory is an Argo CD Application. Common shapes:
|
||
|
||
| Layout | Meaning | Example |
|
||
|--------|---------|---------|
|
||
| `<app>/custom-values.yaml` | Single Helm release; values merged onto a chart from `helm-templates/<chart>` | `argocd/custom-values.yaml`, `etcd/custom-values.yaml`, `contour-internal-0/custom-values.yaml` |
|
||
| `<app>/<sub>.yaml` (no custom-values.yaml) | Argo Application's `path:` points here; raw manifests applied | `computeclass/contour-external-cc.yaml`, `elastic-cluster/argo-launch.yaml`, `mimir-distributed/alertmanager_config.yaml`, `external-dns-services/*.yaml` |
|
||
| Both | Helm release + sidecar raw manifests | rare; check the matching Application in `devops-infra-argo-config` |
|
||
|
||
Multiple Contour instances per cluster is the norm — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-0`, `contour-internal-intra-1`. Each maps to a different node pool / dedicated node taint or compute class. The mapping per cluster is recorded in `contour-nodeselector-tolerations-summary.md` at the repo root — read before adding/changing a Contour instance.
|
||
|
||
### Helm chart inventory
|
||
|
||
| Category | Charts in `helm-templates/` |
|
||
|----------|----------------------------|
|
||
| Argo / GitOps | `argo-cd`, `argo-cd-green` |
|
||
| Ingress / edge | `contour`, `contour-v1.33.3`, `contour-ca-issuer`, `contour-cert-checker`, `ingress-nginx`, `cert-manager`, `external-dns`, `external-secrets` |
|
||
| Observability — metrics | `prometheus-node-exporter`, `prometheus-stackdriver-exporter`, `kube-state-metrics`, `kube-events`, `victoria-metrics-{single,cluster,cluster-latest,agent,agent-latest,alert,alert-stateful,alerts-config,auth,mcp}`, `vm-alert-config`, `mimir-distributed`, `pmm`, `telegraf-operator` |
|
||
| Observability — logs/traces/profiles | `fluentd`, `loki-distributed`, `tempo-distributed`, `pyroscope`, `alloy`, `opentelemetry-collector`, `opentelemetry-collector-latest`, `opentelemetry-operator`, `elastalert2`, `coroot-node-agent`, `deepfence-console`, `deepfence-router` |
|
||
| UI / dashboards | `grafana`, `grafana-edge`, `grafana-mcp`, `kubernetes-dashboard`, `superset`, `uptime-kuma` |
|
||
| Workflow / CI/CD | `jenkins`, `jfrog`, `sonarqube`, `sonarqube-old`, `flagger`, `keda`, `keda-2.17.1`, `kyverno`, `loadtester`, `temporal`, `dind`, `canary-bot-gcp`, `paused-container` |
|
||
| Networking / DNS | `coredns`, `kube-dns`, `bifrost`, `conntrack-adjuster`, `node-thp-config` |
|
||
| Data / search / DB | `clickhouse`, `etcd`, `vault`, `elasticsearch-mcp`, `eck-operator`, `athens-proxy` |
|
||
| AI / 3rd-party | `aurva-dataplane`, `deepgram-onprem`, `rancher` |
|
||
|
||
74 charts total. Each has its own `Chart.yaml`. 19 also have a `Chart.lock` (charts with subchart dependencies that have been resolved with `helm dependency update`). Many of the local "charts" (e.g., `argo-cd/Chart.yaml`) are thin wrappers that declare the upstream chart as a dependency in `Chart.yaml` — the actual templates come from upstream. Others (e.g., `contour/`) carry a full vendored `templates/` tree.
|
||
|
||
### Manifests (singletons)
|
||
|
||
| Path | Scope | What it is |
|
||
|------|-------|------------|
|
||
| `manifests/storageclass/*.yaml` | Cluster-wide | StorageClasses: `pd-standard-retain-dr`, `sc-filestore-standard`, `sc-pd-ssd`, `sc-pd-standard`. Wrong change affects every PVC. |
|
||
| `manifests/priorityclass/<cluster>/*.yaml` | Per-cluster | `priorityclass-high.yaml`, `priorityclass-low.yaml` per BU cluster. Affects scheduling priority for every pod that references them. |
|
||
| `manifests/jenkins-filestore-caching/{dev,prd}/{pv,pvc}.yaml` | Per-env | Jenkins build cache PV/PVC backed by GCP Filestore. |
|
||
| `manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml` | Per-env | Jenkins GCS-backed cache. |
|
||
| `manifests/jfrog-filestore-data/{dev,prd}/` | Per-env | JFrog data PV/PVC. |
|
||
|
||
### Git hooks
|
||
|
||
| Hook | Path | Behavior |
|
||
|------|------|----------|
|
||
| pre-commit, pre-push | `pre-commit-scripts/runner.sh` | Forks every other `*.sh` in the same dir in parallel; fails the commit if any fails. |
|
||
| pre-commit, pre-push | `pre-commit-scripts/trufflehog-hook.sh` | Runs `trufflehog git file://. --since-commit HEAD --branch=$(git rev-parse --abbrev-ref HEAD) --json --results=verified`. On a verified hit: prints the finding and POSTs metadata (no raw secret) to `https://observe.meeshogcp.in/api/webhook`. Exit 1 blocks the commit. **Never bypass.** |
|
||
| pre-commit, pre-push | `pre-commit-scripts/cac-validate.sh` | Gated on `configs/` paths in the staged diff. This repo has no `configs/`, so it always early-exits. Documented for completeness. |
|
||
| pre-commit, pre-push | `pre-commit-scripts/yaakhook.sh` | Gated on `api-collections/` paths. This repo has none — early-exits. |
|
||
| post-commit | `post-commit-scripts/runner.sh` → `commit-metric.sh` | Two-phase: synchronous `start` writes a temp file with commit hash + repo info, spawns detached `continue` background process. `continue` polls the local Cursor SQLite DB up to 120 s for `aiCodeTracking.recentCommit.commitHash` to match HEAD, then POSTs the AI line-edit metrics to `https://cursor-server.meeshogcp.in/api/v1/add-commit-metrics`. Skips rebase/merge/cherry-pick commits. Always exits 0 — never blocks the hook. |
|
||
|
||
### Configuration touch points
|
||
|
||
There is no application config to touch. The only env-equivalent layer is `helm-overrides/<cluster>/<app>/custom-values.yaml` — every cluster × application pair is its own configuration unit. The CAC API at `https://observe.meeshogcp.in/api/cac/repos` would gate config schema validation, but this repo isn't on that allowlist.
|
||
|
||
### Critical invariants & gotchas
|
||
|
||
- **`helm-templates/` is mostly upstream code.** Most charts here are `helm pull`-ed copies of upstream charts (Bitnami, ArgoProj, VictoriaMetrics, Grafana). Editing a `templates/*.yaml` inside one of these is editing upstream — easy to forget on the next chart bump. Treat `helm-templates/<chart>/` as read-only unless you are explicitly forking; if you fork, document why in the chart's `README.md`.
|
||
- **Each cluster is unique on `nodeSelector` / `tolerations` / `computeClass`.** See `contour-nodeselector-tolerations-summary.md` at the repo root. GKE Autopilot clusters use `cloud.google.com/compute-class` keys; standard clusters use `dedicated:` keys. Copying a values block from one cluster to another without rewriting these schedules pods on the wrong nodes — or pending forever.
|
||
- **Versioned siblings are intentional, not duplicates.** `argo-cd` vs `argo-cd-green`, `contour` vs `contour-v1.33.3`, `keda` vs `keda-2.17.1`, `opentelemetry-collector` vs `-latest`, `sonarqube` vs `sonarqube-old`. Don't "consolidate" them. They support blue-green chart upgrades — both versions may be live during a migration.
|
||
- **`fullnameOverride` is load-bearing in dataplane overrides.** `db-*` cluster values use `fullnameOverride: <kind>-dbc-<bu>-prd` to keep release names stable across re-installs. Don't change these — Service DNS, PVC binding, and Argo Application names depend on them.
|
||
- **Argo CD reconciles from `main`.** A merge to `main` is a deploy. There is no staging branch — review the PR as if it ships to production, because it does.
|
||
- **`devops-infra-argo-config` is the routing layer.** A new chart in `helm-templates/` does nothing until an `Application` referencing it is added to the sister repo. A new cluster directory in `helm-overrides/` does nothing until an `ApplicationSet` covers it. Pair the two-repo change.
|
||
|
||
<!-- meesho-init: generated-at=2026-05-05T17:31:45Z base-sha=debd20fcc510e19bfe3123a08247dde39790643b+dirty -->
|