devops-infra-helm-charts-gcp/docs/architecture.md

Architecture

This is a GitOps Helm values repository, not a service repo. There is no application code, no build, no tests — only declarative YAML (Helm charts, value overrides, Kubernetes manifests) and two git-hook shell scripts. Argo CD is the runtime; merging to main is the deployment.

Section 1 — High-level design

Repo purpose

Centralizes (1) cached/forked upstream Helm charts and (2) per-cluster value overrides for every infrastructure tool Meesho runs on its GKE fleet — observability (VictoriaMetrics, Mimir, Loki, Tempo, Grafana, OpenTelemetry, Pyroscope), ingress/edge (Contour/Envoy, ingress-nginx, cert-manager, external-dns, external-secrets), platform (Argo CD, Vault, Keda, Kyverno, Flagger, Jenkins, JFrog, Rancher, SonarQube), and data/AI (ClickHouse, Temporal, Superset, Deepgram, Aurva, Deepfence). It was carved out of gcp-devops-admin and is the source of truth for what gets installed where, with what values.

System context

Edge	Plays
Sister repo devops-infra-argo-config	github.com/Meesho/devops-infra-argo-config. Holds the Argo CD Application / ApplicationSet manifests that point at this repo's helm-overrides/<cluster>/<app>/ paths. Argo Application changes are PRs against that repo, not this one.
Argo CD instance(s)	Reconciles cluster state from this repo + the argo-config repo. One Argo CD per cluster (or per BU); each cluster has an argocd/custom-values.yaml here that configures its own Argo CD.
GKE cluster fleet	All consumers. Standard GKE clusters (k8s-<bu>-prd-ase1) plus auto-named dataplane clusters (db-<numeric-id>-...). Region: asia-southeast1. Project fleet: meesho-admin-prd-0622.
TruffleHog webhook	https://observe.meeshogcp.in/api/webhook — pre-commit hook reports verified secret findings.
CAC API	https://observe.meeshogcp.in/api/cac/repos — pre-commit allowlist; this repo isn't on the list, so the cac validate hook is a no-op.
Cursor metrics API	https://cursor-server.meeshogcp.in/api/v1/... — post-commit hook ships per-commit Cursor AI usage metrics. Local-only side effect.

Module boundaries

Top-level dir	What it owns
helm-templates/<chart>/	Cached or forked upstream Helm chart. Touch only when a deliberate fork update is needed. Most are vanilla upstream — Chart.yaml + templates/ + values.yaml (default upstream values).
helm-templates/<chart>-<variant>/	Versioned/blue-green sibling charts: argo-cd + argo-cd-green, contour + contour-v1.33.3, keda + keda-2.17.1, opentelemetry-collector + -latest, victoria-metrics-cluster + -latest, victoria-metrics-agent + -latest, sonarqube + sonarqube-old. The variant is the target of an in-flight chart upgrade — old version stays until the migration finishes.
helm-overrides/<cluster>/<app>/custom-values.yaml	Cluster × application override values. Argo CD's helm.valueFiles points here; values merge over the chart's own values.yaml (or the upstream subchart's defaults when the local chart is a thin wrapper).
helm-overrides/<cluster>/<app>/<extra>.yaml	Non-custom-values files: extra Kubernetes resources rendered by an Argo Application's path: (e.g., computeclass/*-cc.yaml, elastic-cluster/argo-launch.yaml, mimir-distributed/alertmanager_config.yaml, external-dns-services/*.yaml). These are not Helm values — they are raw manifests applied alongside the Helm release.
manifests/	One-shot, cluster-scoped resources applied outside the Helm flow: storageclass/, priorityclass/<cluster>/, jenkins-filestore-caching/{dev,prd}/, jenkins-gcs-caching/, jfrog-filestore-data/{dev,prd}/. These are singletons — wrong values affect every workload in the cluster.
pre-commit-scripts/	runner.sh (parallel exec), trufflehog-hook.sh (verified-secret scan + webhook), cac-validate.sh (no-op here — gated on configs/ path), yaakhook.sh (no-op here — gated on api-collections/ path).
post-commit-scripts/	runner.sh + commit-metric.sh — Cursor AI commit metric collector (forks to background; never blocks).
repository.yaml	Owners (auto-managed by registry-bootstrap).

Architecture philosophy

Reliability-first, surgical edits. A bad values change can take down ingress, observability, or an entire cluster. Two rules govern every change:

1. Reliability-first — production blast radius is huge; mirror the existing pattern of neighbor cluster overrides; never delete keys without checking what depends on them; preserve explicit limits and HPA bounds.
2. Surgical — touch only what was asked. Don't refactor surrounding values, don't "normalize" across clusters in the same PR, don't drive-by-edit other charts in the same cluster's directory.

Data flow (deployment lifecycle)

edit helm-overrides/<cluster>/<app>/custom-values.yaml
            │
            ▼
git commit  ──► pre-commit hooks (TruffleHog secrets scan)
            │
            ▼
git push    ──► PR → review → merge to main
            │
            ▼
Argo CD on each cluster polls this repo + devops-infra-argo-config
            │
            ▼
Application sync: helm template <chart> -f <override>  →  apply to cluster
            │
            ▼
post-commit hook ships Cursor AI metrics (background, non-blocking)

Argo CD resolves <chart> from helm-templates/ (when the Application uses repoURL of this repo with path: helm-templates/<name>) or pulls upstream by reading the local Chart.yaml dependencies (e.g., argo-cd/Chart.yaml declares argo-cd 7.7.23 from argoproj.github.io/argo-helm).

Cross-cutting concerns

• Secret management — TruffleHog pre-commit hook (pre-commit-scripts/trufflehog-hook.sh) blocks any verified secret. Reports go to the security webhook with content hash + commit + file/line metadata. NEVER bypass. Real secrets are externalised via external-secrets (per-cluster override exists in most clusters) backed by GCP Secret Manager / Vault.
• Per-cluster scheduling — every cluster has its own nodeSelector / tolerations / computeclass topology. The contour-nodeselector-tolerations-summary.md at the repo root documents the current matrix. GKE Autopilot clusters (k8s-central-prd-ase1, k8s-dsgpu-prd-ase1, k8s-shared-int-ase1) use cloud.google.com/compute-class keys; standard clusters use dedicated: keys. Copying values between clusters without rewriting these is a reliable way to schedule pods on the wrong nodes.
• Versioned chart migrations — when upgrading a chart, the new version lives as a sibling dir (argo-cd-green, contour-v1.33.3, keda-2.17.1) until cutover. Both directories may be referenced by Argo Applications during the transition. Don't delete the old sibling without confirming no Application still points at it.
• Image registry — most overrides pin images to asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image> (Meesho's internal Artifact Registry mirror), not upstream Docker Hub.

Section 2 — Low-level details

Cluster fleet (helm-overrides/)

Each top-level dir under helm-overrides/ is one cluster. Two naming conventions:

Convention	Example	Owner / type
k8s-<bu>-prd-ase1 (and -prd-ase1c)	k8s-central-prd-ase1, k8s-supply-prd-ase1, k8s-dataengg-prd-ase1	Standard GKE cluster, BU-owned (central, supply, demand, dataengg, datascience, dengspark, dscispark, dsgpu, farmiso, ml-platform, admin, sec-admin, devops-admin, central-mqkafka). Project varies per BU (e.g., meesho-supply-prd, meesho-datascience-prd). All in asia-southeast1.
k8s-shared-int-ase1	(only)	Shared integration (pre-prod) cluster. The only non-prod cluster in this repo.
k8s-aurva-prd-ase1	(only)	Aurva integration. Limited override set (contour-internal, rancher only).
db-<numeric-id>-...	db-2516183257845181-c-1204-195038-428	Auto-named dataplane / data-tier clusters. Override sets are minimal (typically kube-state-metrics + victoria-metrics-agent only). fullnameOverride values use dbc-<bu>-prd form (e.g., dbc-dsci-prd).
k8s-supply-dev-ase1	(only)	Dev/sandbox supply cluster.

Per-cluster READMEs (where present) say: "This folder contains the custom values.yaml files organized based on specific cluster names. Each subdirectory corresponds to a particular cluster and holds the configurations for the applications and tools deployed within that cluster."

Application directory layout (per cluster)

Inside helm-overrides/<cluster>/, each subdirectory is an Argo CD Application. Common shapes:

Layout	Meaning	Example
<app>/custom-values.yaml	Single Helm release; values merged onto a chart from helm-templates/<chart>	argocd/custom-values.yaml, etcd/custom-values.yaml, contour-internal-0/custom-values.yaml
<app>/<sub>.yaml (no custom-values.yaml)	Argo Application's path: points here; raw manifests applied	computeclass/contour-external-cc.yaml, elastic-cluster/argo-launch.yaml, mimir-distributed/alertmanager_config.yaml, external-dns-services/*.yaml
Both	Helm release + sidecar raw manifests	rare; check the matching Application in devops-infra-argo-config

Multiple Contour instances per cluster is the norm — contour-external, contour-external-1, contour-internal-0, contour-internal-1, contour-internal-intra-0, contour-internal-intra-1. Each maps to a different node pool / dedicated node taint or compute class. The mapping per cluster is recorded in contour-nodeselector-tolerations-summary.md at the repo root — read before adding/changing a Contour instance.

Helm chart inventory

Category	Charts in helm-templates/
Argo / GitOps	argo-cd, argo-cd-green
Ingress / edge	contour, contour-v1.33.3, contour-ca-issuer, contour-cert-checker, ingress-nginx, cert-manager, external-dns, external-secrets
Observability — metrics	prometheus-node-exporter, prometheus-stackdriver-exporter, kube-state-metrics, kube-events, victoria-metrics-{single,cluster,cluster-latest,agent,agent-latest,alert,alert-stateful,alerts-config,auth,mcp}, vm-alert-config, mimir-distributed, pmm, telegraf-operator
Observability — logs/traces/profiles	fluentd, loki-distributed, tempo-distributed, pyroscope, alloy, opentelemetry-collector, opentelemetry-collector-latest, opentelemetry-operator, elastalert2, coroot-node-agent, deepfence-console, deepfence-router
UI / dashboards	grafana, grafana-edge, grafana-mcp, kubernetes-dashboard, superset, uptime-kuma
Workflow / CI/CD	jenkins, jfrog, sonarqube, sonarqube-old, flagger, keda, keda-2.17.1, kyverno, loadtester, temporal, dind, canary-bot-gcp, paused-container
Networking / DNS	coredns, kube-dns, bifrost, conntrack-adjuster, node-thp-config
Data / search / DB	clickhouse, etcd, vault, elasticsearch-mcp, eck-operator, athens-proxy
AI / 3rd-party	aurva-dataplane, deepgram-onprem, rancher

74 charts total. Each has its own Chart.yaml. 19 also have a Chart.lock (charts with subchart dependencies that have been resolved with helm dependency update). Many of the local "charts" (e.g., argo-cd/Chart.yaml) are thin wrappers that declare the upstream chart as a dependency in Chart.yaml — the actual templates come from upstream. Others (e.g., contour/) carry a full vendored templates/ tree.

Manifests (singletons)

Path	Scope	What it is
manifests/storageclass/*.yaml	Cluster-wide	StorageClasses: pd-standard-retain-dr, sc-filestore-standard, sc-pd-ssd, sc-pd-standard. Wrong change affects every PVC.
manifests/priorityclass/<cluster>/*.yaml	Per-cluster	priorityclass-high.yaml, priorityclass-low.yaml per BU cluster. Affects scheduling priority for every pod that references them.
manifests/jenkins-filestore-caching/{dev,prd}/{pv,pvc}.yaml	Per-env	Jenkins build cache PV/PVC backed by GCP Filestore.
manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml	Per-env	Jenkins GCS-backed cache.
manifests/jfrog-filestore-data/{dev,prd}/	Per-env	JFrog data PV/PVC.

Git hooks

Hook	Path	Behavior
pre-commit, pre-push	pre-commit-scripts/runner.sh	Forks every other *.sh in the same dir in parallel; fails the commit if any fails.
pre-commit, pre-push	pre-commit-scripts/trufflehog-hook.sh	Runs trufflehog git file://. --since-commit HEAD --branch=$(git rev-parse --abbrev-ref HEAD) --json --results=verified. On a verified hit: prints the finding and POSTs metadata (no raw secret) to https://observe.meeshogcp.in/api/webhook. Exit 1 blocks the commit. Never bypass.
pre-commit, pre-push	pre-commit-scripts/cac-validate.sh	Gated on configs/ paths in the staged diff. This repo has no configs/, so it always early-exits. Documented for completeness.
pre-commit, pre-push	pre-commit-scripts/yaakhook.sh	Gated on api-collections/ paths. This repo has none — early-exits.
post-commit	post-commit-scripts/runner.sh → commit-metric.sh	Two-phase: synchronous start writes a temp file with commit hash + repo info, spawns detached continue background process. continue polls the local Cursor SQLite DB up to 120 s for aiCodeTracking.recentCommit.commitHash to match HEAD, then POSTs the AI line-edit metrics to https://cursor-server.meeshogcp.in/api/v1/add-commit-metrics. Skips rebase/merge/cherry-pick commits. Always exits 0 — never blocks the hook.

Configuration touch points

There is no application config to touch. The only env-equivalent layer is helm-overrides/<cluster>/<app>/custom-values.yaml — every cluster × application pair is its own configuration unit. The CAC API at https://observe.meeshogcp.in/api/cac/repos would gate config schema validation, but this repo isn't on that allowlist.

Critical invariants & gotchas

• helm-templates/ is mostly upstream code. Most charts here are helm pull-ed copies of upstream charts (Bitnami, ArgoProj, VictoriaMetrics, Grafana). Editing a templates/*.yaml inside one of these is editing upstream — easy to forget on the next chart bump. Treat helm-templates/<chart>/ as read-only unless you are explicitly forking; if you fork, document why in the chart's README.md.
• Each cluster is unique on nodeSelector / tolerations / computeClass. See contour-nodeselector-tolerations-summary.md at the repo root. GKE Autopilot clusters use cloud.google.com/compute-class keys; standard clusters use dedicated: keys. Copying a values block from one cluster to another without rewriting these schedules pods on the wrong nodes — or pending forever.
• Versioned siblings are intentional, not duplicates. argo-cd vs argo-cd-green, contour vs contour-v1.33.3, keda vs keda-2.17.1, opentelemetry-collector vs -latest, sonarqube vs sonarqube-old. Don't "consolidate" them. They support blue-green chart upgrades — both versions may be live during a migration.
• fullnameOverride is load-bearing in dataplane overrides. db-* cluster values use fullnameOverride: <kind>-dbc-<bu>-prd to keep release names stable across re-installs. Don't change these — Service DNS, PVC binding, and Argo Application names depend on them.
• Argo CD reconciles from main. A merge to main is a deploy. There is no staging branch — review the PR as if it ships to production, because it does.
• devops-infra-argo-config is the routing layer. A new chart in helm-templates/ does nothing until an Application referencing it is added to the sister repo. A new cluster directory in helm-overrides/ does nothing until an ApplicationSet covers it. Pair the two-repo change.