devops-infra-argo-config-gcp/wiki/analyses/ADR-I1-generic-chart-per-cluster.md

ADR-I1: Single Generic Helm Chart for All Infra Tool Applications

Status: Accepted

Date: Retroactive (pattern established before documentation)

Decision makers: DevOps / Platform team

---

Context

Meesho manages ~19 Kubernetes clusters, each running 20–50 infrastructure tools (ingress controllers, observability stacks, policy engines, etc.). Each tool needs an ArgoCD Application resource to be deployed and managed.

The question: how should we generate and maintain these ArgoCD Application manifests?

---

Decision

Use a single generic Helm chart (generic-argo-apps-chart/) that renders one ArgoCD Application per entry in an appSpec list. Each cluster gets one values file that defines its appSpec[].

How it works

1. One chart — generic-argo-apps-chart/templates/genericTemplate.yaml iterates over appSpec[] and renders a standard ArgoCD Application for each entry.
2. One values file per cluster — values/<env>/<cluster>-values.yaml defines clusterSpec (routing), teamSpec (source repo/labels), and appSpec[] (list of tools).
3. One incubator Application per cluster — incubator/<env>/<cluster>.yaml is the parent Application that points ArgoCD at the generic chart + the cluster's values file.

The template contract

Every appSpec entry provides four fields: name, namespace, chartDir, valuesDir. The template computes:

• Application name: <name>-<mungedCluster>-<env> (with deterministic cluster name munging)
• Source: repoURL/path/<chartDir> with valueFiles/<valuesDir>/custom-values.yaml
• Destination: clusterSpec.destination.name + namespace
• Sync policy: CreateNamespace=true, finalizer for cleanup

---

Alternatives considered

Alternative 1: One Helm chart per tool

Each tool (keda, contour, victoriametrics, etc.) gets its own chart that wraps the upstream chart AND the ArgoCD Application definition.

Rejected because:

• 74 charts x 19 clusters = massive maintenance surface.
• Tool onboarding requires creating a new chart every time.
• ArgoCD Application boilerplate is duplicated across every chart.
• Changes to Application conventions (labels, sync policy, finalizers) require updating every chart.

Alternative 2: Handwritten Application YAML per tool per cluster

Every tool-on-cluster gets a dedicated YAML file checked in as a static ArgoCD Application.

Rejected because:

• ~19 clusters x ~30 tools = ~570 YAML files, all with near-identical structure.
• Adding a tool to a cluster means creating a new file from a template.
• Drift between files is inevitable; conventions are enforced by review, not by template.
• Bulk operations (change all Application labels) require editing hundreds of files.

Alternative 3: Kustomize overlays

Use Kustomize bases and overlays instead of Helm to generate Applications.

Rejected because:

• The team already uses Helm for chart rendering; adding Kustomize introduces a second rendering engine.
• Kustomize's overlay model works well for patching but poorly for list-based generation (one Application per list entry).
• The Helm approach allows appSpec to be a simple YAML list — easier to review and diff.

---

Consequences

Positive

• Adding a tool to a cluster is a 4-line YAML addition to a values file. No new files, no new charts.
• Consistency is enforced by the template. All Applications get the same labels, finalizers, sync policy, and naming convention.
• Bulk changes to Application conventions (e.g., adding a new label) require editing one template file, not hundreds.
• Reviewing PRs is straightforward — a values file diff clearly shows what tools were added/removed/changed.
• Template rendering is testable — helm template validates the entire cluster's tooling in one command.

Negative

• The generic chart is a single point of failure. A bad merge to the template breaks ALL clusters immediately via auto-sync. This is mitigated by requiring explicit platform-team review for template changes (HIGH RISK operation).
• The template's cluster name munging logic is complex — it uses placeholder-based string replacement to handle edge cases (dp-, backup, -ase1c). This logic is load-bearing and poorly documented in the template itself.
• No per-tool sync policy customization. All tools get the same CreateNamespace=true syncOptions. Tools that need custom sync policies (e.g., prune=true, self-heal=true) can't express this in the current template without adding optional fields to appSpec.
• Cross-repo dependency. The chartDir and valuesDir in appSpec must exist in devops-infra-helm-charts. There's no compile-time check — a typo in these fields is only caught at ArgoCD render time.

---

Verification

To validate that the generic chart correctly renders all Applications for a cluster:

helm template generic-argo-apps-chart/ \
  -f values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml

This produces the full set of ArgoCD Application manifests that ArgoCD would create from the values file.