102 lines
5.0 KiB
Markdown
102 lines
5.0 KiB
Markdown
# ADR-I1: Single Generic Helm Chart for All Infra Tool Applications
|
||
|
||
> **Status:** Accepted
|
||
>
|
||
> **Date:** Retroactive (pattern established before documentation)
|
||
>
|
||
> **Decision makers:** DevOps / Platform team
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
Meesho manages ~19 Kubernetes clusters, each running 20–50 infrastructure tools (ingress controllers, observability stacks, policy engines, etc.). Each tool needs an ArgoCD `Application` resource to be deployed and managed.
|
||
|
||
The question: how should we generate and maintain these ArgoCD Application manifests?
|
||
|
||
---
|
||
|
||
## Decision
|
||
|
||
Use a **single generic Helm chart** (`generic-argo-apps-chart/`) that renders one ArgoCD `Application` per entry in an `appSpec` list. Each cluster gets one values file that defines its `appSpec[]`.
|
||
|
||
### How it works
|
||
|
||
1. **One chart** — `generic-argo-apps-chart/templates/genericTemplate.yaml` iterates over `appSpec[]` and renders a standard ArgoCD Application for each entry.
|
||
2. **One values file per cluster** — `values/<env>/<cluster>-values.yaml` defines `clusterSpec` (routing), `teamSpec` (source repo/labels), and `appSpec[]` (list of tools).
|
||
3. **One incubator Application per cluster** — `incubator/<env>/<cluster>.yaml` is the parent Application that points ArgoCD at the generic chart + the cluster's values file.
|
||
|
||
### The template contract
|
||
|
||
Every `appSpec` entry provides four fields: `name`, `namespace`, `chartDir`, `valuesDir`. The template computes:
|
||
|
||
- **Application name:** `<name>-<mungedCluster>-<env>` (with deterministic cluster name munging)
|
||
- **Source:** `repoURL/path/<chartDir>` with `valueFiles/<valuesDir>/custom-values.yaml`
|
||
- **Destination:** `clusterSpec.destination.name` + `namespace`
|
||
- **Sync policy:** `CreateNamespace=true`, finalizer for cleanup
|
||
|
||
---
|
||
|
||
## Alternatives considered
|
||
|
||
### Alternative 1: One Helm chart per tool
|
||
|
||
Each tool (keda, contour, victoriametrics, etc.) gets its own chart that wraps the upstream chart AND the ArgoCD Application definition.
|
||
|
||
**Rejected because:**
|
||
- 74 charts x 19 clusters = massive maintenance surface.
|
||
- Tool onboarding requires creating a new chart every time.
|
||
- ArgoCD Application boilerplate is duplicated across every chart.
|
||
- Changes to Application conventions (labels, sync policy, finalizers) require updating every chart.
|
||
|
||
### Alternative 2: Handwritten Application YAML per tool per cluster
|
||
|
||
Every tool-on-cluster gets a dedicated YAML file checked in as a static ArgoCD Application.
|
||
|
||
**Rejected because:**
|
||
- ~19 clusters x ~30 tools = ~570 YAML files, all with near-identical structure.
|
||
- Adding a tool to a cluster means creating a new file from a template.
|
||
- Drift between files is inevitable; conventions are enforced by review, not by template.
|
||
- Bulk operations (change all Application labels) require editing hundreds of files.
|
||
|
||
### Alternative 3: Kustomize overlays
|
||
|
||
Use Kustomize bases and overlays instead of Helm to generate Applications.
|
||
|
||
**Rejected because:**
|
||
- The team already uses Helm for chart rendering; adding Kustomize introduces a second rendering engine.
|
||
- Kustomize's overlay model works well for patching but poorly for list-based generation (one Application per list entry).
|
||
- The Helm approach allows `appSpec` to be a simple YAML list — easier to review and diff.
|
||
|
||
---
|
||
|
||
## Consequences
|
||
|
||
### Positive
|
||
|
||
- **Adding a tool to a cluster** is a 4-line YAML addition to a values file. No new files, no new charts.
|
||
- **Consistency** is enforced by the template. All Applications get the same labels, finalizers, sync policy, and naming convention.
|
||
- **Bulk changes** to Application conventions (e.g., adding a new label) require editing one template file, not hundreds.
|
||
- **Reviewing PRs** is straightforward — a values file diff clearly shows what tools were added/removed/changed.
|
||
- **Template rendering is testable** — `helm template` validates the entire cluster's tooling in one command.
|
||
|
||
### Negative
|
||
|
||
- **The generic chart is a single point of failure.** A bad merge to the template breaks ALL clusters immediately via auto-sync. This is mitigated by requiring explicit platform-team review for template changes (HIGH RISK operation).
|
||
- **The template's cluster name munging logic is complex** — it uses placeholder-based string replacement to handle edge cases (`dp-`, `backup`, `-ase1c`). This logic is load-bearing and poorly documented in the template itself.
|
||
- **No per-tool sync policy customization.** All tools get the same `CreateNamespace=true` syncOptions. Tools that need custom sync policies (e.g., prune=true, self-heal=true) can't express this in the current template without adding optional fields to `appSpec`.
|
||
- **Cross-repo dependency.** The `chartDir` and `valuesDir` in `appSpec` must exist in `devops-infra-helm-charts`. There's no compile-time check — a typo in these fields is only caught at ArgoCD render time.
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
To validate that the generic chart correctly renders all Applications for a cluster:
|
||
|
||
```bash
|
||
helm template generic-argo-apps-chart/ \
|
||
-f values/prd/incubator-infra-k8s-central-prd-ase1-values.yaml
|
||
```
|
||
|
||
This produces the full set of ArgoCD Application manifests that ArgoCD would create from the values file.
|