devops-infra-helm-charts-gcp/wiki/analyses/ADR-A4-raw-manifest-sidecars-in-helm-overrides.md

ADR-A4 — Raw Kubernetes manifests alongside Helm values in helm-overrides/

Status: Accepted (de facto — the pattern is widespread). Repo: devops-infra-helm-charts. Related: docs/platform/schemas/raw-manifest-sidecar-schema.md, docs/architecture.md.

---

Context

Most directories under helm-overrides/<cluster>/<app>/ contain a single custom-values.yaml that Argo CD's Application references via helm.valueFiles. But many directories also contain non-custom-values .yaml files that are not Helm values. They are raw Kubernetes resources, applied alongside the Helm release by the same Argo Application:

Path pattern	Resource kind
helm-overrides/<cluster>/<app>/computeclass/<x>-cc.yaml	ComputeClass (GKE Autopilot)
helm-overrides/<cluster>/<app>/external-dns-services/<x>.yaml	Service carrying an external-dns annotation
helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml	ElasticCluster (ECK CRD)
helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml	ConfigMap materialising Alertmanager config
helm-overrides/<cluster>/<app>/external-secrets/*.yaml (in some shapes)	ExternalSecret

Argo CD's directory-source mode (directory.recurse: true or default flat) walks the whole directory; every .yaml file gets applied. The Helm release renders against custom-values.yaml; the other files are treated as raw manifests.

Decision

Use a single helm-overrides/<cluster>/<app>/ directory to hold both the Helm values file and the raw sidecar manifests an app needs alongside its Helm release. Keep them tightly co-located rather than splitting into separate directories.

Rationale

1. Atomic deployment unit. Argo CD applies the directory contents in one Application sync. The Helm release and its supporting ComputeClass / Service / ConfigMap either both appear or neither does — no race between two Applications.

2. Reviewer locality. A PR that "onboards <app> on <cluster>" lives in one directory. The reviewer doesn't have to chase across helm-overrides/, manifests/, and a second sister-repo Application to see the full change.

3. Argo CD doesn't natively support "Helm + raw manifests" in one source declaratively — but it does support a directory source that sweeps everything. Co-locating is the pragmatic way to get atomicity.

4. Lifecycle coupling. A ComputeClass that an app's nodeSelector references is tightly bound to the app — it shouldn't outlive the app, and vice versa. Co-location enforces lifecycle by file proximity.

5. Existing CRDs follow the same shape. ECK's ElasticCluster, External Secrets' ExternalSecret, Pyroscope's launch manifest — all live next to their app's custom-values.yaml. The pattern is consistent.

Consequences

Accepted

• The directory's "shape" is implicit. Argo CD's behaviour depends on whether the matching Application sets helm.valueFiles or directory.recurse. From inside this repo alone, you can't always tell whether <extra>.yaml is a sidecar applied alongside Helm, or whether the directory is a raw-only Application that doesn't render Helm. The matching sister-repo Application is the authoritative source.

• Cross-app cleanup is harder. Removing an app means removing the whole directory; the sidecars come with it. Mostly a feature, occasionally a footgun (a ConfigMap that another app references).

• Schema overlap risk. A file named alertmanager_config.yaml could be either a values-include or a ConfigMap raw manifest. Naming convention matters; review must check.

• Cluster-singleton-vs-app-sidecar boundary. Some resources straddle: a ComputeClass is technically cluster-scoped, but it lives under the app that uses it. A StorageClass (cluster-scoped, fleet-wide) lives in manifests/storageclass/ instead. The split between manifests/ and helm-overrides/<cluster>/<app>/ is "is this resource the app's lifecycle, or is it a long-lived cluster singleton?" — sometimes the answer isn't obvious.

Mitigated

• Schema doc (raw-manifest-sidecar-schema.md) documents the common kinds and the "always pin apiVersion and metadata.namespace" rule.
• manifests/ is reserved for cluster-wide singletons explicitly, with storageclass-priorityclass-schema.md documenting the boundary.

Open

• No formal indicator in this repo of whether a given directory is "Helm + sidecars" or "raw only." The user has to read the sister-repo Application to know.
• Naming for sub-directories (computeclass/, external-dns-services/, external-secrets/) is conventional but not enforced. New patterns get added ad-hoc.
• Some manifests/ content arguably should be in helm-overrides/<cluster>/<app>/ (e.g. per-cluster Jenkins filestore PV/PVCs are tied to a Jenkins release). The current split was historical; revisiting it is open.

Alternatives considered

Alternative	Why not
Two Argo Applications per app — one Helm, one raw.	Loses atomicity; introduces sync-ordering races.
Render every sidecar through Helm by inlining it as a templates/ file in a forked chart.	Forks a chart we'd otherwise leave vanilla; conflicts with ADR-A1.
Move sidecars into a separate cluster-resources/<cluster>/ tree.	Loses lifecycle coupling; a separate directory tree to maintain. Reviewer must cross-reference.
Use Helm's post-renderer hooks to inject sidecars into the Helm release.	Adds tooling complexity; doesn't help when the sidecar is a different apiVersion than the chart understands.

References

• Schema: raw-manifest-sidecar-schema.md.
• Schema: storageclass-priorityclass-schema.md — for the manifests/ boundary.
• Procedure: onboard-app-to-cluster.md.