added repo
This commit is contained in:
Your Name
@@ -0,0 +1,71 @@
|
||||
# ADR-A1 — Cache upstream charts in `helm-templates/` (vs. pull on the fly)
|
||||
|
||||
> **Status:** Accepted (de facto — current state of the repo).
|
||||
> **Repo:** `devops-infra-helm-charts`.
|
||||
> **Related:** [docs/architecture.md](../../docs/architecture.md), [SANCTITY_RULES R7](../../docs/global/SANCTITY_RULES.md), [update-chart-version.md](../../docs/platform/procedures/update-chart-version.md), [fork-upstream-chart.md](../../docs/platform/procedures/fork-upstream-chart.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Argo CD can render a Helm release in two ways:
|
||||
|
||||
1. **Pull on the fly** — `Application.spec.source.repoURL` points at an upstream Helm registry; `chart:` names the chart. Argo CD pulls the chart at sync time.
|
||||
2. **Cache locally** — the chart lives in a git repo (this one), and `Application.spec.source.path` points at it. Argo CD reads the chart files from git directly.
|
||||
|
||||
This repo has chosen **caching**. Every chart consumed by Meesho's GKE infra fleet has a directory in `helm-templates/<chart>/` — usually a thin wrapper whose `Chart.yaml` declares the upstream chart as a dependency, with the resolved subchart materialised in `Chart.lock` and `charts/<sub>-<ver>.tgz`.
|
||||
|
||||
## Decision
|
||||
|
||||
Cache upstream charts in `helm-templates/` as thin wrapper directories with pinned `dependencies[].version` and a committed `Chart.lock`. Do not configure Argo CD to pull charts directly from upstream registries.
|
||||
|
||||
## Rationale
|
||||
|
||||
1. **Repeatable renders.** A chart-version bump in this repo is a git diff; a chart-version "bump" via upstream pull is whatever the registry returns at sync time. Reproducibility on rollback requires the chart bytes to be in git.
|
||||
|
||||
2. **Air-gapped reviewability.** A reviewer can read the `Chart.yaml`, the `Chart.lock`, and the subchart `.tgz` to know exactly what will render. With on-the-fly pulls, the reviewer trusts the upstream registry hasn't moved a tag.
|
||||
|
||||
3. **Network independence at sync time.** Argo CD's reconcile loop doesn't need outbound network to upstream registries. If GitHub is reachable, sync works; if it isn't, nothing's deploying anyway.
|
||||
|
||||
4. **Supply-chain control.** Pinning `dependencies[].version` plus committing `Chart.lock` means the *digest* of each subchart `.tgz` is recorded. A registry compromise that re-publishes a tag with new content is detected by the lockfile mismatch.
|
||||
|
||||
5. **Forking is local.** When upstream lacks a feature or has a bug, [fork-upstream-chart](../../docs/platform/procedures/fork-upstream-chart.md) is a local edit. No "wait for upstream merge"; the patch lives in our repo until upstream catches up.
|
||||
|
||||
6. **Render-tooling unchanged.** Reviewer-side `helm template helm-templates/<chart> -f overrides.yaml` works locally without any registry config. Pre-merge validation is just `helm template` against the directory.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Accepted
|
||||
|
||||
- **Repo bigger.** 74 chart directories totalling tens of MBs of subchart `.tgz` files.
|
||||
- **Manual update cadence.** A new upstream release isn't picked up automatically. Someone has to bump `dependencies[].version` and run `helm dependency update`. ([update-chart-version](../../docs/platform/procedures/update-chart-version.md))
|
||||
- **Forking risk.** Editing `helm-templates/<chart>/templates/` casually creates an accidental fork that gets clobbered next `helm dependency update`. ([SANCTITY_RULES R7](../../docs/global/SANCTITY_RULES.md))
|
||||
- **Lock-step requirement.** A `Chart.yaml` bump without a refreshed `Chart.lock` is incomplete — Argo CD reads the lockfile, so the version change silently no-ops.
|
||||
|
||||
### Mitigated
|
||||
|
||||
- **Update-chart-version procedure** documents the lock-step requirement explicitly.
|
||||
- **Versioned siblings** ([ADR-A2](ADR-A2-blue-green-sibling-pattern.md)) handle major-version bumps without losing the old chart.
|
||||
- **Pre-merge `helm template`** catches values incompatibilities before merge.
|
||||
|
||||
### Open
|
||||
|
||||
- **Stale charts.** Some directories in `helm-templates/` have no current consumer (`grep -rl '<chart>' helm-overrides` empty). A periodic clean-up exercise has not been formalised.
|
||||
- **Subchart `.tgz` size pollution in git history.** Every `helm dependency update` writes a new `.tgz` to `charts/`. Over years, the repo's history grows accordingly. Whether to switch to a Helm-OCI-pull model is an open question.
|
||||
- **Chart-bump notification.** Nothing currently alerts the team when an upstream advisory affects a chart we have pinned at an old version. Manual diligence today.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Alternative | Why not |
|
||||
|-------------|---------|
|
||||
| **Pull-on-the-fly from upstream registries.** | Reproducibility, network dependency, supply-chain risk all worse. |
|
||||
| **Fully vendor every chart's `templates/`** (no `dependencies[]`, no subchart `.tgz`). | Massive diff churn on every upstream release; worse fork hygiene. |
|
||||
| **Use Helm OCI registries** as a middle ground (pull `.tgz` from a private registry instead of git). | Plausible Phase-2 work. Adds an extra service to maintain; doesn't solve forking. Not done today. |
|
||||
| **Use `kustomize` instead of Helm.** | Most upstream charts are Helm; rewriting every chart's templates as Kustomize patches would be enormous. |
|
||||
|
||||
## References
|
||||
|
||||
- Argo CD Helm chart source docs: <https://argo-cd.readthedocs.io/en/stable/user-guide/helm/>
|
||||
- This repo's chart directory layout: [docs/architecture.md §Module boundaries](../../docs/architecture.md).
|
||||
- Bump procedure: [update-chart-version](../../docs/platform/procedures/update-chart-version.md).
|
||||
- Fork procedure: [fork-upstream-chart](../../docs/platform/procedures/fork-upstream-chart.md).
|
||||
@@ -0,0 +1,83 @@
|
||||
# ADR-A2 — Versioned chart siblings for blue-green migrations
|
||||
|
||||
> **Status:** Accepted (in production — multiple sibling pairs exist today).
|
||||
> **Repo:** `devops-infra-helm-charts`.
|
||||
> **Related:** [blue-green-chart-migration.md](../../docs/platform/procedures/blue-green-chart-migration.md), [SANCTITY_RULES R8](../../docs/global/SANCTITY_RULES.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
When a chart needs an upgrade with breaking template changes — immutable selector mismatches, removed values keys, major-version semantics — you cannot just bump `dependencies[].version` and call it done. The bump renders a different shape against the same overrides; on every cluster, the next sync would push a Helm upgrade that may fail mid-flight (immutable field) or succeed in ways that surprise the operator.
|
||||
|
||||
The team has chosen a **versioned-sibling** pattern: keep the old chart directory live as `<chart>` and introduce the new version as `<chart>-<variant>` where the variant is one of:
|
||||
|
||||
| Variant | Convention |
|
||||
|---------|------------|
|
||||
| `<chart>-green` | Blue-green pair (the new is "green") |
|
||||
| `<chart>-vX.Y.Z` | Pinned target version |
|
||||
| `<chart>-latest` | Work-in-progress, soon to subsume the old |
|
||||
| `<chart>-old` | Reverse pattern — `<chart>` is the new; `-old` is kept for rollback |
|
||||
|
||||
Live examples in the repo today:
|
||||
|
||||
- `argo-cd` ↔ `argo-cd-green`
|
||||
- `contour` ↔ `contour-v1.33.3`
|
||||
- `keda` ↔ `keda-2.17.1`
|
||||
- `opentelemetry-collector` ↔ `opentelemetry-collector-latest`
|
||||
- `victoria-metrics-cluster` ↔ `victoria-metrics-cluster-latest`
|
||||
- `victoria-metrics-agent` ↔ `victoria-metrics-agent-latest`
|
||||
- `sonarqube` ↔ `sonarqube-old` (reverse — sonarqube is new)
|
||||
|
||||
## Decision
|
||||
|
||||
For chart upgrades that involve breaking changes, create a sibling directory `<chart>-<variant>` and migrate cluster-by-cluster by repointing the `Application.spec.source.path` in the sister repo. Both directories remain live for the duration of the migration.
|
||||
|
||||
Do not "consolidate" siblings as a maintenance PR — the split is intentional.
|
||||
|
||||
## Rationale
|
||||
|
||||
1. **Per-cluster cutover with rollback.** Each cluster's Argo `Application` flips one path; if the flip fails, that cluster's revert is a one-line PR in the sister repo. Other clusters are untouched.
|
||||
|
||||
2. **No values-shape coupling.** When the new chart uses different values keys, the new chart's overrides can be authored at leisure and tested before any cluster cuts over. The old chart keeps rendering the old shape against the old overrides.
|
||||
|
||||
3. **Immutable-field changes get a clean exit.** A bump in place that changes `Deployment.spec.selector` fails to apply (immutable). The sibling pattern lets you delete-and-recreate the workload as a one-time per-cluster event during cutover, rather than a fleet-wide failure mode.
|
||||
|
||||
4. **Supports staged rollouts.** Some clusters cut over in week 1, others in week 4. The sister repo can hold both states simultaneously without forcing a full-fleet flip.
|
||||
|
||||
5. **Tooling unchanged.** Argo CD, `helm template`, pre-commit hooks all see two parallel directories; no special handling.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Accepted
|
||||
|
||||
- **Repo bigger** during migration windows. A migration in flight has both `<chart>` and `<chart>-<variant>` live.
|
||||
- **Two charts to maintain** during the window. A CVE patch landing on upstream during the migration may need to be applied to both.
|
||||
- **Sister repo carries the routing decision.** This repo doesn't know which clusters have cut over; that info lives in `devops-infra-argo-config`.
|
||||
- **Naming inconsistency.** `-green`, `-vX.Y.Z`, `-latest`, `-old` aren't unified — different migrations chose different conventions. New migrations should pick the most descriptive (`-vX.Y.Z` if the target version is known; `-green` if the migration is a blue-green flip).
|
||||
|
||||
### Mitigated
|
||||
|
||||
- **Procedure** ([blue-green-chart-migration](../../docs/platform/procedures/blue-green-chart-migration.md)) names the steps explicitly: introduce sibling, render-and-diff, per-cluster cutover, retire old.
|
||||
- **Sanctity rule** ([R8](../../docs/global/SANCTITY_RULES.md)) prevents accidental deletion before all clusters have cut over.
|
||||
|
||||
### Open
|
||||
|
||||
- **Naming convention.** Should the team standardise on `<chart>-vX.Y.Z` for all future migrations? Today the choice is ad-hoc.
|
||||
- **CI/automation** to flag long-running migrations (siblings live > N weeks). Today migrations stall sometimes; nothing alerts.
|
||||
- **Per-cluster cutover tracking.** Today, knowing "which clusters still point at the old chart" requires `grep` against the sister repo. A small dashboard would help.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Alternative | Why not |
|
||||
|-------------|---------|
|
||||
| **In-place bump.** Just change `dependencies[].version` and merge. | Works for compatible bumps; for breaking bumps, fails on the first immutable-field mismatch and may leave the cluster broken. |
|
||||
| **Branch-as-environment** (e.g. an `int` branch with the new chart). | Doesn't help — Argo CD reads `main`. The sibling pattern is more flexible; per-cluster paths beat branches for this. |
|
||||
| **Helm `--atomic` upgrades.** Argo can pass `--atomic` to roll back failed upgrades. | Doesn't address breaking values-shape changes that succeed-but-render-wrong. |
|
||||
| **One big PR that bumps the chart and updates every override.** | Untestable; impossible to roll back one cluster. |
|
||||
|
||||
## References
|
||||
|
||||
- Procedure: [blue-green-chart-migration](../../docs/platform/procedures/blue-green-chart-migration.md).
|
||||
- Sanctity rule: [R8](../../docs/global/SANCTITY_RULES.md).
|
||||
- Live siblings (today): [docs/architecture.md §Cross-cutting concerns](../../docs/architecture.md).
|
||||
@@ -0,0 +1,78 @@
|
||||
# ADR-A3 — Per-cluster `nodeSelector` / `tolerations` / `computeClass`
|
||||
|
||||
> **Status:** Accepted (status quo — every cluster has bespoke scheduling).
|
||||
> **Repo:** `devops-infra-helm-charts`.
|
||||
> **Related:** [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md), [SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md), [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Meesho's GKE fleet has two cluster types:
|
||||
|
||||
| Type | Scheduling primitives |
|
||||
|------|------------------------|
|
||||
| **Standard GKE** | Node pools with `dedicated:` taints and matching node labels |
|
||||
| **GKE Autopilot** (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `ComputeClass` resources with `cloud.google.com/compute-class:` keys |
|
||||
|
||||
Within each type, individual clusters have their own node-pool / compute-class topology, designed for the workloads that cluster runs:
|
||||
|
||||
- `k8s-central-mqkafka-prd-ase1` has Kafka-optimised pools.
|
||||
- `k8s-dsgpu-prd-ase1` has GPU-equipped Autopilot classes.
|
||||
- `k8s-dengspark-prd-ase1` has Spark-executor pools.
|
||||
- BU clusters (`k8s-supply-prd-ase1`, `k8s-demand-prd-ase1`, etc.) have per-app pools (`contour-external`, `contour-internal-0`, `monitoring`, …).
|
||||
|
||||
The `helm-overrides/<cluster>/<app>/custom-values.yaml` files reflect this — each cluster's values for the same app are different.
|
||||
|
||||
## Decision
|
||||
|
||||
`nodeSelector` / `tolerations` / `affinity` / `topologySpreadConstraints` / `cloud.google.com/compute-class` keys in this repo are **per-cluster, hand-authored, never copied**. The matrix of which Contour instance uses which key on which cluster is recorded in [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md). For non-Contour apps, sample sibling apps on the same cluster.
|
||||
|
||||
## Rationale
|
||||
|
||||
1. **GKE Autopilot vs Standard isn't optional.** Autopilot's `ComputeClass` mechanism is mutually exclusive with standard `dedicated:` taints. A values block written for one type has no scheduling effect on the other — pods stay `Pending`.
|
||||
|
||||
2. **Per-cluster pool naming is intentional.** `contour-internal-0` on `k8s-supply-prd-ase1` is not the same node pool as `contour-internal-0` on `k8s-demand-prd-ase1` even if they share the name. The pool is sized differently, may have different machine types, may have different anti-affinity rules. Copying values across clusters works *by accident* sometimes; it fails *deliberately* the rest of the time.
|
||||
|
||||
3. **Multi-Contour-per-cluster pattern.** Most BU clusters run 5–6 Contour releases (`contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`). Each has its own pool. Cross-instance copying within the same cluster is also wrong.
|
||||
|
||||
4. **Operational reality.** When a cluster's node pool changes (a new pool added, an old one renamed), only that cluster's overrides need editing. Centralising scheduling values would mean every node-pool change becomes a fleet-wide PR.
|
||||
|
||||
5. **Reviewability.** A reviewer of a values diff can compare against the same file's git history (this cluster's previous state) without needing to know what other clusters look like. Cross-cluster consistency, when it exists, is incidental.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Accepted
|
||||
|
||||
- **The most common silent bug** in this repo is `nodeSelector` / `tolerations` / `computeClass` copied from another cluster. ([SANCTITY_RULES R5](../../docs/global/SANCTITY_RULES.md))
|
||||
- **Cross-cluster cleanup is hard.** Renaming a pool (e.g. `monitoring` → `obs-shared`) is N PRs, one per cluster.
|
||||
- **Onboarding a new cluster** is bespoke per app — every app needs its scheduling block authored from scratch ([onboard-new-cluster](../../docs/platform/procedures/onboard-new-cluster.md)).
|
||||
- **Reasoning over the fleet** ("which apps are on which pool, fleet-wide?") requires `grep` across cluster directories.
|
||||
|
||||
### Mitigated
|
||||
|
||||
- **The Contour matrix file** ([contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md)) is the single source of truth for the Contour scheduling. **Read before editing any Contour values.**
|
||||
- **The runbook** ([pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md)) walks the diagnosis when scheduling fails.
|
||||
- **The skill** ([diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md)) gives an agent a deterministic diagnosis path.
|
||||
|
||||
### Open
|
||||
|
||||
- **A non-Contour scheduling matrix** has not been formalised. Sample-sibling-on-same-cluster is the working approach but isn't written down.
|
||||
- **Auto-detection of "values copied from another cluster"** is plausible (compare a new file's `nodeSelector` against the cluster's own labels via kubectl). Not implemented.
|
||||
- **Per-cluster topology drift over time** — when a cluster's underlying pools change in Terraform, the values here need a corresponding update. Today it's manual; ideally a Terraform-side hook would notify.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Alternative | Why not |
|
||||
|-------------|---------|
|
||||
| **A shared `_scheduling.yaml`** at the repo root or per-cluster, included via Helm subchart values. | Charts here mostly don't support arbitrary value-file inclusion (Argo CD's `valueFiles` does, but the structure would still need to map per-cluster). Would add a templating step that doesn't exist today. |
|
||||
| **Centralised "platform values" subchart** that every release inherits. | Requires every chart to be a wrapper that depends on the platform subchart. Most upstream charts aren't structured for this. |
|
||||
| **Programmatic generation** (a script that emits per-cluster overrides from a topology spec). | Plausible Phase-2 work — the topology spec would need to live somewhere (likely Terraform output), and the generator would need to handle every chart's idiosyncratic values shape. Not done today. |
|
||||
| **Argo CD `ApplicationSet` with cluster generator + matrix.** | Would centralise routing but doesn't help author the scheduling values. The values still need to be cluster-specific somewhere. |
|
||||
|
||||
## References
|
||||
|
||||
- The matrix: [contour-nodeselector-tolerations-summary.md](../../contour-nodeselector-tolerations-summary.md).
|
||||
- Sanctity rule: [R5](../../docs/global/SANCTITY_RULES.md).
|
||||
- Runbook: [pod-pending-scheduling.md](../../docs/platform/runbooks/pod-pending-scheduling.md).
|
||||
- Skill: [diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md).
|
||||
@@ -0,0 +1,75 @@
|
||||
# ADR-A4 — Raw Kubernetes manifests alongside Helm values in `helm-overrides/`
|
||||
|
||||
> **Status:** Accepted (de facto — the pattern is widespread).
|
||||
> **Repo:** `devops-infra-helm-charts`.
|
||||
> **Related:** [docs/platform/schemas/raw-manifest-sidecar-schema.md](../../docs/platform/schemas/raw-manifest-sidecar-schema.md), [docs/architecture.md](../../docs/architecture.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Most directories under `helm-overrides/<cluster>/<app>/` contain a single `custom-values.yaml` that Argo CD's Application references via `helm.valueFiles`. But many directories also contain **non-`custom-values` `.yaml` files** that are *not* Helm values. They are raw Kubernetes resources, applied alongside the Helm release by the same Argo Application:
|
||||
|
||||
| Path pattern | Resource kind |
|
||||
|--------------|---------------|
|
||||
| `helm-overrides/<cluster>/<app>/computeclass/<x>-cc.yaml` | `ComputeClass` (GKE Autopilot) |
|
||||
| `helm-overrides/<cluster>/<app>/external-dns-services/<x>.yaml` | `Service` carrying an `external-dns` annotation |
|
||||
| `helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml` | `ElasticCluster` (ECK CRD) |
|
||||
| `helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml` | `ConfigMap` materialising Alertmanager config |
|
||||
| `helm-overrides/<cluster>/<app>/external-secrets/*.yaml` (in some shapes) | `ExternalSecret` |
|
||||
|
||||
Argo CD's directory-source mode (`directory.recurse: true` or default flat) walks the whole directory; every `.yaml` file gets applied. The Helm release renders against `custom-values.yaml`; the other files are treated as raw manifests.
|
||||
|
||||
## Decision
|
||||
|
||||
Use a single `helm-overrides/<cluster>/<app>/` directory to hold both the Helm values file *and* the raw sidecar manifests an app needs alongside its Helm release. Keep them tightly co-located rather than splitting into separate directories.
|
||||
|
||||
## Rationale
|
||||
|
||||
1. **Atomic deployment unit.** Argo CD applies the directory contents in one Application sync. The Helm release and its supporting `ComputeClass` / `Service` / `ConfigMap` either both appear or neither does — no race between two Applications.
|
||||
|
||||
2. **Reviewer locality.** A PR that "onboards `<app>` on `<cluster>`" lives in one directory. The reviewer doesn't have to chase across `helm-overrides/`, `manifests/`, and a second sister-repo `Application` to see the full change.
|
||||
|
||||
3. **Argo CD doesn't natively support "Helm + raw manifests" in one source declaratively** — but it does support a directory source that sweeps everything. Co-locating is the pragmatic way to get atomicity.
|
||||
|
||||
4. **Lifecycle coupling.** A `ComputeClass` that an app's `nodeSelector` references is tightly bound to the app — it shouldn't outlive the app, and vice versa. Co-location enforces lifecycle by file proximity.
|
||||
|
||||
5. **Existing CRDs follow the same shape.** ECK's `ElasticCluster`, External Secrets' `ExternalSecret`, Pyroscope's launch manifest — all live next to their app's `custom-values.yaml`. The pattern is consistent.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Accepted
|
||||
|
||||
- **The directory's "shape" is implicit.** Argo CD's behaviour depends on whether the matching `Application` sets `helm.valueFiles` or `directory.recurse`. From inside this repo alone, you can't always tell whether `<extra>.yaml` is a sidecar applied alongside Helm, or whether the directory is a raw-only Application that doesn't render Helm. **The matching sister-repo `Application` is the authoritative source.**
|
||||
|
||||
- **Cross-app cleanup is harder.** Removing an app means removing the whole directory; the sidecars come with it. Mostly a feature, occasionally a footgun (a `ConfigMap` that another app references).
|
||||
|
||||
- **Schema overlap risk.** A file named `alertmanager_config.yaml` could be either a values-include or a `ConfigMap` raw manifest. Naming convention matters; review must check.
|
||||
|
||||
- **Cluster-singleton-vs-app-sidecar boundary.** Some resources straddle: a `ComputeClass` is technically cluster-scoped, but it lives under the app that uses it. A `StorageClass` (cluster-scoped, fleet-wide) lives in `manifests/storageclass/` instead. The split between `manifests/` and `helm-overrides/<cluster>/<app>/` is "is this resource the app's lifecycle, or is it a long-lived cluster singleton?" — sometimes the answer isn't obvious.
|
||||
|
||||
### Mitigated
|
||||
|
||||
- **Schema doc** ([raw-manifest-sidecar-schema.md](../../docs/platform/schemas/raw-manifest-sidecar-schema.md)) documents the common kinds and the "always pin `apiVersion` and `metadata.namespace`" rule.
|
||||
- **`manifests/`** is reserved for cluster-wide singletons explicitly, with [storageclass-priorityclass-schema.md](../../docs/platform/schemas/storageclass-priorityclass-schema.md) documenting the boundary.
|
||||
|
||||
### Open
|
||||
|
||||
- **No formal indicator in this repo** of whether a given directory is "Helm + sidecars" or "raw only." The user has to read the sister-repo `Application` to know.
|
||||
- **Naming for sub-directories** (`computeclass/`, `external-dns-services/`, `external-secrets/`) is conventional but not enforced. New patterns get added ad-hoc.
|
||||
- **Some `manifests/` content arguably should be in `helm-overrides/<cluster>/<app>/`** (e.g. per-cluster Jenkins filestore PV/PVCs are tied to a Jenkins release). The current split was historical; revisiting it is open.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Alternative | Why not |
|
||||
|-------------|---------|
|
||||
| **Two Argo Applications per app — one Helm, one raw.** | Loses atomicity; introduces sync-ordering races. |
|
||||
| **Render every sidecar through Helm** by inlining it as a `templates/` file in a forked chart. | Forks a chart we'd otherwise leave vanilla; conflicts with [ADR-A1](ADR-A1-cache-vs-upstream-charts.md). |
|
||||
| **Move sidecars into a separate `cluster-resources/<cluster>/` tree.** | Loses lifecycle coupling; a separate directory tree to maintain. Reviewer must cross-reference. |
|
||||
| **Use Helm's post-renderer hooks** to inject sidecars into the Helm release. | Adds tooling complexity; doesn't help when the sidecar is a different `apiVersion` than the chart understands. |
|
||||
|
||||
## References
|
||||
|
||||
- Schema: [raw-manifest-sidecar-schema.md](../../docs/platform/schemas/raw-manifest-sidecar-schema.md).
|
||||
- Schema: [storageclass-priorityclass-schema.md](../../docs/platform/schemas/storageclass-priorityclass-schema.md) — for the `manifests/` boundary.
|
||||
- Procedure: [onboard-app-to-cluster.md](../../docs/platform/procedures/onboard-app-to-cluster.md).
|
||||
@@ -0,0 +1,66 @@
|
||||
# ADR-A5 — Manual sync is the default for infra Applications
|
||||
|
||||
> **Status:** Accepted (status quo — most infra Applications lack `automated`).
|
||||
> **Repo:** `devops-infra-helm-charts` (consumer of the decision; the `Application` shapes that enact it live in the sister repo).
|
||||
> **Related:** [SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md) (analogue from the application-side repo), [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md), [deboard-app.md](../../docs/platform/procedures/deboard-app.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Argo CD `Application` resources can have a `spec.syncPolicy.automated` block that auto-applies any diff between git and the cluster on every reconcile cycle. With it: a merge to `main` deploys immediately. Without it: a merge updates the Application's *desired state*, but the cluster doesn't change until a human (or external trigger) clicks **Sync** in the Argo CD UI (or runs `argocd app sync`).
|
||||
|
||||
Most Applications routing to this repo (`devops-infra-helm-charts`) **do not have `automated`** set. A small minority of infra Applications — typically things that should self-heal aggressively (canary-bot, statsd-exporter, vmextractor) — do.
|
||||
|
||||
## Decision
|
||||
|
||||
For infra Applications routed by `github.com/Meesho/devops-infra-argo-config`, the default is **manual sync** — `spec.syncPolicy` contains only `syncOptions: [CreateNamespace=true]`, with no `automated` block. Adding `automated.{prune,selfHeal}: true` to a service-tier Application is a deliberate, headline-of-the-PR change.
|
||||
|
||||
## Rationale
|
||||
|
||||
1. **Production blast radius.** A merge here can cascade across many clusters. If a values change is wrong, the manual-sync default means an operator has a chance to spot it (in Argo CD's Diff view) before clicking through. Auto-sync would push the broken change to every cluster simultaneously on the next reconcile.
|
||||
|
||||
2. **Per-cluster cutover.** A typical chart bump or values change rolls out cluster-by-cluster. The operator clicks Sync on cluster A, watches, then proceeds to cluster B. Auto-sync forces a fleet-wide flip with no soak window.
|
||||
|
||||
3. **Out-of-band drift detection.** Manual sync makes drift visible — when someone `kubectl edit`-s a release on a cluster, Argo CD shows `OutOfSync` and surfaces the diff. With auto-sync, the drift is silently overwritten on the next reconcile, hiding the fact that someone made an out-of-band change.
|
||||
|
||||
4. **Sync click is the agent's hard stop.** [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md) classifies "click Sync" as Layer 2 advisory — the agent recommends the command but never executes. The default of manual sync makes this enforceable: the agent literally cannot deploy without a human in the loop.
|
||||
|
||||
5. **Safe by default; opt in for the loop closures.** Apps that genuinely should self-heal (canary-bot — purpose is to test traffic; statsd-exporter — purpose is fleet-wide telemetry) can be opted in via `automated.prune: true`. The opt-in is a deliberate decision, not a side-effect.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Accepted
|
||||
|
||||
- **Operational tax.** Every PR merge creates an `OutOfSync` Application that someone has to click through. With ~30 clusters × dozens of apps, this can pile up on busy days.
|
||||
- **Drift accumulation.** A PR that nobody clicks Sync on sits as `OutOfSync` indefinitely. Sometimes this is intentional (the PR was speculative); sometimes it's forgotten. Periodic audits ("which Applications have been `OutOfSync` for > 7 days?") aren't yet automated.
|
||||
- **Manual-sync bias** can mask incidents — an Application failing to sync (because of a values regression) may sit `OutOfSync` for a while before someone notices. Auto-sync would have surfaced it loudly via failed reconciles.
|
||||
|
||||
### Mitigated
|
||||
|
||||
- **The runbook** ([argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md)) explicitly handles the "OutOfSync only, no error, sync hasn't run" branch as §4 — its own diagnostic path.
|
||||
- **Sanctity rule analogue** in the application-side `devops-argo-config` repo names this explicitly (R3); we inherit the principle.
|
||||
- **Skill** ([diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md)) outputs Layer 2 advisories ("recommend operator clicks Sync") rather than auto-Sync triggers.
|
||||
|
||||
### Open
|
||||
|
||||
- **Notification on long-`OutOfSync` Applications** — periodic alert / dashboard. Today operators just see this in the Argo UI.
|
||||
- **Should some infra apps move to auto-sync?** Specifically, sidecars whose blast radius is tiny (telemetry collectors, log agents). A periodic review hasn't been done.
|
||||
- **The opt-in list of currently auto-synced apps** isn't documented in this repo. Has to be inferred from the sister repo's `Application` files.
|
||||
- **Cluster-specific opt-ins** — auto-sync on dev clusters but manual on prod — would be a reasonable refinement but adds per-cluster `Application` divergence.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Alternative | Why not |
|
||||
|-------------|---------|
|
||||
| **Auto-sync everywhere by default.** | Loses the per-cluster operator gate; a bad merge cascades fleet-wide. |
|
||||
| **Auto-sync with `selfHeal: false` but `prune: true`.** | Still applies values changes immediately; doesn't help. |
|
||||
| **Manual sync but with auto-fallback after N hours.** | Argo CD doesn't offer this natively. Building it would require a controller. |
|
||||
| **Per-environment policy** (auto-sync on int, manual on prod). | Reasonable refinement; adds policy state to the sister repo. Could be future work. |
|
||||
|
||||
## References
|
||||
|
||||
- Sister-repo `Application` shapes: `github.com/Meesho/devops-infra-argo-config`.
|
||||
- Runbook §4: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
|
||||
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md) (Layer 2 row "Sync `<release>` now in Argo CD").
|
||||
- Application-side analogue: `devops-argo-config`'s `SANCTITY_RULES.md R3`.
|
||||
Reference in New Issue
Block a user