devops-infra-helm-charts-gcp/wiki/analyses/ADR-A5-manual-sync-default-for-infra.md

# ADR-A5 — Manual sync is the default for infra Applications

> **Status:** Accepted (status quo — most infra Applications lack `automated`).
> **Repo:** `devops-infra-helm-charts` (consumer of the decision; the `Application` shapes that enact it live in the sister repo).
> **Related:** [SANCTITY_RULES R3](../../docs/global/SANCTITY_RULES.md) (analogue from the application-side repo), [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md), [deboard-app.md](../../docs/platform/procedures/deboard-app.md).

---

## Context

Argo CD `Application` resources can have a `spec.syncPolicy.automated` block that auto-applies any diff between git and the cluster on every reconcile cycle. With it: a merge to `main` deploys immediately. Without it: a merge updates the Application's *desired state*, but the cluster doesn't change until a human (or external trigger) clicks **Sync** in the Argo CD UI (or runs `argocd app sync`).

Most Applications routing to this repo (`devops-infra-helm-charts`) **do not have `automated`** set. A small minority of infra Applications — typically things that should self-heal aggressively (canary-bot, statsd-exporter, vmextractor) — do.

## Decision

For infra Applications routed by `github.com/Meesho/devops-infra-argo-config`, the default is **manual sync** — `spec.syncPolicy` contains only `syncOptions: [CreateNamespace=true]`, with no `automated` block. Adding `automated.{prune,selfHeal}: true` to a service-tier Application is a deliberate, headline-of-the-PR change.

## Rationale

1. **Production blast radius.** A merge here can cascade across many clusters. If a values change is wrong, the manual-sync default means an operator has a chance to spot it (in Argo CD's Diff view) before clicking through. Auto-sync would push the broken change to every cluster simultaneously on the next reconcile.

2. **Per-cluster cutover.** A typical chart bump or values change rolls out cluster-by-cluster. The operator clicks Sync on cluster A, watches, then proceeds to cluster B. Auto-sync forces a fleet-wide flip with no soak window.

3. **Out-of-band drift detection.** Manual sync makes drift visible — when someone `kubectl edit`-s a release on a cluster, Argo CD shows `OutOfSync` and surfaces the diff. With auto-sync, the drift is silently overwritten on the next reconcile, hiding the fact that someone made an out-of-band change.

4. **Sync click is the agent's hard stop.** [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md) classifies "click Sync" as Layer 2 advisory — the agent recommends the command but never executes. The default of manual sync makes this enforceable: the agent literally cannot deploy without a human in the loop.

5. **Safe by default; opt in for the loop closures.** Apps that genuinely should self-heal (canary-bot — purpose is to test traffic; statsd-exporter — purpose is fleet-wide telemetry) can be opted in via `automated.prune: true`. The opt-in is a deliberate decision, not a side-effect.

## Consequences

### Accepted

- **Operational tax.** Every PR merge creates an `OutOfSync` Application that someone has to click through. With ~30 clusters × dozens of apps, this can pile up on busy days.
- **Drift accumulation.** A PR that nobody clicks Sync on sits as `OutOfSync` indefinitely. Sometimes this is intentional (the PR was speculative); sometimes it's forgotten. Periodic audits ("which Applications have been `OutOfSync` for > 7 days?") aren't yet automated.
- **Manual-sync bias** can mask incidents — an Application failing to sync (because of a values regression) may sit `OutOfSync` for a while before someone notices. Auto-sync would have surfaced it loudly via failed reconciles.

### Mitigated

- **The runbook** ([argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md)) explicitly handles the "OutOfSync only, no error, sync hasn't run" branch as §4 — its own diagnostic path.
- **Sanctity rule analogue** in the application-side `devops-argo-config` repo names this explicitly (R3); we inherit the principle.
- **Skill** ([diagnose-scheduling.md](../../skills/infra/diagnose-scheduling.md)) outputs Layer 2 advisories ("recommend operator clicks Sync") rather than auto-Sync triggers.

### Open

- **Notification on long-`OutOfSync` Applications** — periodic alert / dashboard. Today operators just see this in the Argo UI.
- **Should some infra apps move to auto-sync?** Specifically, sidecars whose blast radius is tiny (telemetry collectors, log agents). A periodic review hasn't been done.
- **The opt-in list of currently auto-synced apps** isn't documented in this repo. Has to be inferred from the sister repo's `Application` files.
- **Cluster-specific opt-ins** — auto-sync on dev clusters but manual on prod — would be a reasonable refinement but adds per-cluster `Application` divergence.

## Alternatives considered

| Alternative | Why not |
|-------------|---------|
| **Auto-sync everywhere by default.** | Loses the per-cluster operator gate; a bad merge cascades fleet-wide. |
| **Auto-sync with `selfHeal: false` but `prune: true`.** | Still applies values changes immediately; doesn't help. |
| **Manual sync but with auto-fallback after N hours.** | Argo CD doesn't offer this natively. Building it would require a controller. |
| **Per-environment policy** (auto-sync on int, manual on prod). | Reasonable refinement; adds policy state to the sister repo. Could be future work. |

## References

- Sister-repo `Application` shapes: `github.com/Meesho/devops-infra-argo-config`.
- Runbook §4: [argocd-sync-failure.md](../../docs/platform/runbooks/argocd-sync-failure.md).
- Boundaries: [AGENT_BOUNDARIES.md](../../docs/global/AGENT_BOUNDARIES.md) (Layer 2 row "Sync `<release>` now in Argo CD").
- Application-side analogue: `devops-argo-config`'s `SANCTITY_RULES.md R3`.