ADR-A5 — Manual sync is the default for infra Applications

Status: Accepted (status quo — most infra Applications lack automated). Repo: devops-infra-helm-charts (consumer of the decision; the Application shapes that enact it live in the sister repo). Related: SANCTITY_RULES R3 (analogue from the application-side repo), argocd-sync-failure.md, deboard-app.md.

Context

Argo CD Application resources can have a spec.syncPolicy.automated block that auto-applies any diff between git and the cluster on every reconcile cycle. With it: a merge to main deploys immediately. Without it: a merge updates the Application's desired state, but the cluster doesn't change until a human (or external trigger) clicks Sync in the Argo CD UI (or runs argocd app sync).

Most Applications routing to this repo (devops-infra-helm-charts) do not have automated set. A small minority of infra Applications — typically things that should self-heal aggressively (canary-bot, statsd-exporter, vmextractor) — do.

Decision

For infra Applications routed by github.com/Meesho/devops-infra-argo-config, the default is manual sync — spec.syncPolicy contains only syncOptions: [CreateNamespace=true], with no automated block. Adding automated.{prune,selfHeal}: true to a service-tier Application is a deliberate, headline-of-the-PR change.

Rationale

Production blast radius. A merge here can cascade across many clusters. If a values change is wrong, the manual-sync default means an operator has a chance to spot it (in Argo CD's Diff view) before clicking through. Auto-sync would push the broken change to every cluster simultaneously on the next reconcile.
Per-cluster cutover. A typical chart bump or values change rolls out cluster-by-cluster. The operator clicks Sync on cluster A, watches, then proceeds to cluster B. Auto-sync forces a fleet-wide flip with no soak window.
Out-of-band drift detection. Manual sync makes drift visible — when someone kubectl edit-s a release on a cluster, Argo CD shows OutOfSync and surfaces the diff. With auto-sync, the drift is silently overwritten on the next reconcile, hiding the fact that someone made an out-of-band change.
Sync click is the agent's hard stop. AGENT_BOUNDARIES.md classifies "click Sync" as Layer 2 advisory — the agent recommends the command but never executes. The default of manual sync makes this enforceable: the agent literally cannot deploy without a human in the loop.
Safe by default; opt in for the loop closures. Apps that genuinely should self-heal (canary-bot — purpose is to test traffic; statsd-exporter — purpose is fleet-wide telemetry) can be opted in via automated.prune: true. The opt-in is a deliberate decision, not a side-effect.

Consequences

Accepted

Operational tax. Every PR merge creates an OutOfSync Application that someone has to click through. With ~30 clusters × dozens of apps, this can pile up on busy days.
Drift accumulation. A PR that nobody clicks Sync on sits as OutOfSync indefinitely. Sometimes this is intentional (the PR was speculative); sometimes it's forgotten. Periodic audits ("which Applications have been OutOfSync for > 7 days?") aren't yet automated.
Manual-sync bias can mask incidents — an Application failing to sync (because of a values regression) may sit OutOfSync for a while before someone notices. Auto-sync would have surfaced it loudly via failed reconciles.

Mitigated

The runbook (argocd-sync-failure.md) explicitly handles the "OutOfSync only, no error, sync hasn't run" branch as §4 — its own diagnostic path.
Sanctity rule analogue in the application-side devops-argo-config repo names this explicitly (R3); we inherit the principle.
Skill (diagnose-scheduling.md) outputs Layer 2 advisories ("recommend operator clicks Sync") rather than auto-Sync triggers.

Open

Notification on long-OutOfSync Applications — periodic alert / dashboard. Today operators just see this in the Argo UI.
Should some infra apps move to auto-sync? Specifically, sidecars whose blast radius is tiny (telemetry collectors, log agents). A periodic review hasn't been done.
The opt-in list of currently auto-synced apps isn't documented in this repo. Has to be inferred from the sister repo's Application files.
Cluster-specific opt-ins — auto-sync on dev clusters but manual on prod — would be a reasonable refinement but adds per-cluster Application divergence.

Alternatives considered

Alternative	Why not
Auto-sync everywhere by default.	Loses the per-cluster operator gate; a bad merge cascades fleet-wide.
Auto-sync with `selfHeal: false` but `prune: true`.	Still applies values changes immediately; doesn't help.
Manual sync but with auto-fallback after N hours.	Argo CD doesn't offer this natively. Building it would require a controller.
Per-environment policy (auto-sync on int, manual on prod).	Reasonable refinement; adds policy state to the sister repo. Could be future work.

References

Sister-repo Application shapes: github.com/Meesho/devops-infra-argo-config.
Runbook §4: argocd-sync-failure.md.
Boundaries: AGENT_BOUNDARIES.md (Layer 2 row "Sync <release> now in Argo CD").
Application-side analogue: devops-argo-config's SANCTITY_RULES.md R3.

5.7 KiB Raw Blame History Unescape Escape