05 — Deploy lifecycle

End-to-end story of how a values change reaches a live cluster, where the safety gates are, and what happens when they fail.

The path

edit override (branch) → commit (TruffleHog runs) → push → PR
   → reviewer approves → merge to main
   → Argo CD on cluster reconciles (auto-sync OR human Sync click)
   → manifests applied → workload changes

Stage 1 — Branch and edit

Branch off main. Never push to main directly.
Edit one helm-overrides/<cluster>/<app>/custom-values.yaml (or its raw sidecars).
No drive-by edits, no cross-cluster normalization in the same PR. See ../docs/global/SANCTITY_RULES.md.

Stage 2 — Local validation

helm template against the override — confirms render succeeds.
For wrappers: helm dependency update first.
For raw sidecars: kubectl apply --dry-run=client.

If render fails locally, it will fail in Argo CD's OutOfSync → SyncFailed. Fix before pushing.

Stage 3 — Commit

git commit triggers pre-commit hooks:

TruffleHog — blocking. Real secrets bounce. Never --no-verify.
CAC, Yaak — gated on paths this repo doesn't have, no-op.

Post-commit:

Cursor metric collector — background, non-blocking. Pings observe.meeshogcp.in with commit telemetry. Failure here does not block.

Detail: ./08-pre-commit-and-hooks.md.

Stage 4 — PR + review (safety gate 1)

The reviewer's job:

Confirm the change touches only the cluster × app named in the PR.
Confirm any per-cluster scheduling fields were rewritten, not copy-pasted.
Confirm fullnameOverride is unchanged.
Confirm no secret materializes in the file.
Confirm chart Chart.yaml dep bumps came with Chart.lock refresh and a changelog reference.
Confirm versioned-sibling deletes have no sister-repo references.

If a chart fork is suspected, escalate per ../docs/global/escalation-matrix.md row 1.

Stage 5 — Merge to `main`

Merging is the deploy event. Argo CD on every cluster whose Application points at the changed path will move to OutOfSync.

Stage 6 — Argo CD reconcile (safety gate 2)

Two reconciliation modes, set per Application in the sister repo:

Manual sync (prod default for infra) — Argo waits for a human Sync click. Engineer reviews the diff in the Argo UI before applying.
Auto-sync — Argo applies on its own. Reserved for low-risk leaves (kube-state-metrics, monitoring agents).

Background on the manual-sync default: ../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md.

The Argo Application also defines:

syncPolicy.automated.prune — whether Argo deletes objects no longer in Git.
syncPolicy.automated.selfHeal — whether Argo reverts manual cluster edits.
syncOptions — CreateNamespace, ServerSideApply, RespectIgnoreDifferences, retry/backoff.
Sync waves via annotations (in chart templates or sidecars).

These all live in the sister repo, not here. See ../docs/global/coding-guidelines/argocd.md.

Failure modes

Failure	Where it surfaces	Read
Render error in `helm template`	Argo Application status `ComparisonError`	Re-render locally; fix values
`OutOfSync → SyncFailed` after Sync click	Argo UI events	`../docs/platform/runbooks/argocd-sync-failure.md`
Pods land but stay `Pending`	`kubectl get pods` on target cluster	`../docs/platform/runbooks/pod-pending-scheduling.md`
Ingress 5xx after Contour change	`contour-external` envoy logs / synthetic probes	`../docs/platform/runbooks/ingress-down.md`
Drift reappears after `kubectl edit`	`selfHeal: true` doing its job	Edit Git, not the cluster

Sister-repo coupling

Almost every non-trivial change is a paired PR:

New app on cluster: PR here (override) + PR in sister repo (Application).
New cluster: PR here (cluster directory) + PR in sister repo (ApplicationSet cluster generator).
Blue-green sibling cutover: PR here (sibling values) + PR in sister repo (Application targetRevision / chart path).

Procedures: ../docs/platform/procedures/onboard-app-to-cluster.md, ../docs/platform/procedures/onboard-new-cluster.md, ../docs/platform/procedures/blue-green-chart-migration.md, ../docs/platform/procedures/deboard-app.md.

5.1 KiB Raw Blame History Unescape Escape