104 lines
5.1 KiB
Markdown
104 lines
5.1 KiB
Markdown
> Per AI Blitz Plan §claude. Layer: 1. Repo: devops-infra-helm-charts.
|
||
|
||
# 05 — Deploy lifecycle
|
||
|
||
End-to-end story of how a values change reaches a live cluster, where the safety gates are, and what happens when they fail.
|
||
|
||
## The path
|
||
|
||
```
|
||
edit override (branch) → commit (TruffleHog runs) → push → PR
|
||
→ reviewer approves → merge to main
|
||
→ Argo CD on cluster reconciles (auto-sync OR human Sync click)
|
||
→ manifests applied → workload changes
|
||
```
|
||
|
||
## Stage 1 — Branch and edit
|
||
|
||
- Branch off `main`. Never push to `main` directly.
|
||
- Edit one `helm-overrides/<cluster>/<app>/custom-values.yaml` (or its raw sidecars).
|
||
- No drive-by edits, no cross-cluster normalization in the same PR. See [`../docs/global/SANCTITY_RULES.md`](../docs/global/SANCTITY_RULES.md).
|
||
|
||
## Stage 2 — Local validation
|
||
|
||
- `helm template` against the override — confirms render succeeds.
|
||
- For wrappers: `helm dependency update` first.
|
||
- For raw sidecars: `kubectl apply --dry-run=client`.
|
||
|
||
If render fails locally, it will fail in Argo CD's `OutOfSync → SyncFailed`. Fix before pushing.
|
||
|
||
## Stage 3 — Commit
|
||
|
||
`git commit` triggers pre-commit hooks:
|
||
|
||
- **TruffleHog** — blocking. Real secrets bounce. Never `--no-verify`.
|
||
- **CAC, Yaak** — gated on paths this repo doesn't have, no-op.
|
||
|
||
Post-commit:
|
||
|
||
- **Cursor metric collector** — background, non-blocking. Pings `observe.meeshogcp.in` with commit telemetry. Failure here does not block.
|
||
|
||
Detail: [`./08-pre-commit-and-hooks.md`](./08-pre-commit-and-hooks.md).
|
||
|
||
## Stage 4 — PR + review (safety gate 1)
|
||
|
||
The reviewer's job:
|
||
|
||
1. Confirm the change touches only the cluster × app named in the PR.
|
||
2. Confirm any per-cluster scheduling fields were rewritten, not copy-pasted.
|
||
3. Confirm `fullnameOverride` is unchanged.
|
||
4. Confirm no secret materializes in the file.
|
||
5. Confirm chart `Chart.yaml` dep bumps came with `Chart.lock` refresh and a changelog reference.
|
||
6. Confirm versioned-sibling deletes have no sister-repo references.
|
||
|
||
If a chart fork is suspected, escalate per [`../docs/global/escalation-matrix.md`](../docs/global/escalation-matrix.md) row 1.
|
||
|
||
## Stage 5 — Merge to `main`
|
||
|
||
Merging is the deploy event. Argo CD on every cluster whose Application points at the changed path will move to `OutOfSync`.
|
||
|
||
## Stage 6 — Argo CD reconcile (safety gate 2)
|
||
|
||
Two reconciliation modes, set per Application in the sister repo:
|
||
|
||
- **Manual sync** (prod default for infra) — Argo waits for a human Sync click. Engineer reviews the diff in the Argo UI before applying.
|
||
- **Auto-sync** — Argo applies on its own. Reserved for low-risk leaves (`kube-state-metrics`, monitoring agents).
|
||
|
||
Background on the manual-sync default: [`../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md`](../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||
|
||
The Argo Application also defines:
|
||
|
||
- `syncPolicy.automated.prune` — whether Argo deletes objects no longer in Git.
|
||
- `syncPolicy.automated.selfHeal` — whether Argo reverts manual cluster edits.
|
||
- `syncOptions` — `CreateNamespace`, `ServerSideApply`, `RespectIgnoreDifferences`, retry/backoff.
|
||
- Sync waves via annotations (in chart templates or sidecars).
|
||
|
||
These all live in the **sister repo**, not here. See [`../docs/global/coding-guidelines/argocd.md`](../docs/global/coding-guidelines/argocd.md).
|
||
|
||
## Failure modes
|
||
|
||
| Failure | Where it surfaces | Read |
|
||
|---------|------------------|------|
|
||
| Render error in `helm template` | Argo Application status `ComparisonError` | Re-render locally; fix values |
|
||
| `OutOfSync → SyncFailed` after Sync click | Argo UI events | [`../docs/platform/runbooks/argocd-sync-failure.md`](../docs/platform/runbooks/argocd-sync-failure.md) |
|
||
| Pods land but stay `Pending` | `kubectl get pods` on target cluster | [`../docs/platform/runbooks/pod-pending-scheduling.md`](../docs/platform/runbooks/pod-pending-scheduling.md) |
|
||
| Ingress 5xx after Contour change | `contour-external` envoy logs / synthetic probes | [`../docs/platform/runbooks/ingress-down.md`](../docs/platform/runbooks/ingress-down.md) |
|
||
| Drift reappears after `kubectl edit` | `selfHeal: true` doing its job | Edit Git, not the cluster |
|
||
|
||
## Sister-repo coupling
|
||
|
||
Almost every non-trivial change is a **paired PR**:
|
||
|
||
- New app on cluster: PR here (override) + PR in sister repo (Application).
|
||
- New cluster: PR here (cluster directory) + PR in sister repo (`ApplicationSet` cluster generator).
|
||
- Blue-green sibling cutover: PR here (sibling values) + PR in sister repo (Application `targetRevision` / chart path).
|
||
|
||
Procedures: [`../docs/platform/procedures/onboard-app-to-cluster.md`](../docs/platform/procedures/onboard-app-to-cluster.md), [`../docs/platform/procedures/onboard-new-cluster.md`](../docs/platform/procedures/onboard-new-cluster.md), [`../docs/platform/procedures/blue-green-chart-migration.md`](../docs/platform/procedures/blue-green-chart-migration.md), [`../docs/platform/procedures/deboard-app.md`](../docs/platform/procedures/deboard-app.md).
|
||
|
||
## See also
|
||
|
||
- [`./00-overview.md`](./00-overview.md)
|
||
- [`./04-override-hierarchy.md`](./04-override-hierarchy.md)
|
||
- [`../docs/architecture.md`](../docs/architecture.md)
|
||
- [`../docs/global/agent-operations-guide.md`](../docs/global/agent-operations-guide.md)
|