added repo
This commit is contained in:
Your Name
@@ -0,0 +1,231 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Add or modify a Contour HTTPProxy route
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** one Contour release × one cluster (a misrouted host can break ingress for the whole BU).
|
||||
> **Approval:** consuming-app owner + cluster owner. Platform team if the change touches `contour-external*`.
|
||||
|
||||
This procedure covers HTTPProxy route changes that flow through one of the cluster's Contour releases. Most clusters run **multiple Contour instances** — read the [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) (repo root) authoritative scheduling matrix before editing **any** Contour values.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Contour topology
|
||||
|
||||
| Release name | Plane | Typical purpose |
|
||||
|--------------|-------|-----------------|
|
||||
| `contour-external` | North-south | Public / external traffic (terminates at GCP external LB) |
|
||||
| `contour-external-1` | North-south | Second external Contour (blue-green or capacity split) |
|
||||
| `contour-internal-0` | East-west | Internal traffic, plane 0 |
|
||||
| `contour-internal-1` | East-west | Internal traffic, plane 1 |
|
||||
| `contour-internal-intra-0` | Intra-cluster | Cluster-local east-west, plane 0 |
|
||||
| `contour-internal-intra-1` | Intra-cluster | Cluster-local east-west, plane 1 |
|
||||
|
||||
Each is a separate Helm release, on a separate node pool, with its own `nodeSelector` / `tolerations` / `computeClass`. Picking the wrong release is the most common authoring mistake.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding a new HTTPProxy / route for a service.
|
||||
- Changing TLS, retry, timeout, or rate-limit policy on an existing route.
|
||||
- Re-pointing an HTTPProxy at a different upstream Service.
|
||||
- Adding a new host to an existing route's `virtualhost.fqdn`.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- Changing Contour itself (sizing, scheduling, image) — that's [modify-observability-config.md](modify-observability-config.md)-style infra editing on the Contour release.
|
||||
- Bumping Contour's chart version — use [update-chart-version.md](update-chart-version.md) and consult the versioned sibling (`contour-v1.33.3`).
|
||||
- Curling production hostnames to test — [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md) forbids it. Test from inside the cluster with a `curl` Pod.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target Contour release | `contour-internal-0` |
|
||||
| HTTPProxy host | `api-foo.internal.meeshogcp.in` |
|
||||
| Upstream Service | `foo-svc.foo-ns:8080` |
|
||||
| TLS source | `cert-manager` / `external-secret` / `none` |
|
||||
| Path prefixes | `/v1`, `/health` |
|
||||
| Approval ticket | CMR-… (if BU policy) |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You know which Contour release is the right one for this host (north-south = `external*`; east-west between BUs = `internal-{0,1}`; cluster-local = `internal-intra-{0,1}`). When in doubt, sample existing HTTPProxies in the same namespace.
|
||||
- [ ] The upstream Service exists or will exist by the time of Sync.
|
||||
- [ ] The DNS name follows the cluster's `external-dns` pattern.
|
||||
- [ ] The TLS source (Secret, Issuer, etc.) exists on the cluster.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Identify the right Contour release
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/ | grep '^contour'
|
||||
```
|
||||
|
||||
Pick the release whose plane matches the new route's traffic class. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). If unsure, look at existing HTTPProxies on the cluster:
|
||||
|
||||
```bash
|
||||
# Read existing routes already shipped via this repo
|
||||
yq e '.. | select(has("httpproxies"))' \
|
||||
helm-overrides/<cluster>/contour-internal-0/custom-values.yaml
|
||||
```
|
||||
|
||||
### 2. Decide where the HTTPProxy lives
|
||||
|
||||
Two patterns exist:
|
||||
|
||||
| Pattern | When | Where to author |
|
||||
|---------|------|-----------------|
|
||||
| **Inline in Contour values** | Routes shared by the cluster's infra layer (e.g. Grafana, Argo CD). | `helm-overrides/<cluster>/<contour-release>/custom-values.yaml` under `httpproxies:` (if the chart supports it) or as a sidecar manifest in the same dir. |
|
||||
| **In the consuming app's repo** | Routes for a specific service. | The service's own deployment artifacts. **Out of scope for this repo.** |
|
||||
|
||||
If the route is service-owned, **redirect** to the consuming team's repo and stop. This procedure only covers infra-layer HTTPProxies.
|
||||
|
||||
### 3. Author the HTTPProxy
|
||||
|
||||
Skeleton (sidecar manifest pattern):
|
||||
|
||||
```yaml
|
||||
apiVersion: projectcontour.io/v1
|
||||
kind: HTTPProxy
|
||||
metadata:
|
||||
name: <name>
|
||||
namespace: <ns>
|
||||
spec:
|
||||
virtualhost:
|
||||
fqdn: <fqdn>
|
||||
tls:
|
||||
secretName: <tls-secret> # cert-manager-managed Secret
|
||||
routes:
|
||||
- conditions:
|
||||
- prefix: /
|
||||
services:
|
||||
- name: <upstream-service>
|
||||
port: <port>
|
||||
timeoutPolicy:
|
||||
response: 30s
|
||||
retryPolicy:
|
||||
count: 2
|
||||
retryOn: 5xx
|
||||
```
|
||||
|
||||
Drop into `helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml` (sidecar manifest) — see [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
### 4. Lint the HTTPProxy
|
||||
|
||||
```bash
|
||||
# Schema check (kubectl --dry-run against the cluster's CRD)
|
||||
kubectl --context=<ctx> --dry-run=server -f helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml apply
|
||||
|
||||
# Or local: validate against the projectcontour CRD schema
|
||||
kubeconform -schema-location default -schema-location \
|
||||
'https://raw.githubusercontent.com/projectcontour/contour/main/examples/contour/01-crds.yaml' \
|
||||
helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml
|
||||
```
|
||||
|
||||
### 5. Render the chart
|
||||
|
||||
```bash
|
||||
helm template <contour-release> helm-templates/<contour-chart> \
|
||||
-f helm-overrides/<cluster>/<contour-release>/custom-values.yaml > /tmp/contour.yaml
|
||||
```
|
||||
|
||||
The render must succeed; the HTTPProxy sidecar is applied alongside, not through Helm — but rendering catches values-side errors that would block sync of the same Argo Application.
|
||||
|
||||
### 6. Test from inside the cluster (post-merge, not pre-merge)
|
||||
|
||||
**Do not** curl `<fqdn>.meeshogcp.in` from your laptop / a build agent. Run a curl Pod on-cluster:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> run -it --rm curl-test \
|
||||
--image=curlimages/curl --restart=Never -- \
|
||||
curl -v -H "Host: <fqdn>" http://<contour-svc>.projectcontour.svc.cluster.local
|
||||
```
|
||||
|
||||
### 7. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b contour/<cluster>-<route-name>
|
||||
git add helm-overrides/<cluster>/<contour-release>/
|
||||
git commit -m "contour(<cluster>/<contour-release>): add route <name>"
|
||||
git push origin contour/<cluster>-<route-name>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
Adds (or modifies) HTTPProxy `<name>` on `<cluster>` via `<contour-release>` for FQDN `<fqdn>`.
|
||||
|
||||
## Why
|
||||
<1-2 sentences>
|
||||
|
||||
## Topology
|
||||
- Cluster: `<cluster>`
|
||||
- Contour release: `<contour-release>` (plane: external / internal / intra)
|
||||
- FQDN: `<fqdn>`
|
||||
- Upstream Service: `<svc>:<port>` in namespace `<ns>`
|
||||
- TLS: cert-manager Secret `<tls-secret>`
|
||||
|
||||
## Validation
|
||||
- [ ] HTTPProxy CRD schema validation passed (kubeconform / kubectl --dry-run)
|
||||
- [ ] `helm template` rendered cleanly
|
||||
- [ ] On-cluster curl from a curl Pod returns expected status
|
||||
- [ ] DNS / external-dns plumbed (existing wildcard or new external-dns Service)
|
||||
|
||||
## Approvers
|
||||
- App owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
- Platform (if `contour-external*`): <handle>
|
||||
```
|
||||
|
||||
### 8. After merge — Sync
|
||||
|
||||
Open the cluster's Argo CD UI, find the Contour release's Application, **Sync**. Verify:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> -n projectcontour get httpproxy <name>
|
||||
kubectl --context=<ctx> -n projectcontour describe httpproxy <name> | grep -A5 'Status:'
|
||||
```
|
||||
|
||||
A `Valid: true` status means Contour accepted the route. `Valid: false` with a reason → fix the values and re-PR.
|
||||
|
||||
If the HTTPProxy is `Valid` but traffic still 5xx → [../runbooks/ingress-down.md](../runbooks/ingress-down.md) §4.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Wrong Contour release.** A route mounted on `contour-internal-intra-0` is unreachable from outside the cluster. Cross-check the matrix.
|
||||
2. **Curling the production FQDN** from a developer machine to "test." Forbidden — see [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md).
|
||||
3. **Hand-edited values for a single host across all Contour releases** — pick one release, justify it.
|
||||
4. **`tls.passthrough` for plain HTTP services** — silent: TLS terminates upstream, doesn't.
|
||||
5. **Wildcard hosts that overlap an existing HTTPProxy** — Contour status will mark one of them invalid; check before merging.
|
||||
6. **Editing `helm-templates/contour*/`** to "tweak the chart." Layer 1 forbids casual chart edits — the chart is vanilla upstream.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the values PR.
|
||||
- Sync the Contour Application — Argo will prune the HTTPProxy or restore the prior values.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference (authoritative scheduling): [../../../contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [../runbooks/ingress-down.md](../runbooks/ingress-down.md).
|
||||
- Schema: [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- ADR: [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,206 @@
|
||||
# Procedure — Blue-green chart migration (versioned siblings)
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** the chart upgrade ships in a new sibling directory; no cluster cuts over until its `Application` is repointed. Each cluster's cutover is its own decision.
|
||||
> **Approval:** platform team — multi-reviewer.
|
||||
|
||||
This procedure handles the case where you can't safely bump a chart's pinned `dependencies[].version` in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as **sibling chart directories** until every consuming cluster has migrated.
|
||||
|
||||
The repo already shows this pattern at work:
|
||||
|
||||
| Original | Migration target |
|
||||
|----------|------------------|
|
||||
| `argo-cd` | `argo-cd-green` |
|
||||
| `contour` | `contour-v1.33.3` |
|
||||
| `keda` | `keda-2.17.1` |
|
||||
| `opentelemetry-collector` | `opentelemetry-collector-latest` |
|
||||
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` |
|
||||
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` |
|
||||
| `sonarqube` | `sonarqube-old` *(reverse — `sonarqube` is the new one; `-old` retained for rollback)* |
|
||||
|
||||
See [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) for the rationale.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- A major-version bump with breaking template changes.
|
||||
- A bump with an immutable-field change (Deployment `spec.selector`, StatefulSet `volumeClaimTemplates`).
|
||||
- A migration that needs cluster-by-cluster cutover with rollback windows.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- A patch / minor bump that's a clean drop-in → use [update-chart-version.md](update-chart-version.md).
|
||||
- Forking templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] Platform team agrees a blue-green migration is required (not just a bump).
|
||||
- [ ] You've identified the variant naming convention (`-green`, `-vX.Y.Z`, `-latest`, etc.).
|
||||
- [ ] You've identified every consuming cluster: `grep -rl '<chart>' helm-overrides`.
|
||||
- [ ] CMR open.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Create the sibling chart directory
|
||||
|
||||
```bash
|
||||
cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
|
||||
cd helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
Update `Chart.yaml`:
|
||||
|
||||
```diff
|
||||
apiVersion: v2
|
||||
-name: <chart>
|
||||
+name: <chart>-<variant>
|
||||
…
|
||||
dependencies:
|
||||
- name: <sub>
|
||||
- version: <old-version>
|
||||
+ version: <new-version>
|
||||
repository: <upstream-repo>
|
||||
```
|
||||
|
||||
```bash
|
||||
helm dependency update
|
||||
```
|
||||
|
||||
### 2. Render against a representative cluster's overrides — old variant
|
||||
|
||||
The point of blue-green is *no surprises*. Render the **old** chart with each consuming cluster's override and capture the output.
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 3. Render against the same overrides — new sibling variant
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 4. Diff old → new per cluster
|
||||
|
||||
```bash
|
||||
for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
|
||||
echo "=== $cluster ==="
|
||||
diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
|
||||
done
|
||||
```
|
||||
|
||||
For each cluster, decide:
|
||||
|
||||
- Is the diff what you expected?
|
||||
- Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
|
||||
- Is the cutover safe to do without the workload owner present? (If no → schedule.)
|
||||
|
||||
### 5. Open PR-1: introduce the sibling chart
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-to-<variant>-introduce
|
||||
git add helm-templates/<chart>-<variant>/
|
||||
git commit
|
||||
git push origin migrate/<chart>-to-<variant>-introduce
|
||||
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"
|
||||
```
|
||||
|
||||
After merge, the new chart exists in the repo but **no cluster uses it yet** — the existing Argo `Application`s still point at `helm-templates/<chart>`.
|
||||
|
||||
### 6. Per-cluster cutover (one PR pair per cluster)
|
||||
|
||||
For each consuming cluster:
|
||||
|
||||
a. **Update the cluster's `custom-values.yaml`** if the new chart needs different values. Open as a values-side PR (this repo).
|
||||
|
||||
b. **Update the sister-repo `Application`** to repoint:
|
||||
|
||||
```diff
|
||||
spec:
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
- path: helm-templates/<chart>
|
||||
+ path: helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
c. **After both merge**, click Sync in the cluster's Argo CD UI.
|
||||
|
||||
d. **Soak** — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.
|
||||
|
||||
### 7. Open PR-N: retire the old sibling
|
||||
|
||||
After every cluster has cut over and soaked:
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-retire-old
|
||||
git rm -r helm-templates/<chart>
|
||||
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
|
||||
git commit
|
||||
git push origin migrate/<chart>-retire-old
|
||||
gh pr create --base main --title "migrate: retire old <chart> sibling"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Confirmation every cluster has cut over (`grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config` returns nothing).
|
||||
- Confirmation soak period elapsed.
|
||||
- Decision: delete vs rename to `-old` (kept for rollback).
|
||||
|
||||
---
|
||||
|
||||
## Why two PRs at the start, then per-cluster pairs, then a final retirement
|
||||
|
||||
| PR | Effect |
|
||||
|----|--------|
|
||||
| **PR-1: introduce sibling** | Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them. |
|
||||
| **PR-2..N-1: per-cluster cutover (paired with sister repo)** | One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart. |
|
||||
| **PR-N: retire old sibling** | Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible. |
|
||||
|
||||
Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Cutting over multiple clusters in one PR.** Bundle = no per-cluster rollback.
|
||||
2. **Deleting the old sibling before every cluster has cut over.** Cluster N+1 has its `Application` pointing at a path that no longer exists; sync fails immediately.
|
||||
3. **Cutting over without rendering first.** Surprises after merge.
|
||||
4. **Skipping the soak period.** "It looked fine in the first 5 minutes" is not soak.
|
||||
5. **Renaming the *new* sibling to drop the suffix** (e.g. `argo-cd-green` → `argo-cd`) before the old chart is retired. The path collision will break Argo CD's caching.
|
||||
|
||||
---
|
||||
|
||||
## Rollback (per cluster)
|
||||
|
||||
If a cluster's cutover fails:
|
||||
|
||||
1. Revert the sister-repo PR for that cluster.
|
||||
2. Click Sync in the cluster's Argo CD — the `Application` re-renders against `<chart>` (the old sibling). The rollback is one cluster only.
|
||||
3. Investigate; iterate.
|
||||
|
||||
If the new sibling has a fundamental problem affecting every cluster:
|
||||
|
||||
1. Open PR-X: revert PR-1 (delete the sibling).
|
||||
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
|
||||
3. Schedule a postmortem before re-attempting.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md).
|
||||
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
- ADR: [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- [SANCTITY_RULES R8](../../global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,119 @@
|
||||
# Procedure — Deboard a retired app from a cluster
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
|
||||
> **Approval:** app owner + cluster owner. CMR mandatory.
|
||||
|
||||
This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions (block deboarding if any are FALSE)
|
||||
|
||||
- [ ] App is genuinely retired on this cluster — confirmed by app owner.
|
||||
- [ ] No traffic / scrape job / dependency is still hitting it.
|
||||
- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
|
||||
- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
|
||||
- [ ] CMR approved.
|
||||
|
||||
```bash
|
||||
# Where does this app exist today?
|
||||
find helm-overrides -maxdepth 2 -type d -name '<app>'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Decide the scope
|
||||
|
||||
| Scope | Then |
|
||||
|-------|------|
|
||||
| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
|
||||
| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
|
||||
| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |
|
||||
|
||||
### 2. Open the sister-repo `Application` removal PR FIRST
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`:
|
||||
|
||||
- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
|
||||
- Merge.
|
||||
- The cluster's Argo CD will mark the `Application` for removal on next reconcile.
|
||||
|
||||
This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.
|
||||
|
||||
### 3. Manually clean up the workload (if `automated.prune` was not set)
|
||||
|
||||
For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:
|
||||
|
||||
```bash
|
||||
kubectl --context=<cluster> delete <kind>/<name> -n <ns>
|
||||
# OR, if the whole namespace is dedicated to this release:
|
||||
kubectl --context=<cluster> delete namespace <ns>
|
||||
```
|
||||
|
||||
Coordinate with the app owner — wholesale namespace deletion is irreversible.
|
||||
|
||||
### 4. Open the values-side removal PR
|
||||
|
||||
```bash
|
||||
git checkout -b deboard/<app>-from-<cluster>
|
||||
git rm -r helm-overrides/<cluster>/<app>/
|
||||
git commit
|
||||
git push origin deboard/<app>-from-<cluster>
|
||||
gh pr create --base main --title "deboard: <app> from <cluster>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Confirmation pre-conditions are TRUE.
|
||||
- Sister-repo PR (already merged).
|
||||
- CMR ticket reference.
|
||||
- Confirmation the workload was manually cleaned up (or scheduled).
|
||||
|
||||
### 5. (Optional) If this was the last cluster running the app
|
||||
|
||||
If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:
|
||||
|
||||
- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).
|
||||
|
||||
---
|
||||
|
||||
## "What if the app might come back?"
|
||||
|
||||
If retirement is provisional:
|
||||
|
||||
- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
|
||||
- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
|
||||
2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
|
||||
3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
|
||||
4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
|
||||
5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If you deboarded by mistake:
|
||||
|
||||
1. Revert the values-side PR (`git revert <merge-sha>`).
|
||||
2. Revert the sister-repo PR.
|
||||
3. Click Sync in the cluster's Argo CD UI.
|
||||
4. The `Application` is recreated and renders the chart.
|
||||
5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
|
||||
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
|
||||
- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Procedure — Intentionally fork an upstream chart
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** every cluster that consumes this chart will render against the forked templates on next sync.
|
||||
> **Approval:** platform team — multi-reviewer.
|
||||
|
||||
Most charts in `helm-templates/` are **vanilla upstream**, pulled via `helm pull`. The `Chart.yaml` is a thin wrapper declaring the upstream chart as a dependency; the actual templates come from the subchart in `charts/`. Editing a `templates/*.yaml` *under the wrapper* silently forks the chart, and the fork is clobbered the next time someone runs `helm dependency update`.
|
||||
|
||||
This procedure is the supported path when a fork is **intentional**.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Upstream chart lacks a feature you need (e.g. a values key that doesn't exist).
|
||||
- Upstream behaviour conflicts with Meesho's environment (e.g. probe path doesn't work behind Contour).
|
||||
- Upstream chart has a bug awaiting an upstream fix.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- Adding a values knob — file a PR upstream, or override behaviour through existing knobs.
|
||||
- Pinning to an old version — use [update-chart-version.md](update-chart-version.md) (or just don't bump).
|
||||
- Style preferences — leave the upstream chart alone.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You've confirmed the desired behaviour cannot be achieved via the chart's existing values.
|
||||
- [ ] You've checked whether an upstream PR / issue already covers this.
|
||||
- [ ] Platform team agrees the fork is justified.
|
||||
- [ ] The fork's rationale will be documented in the chart's `README.md`.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Vendor the chart's templates
|
||||
|
||||
If the chart is currently a thin wrapper (templates come from a subchart in `charts/<sub>`), you must first promote the subchart's templates into the wrapper.
|
||||
|
||||
```bash
|
||||
cd helm-templates/<chart>
|
||||
ls charts/ # find the subchart .tgz
|
||||
helm dependency update # ensure it's resolved
|
||||
|
||||
# Extract templates from the subchart .tgz
|
||||
tar -xzf charts/<sub>-<ver>.tgz -C /tmp/
|
||||
cp -R /tmp/<sub>/templates ./templates
|
||||
cp -R /tmp/<sub>/values.yaml ./values.yaml.upstream
|
||||
```
|
||||
|
||||
Now the wrapper has its own `templates/` — Helm will use those instead of the subchart's.
|
||||
|
||||
### 2. Edit `Chart.yaml` to reflect the fork
|
||||
|
||||
Remove the dependency (since the templates are now local), and bump `version:` (the chart's own version, not the subchart's):
|
||||
|
||||
```diff
|
||||
apiVersion: v2
|
||||
name: <chart>
|
||||
-version: 0.1.0
|
||||
+version: 0.1.0+fork.1
|
||||
description: <description> — Meesho-forked from upstream <chart> <upstream-version>
|
||||
-dependencies:
|
||||
- - name: <sub>
|
||||
- version: <upstream-version>
|
||||
- repository: <upstream-repo>
|
||||
```
|
||||
|
||||
### 3. Make the fork edits
|
||||
|
||||
Edit `templates/*.yaml` or `values.yaml` to apply the fix. Keep the diff minimal — every line away from upstream is technical debt.
|
||||
|
||||
### 4. Document the fork
|
||||
|
||||
Edit `helm-templates/<chart>/README.md` (create if missing). Use this template:
|
||||
|
||||
```markdown
|
||||
# <chart>
|
||||
|
||||
**Forked from upstream <chart> <upstream-version>** at <date>.
|
||||
|
||||
## Why
|
||||
|
||||
<one-paragraph rationale — what behaviour the fork changes and why we couldn't achieve
|
||||
it through values alone>
|
||||
|
||||
## What's changed
|
||||
|
||||
- `templates/<file>.yaml` — <one-line diff summary>
|
||||
- `values.yaml` — <one-line diff summary>
|
||||
|
||||
## Upstream tracking
|
||||
|
||||
- Upstream PR: <link, if you've sent one>
|
||||
- Upstream issue: <link>
|
||||
- When upstream merges: revert this fork via [unfork procedure].
|
||||
```
|
||||
|
||||
### 5. Render and diff
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
|
||||
|
||||
# Spot-check helm diff against a live cluster
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml --kube-context=<context>
|
||||
```
|
||||
|
||||
### 6. Commit and open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b fork/<chart>-<purpose>
|
||||
git add helm-templates/<chart>/
|
||||
git commit
|
||||
git push origin fork/<chart>-<purpose>
|
||||
gh pr create --base main --title "fork: <chart> — <one-line purpose>"
|
||||
```
|
||||
|
||||
PR description must include:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Why the fork is necessary (link to upstream issue/PR if any).
|
||||
- What templates / values are changed and why.
|
||||
- Platform team approver(s).
|
||||
- Plan for unforking when upstream lands the fix.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Editing `templates/` without first vendoring** (templates from a subchart). Your edits live in `helm-templates/<chart>/templates/` but Helm renders from `charts/<sub>/templates/` — the edits do nothing, then get clobbered.
|
||||
2. **Forking a chart and not documenting why.** Six months later nobody remembers; the fork looks like accidental drift.
|
||||
3. **Forking instead of overriding via values.** Always check the chart's existing values surface first.
|
||||
4. **Forking and then bumping the upstream version.** A bump runs `helm dependency update`, which can clobber the fork. The fork must be re-applied or the bump must explicitly re-vendor.
|
||||
5. **Sweeping cleanup of upstream code** alongside the fork edit. The diff should be exactly the change you intended; everything else is upstream.
|
||||
|
||||
---
|
||||
|
||||
## Rollback / unforking
|
||||
|
||||
When upstream lands the fix and you want to return to vanilla:
|
||||
|
||||
1. Restore `Chart.yaml` to declare the upstream as a dependency (with the new upstream version that includes the fix).
|
||||
2. Delete the local `templates/` and `values.yaml` (or rename them as `.upstream` for reference).
|
||||
3. Run `helm dependency update`.
|
||||
4. Update `helm-templates/<chart>/README.md` to remove the "forked" status.
|
||||
5. Render against a sibling override and confirm the resulting manifests match what the fork was producing.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md).
|
||||
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
- [SANCTITY_RULES R7](../../global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,227 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify Prometheus / VictoriaMetrics alert rules
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** PagerDuty / on-call paging across the cluster's tenants. A wrong threshold pages everyone; a missing rule masks an outage.
|
||||
> **Approval:** observability owner + on-call lead for the affected severity.
|
||||
|
||||
This procedure is the narrow alert-rule slice of [modify-observability-config.md](modify-observability-config.md). Use this when the entire intent of the PR is alert-rule editing — adding, removing, retuning thresholds, changing `for:` windows, or rewriting `expr:` PromQL.
|
||||
|
||||
---
|
||||
|
||||
## Where alert rules live
|
||||
|
||||
| File | Engine | Notes |
|
||||
|------|--------|-------|
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert | Most clusters use vmalert against VM cluster as the primary alerting engine. Rules typically under `config.rules` or rendered as a `VMRule` CRD. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmalert (bundled) / VM ruler | Some clusters carry rules in the cluster chart's `vmalert.config` block. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` | Prometheus / `kube-prometheus-stack` | Rules under `additionalPrometheusRulesMap` or `serverFiles.alerting_rules.yml`. |
|
||||
| `helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml` | Prometheus | Same as above; older clusters use this chart name. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` | Mimir ruler | Rules go through the ruler API; YAML in `runtimeConfig` or via a sidecar `ConfigMap`. |
|
||||
|
||||
Confirm which cluster runs which engine before editing — sample existing rules in the override.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing one or more alert rules in the cluster's primary alerting engine.
|
||||
- Tuning `expr:` PromQL or `for:` windows on existing rules.
|
||||
- Re-routing alerts (changing `severity`, `team`, or other routing labels).
|
||||
- Adding/removing recording rules for downstream alert efficiency.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- Touching scrape configs, retention, datasources, or sizing in the same PR — split it. See [modify-observability-config.md](modify-observability-config.md).
|
||||
- Bumping the alerting chart's version — see [update-chart-version.md](update-chart-version.md).
|
||||
- Writing app-specific alert rules that belong in the consuming service's own values — those go in the service's repo.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `vmalert` / `prometheus` / `kube-prometheus-stack` / `mimir` |
|
||||
| Rule name(s) added/changed | `HighErrorRate`, `KafkaConsumerLag` |
|
||||
| `expr:` PromQL | `sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 0.05` |
|
||||
| `for:` window | `5m` |
|
||||
| Severity / routing labels | `severity: critical`, `team: supply-platform` |
|
||||
| Approval ticket | CMR-… |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You have `promtool` and/or `vmalert` available locally.
|
||||
- [ ] You know the cluster's Alertmanager / vmalert notifier routing (which `severity` / `team` label routes where).
|
||||
- [ ] You have run the new PromQL expression against the cluster's VM/Prometheus to confirm it returns sensible values *before* writing the rule — open Grafana Explore.
|
||||
- [ ] You have on-call sign-off if this rule pages on-call.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Locate the rules block
|
||||
|
||||
```bash
|
||||
yq e '.config.rules // .additionalPrometheusRulesMap // .serverFiles' \
|
||||
helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
Identify the YAML path the engine expects:
|
||||
|
||||
- **vmalert**: `config.groups[].rules[]`.
|
||||
- **kube-prometheus-stack**: `additionalPrometheusRulesMap.<group-name>.groups[].rules[]`.
|
||||
- **prometheus**: `serverFiles.alerting_rules.yml.groups[].rules[]`.
|
||||
- **mimir ruler**: per-tenant config — confirm the cluster's tenancy setup.
|
||||
|
||||
### 2. Author the rule
|
||||
|
||||
```yaml
|
||||
- alert: <RuleName> # PascalCase, no spaces; surfaces in pages and dashboards
|
||||
expr: |
|
||||
<PromQL expression>
|
||||
for: <duration> # e.g. 5m, 10m
|
||||
labels:
|
||||
severity: <warning|critical|info>
|
||||
team: <owning-team> # routes to the right PagerDuty service
|
||||
annotations:
|
||||
summary: "<one-line>"
|
||||
description: "<longer text, can reference $labels.* and $value>"
|
||||
runbook_url: <link to runbook>
|
||||
```
|
||||
|
||||
#### Threshold discipline
|
||||
|
||||
- Express thresholds as `rate(...) > 0.05` rather than `> 5%`. Ensure units in the expression match the units in `summary` / `description`.
|
||||
- Avoid `for:` windows shorter than the scrape interval × 2. Sub-`1m` windows are flap factories.
|
||||
- Avoid alerting on absolute counts (`sum(http_requests_total) > 1000`); prefer rates / ratios.
|
||||
- For SLO-style alerts, use multi-window multi-burn-rate (Google SRE workbook). Single-window threshold alerts are a known anti-pattern.
|
||||
|
||||
#### Routing implications
|
||||
|
||||
Changing `severity` from `warning` to `critical` (or vice versa) **silently re-routes the page** — the rule may now wake on-call where it previously emailed (or vice versa). Same for `team:` — a typo here drops alerts into the wrong PagerDuty service. Cross-check the cluster's Alertmanager / vmalert routing config:
|
||||
|
||||
```bash
|
||||
yq e '.config.route // .alertmanagerConfig' \
|
||||
helm-overrides/<cluster>/<alertmanager-chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
### 3. Render the chart
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
### 4. Validate PromQL
|
||||
|
||||
For Prometheus / `kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
For Mimir ruler — `mimirtool rules check` against the rendered ruler config.
|
||||
|
||||
If validation fails: fix the PromQL. **Do not** merge a rule that fails parse.
|
||||
|
||||
### 5. Sanity-check expressions in Grafana Explore
|
||||
|
||||
In a non-production manner: open Grafana on the cluster, paste the `expr:` into Explore, run over the last 6h. Confirm:
|
||||
|
||||
- The series returned makes sense for the rule.
|
||||
- The threshold isn't always-firing or never-firing in normal operation.
|
||||
- Empty result is allowed (`absent_over_time(...)` style).
|
||||
|
||||
### 6. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b alerts/<cluster>-<rule-name>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "alerts(<cluster>/<chart>): <add|tune|remove> <RuleName>"
|
||||
git push origin alerts/<cluster>-<rule-name>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<add|tune|remove> alert rule(s) on `<cluster>` × `<chart>`.
|
||||
|
||||
## Rules touched
|
||||
| Rule | Change | New expr | New `for:` | Severity | Team |
|
||||
|------|--------|----------|-----------|----------|------|
|
||||
| `<RuleName>` | added | `<expr>` | `5m` | `critical` | `supply-platform` |
|
||||
|
||||
## Validation
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed
|
||||
- [ ] PromQL sanity-checked in Grafana Explore over last 6h
|
||||
- [ ] Threshold appropriate (no flap risk, no always-firing)
|
||||
- [ ] Routing labels (`severity`, `team`) verified against Alertmanager/vmalert routes
|
||||
- [ ] Runbook URL added (`annotations.runbook_url`)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- On-call lead (severity=<x>): <handle>
|
||||
```
|
||||
|
||||
### 7. After merge — Sync and observe
|
||||
|
||||
Sync the Argo Application. Watch:
|
||||
|
||||
```bash
|
||||
# vmalert
|
||||
kubectl --context=<ctx> -n monitoring logs deploy/vmalert | grep -i "<RuleName>\|error"
|
||||
|
||||
# kube-prometheus-stack
|
||||
kubectl --context=<ctx> -n monitoring logs prometheus-k8s-0 | grep -i "<RuleName>\|error"
|
||||
```
|
||||
|
||||
Open the alerting engine's UI (vmalert UI / Prometheus `/alerts`) to confirm the rule loaded.
|
||||
|
||||
### Watch for false positives
|
||||
|
||||
Stay on the alert for at least one full `for:` window after sync. If the rule pages immediately and was not expected to → revert.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Sub-1m `for:` windows** — flap-prone.
|
||||
2. **Alerting on absolute counts** instead of rates/ratios.
|
||||
3. **`severity` flips without on-call sign-off** — silent re-route.
|
||||
4. **Missing `runbook_url`** — pages without remediation steps waste on-call cycles.
|
||||
5. **PromQL that uses `{job=~"foo.*"}` regex without anchors** — slow query, sometimes false matches.
|
||||
6. **Inlining the rule body in the chart `templates/`** to "make it sticky" — that's a chart fork; see NEVER-DO.
|
||||
7. **Adding the rule on every cluster in one PR** — surgical only.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the PR. Sync.
|
||||
- vmalert/Prometheus reload picks up the previous ruleset within seconds.
|
||||
- Any in-flight alerts resolve on next evaluation.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Procedure: [modify-observability-config.md](modify-observability-config.md) — broader observability edits.
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md) — when an alert is silent because the metric is gone.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
@@ -0,0 +1,210 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify observability stack config
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU).
|
||||
> **Approval:** observability owner (`siddharth.pal@meesho.com`) + cluster owner.
|
||||
|
||||
This procedure covers edits to the cluster-by-cluster observability override files:
|
||||
|
||||
| Override path | What it controls |
|
||||
|---------------|------------------|
|
||||
| `helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml` | Scrape targets, relabel rules, remote-write tenancy. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmstorage retention, vmselect/vminsert sizing, ruler config. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` (also `mimir-distributed`) | Mimir distributor/ingester/ruler config and tenancy. |
|
||||
| `helm-overrides/<cluster>/loki/custom-values.yaml` (also `loki-distributed`) | Loki retention, ingestion limits, multi-tenant config. |
|
||||
| `helm-overrides/<cluster>/tempo/custom-values.yaml` (also `tempo-distributed`) | Tempo block retention, distributor sizing. |
|
||||
| `helm-overrides/<cluster>/grafana/custom-values.yaml` | Datasources, dashboard providers, plugins. |
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert alerting rules and notifier config. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` (or `kube-prometheus-stack`) | Prometheus alert rules, scrape config, retention. |
|
||||
|
||||
All of these are **versioned-sibling-aware** — confirm whether the cluster's Argo Application points at `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`, etc., before editing. See [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md) for stack conventions.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing scrape targets, relabel rules, recording rules.
|
||||
- Changing retention (`vmstorage.retentionPeriod`, Loki `retention_period`, Tempo `compaction.block_retention`).
|
||||
- Adding/removing/tuning alert rules in `vmalert` or `kube-prometheus-stack`.
|
||||
- Adding/removing Grafana datasources, dashboards, plugins.
|
||||
- Tuning ingester/distributor sizing for Mimir/Loki/Tempo.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- **Bumping the chart version** of an observability tool — use [update-chart-version.md](update-chart-version.md).
|
||||
- **Onboarding a brand-new observability tool** to a cluster — use [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
||||
- **A blue-green migration** between sibling charts (`-latest`) — use [blue-green-chart-migration.md](blue-green-chart-migration.md).
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `victoria-metrics-cluster` |
|
||||
| Versioned-sibling target (if applicable) | `victoria-metrics-cluster-latest` |
|
||||
| Change kind | `retention bump`, `new scrape target`, `new alert rule`, `dashboard add` |
|
||||
| Promql expression(s) touched (if any) | `sum(rate(...)) by (job) > 0.05` |
|
||||
| Approval ticket | CMR-1234 (if BU policy requires) |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The cluster directory and chart override exist.
|
||||
- [ ] The cluster's Argo `Application` points at the chart you're editing (and not its sibling). Check `github.com/Meesho/devops-infra-argo-config`.
|
||||
- [ ] You have the previous PR diff for context (most observability edits are touching a known knob).
|
||||
- [ ] You have access to PromQL/promtool (or vmalert binary) locally for rule validation.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Confirm which sibling the cluster runs
|
||||
|
||||
```bash
|
||||
# In the sister repo
|
||||
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*
|
||||
```
|
||||
|
||||
Find the `Application` whose `spec.source.path` points at this repo. Note whether it's `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`. **Editing the wrong one is silent** — the file diff merges, but no cluster picks it up.
|
||||
|
||||
### 2. Read the current values
|
||||
|
||||
```bash
|
||||
yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less
|
||||
```
|
||||
|
||||
Note current retention, scrape targets, and any inline alert rules.
|
||||
|
||||
### 3. Author the change
|
||||
|
||||
Edit `helm-overrides/<cluster>/<chart>/custom-values.yaml`. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.
|
||||
|
||||
#### Cardinality discipline
|
||||
|
||||
If the change adds a label, scrape target, or `relabel_configs` rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See [observability.md §Cardinality discipline](../../global/coding-guidelines/observability.md).
|
||||
|
||||
#### Retention changes
|
||||
|
||||
Increasing `vmstorage.retentionPeriod`, Loki `retention_period`, or Tempo `block_retention` grows the backing PVC. Confirm `persistence.size` (or `vmstorage.persistentVolume.size`) has been bumped to match — otherwise vmstorage runs out of disk silently.
|
||||
|
||||
#### Alert-rule edits
|
||||
|
||||
If the rule's `for:` window or `severity` label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.
|
||||
|
||||
### 4. Render the chart locally
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
The render must succeed. If it errors, fix the values before continuing.
|
||||
|
||||
### 5. Validate any PromQL touched
|
||||
|
||||
For Prometheus/`kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
# Extract PrometheusRule objects from rendered output
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
|
||||
# Validate
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
# Extract VMRule (or ConfigMap with rules)
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
|
||||
# vmalert dry-run
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
If `promtool` / `vmalert` is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.
|
||||
|
||||
### 6. (Grafana) Validate datasource URL is in-cluster
|
||||
|
||||
Datasources should point at in-cluster Service DNS (e.g. `http://victoria-metrics-cluster-vmselect:8481`), not external endpoints. **Never** pin a Grafana datasource at `*.meeshogcp.in`, `prd.meesho.int`, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.
|
||||
|
||||
### 7. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b obs/<cluster>-<chart>-<short-desc>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "obs(<cluster>/<chart>): <short desc>"
|
||||
git push origin obs/<cluster>-<chart>-<short-desc>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<one-line: what changed and why>
|
||||
|
||||
## Cluster × chart
|
||||
- Cluster: `<cluster>`
|
||||
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
|
||||
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`
|
||||
|
||||
## Validation
|
||||
- [ ] `helm template` renders cleanly
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
|
||||
- [ ] Cardinality bounded (no unbounded label introduced)
|
||||
- [ ] Retention/PVC headroom confirmed (if retention changed)
|
||||
- [ ] PagerDuty routing unchanged (if alert rule changed)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
|
||||
## Sister repo
|
||||
- N/A (no Application change required)
|
||||
```
|
||||
|
||||
### 8. After merge
|
||||
|
||||
Argo CD on the target cluster reconciles. Most observability charts use **manual sync** (see [observability.md](../../global/coding-guidelines/observability.md)) — open the cluster's Argo CD UI, find the Application, click **Sync**. Watch:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> -n <observability-ns> get pods -w
|
||||
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50
|
||||
```
|
||||
|
||||
If a metric goes missing post-sync → [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Editing the wrong sibling.** The Argo Application points at `*-latest`; you edit the stable chart's values. Diff merges, nothing applies.
|
||||
2. **Adding `pod_name` / `request_id` / `user_id` as a label** without aggregation — explodes cardinality.
|
||||
3. **Bumping retention without bumping PVC.** vmstorage runs out of disk; ingest fails silently.
|
||||
4. **Silent PagerDuty re-route.** Changing `severity:` from `warning` to `critical` (or vice versa) without coordinating on-call.
|
||||
5. **Inlining production hostnames** as Grafana datasource URLs — see [SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3.
|
||||
6. **Cross-cluster normalising** — touching every cluster's `custom-values.yaml` in one PR. Surgical only; one cluster per PR.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the values PR. Argo CD will re-render with the previous values; click Sync.
|
||||
- For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the *config* but not the *data*.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
- Procedure: [modify-alert-rules.md](modify-alert-rules.md) — alert-only edits.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- ADR: [../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- Wiki: [../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md](../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md).
|
||||
@@ -0,0 +1,204 @@
|
||||
# Procedure — Onboard a new app to an existing cluster
|
||||
|
||||
> **Layer:** Layer 1 — Agent-Writable.
|
||||
> **Blast radius:** one new Helm release on one cluster.
|
||||
> **Approval:** app owner + cluster owner.
|
||||
|
||||
This procedure adds a new infrastructure-tooling Helm release to a cluster that already exists in `helm-overrides/`. It covers the slice owned by `devops-infra-helm-charts`. The matching Argo `Application` lives in `github.com/Meesho/devops-infra-argo-config` and must be paired.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Chart name (must exist in `helm-templates/`) | `kube-state-metrics` |
|
||||
| Target cluster directory | `k8s-supply-prd-ase1` |
|
||||
| Release name | `kube-state-metrics` (matches chart name; may be different for variants) |
|
||||
| Workload namespace | `monitoring` |
|
||||
| Image tag | `v2.10.1` |
|
||||
| Sized resources (CPU/memory requests + limits) | `250m / 512Mi` |
|
||||
| Node-pool key (per cluster) | `dedicated: monitoring` |
|
||||
| Whether this needs a sidecar `external-dns` Service | yes / no |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The chart exists in `helm-templates/<chart>/` with a current `Chart.yaml`.
|
||||
- [ ] The cluster directory exists in `helm-overrides/<cluster>/`.
|
||||
- [ ] The chart is appropriate for an *infra* release (services don't go here — they live in `devops-argo-config`).
|
||||
- [ ] The matching sister-repo `Application` PR is drafted (or will be drafted in parallel).
|
||||
- [ ] CMR ticket open if required by BU policy.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Verify the chart renders with a sibling cluster's values
|
||||
|
||||
Pick a cluster that already runs this chart and use its values as a starting point:
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -60
|
||||
```
|
||||
|
||||
If the render errors → the chart's dependencies may be unresolved. Run `helm dependency update helm-templates/<chart>` first.
|
||||
|
||||
### 2. Identify the cluster's scheduling profile
|
||||
|
||||
```bash
|
||||
# Standard GKE: uses 'dedicated:' keys
|
||||
grep -rh 'dedicated:' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
|
||||
|
||||
# GKE Autopilot: uses 'cloud.google.com/compute-class' keys
|
||||
grep -rh 'cloud.google.com/compute-class' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
|
||||
```
|
||||
|
||||
For Contour, cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). For others, copy from a sibling app on the **same** cluster — never from the same app on a different cluster ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md)).
|
||||
|
||||
### 3. Create the directory and `custom-values.yaml`
|
||||
|
||||
```bash
|
||||
mkdir -p helm-overrides/<cluster>/<app>
|
||||
$EDITOR helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
```
|
||||
|
||||
Author from scratch using [custom-values-schema.md](../schemas/custom-values-schema.md) and the cluster's scheduling profile from step 2. Do **not** copy a sibling cluster's values verbatim.
|
||||
|
||||
Skeleton:
|
||||
|
||||
```yaml
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<image>
|
||||
tag: <tag>
|
||||
|
||||
replicaCount: <N>
|
||||
|
||||
resources:
|
||||
requests: {cpu: <c>, memory: <m>}
|
||||
limits: {cpu: <c>, memory: <m>}
|
||||
|
||||
nodeSelector:
|
||||
<pool-key>: <pool-value>
|
||||
tolerations:
|
||||
- {key: <pool-key>, value: <pool-value>, effect: NoSchedule}
|
||||
```
|
||||
|
||||
### 4. (If needed) Add sidecar raw manifests
|
||||
|
||||
If the app needs sidecar resources (e.g. `external-dns` `Service`, `ComputeClass`, `ExternalSecret`), drop them in the same directory under a subfolder:
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<app>/
|
||||
custom-values.yaml
|
||||
external-dns-services/<svc>.yaml
|
||||
computeclass/<name>-cc.yaml
|
||||
```
|
||||
|
||||
See [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
### 5. Validate locally
|
||||
|
||||
```bash
|
||||
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
|
||||
# Render
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -60
|
||||
|
||||
# Optional: dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
|
||||
--kube-context=<context>
|
||||
```
|
||||
|
||||
### 6. Open the sister-repo PR
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`, draft an `Application` (or add to an existing `ApplicationSet`):
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: <release>-<cluster>
|
||||
spec:
|
||||
destination:
|
||||
name: <cluster>
|
||||
namespace: <workload-namespace>
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
path: helm-overrides/<cluster>/<app>
|
||||
helm:
|
||||
valueFiles: [custom-values.yaml]
|
||||
syncPolicy:
|
||||
syncOptions: [CreateNamespace=true]
|
||||
# Most infra apps DO NOT use automated sync — see ADR-A5
|
||||
```
|
||||
|
||||
### 7. Open the values-side PR (this repo)
|
||||
|
||||
```bash
|
||||
git checkout -b onboard/<chart>-on-<cluster>
|
||||
git add helm-overrides/<cluster>/<app>/
|
||||
git commit
|
||||
# pre-commit hook runs TruffleHog
|
||||
git push origin onboard/<chart>-on-<cluster>
|
||||
gh pr create --base main --title "Onboard <chart> to <cluster>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- App owner approver tag.
|
||||
- Cluster owner approver tag.
|
||||
- Sister-repo PR link (`devops-infra-argo-config#<n>`).
|
||||
- CMR ticket reference (if applicable).
|
||||
- Confirmation that the chart renders cleanly and `helm diff` (if run) showed only additions.
|
||||
|
||||
### 8. After both merge — sync in Argo CD
|
||||
|
||||
`devops-infra-argo-config`'s reconciler will create the `Application` resource on the cluster's Argo CD. Most infra apps are **manual sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)), so the workload deploy is a separate step:
|
||||
|
||||
1. Open the cluster's Argo CD UI.
|
||||
2. Search for the new `Application`.
|
||||
3. Verify the manifest renders (Diff view shows the chart's resources).
|
||||
4. Click **Sync**.
|
||||
5. Watch the rollout: `kubectl --context=<context> get pods -n <ns> -w`.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Copying the entire `helm-overrides/<sibling-cluster>/<app>/` directory** verbatim. Per-cluster scheduling differs.
|
||||
2. **Bundling onboarding with a chart-version bump.** Two separate PRs.
|
||||
3. **Skipping the sister-repo PR.** Without an `Application`, the values do nothing.
|
||||
4. **Setting `automated.{prune,selfHeal}: true`** in the sister-repo `Application` "to make life easier." Manual sync is the default safety property.
|
||||
5. **Inlining secrets** in `custom-values.yaml`. Use `ExternalSecret`.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the merge causes a problem before Sync:
|
||||
|
||||
- Revert the values-side PR (and the sister-repo PR).
|
||||
- Argo CD will prune the `Application` resource on next reconcile of the sister repo.
|
||||
|
||||
If Sync was clicked and the workload broke:
|
||||
|
||||
- Click **Rollback** in Argo CD UI to the previous synced revision (if there is one).
|
||||
- Or revert both PRs and re-Sync — the previous state had no `Application`, so the workload is removed.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md) for bumping after onboarding.
|
||||
- Procedure: [deboard-app.md](deboard-app.md) for retirement.
|
||||
- Skill: [skills/infra/onboard-app.md](../../../skills/infra/onboard-app.md) — agent-callable wrapper.
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,140 @@
|
||||
# Procedure — Onboard a new cluster's overrides
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded.
|
||||
> **Approval:** platform team + cluster owner. CMR mandatory.
|
||||
|
||||
When a new GKE cluster is provisioned (typically by `terraform-gcp-infra` or its successor), this repo gains a new `helm-overrides/<cluster>/` directory and the sister repo gains the `ApplicationSet` (or per-cluster Applications) that route to it. This procedure covers the slice owned by `devops-infra-helm-charts`.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The cluster exists in GCP — confirmed by the platform team.
|
||||
- [ ] The cluster is registered in the relevant Argo CD instance(s).
|
||||
- [ ] The cluster's `nodeSelector` / `tolerations` / `computeClass` topology is documented (node pool names + taints).
|
||||
- [ ] The cluster has a `SecretStore` / `ClusterSecretStore` for `external-secrets` — or onboarding `external-secrets` is part of this PR.
|
||||
- [ ] CMR open.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Confirm the naming convention
|
||||
|
||||
| Pattern | Use |
|
||||
|---------|-----|
|
||||
| `k8s-<bu>-prd-ase1[c]` | Standard BU prod cluster (GCP zone-a or zone-c) |
|
||||
| `k8s-shared-int-ase1` | Shared int |
|
||||
| `k8s-aurva-prd-ase1` | Aurva |
|
||||
| `k8s-supply-dev-ase1` | Dev/sandbox |
|
||||
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier |
|
||||
|
||||
The directory name is a **contract** — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.
|
||||
|
||||
### 2. Determine the cluster's scheduling profile
|
||||
|
||||
| Cluster type | Scheduling key |
|
||||
|--------------|----------------|
|
||||
| GKE Autopilot | `cloud.google.com/compute-class` |
|
||||
| Standard GKE | `dedicated:` |
|
||||
|
||||
Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of `ComputeClass` resources that need to land in `helm-overrides/<cluster>/<app>/computeclass/`.
|
||||
|
||||
### 3. Decide the minimal app set
|
||||
|
||||
Most clusters need at least:
|
||||
|
||||
- `external-secrets` — to materialise secrets from GCP Secret Manager
|
||||
- `kube-state-metrics` — for fleet observability
|
||||
- `victoria-metrics-agent` — to ship metrics to the central VM
|
||||
- `fluentd` (or equivalent log shipper)
|
||||
|
||||
Plus per-cluster role:
|
||||
|
||||
- BU clusters: `contour-internal-0`, `contour-internal-1`, sometimes `contour-external` — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) for the convention
|
||||
- DSGPU / Spark clusters: GPU operators, `keda` for queue autoscaling
|
||||
- Dataplane (`db-*`) clusters: minimal — `kube-state-metrics` + `victoria-metrics-agent` only
|
||||
|
||||
### 4. Create the directory and minimal apps
|
||||
|
||||
```bash
|
||||
mkdir -p helm-overrides/<cluster>
|
||||
cd helm-overrides/<cluster>
|
||||
```
|
||||
|
||||
For each app in the minimal set, follow [onboard-app-to-cluster.md](onboard-app-to-cluster.md) (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).
|
||||
|
||||
### 5. Update the per-cluster Contour matrix
|
||||
|
||||
If the cluster runs Contour, add a section to [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.
|
||||
|
||||
### 6. (If GKE Autopilot) Add the `ComputeClass` resources
|
||||
|
||||
Each Autopilot Contour / pool needs a corresponding `ComputeClass`. Drop them in:
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml
|
||||
```
|
||||
|
||||
Schema: [raw-manifest-sidecar-schema.md §ComputeClass](../schemas/raw-manifest-sidecar-schema.md). The `metadata.name` of each `ComputeClass` must equal the value the corresponding app's `nodeSelector: {cloud.google.com/compute-class: <name>}` references.
|
||||
|
||||
### 7. Open the base PR
|
||||
|
||||
```bash
|
||||
git checkout -b cluster/<cluster>-base
|
||||
git add helm-overrides/<cluster>/
|
||||
git add contour-nodeselector-tolerations-summary.md # if updated
|
||||
git commit
|
||||
git push origin cluster/<cluster>-base
|
||||
gh pr create --base main --title "Onboard <cluster> — base override set"
|
||||
```
|
||||
|
||||
PR description must include:
|
||||
|
||||
- Cluster owner approver tag.
|
||||
- Platform team approver tag.
|
||||
- CMR ticket.
|
||||
- Sister-repo PR for the `ApplicationSet` / per-cluster `Application` set.
|
||||
- The minimal app set being shipped, with per-app rationale.
|
||||
- The cluster's scheduling profile (key style + sample `nodeSelector`).
|
||||
|
||||
### 8. Pair the sister-repo PR
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`, draft the `ApplicationSet` (or per-cluster Application set) that walks `helm-overrides/<cluster>/`. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).
|
||||
|
||||
### 9. Stage subsequent app additions
|
||||
|
||||
After the base PR merges:
|
||||
|
||||
- Each additional app (`contour-internal-0`, `flagger`, etc.) is its own PR via [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
||||
- Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Naming the directory `<cluster-name>` slightly differently** from how Argo CD registered it. Silent bind failure.
|
||||
2. **Cloning another cluster's directory verbatim.** Per-cluster scheduling differs; `external-secrets` `SecretStore` references differ; image-tag pinning differs.
|
||||
3. **Onboarding 30 apps in the base PR.** Stage them — base + per-app.
|
||||
4. **Forgetting the Contour matrix update.** Future Contour edits then have no reference and silently mis-schedule.
|
||||
5. **Skipping the `ComputeClass` resources** on a GKE Autopilot cluster. `nodeSelector` references a class that doesn't exist; pods Pending.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the cluster onboarding turns out to be premature:
|
||||
|
||||
- Revert the values-side PR.
|
||||
- Revert the sister-repo `ApplicationSet` PR.
|
||||
- The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't `prune` what's not in its scope after the `Application` is removed); for a clean wipe, manually `kubectl delete ns` the affected namespaces.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — for each app after the base PR.
|
||||
- Schema: [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md) — for `ComputeClass` and `ExternalSecret` sidecars.
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,176 @@
|
||||
# Procedure — Update a chart's pinned dependency version
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK (charts in `helm-templates/` are consumed by every cluster that runs them).
|
||||
> **Blast radius:** every cluster that has an Argo `Application` referencing this chart will pick up the new version on next sync.
|
||||
> **Approval:** platform team. CMR mandatory for prod-fleet charts.
|
||||
|
||||
In this repo, charts in `helm-templates/<chart>/` are typically thin wrappers — `Chart.yaml` declares an upstream chart as a dependency, and `Chart.lock` pins the resolved subchart. "Bumping the chart version" means:
|
||||
|
||||
1. Update `dependencies[].version` in `Chart.yaml`.
|
||||
2. Run `helm dependency update` to refresh `Chart.lock` (and re-pull the subchart).
|
||||
3. Test render with representative cluster overrides.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Upgrading a wrapper chart's pinned subchart version (e.g. `argo-cd 7.7.23` → `7.8.0`).
|
||||
- Following a security advisory in an upstream chart.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- A blue-green migration to a new major version → use [blue-green-chart-migration.md](blue-green-chart-migration.md).
|
||||
- Editing chart templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
- Changing values without a chart bump → that's per-cluster `custom-values.yaml` work, not this.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The new upstream version exists and has a published changelog.
|
||||
- [ ] You have read the changelog for breaking template / values changes.
|
||||
- [ ] The bump is the **explicit headline** of the PR (not a side-effect of another change).
|
||||
- [ ] CMR open if this is a prod-fleet chart (Argo CD, Contour, VictoriaMetrics, ingress, cert-manager).
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Read the changelog
|
||||
|
||||
```bash
|
||||
# For most upstream charts:
|
||||
helm repo update
|
||||
helm search repo <repo>/<chart> --versions | head -20
|
||||
# Read the chart's CHANGELOG.md or release notes on GitHub.
|
||||
```
|
||||
|
||||
Look specifically for:
|
||||
|
||||
- **Removed values keys** — would silently no-op overrides.
|
||||
- **Renamed values keys** — same.
|
||||
- **CRD changes** — might require manual `kubectl apply` of new CRDs.
|
||||
- **Breaking template changes** (e.g. label selector immutability on Deployments).
|
||||
- **Required Kubernetes version bumps**.
|
||||
|
||||
If any of these apply, this is not a simple version bump — escalate to a blue-green migration ([blue-green-chart-migration.md](blue-green-chart-migration.md)).
|
||||
|
||||
### 2. Update `Chart.yaml`
|
||||
|
||||
```bash
|
||||
$EDITOR helm-templates/<chart>/Chart.yaml
|
||||
```
|
||||
|
||||
Find the `dependencies[]` entry and bump `version:`. Example:
|
||||
|
||||
```diff
|
||||
dependencies:
|
||||
- name: argo-cd
|
||||
- version: 7.7.23
|
||||
+ version: 7.8.0
|
||||
repository: https://argoproj.github.io/argo-helm
|
||||
```
|
||||
|
||||
### 3. Refresh `Chart.lock`
|
||||
|
||||
```bash
|
||||
helm dependency update helm-templates/<chart>
|
||||
```
|
||||
|
||||
This:
|
||||
|
||||
- Verifies the new version resolves.
|
||||
- Updates `Chart.lock` with the new digest.
|
||||
- Re-pulls the subchart `.tgz` into `helm-templates/<chart>/charts/`.
|
||||
|
||||
The lockfile must be committed alongside `Chart.yaml`. A bump without a refreshed lockfile is incomplete.
|
||||
|
||||
### 4. Render against a representative cluster's overrides
|
||||
|
||||
Pick a cluster that runs this chart with a non-trivial override set:
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' \
|
||||
| grep -v '^helm-overrides/db-' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
|
||||
```
|
||||
|
||||
If the render errors → the chart bump introduced a values incompatibility. **Stop.** Either:
|
||||
|
||||
- Find the new values shape and update affected `custom-values.yaml` files in this PR, OR
|
||||
- Defer the bump until those updates are scoped.
|
||||
|
||||
### 5. Spot-check `helm diff` against a live cluster (recommended)
|
||||
|
||||
```bash
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
|
||||
--kube-context=<cluster-context>
|
||||
```
|
||||
|
||||
What you want to see:
|
||||
|
||||
- Image tag bumps.
|
||||
- Label updates (often `helm.sh/chart`).
|
||||
- Possibly new CRDs or RBAC.
|
||||
|
||||
What's a red flag:
|
||||
|
||||
- `Deployment` selector changes (immutable; will fail to apply).
|
||||
- `StatefulSet` `volumeClaimTemplates` changes.
|
||||
- Resource removals you didn't expect.
|
||||
|
||||
### 6. Commit and open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b chart-bump/<chart>-<new-version>
|
||||
git add helm-templates/<chart>/Chart.yaml helm-templates/<chart>/Chart.lock
|
||||
git add helm-templates/<chart>/charts/ # if the .tgz was re-pulled
|
||||
git commit
|
||||
git push origin chart-bump/<chart>-<new-version>
|
||||
gh pr create --base main --title "chart bump: <chart> <old> → <new>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Old → new with a link to the upstream changelog.
|
||||
- The list of clusters that consume this chart (`grep -rl '<chart>' helm-overrides | head`).
|
||||
- The render(s) and diff(s) you ran.
|
||||
- Whether any breaking-change fallout is bundled (preferably not — if so, scope to a separate PR).
|
||||
|
||||
### 7. After merge — Argo CD picks up the new chart on next sync
|
||||
|
||||
For each cluster running this chart:
|
||||
|
||||
- The cluster's Argo CD detects the chart hash change.
|
||||
- The `Application` becomes `OutOfSync`.
|
||||
- A human clicks **Sync** per [ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||||
- Watch rollout per cluster — don't sync 20 clusters simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Bumping `Chart.yaml` without refreshing `Chart.lock`.** Argo CD uses the lockfile; the bump silently no-ops.
|
||||
2. **Bundling a chart bump with values changes.** Two separate PRs — one for the bump, one for any values that need to change because of the bump.
|
||||
3. **Bumping a major version (e.g. 7.x → 8.x) in place.** Use [blue-green-chart-migration.md](blue-green-chart-migration.md) — give yourself a sibling chart and migrate cluster-by-cluster.
|
||||
4. **Syncing every cluster simultaneously after merge.** Stagger; one cluster at a time, with a soak between.
|
||||
5. **Skipping the changelog read.** Surprises you'll regret.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
Open a follow-up PR that reverts `Chart.yaml` and `Chart.lock` to the previous versions. After merge, each cluster's Argo CD shows `OutOfSync` against the old chart; click **Sync** to roll back per cluster.
|
||||
|
||||
If the chart bump landed CRDs that are now incompatible with the old version, the rollback may require manual CRD cleanup — coordinate with the platform team.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — for major-version bumps.
|
||||
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md) — when the bump requires template edits.
|
||||
- Skill: [skills/infra/bump-chart-version.md](../../../skills/infra/bump-chart-version.md).
|
||||
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
Reference in New Issue
Block a user