added repo
This commit is contained in:
Your Name
@@ -0,0 +1,231 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Add or modify a Contour HTTPProxy route
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** one Contour release × one cluster (a misrouted host can break ingress for the whole BU).
|
||||
> **Approval:** consuming-app owner + cluster owner. Platform team if the change touches `contour-external*`.
|
||||
|
||||
This procedure covers HTTPProxy route changes that flow through one of the cluster's Contour releases. Most clusters run **multiple Contour instances** — read the [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) (repo root) authoritative scheduling matrix before editing **any** Contour values.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Contour topology
|
||||
|
||||
| Release name | Plane | Typical purpose |
|
||||
|--------------|-------|-----------------|
|
||||
| `contour-external` | North-south | Public / external traffic (terminates at GCP external LB) |
|
||||
| `contour-external-1` | North-south | Second external Contour (blue-green or capacity split) |
|
||||
| `contour-internal-0` | East-west | Internal traffic, plane 0 |
|
||||
| `contour-internal-1` | East-west | Internal traffic, plane 1 |
|
||||
| `contour-internal-intra-0` | Intra-cluster | Cluster-local east-west, plane 0 |
|
||||
| `contour-internal-intra-1` | Intra-cluster | Cluster-local east-west, plane 1 |
|
||||
|
||||
Each is a separate Helm release, on a separate node pool, with its own `nodeSelector` / `tolerations` / `computeClass`. Picking the wrong release is the most common authoring mistake.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding a new HTTPProxy / route for a service.
|
||||
- Changing TLS, retry, timeout, or rate-limit policy on an existing route.
|
||||
- Re-pointing an HTTPProxy at a different upstream Service.
|
||||
- Adding a new host to an existing route's `virtualhost.fqdn`.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- Changing Contour itself (sizing, scheduling, image) — that's [modify-observability-config.md](modify-observability-config.md)-style infra editing on the Contour release.
|
||||
- Bumping Contour's chart version — use [update-chart-version.md](update-chart-version.md) and consult the versioned sibling (`contour-v1.33.3`).
|
||||
- Curling production hostnames to test — [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md) forbids it. Test from inside the cluster with a `curl` Pod.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target Contour release | `contour-internal-0` |
|
||||
| HTTPProxy host | `api-foo.internal.meeshogcp.in` |
|
||||
| Upstream Service | `foo-svc.foo-ns:8080` |
|
||||
| TLS source | `cert-manager` / `external-secret` / `none` |
|
||||
| Path prefixes | `/v1`, `/health` |
|
||||
| Approval ticket | CMR-… (if BU policy) |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You know which Contour release is the right one for this host (north-south = `external*`; east-west between BUs = `internal-{0,1}`; cluster-local = `internal-intra-{0,1}`). When in doubt, sample existing HTTPProxies in the same namespace.
|
||||
- [ ] The upstream Service exists or will exist by the time of Sync.
|
||||
- [ ] The DNS name follows the cluster's `external-dns` pattern.
|
||||
- [ ] The TLS source (Secret, Issuer, etc.) exists on the cluster.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Identify the right Contour release
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/ | grep '^contour'
|
||||
```
|
||||
|
||||
Pick the release whose plane matches the new route's traffic class. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). If unsure, look at existing HTTPProxies on the cluster:
|
||||
|
||||
```bash
|
||||
# Read existing routes already shipped via this repo
|
||||
yq e '.. | select(has("httpproxies"))' \
|
||||
helm-overrides/<cluster>/contour-internal-0/custom-values.yaml
|
||||
```
|
||||
|
||||
### 2. Decide where the HTTPProxy lives
|
||||
|
||||
Two patterns exist:
|
||||
|
||||
| Pattern | When | Where to author |
|
||||
|---------|------|-----------------|
|
||||
| **Inline in Contour values** | Routes shared by the cluster's infra layer (e.g. Grafana, Argo CD). | `helm-overrides/<cluster>/<contour-release>/custom-values.yaml` under `httpproxies:` (if the chart supports it) or as a sidecar manifest in the same dir. |
|
||||
| **In the consuming app's repo** | Routes for a specific service. | The service's own deployment artifacts. **Out of scope for this repo.** |
|
||||
|
||||
If the route is service-owned, **redirect** to the consuming team's repo and stop. This procedure only covers infra-layer HTTPProxies.
|
||||
|
||||
### 3. Author the HTTPProxy
|
||||
|
||||
Skeleton (sidecar manifest pattern):
|
||||
|
||||
```yaml
|
||||
apiVersion: projectcontour.io/v1
|
||||
kind: HTTPProxy
|
||||
metadata:
|
||||
name: <name>
|
||||
namespace: <ns>
|
||||
spec:
|
||||
virtualhost:
|
||||
fqdn: <fqdn>
|
||||
tls:
|
||||
secretName: <tls-secret> # cert-manager-managed Secret
|
||||
routes:
|
||||
- conditions:
|
||||
- prefix: /
|
||||
services:
|
||||
- name: <upstream-service>
|
||||
port: <port>
|
||||
timeoutPolicy:
|
||||
response: 30s
|
||||
retryPolicy:
|
||||
count: 2
|
||||
retryOn: 5xx
|
||||
```
|
||||
|
||||
Drop into `helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml` (sidecar manifest) — see [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
### 4. Lint the HTTPProxy
|
||||
|
||||
```bash
|
||||
# Schema check (kubectl --dry-run against the cluster's CRD)
|
||||
kubectl --context=<ctx> --dry-run=server -f helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml apply
|
||||
|
||||
# Or local: validate against the projectcontour CRD schema
|
||||
kubeconform -schema-location default -schema-location \
|
||||
'https://raw.githubusercontent.com/projectcontour/contour/main/examples/contour/01-crds.yaml' \
|
||||
helm-overrides/<cluster>/<contour-release>/httpproxies/<name>.yaml
|
||||
```
|
||||
|
||||
### 5. Render the chart
|
||||
|
||||
```bash
|
||||
helm template <contour-release> helm-templates/<contour-chart> \
|
||||
-f helm-overrides/<cluster>/<contour-release>/custom-values.yaml > /tmp/contour.yaml
|
||||
```
|
||||
|
||||
The render must succeed; the HTTPProxy sidecar is applied alongside, not through Helm — but rendering catches values-side errors that would block sync of the same Argo Application.
|
||||
|
||||
### 6. Test from inside the cluster (post-merge, not pre-merge)
|
||||
|
||||
**Do not** curl `<fqdn>.meeshogcp.in` from your laptop / a build agent. Run a curl Pod on-cluster:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> run -it --rm curl-test \
|
||||
--image=curlimages/curl --restart=Never -- \
|
||||
curl -v -H "Host: <fqdn>" http://<contour-svc>.projectcontour.svc.cluster.local
|
||||
```
|
||||
|
||||
### 7. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b contour/<cluster>-<route-name>
|
||||
git add helm-overrides/<cluster>/<contour-release>/
|
||||
git commit -m "contour(<cluster>/<contour-release>): add route <name>"
|
||||
git push origin contour/<cluster>-<route-name>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
Adds (or modifies) HTTPProxy `<name>` on `<cluster>` via `<contour-release>` for FQDN `<fqdn>`.
|
||||
|
||||
## Why
|
||||
<1-2 sentences>
|
||||
|
||||
## Topology
|
||||
- Cluster: `<cluster>`
|
||||
- Contour release: `<contour-release>` (plane: external / internal / intra)
|
||||
- FQDN: `<fqdn>`
|
||||
- Upstream Service: `<svc>:<port>` in namespace `<ns>`
|
||||
- TLS: cert-manager Secret `<tls-secret>`
|
||||
|
||||
## Validation
|
||||
- [ ] HTTPProxy CRD schema validation passed (kubeconform / kubectl --dry-run)
|
||||
- [ ] `helm template` rendered cleanly
|
||||
- [ ] On-cluster curl from a curl Pod returns expected status
|
||||
- [ ] DNS / external-dns plumbed (existing wildcard or new external-dns Service)
|
||||
|
||||
## Approvers
|
||||
- App owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
- Platform (if `contour-external*`): <handle>
|
||||
```
|
||||
|
||||
### 8. After merge — Sync
|
||||
|
||||
Open the cluster's Argo CD UI, find the Contour release's Application, **Sync**. Verify:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> -n projectcontour get httpproxy <name>
|
||||
kubectl --context=<ctx> -n projectcontour describe httpproxy <name> | grep -A5 'Status:'
|
||||
```
|
||||
|
||||
A `Valid: true` status means Contour accepted the route. `Valid: false` with a reason → fix the values and re-PR.
|
||||
|
||||
If the HTTPProxy is `Valid` but traffic still 5xx → [../runbooks/ingress-down.md](../runbooks/ingress-down.md) §4.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Wrong Contour release.** A route mounted on `contour-internal-intra-0` is unreachable from outside the cluster. Cross-check the matrix.
|
||||
2. **Curling the production FQDN** from a developer machine to "test." Forbidden — see [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md).
|
||||
3. **Hand-edited values for a single host across all Contour releases** — pick one release, justify it.
|
||||
4. **`tls.passthrough` for plain HTTP services** — silent: TLS terminates upstream, doesn't.
|
||||
5. **Wildcard hosts that overlap an existing HTTPProxy** — Contour status will mark one of them invalid; check before merging.
|
||||
6. **Editing `helm-templates/contour*/`** to "tweak the chart." Layer 1 forbids casual chart edits — the chart is vanilla upstream.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the values PR.
|
||||
- Sync the Contour Application — Argo will prune the HTTPProxy or restore the prior values.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference (authoritative scheduling): [../../../contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [../runbooks/ingress-down.md](../runbooks/ingress-down.md).
|
||||
- Schema: [../schemas/raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- ADR: [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,206 @@
|
||||
# Procedure — Blue-green chart migration (versioned siblings)
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** the chart upgrade ships in a new sibling directory; no cluster cuts over until its `Application` is repointed. Each cluster's cutover is its own decision.
|
||||
> **Approval:** platform team — multi-reviewer.
|
||||
|
||||
This procedure handles the case where you can't safely bump a chart's pinned `dependencies[].version` in place — typically because of breaking template changes, immutable selector mismatches, or major-version semantics. The pattern is to keep both versions live as **sibling chart directories** until every consuming cluster has migrated.
|
||||
|
||||
The repo already shows this pattern at work:
|
||||
|
||||
| Original | Migration target |
|
||||
|----------|------------------|
|
||||
| `argo-cd` | `argo-cd-green` |
|
||||
| `contour` | `contour-v1.33.3` |
|
||||
| `keda` | `keda-2.17.1` |
|
||||
| `opentelemetry-collector` | `opentelemetry-collector-latest` |
|
||||
| `victoria-metrics-cluster` | `victoria-metrics-cluster-latest` |
|
||||
| `victoria-metrics-agent` | `victoria-metrics-agent-latest` |
|
||||
| `sonarqube` | `sonarqube-old` *(reverse — `sonarqube` is the new one; `-old` retained for rollback)* |
|
||||
|
||||
See [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md) for the rationale.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- A major-version bump with breaking template changes.
|
||||
- A bump with an immutable-field change (Deployment `spec.selector`, StatefulSet `volumeClaimTemplates`).
|
||||
- A migration that needs cluster-by-cluster cutover with rollback windows.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- A patch / minor bump that's a clean drop-in → use [update-chart-version.md](update-chart-version.md).
|
||||
- Forking templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] Platform team agrees a blue-green migration is required (not just a bump).
|
||||
- [ ] You've identified the variant naming convention (`-green`, `-vX.Y.Z`, `-latest`, etc.).
|
||||
- [ ] You've identified every consuming cluster: `grep -rl '<chart>' helm-overrides`.
|
||||
- [ ] CMR open.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Create the sibling chart directory
|
||||
|
||||
```bash
|
||||
cp -R helm-templates/<chart> helm-templates/<chart>-<variant>
|
||||
cd helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
Update `Chart.yaml`:
|
||||
|
||||
```diff
|
||||
apiVersion: v2
|
||||
-name: <chart>
|
||||
+name: <chart>-<variant>
|
||||
…
|
||||
dependencies:
|
||||
- name: <sub>
|
||||
- version: <old-version>
|
||||
+ version: <new-version>
|
||||
repository: <upstream-repo>
|
||||
```
|
||||
|
||||
```bash
|
||||
helm dependency update
|
||||
```
|
||||
|
||||
### 2. Render against a representative cluster's overrides — old variant
|
||||
|
||||
The point of blue-green is *no surprises*. Render the **old** chart with each consuming cluster's override and capture the output.
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart> helm-templates/<chart> -f "$f" > /tmp/old-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 3. Render against the same overrides — new sibling variant
|
||||
|
||||
```bash
|
||||
for f in $(find helm-overrides -maxdepth 2 -name custom-values.yaml -path "*/<chart>/*"); do
|
||||
cluster=$(echo "$f" | awk -F/ '{print $2}')
|
||||
helm template <chart>-<variant> helm-templates/<chart>-<variant> -f "$f" > /tmp/new-${cluster}.yaml
|
||||
done
|
||||
```
|
||||
|
||||
### 4. Diff old → new per cluster
|
||||
|
||||
```bash
|
||||
for cluster in $(ls /tmp/old-*.yaml | sed 's:/tmp/old-::; s:.yaml::'); do
|
||||
echo "=== $cluster ==="
|
||||
diff /tmp/old-${cluster}.yaml /tmp/new-${cluster}.yaml | head -30
|
||||
done
|
||||
```
|
||||
|
||||
For each cluster, decide:
|
||||
|
||||
- Is the diff what you expected?
|
||||
- Will any value need updating to make the new chart render correctly? (If yes → that's a separate per-cluster PR after the sibling lands.)
|
||||
- Is the cutover safe to do without the workload owner present? (If no → schedule.)
|
||||
|
||||
### 5. Open PR-1: introduce the sibling chart
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-to-<variant>-introduce
|
||||
git add helm-templates/<chart>-<variant>/
|
||||
git commit
|
||||
git push origin migrate/<chart>-to-<variant>-introduce
|
||||
gh pr create --base main --title "migrate: introduce <chart>-<variant> sibling"
|
||||
```
|
||||
|
||||
After merge, the new chart exists in the repo but **no cluster uses it yet** — the existing Argo `Application`s still point at `helm-templates/<chart>`.
|
||||
|
||||
### 6. Per-cluster cutover (one PR pair per cluster)
|
||||
|
||||
For each consuming cluster:
|
||||
|
||||
a. **Update the cluster's `custom-values.yaml`** if the new chart needs different values. Open as a values-side PR (this repo).
|
||||
|
||||
b. **Update the sister-repo `Application`** to repoint:
|
||||
|
||||
```diff
|
||||
spec:
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
- path: helm-templates/<chart>
|
||||
+ path: helm-templates/<chart>-<variant>
|
||||
```
|
||||
|
||||
c. **After both merge**, click Sync in the cluster's Argo CD UI.
|
||||
|
||||
d. **Soak** — leave it for the agreed soak period (often 24–72 h) before moving to the next cluster.
|
||||
|
||||
### 7. Open PR-N: retire the old sibling
|
||||
|
||||
After every cluster has cut over and soaked:
|
||||
|
||||
```bash
|
||||
git checkout -b migrate/<chart>-retire-old
|
||||
git rm -r helm-templates/<chart>
|
||||
# OR rename: git mv helm-templates/<chart> helm-templates/<chart>-old
|
||||
git commit
|
||||
git push origin migrate/<chart>-retire-old
|
||||
gh pr create --base main --title "migrate: retire old <chart> sibling"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Confirmation every cluster has cut over (`grep -rl 'helm-templates/<chart>$' /path/to/devops-infra-argo-config` returns nothing).
|
||||
- Confirmation soak period elapsed.
|
||||
- Decision: delete vs rename to `-old` (kept for rollback).
|
||||
|
||||
---
|
||||
|
||||
## Why two PRs at the start, then per-cluster pairs, then a final retirement
|
||||
|
||||
| PR | Effect |
|
||||
|----|--------|
|
||||
| **PR-1: introduce sibling** | Adds the new chart. No cluster cuts over. Worst case: render errors caught before any cluster sees them. |
|
||||
| **PR-2..N-1: per-cluster cutover (paired with sister repo)** | One cluster moves. Worst case: that cluster's release breaks; revert the sister-repo PR and Sync to the old chart. |
|
||||
| **PR-N: retire old sibling** | Removes the old chart. Worst case: a cluster you missed becomes broken — but you grep'd, so this should be impossible. |
|
||||
|
||||
Bundling any of these violates the "rollback one cluster at a time" property that's the whole point of the blue-green pattern.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Cutting over multiple clusters in one PR.** Bundle = no per-cluster rollback.
|
||||
2. **Deleting the old sibling before every cluster has cut over.** Cluster N+1 has its `Application` pointing at a path that no longer exists; sync fails immediately.
|
||||
3. **Cutting over without rendering first.** Surprises after merge.
|
||||
4. **Skipping the soak period.** "It looked fine in the first 5 minutes" is not soak.
|
||||
5. **Renaming the *new* sibling to drop the suffix** (e.g. `argo-cd-green` → `argo-cd`) before the old chart is retired. The path collision will break Argo CD's caching.
|
||||
|
||||
---
|
||||
|
||||
## Rollback (per cluster)
|
||||
|
||||
If a cluster's cutover fails:
|
||||
|
||||
1. Revert the sister-repo PR for that cluster.
|
||||
2. Click Sync in the cluster's Argo CD — the `Application` re-renders against `<chart>` (the old sibling). The rollback is one cluster only.
|
||||
3. Investigate; iterate.
|
||||
|
||||
If the new sibling has a fundamental problem affecting every cluster:
|
||||
|
||||
1. Open PR-X: revert PR-1 (delete the sibling).
|
||||
2. Any cluster that was already cutover gets reverted via its own per-cluster sister-repo revert.
|
||||
3. Schedule a postmortem before re-attempting.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md).
|
||||
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
- ADR: [ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- [SANCTITY_RULES R8](../../global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,119 @@
|
||||
# Procedure — Deboard a retired app from a cluster
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** the Helm release is removed from the cluster. If still in use, that's an outage.
|
||||
> **Approval:** app owner + cluster owner. CMR mandatory.
|
||||
|
||||
This procedure removes an app's `helm-overrides/<cluster>/<app>/` directory. It does **not** delete the workload directly — but the matching sister-repo `Application` PR (which must be paired) tells Argo CD to stop tracking the release.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions (block deboarding if any are FALSE)
|
||||
|
||||
- [ ] App is genuinely retired on this cluster — confirmed by app owner.
|
||||
- [ ] No traffic / scrape job / dependency is still hitting it.
|
||||
- [ ] All upstream consumers (alerts, dashboards, log collectors) have been migrated or notified.
|
||||
- [ ] You have inventoried every cluster that runs the app and decided whether this is a single-cluster or fleet-wide deboard.
|
||||
- [ ] CMR approved.
|
||||
|
||||
```bash
|
||||
# Where does this app exist today?
|
||||
find helm-overrides -maxdepth 2 -type d -name '<app>'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Decide the scope
|
||||
|
||||
| Scope | Then |
|
||||
|-------|------|
|
||||
| Single cluster | Remove `helm-overrides/<cluster>/<app>/` only. Leave other clusters running. |
|
||||
| Fleet-wide | Multiple PRs — one cluster per PR. Don't bundle. |
|
||||
| App is being replaced (e.g. `victoria-metrics-cluster` → `victoria-metrics-cluster-latest`) | This is a **migration**, not a deboard — use [blue-green-chart-migration.md](blue-green-chart-migration.md). |
|
||||
|
||||
### 2. Open the sister-repo `Application` removal PR FIRST
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`:
|
||||
|
||||
- Remove (or scope out of) the `Application` / `ApplicationSet` entry that targets this cluster × app.
|
||||
- Merge.
|
||||
- The cluster's Argo CD will mark the `Application` for removal on next reconcile.
|
||||
|
||||
This step **must precede** the values-side removal. If you delete the values-side first, the `Application` will fail to render and the workload may go into an `Errored` state on the cluster.
|
||||
|
||||
### 3. Manually clean up the workload (if `automated.prune` was not set)
|
||||
|
||||
For most infra apps, the `Application` does not have `automated.prune: true` — so removing the `Application` does not delete the workload. Manually:
|
||||
|
||||
```bash
|
||||
kubectl --context=<cluster> delete <kind>/<name> -n <ns>
|
||||
# OR, if the whole namespace is dedicated to this release:
|
||||
kubectl --context=<cluster> delete namespace <ns>
|
||||
```
|
||||
|
||||
Coordinate with the app owner — wholesale namespace deletion is irreversible.
|
||||
|
||||
### 4. Open the values-side removal PR
|
||||
|
||||
```bash
|
||||
git checkout -b deboard/<app>-from-<cluster>
|
||||
git rm -r helm-overrides/<cluster>/<app>/
|
||||
git commit
|
||||
git push origin deboard/<app>-from-<cluster>
|
||||
gh pr create --base main --title "deboard: <app> from <cluster>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Confirmation pre-conditions are TRUE.
|
||||
- Sister-repo PR (already merged).
|
||||
- CMR ticket reference.
|
||||
- Confirmation the workload was manually cleaned up (or scheduled).
|
||||
|
||||
### 5. (Optional) If this was the last cluster running the app
|
||||
|
||||
If `find helm-overrides -maxdepth 2 -type d -name '<app>'` now returns nothing, consider:
|
||||
|
||||
- **Should `helm-templates/<chart>/` also be removed?** Probably not — keeping the chart cached lets a future re-onboarding be cheap. But if it's a stale chart with security advisories you don't want to maintain, schedule a separate retirement PR for it (with platform-team review).
|
||||
|
||||
---
|
||||
|
||||
## "What if the app might come back?"
|
||||
|
||||
If retirement is provisional:
|
||||
|
||||
- **Do not** scale the workload to zero replicas via values "to deactivate it." Either it's running or it's not. Half-states are operational debt.
|
||||
- **Do** keep the values directory in place but document a 30-day decision deadline in a TODO comment. If the deadline passes without a re-decision, deboard for real.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Deleting the values-side first** before the sister-repo `Application` is removed. The `Application` errors on next reconcile.
|
||||
2. **Bulk-deleting multiple clusters in one PR.** One cluster per PR. Rollback granularity.
|
||||
3. **Forgetting to clean up the workload manually.** The Helm release lingers on the cluster after the `Application` is removed (because most infra apps lack `automated.prune`).
|
||||
4. **Forgetting `external-secrets` cleanup.** If the app referenced an `ExternalSecret`, the corresponding `ExternalSecret` resource (in the cluster's `external-secrets/` directory) often outlives the app. Either repurpose it or delete it in the same PR.
|
||||
5. **Forgetting alert / dashboard cleanup.** Alerts firing on a workload that no longer exists cause noise; dashboards showing nothing cause confusion.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If you deboarded by mistake:
|
||||
|
||||
1. Revert the values-side PR (`git revert <merge-sha>`).
|
||||
2. Revert the sister-repo PR.
|
||||
3. Click Sync in the cluster's Argo CD UI.
|
||||
4. The `Application` is recreated and renders the chart.
|
||||
5. **However**, if you also manually deleted the workload (step 3), the Sync recreates it from scratch — make sure the underlying chart and values are still intact and any data PVCs were retained (see `manifests/storageclass/pd-standard-retain-dr.yaml`).
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — the inverse.
|
||||
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — if "deboard" is actually a migration.
|
||||
- ADR: [ADR-A5-manual-sync-default-for-infra.md](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md) — why the workload doesn't auto-delete.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Procedure — Intentionally fork an upstream chart
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** every cluster that consumes this chart will render against the forked templates on next sync.
|
||||
> **Approval:** platform team — multi-reviewer.
|
||||
|
||||
Most charts in `helm-templates/` are **vanilla upstream**, pulled via `helm pull`. The `Chart.yaml` is a thin wrapper declaring the upstream chart as a dependency; the actual templates come from the subchart in `charts/`. Editing a `templates/*.yaml` *under the wrapper* silently forks the chart, and the fork is clobbered the next time someone runs `helm dependency update`.
|
||||
|
||||
This procedure is the supported path when a fork is **intentional**.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Upstream chart lacks a feature you need (e.g. a values key that doesn't exist).
|
||||
- Upstream behaviour conflicts with Meesho's environment (e.g. probe path doesn't work behind Contour).
|
||||
- Upstream chart has a bug awaiting an upstream fix.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- Adding a values knob — file a PR upstream, or override behaviour through existing knobs.
|
||||
- Pinning to an old version — use [update-chart-version.md](update-chart-version.md) (or just don't bump).
|
||||
- Style preferences — leave the upstream chart alone.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You've confirmed the desired behaviour cannot be achieved via the chart's existing values.
|
||||
- [ ] You've checked whether an upstream PR / issue already covers this.
|
||||
- [ ] Platform team agrees the fork is justified.
|
||||
- [ ] The fork's rationale will be documented in the chart's `README.md`.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Vendor the chart's templates
|
||||
|
||||
If the chart is currently a thin wrapper (templates come from a subchart in `charts/<sub>`), you must first promote the subchart's templates into the wrapper.
|
||||
|
||||
```bash
|
||||
cd helm-templates/<chart>
|
||||
ls charts/ # find the subchart .tgz
|
||||
helm dependency update # ensure it's resolved
|
||||
|
||||
# Extract templates from the subchart .tgz
|
||||
tar -xzf charts/<sub>-<ver>.tgz -C /tmp/
|
||||
cp -R /tmp/<sub>/templates ./templates
|
||||
cp -R /tmp/<sub>/values.yaml ./values.yaml.upstream
|
||||
```
|
||||
|
||||
Now the wrapper has its own `templates/` — Helm will use those instead of the subchart's.
|
||||
|
||||
### 2. Edit `Chart.yaml` to reflect the fork
|
||||
|
||||
Remove the dependency (since the templates are now local), and bump `version:` (the chart's own version, not the subchart's):
|
||||
|
||||
```diff
|
||||
apiVersion: v2
|
||||
name: <chart>
|
||||
-version: 0.1.0
|
||||
+version: 0.1.0+fork.1
|
||||
description: <description> — Meesho-forked from upstream <chart> <upstream-version>
|
||||
-dependencies:
|
||||
- - name: <sub>
|
||||
- version: <upstream-version>
|
||||
- repository: <upstream-repo>
|
||||
```
|
||||
|
||||
### 3. Make the fork edits
|
||||
|
||||
Edit `templates/*.yaml` or `values.yaml` to apply the fix. Keep the diff minimal — every line away from upstream is technical debt.
|
||||
|
||||
### 4. Document the fork
|
||||
|
||||
Edit `helm-templates/<chart>/README.md` (create if missing). Use this template:
|
||||
|
||||
```markdown
|
||||
# <chart>
|
||||
|
||||
**Forked from upstream <chart> <upstream-version>** at <date>.
|
||||
|
||||
## Why
|
||||
|
||||
<one-paragraph rationale — what behaviour the fork changes and why we couldn't achieve
|
||||
it through values alone>
|
||||
|
||||
## What's changed
|
||||
|
||||
- `templates/<file>.yaml` — <one-line diff summary>
|
||||
- `values.yaml` — <one-line diff summary>
|
||||
|
||||
## Upstream tracking
|
||||
|
||||
- Upstream PR: <link, if you've sent one>
|
||||
- Upstream issue: <link>
|
||||
- When upstream merges: revert this fork via [unfork procedure].
|
||||
```
|
||||
|
||||
### 5. Render and diff
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
|
||||
|
||||
# Spot-check helm diff against a live cluster
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml --kube-context=<context>
|
||||
```
|
||||
|
||||
### 6. Commit and open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b fork/<chart>-<purpose>
|
||||
git add helm-templates/<chart>/
|
||||
git commit
|
||||
git push origin fork/<chart>-<purpose>
|
||||
gh pr create --base main --title "fork: <chart> — <one-line purpose>"
|
||||
```
|
||||
|
||||
PR description must include:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Why the fork is necessary (link to upstream issue/PR if any).
|
||||
- What templates / values are changed and why.
|
||||
- Platform team approver(s).
|
||||
- Plan for unforking when upstream lands the fix.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Editing `templates/` without first vendoring** (templates from a subchart). Your edits live in `helm-templates/<chart>/templates/` but Helm renders from `charts/<sub>/templates/` — the edits do nothing, then get clobbered.
|
||||
2. **Forking a chart and not documenting why.** Six months later nobody remembers; the fork looks like accidental drift.
|
||||
3. **Forking instead of overriding via values.** Always check the chart's existing values surface first.
|
||||
4. **Forking and then bumping the upstream version.** A bump runs `helm dependency update`, which can clobber the fork. The fork must be re-applied or the bump must explicitly re-vendor.
|
||||
5. **Sweeping cleanup of upstream code** alongside the fork edit. The diff should be exactly the change you intended; everything else is upstream.
|
||||
|
||||
---
|
||||
|
||||
## Rollback / unforking
|
||||
|
||||
When upstream lands the fix and you want to return to vanilla:
|
||||
|
||||
1. Restore `Chart.yaml` to declare the upstream as a dependency (with the new upstream version that includes the fix).
|
||||
2. Delete the local `templates/` and `values.yaml` (or rename them as `.upstream` for reference).
|
||||
3. Run `helm dependency update`.
|
||||
4. Update `helm-templates/<chart>/README.md` to remove the "forked" status.
|
||||
5. Render against a sibling override and confirm the resulting manifests match what the fork was producing.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md).
|
||||
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
- [SANCTITY_RULES R7](../../global/SANCTITY_RULES.md).
|
||||
@@ -0,0 +1,227 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify Prometheus / VictoriaMetrics alert rules
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** PagerDuty / on-call paging across the cluster's tenants. A wrong threshold pages everyone; a missing rule masks an outage.
|
||||
> **Approval:** observability owner + on-call lead for the affected severity.
|
||||
|
||||
This procedure is the narrow alert-rule slice of [modify-observability-config.md](modify-observability-config.md). Use this when the entire intent of the PR is alert-rule editing — adding, removing, retuning thresholds, changing `for:` windows, or rewriting `expr:` PromQL.
|
||||
|
||||
---
|
||||
|
||||
## Where alert rules live
|
||||
|
||||
| File | Engine | Notes |
|
||||
|------|--------|-------|
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert | Most clusters use vmalert against VM cluster as the primary alerting engine. Rules typically under `config.rules` or rendered as a `VMRule` CRD. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmalert (bundled) / VM ruler | Some clusters carry rules in the cluster chart's `vmalert.config` block. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` | Prometheus / `kube-prometheus-stack` | Rules under `additionalPrometheusRulesMap` or `serverFiles.alerting_rules.yml`. |
|
||||
| `helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml` | Prometheus | Same as above; older clusters use this chart name. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` | Mimir ruler | Rules go through the ruler API; YAML in `runtimeConfig` or via a sidecar `ConfigMap`. |
|
||||
|
||||
Confirm which cluster runs which engine before editing — sample existing rules in the override.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing one or more alert rules in the cluster's primary alerting engine.
|
||||
- Tuning `expr:` PromQL or `for:` windows on existing rules.
|
||||
- Re-routing alerts (changing `severity`, `team`, or other routing labels).
|
||||
- Adding/removing recording rules for downstream alert efficiency.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- Touching scrape configs, retention, datasources, or sizing in the same PR — split it. See [modify-observability-config.md](modify-observability-config.md).
|
||||
- Bumping the alerting chart's version — see [update-chart-version.md](update-chart-version.md).
|
||||
- Writing app-specific alert rules that belong in the consuming service's own values — those go in the service's repo.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `vmalert` / `prometheus` / `kube-prometheus-stack` / `mimir` |
|
||||
| Rule name(s) added/changed | `HighErrorRate`, `KafkaConsumerLag` |
|
||||
| `expr:` PromQL | `sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 0.05` |
|
||||
| `for:` window | `5m` |
|
||||
| Severity / routing labels | `severity: critical`, `team: supply-platform` |
|
||||
| Approval ticket | CMR-… |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You have `promtool` and/or `vmalert` available locally.
|
||||
- [ ] You know the cluster's Alertmanager / vmalert notifier routing (which `severity` / `team` label routes where).
|
||||
- [ ] You have run the new PromQL expression against the cluster's VM/Prometheus to confirm it returns sensible values *before* writing the rule — open Grafana Explore.
|
||||
- [ ] You have on-call sign-off if this rule pages on-call.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Locate the rules block
|
||||
|
||||
```bash
|
||||
yq e '.config.rules // .additionalPrometheusRulesMap // .serverFiles' \
|
||||
helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
Identify the YAML path the engine expects:
|
||||
|
||||
- **vmalert**: `config.groups[].rules[]`.
|
||||
- **kube-prometheus-stack**: `additionalPrometheusRulesMap.<group-name>.groups[].rules[]`.
|
||||
- **prometheus**: `serverFiles.alerting_rules.yml.groups[].rules[]`.
|
||||
- **mimir ruler**: per-tenant config — confirm the cluster's tenancy setup.
|
||||
|
||||
### 2. Author the rule
|
||||
|
||||
```yaml
|
||||
- alert: <RuleName> # PascalCase, no spaces; surfaces in pages and dashboards
|
||||
expr: |
|
||||
<PromQL expression>
|
||||
for: <duration> # e.g. 5m, 10m
|
||||
labels:
|
||||
severity: <warning|critical|info>
|
||||
team: <owning-team> # routes to the right PagerDuty service
|
||||
annotations:
|
||||
summary: "<one-line>"
|
||||
description: "<longer text, can reference $labels.* and $value>"
|
||||
runbook_url: <link to runbook>
|
||||
```
|
||||
|
||||
#### Threshold discipline
|
||||
|
||||
- Express thresholds as `rate(...) > 0.05` rather than `> 5%`. Ensure units in the expression match the units in `summary` / `description`.
|
||||
- Avoid `for:` windows shorter than the scrape interval × 2. Sub-`1m` windows are flap factories.
|
||||
- Avoid alerting on absolute counts (`sum(http_requests_total) > 1000`); prefer rates / ratios.
|
||||
- For SLO-style alerts, use multi-window multi-burn-rate (Google SRE workbook). Single-window threshold alerts are a known anti-pattern.
|
||||
|
||||
#### Routing implications
|
||||
|
||||
Changing `severity` from `warning` to `critical` (or vice versa) **silently re-routes the page** — the rule may now wake on-call where it previously emailed (or vice versa). Same for `team:` — a typo here drops alerts into the wrong PagerDuty service. Cross-check the cluster's Alertmanager / vmalert routing config:
|
||||
|
||||
```bash
|
||||
yq e '.config.route // .alertmanagerConfig' \
|
||||
helm-overrides/<cluster>/<alertmanager-chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
### 3. Render the chart
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
### 4. Validate PromQL
|
||||
|
||||
For Prometheus / `kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
For Mimir ruler — `mimirtool rules check` against the rendered ruler config.
|
||||
|
||||
If validation fails: fix the PromQL. **Do not** merge a rule that fails parse.
|
||||
|
||||
### 5. Sanity-check expressions in Grafana Explore
|
||||
|
||||
In a non-production manner: open Grafana on the cluster, paste the `expr:` into Explore, run over the last 6h. Confirm:
|
||||
|
||||
- The series returned makes sense for the rule.
|
||||
- The threshold isn't always-firing or never-firing in normal operation.
|
||||
- Empty result is allowed (`absent_over_time(...)` style).
|
||||
|
||||
### 6. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b alerts/<cluster>-<rule-name>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "alerts(<cluster>/<chart>): <add|tune|remove> <RuleName>"
|
||||
git push origin alerts/<cluster>-<rule-name>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<add|tune|remove> alert rule(s) on `<cluster>` × `<chart>`.
|
||||
|
||||
## Rules touched
|
||||
| Rule | Change | New expr | New `for:` | Severity | Team |
|
||||
|------|--------|----------|-----------|----------|------|
|
||||
| `<RuleName>` | added | `<expr>` | `5m` | `critical` | `supply-platform` |
|
||||
|
||||
## Validation
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed
|
||||
- [ ] PromQL sanity-checked in Grafana Explore over last 6h
|
||||
- [ ] Threshold appropriate (no flap risk, no always-firing)
|
||||
- [ ] Routing labels (`severity`, `team`) verified against Alertmanager/vmalert routes
|
||||
- [ ] Runbook URL added (`annotations.runbook_url`)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- On-call lead (severity=<x>): <handle>
|
||||
```
|
||||
|
||||
### 7. After merge — Sync and observe
|
||||
|
||||
Sync the Argo Application. Watch:
|
||||
|
||||
```bash
|
||||
# vmalert
|
||||
kubectl --context=<ctx> -n monitoring logs deploy/vmalert | grep -i "<RuleName>\|error"
|
||||
|
||||
# kube-prometheus-stack
|
||||
kubectl --context=<ctx> -n monitoring logs prometheus-k8s-0 | grep -i "<RuleName>\|error"
|
||||
```
|
||||
|
||||
Open the alerting engine's UI (vmalert UI / Prometheus `/alerts`) to confirm the rule loaded.
|
||||
|
||||
### Watch for false positives
|
||||
|
||||
Stay on the alert for at least one full `for:` window after sync. If the rule pages immediately and was not expected to → revert.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Sub-1m `for:` windows** — flap-prone.
|
||||
2. **Alerting on absolute counts** instead of rates/ratios.
|
||||
3. **`severity` flips without on-call sign-off** — silent re-route.
|
||||
4. **Missing `runbook_url`** — pages without remediation steps waste on-call cycles.
|
||||
5. **PromQL that uses `{job=~"foo.*"}` regex without anchors** — slow query, sometimes false matches.
|
||||
6. **Inlining the rule body in the chart `templates/`** to "make it sticky" — that's a chart fork; see NEVER-DO.
|
||||
7. **Adding the rule on every cluster in one PR** — surgical only.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the PR. Sync.
|
||||
- vmalert/Prometheus reload picks up the previous ruleset within seconds.
|
||||
- Any in-flight alerts resolve on next evaluation.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Procedure: [modify-observability-config.md](modify-observability-config.md) — broader observability edits.
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md) — when an alert is silent because the metric is gone.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
@@ -0,0 +1,210 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify observability stack config
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** one chart × one cluster (a metrics gap, retention change, or alert-rule edit can ripple to paging and dashboards across the BU).
|
||||
> **Approval:** observability owner (`siddharth.pal@meesho.com`) + cluster owner.
|
||||
|
||||
This procedure covers edits to the cluster-by-cluster observability override files:
|
||||
|
||||
| Override path | What it controls |
|
||||
|---------------|------------------|
|
||||
| `helm-overrides/<cluster>/victoria-metrics-agent/custom-values.yaml` | Scrape targets, relabel rules, remote-write tenancy. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmstorage retention, vmselect/vminsert sizing, ruler config. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` (also `mimir-distributed`) | Mimir distributor/ingester/ruler config and tenancy. |
|
||||
| `helm-overrides/<cluster>/loki/custom-values.yaml` (also `loki-distributed`) | Loki retention, ingestion limits, multi-tenant config. |
|
||||
| `helm-overrides/<cluster>/tempo/custom-values.yaml` (also `tempo-distributed`) | Tempo block retention, distributor sizing. |
|
||||
| `helm-overrides/<cluster>/grafana/custom-values.yaml` | Datasources, dashboard providers, plugins. |
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert alerting rules and notifier config. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` (or `kube-prometheus-stack`) | Prometheus alert rules, scrape config, retention. |
|
||||
|
||||
All of these are **versioned-sibling-aware** — confirm whether the cluster's Argo Application points at `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`, etc., before editing. See [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md) for stack conventions.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing scrape targets, relabel rules, recording rules.
|
||||
- Changing retention (`vmstorage.retentionPeriod`, Loki `retention_period`, Tempo `compaction.block_retention`).
|
||||
- Adding/removing/tuning alert rules in `vmalert` or `kube-prometheus-stack`.
|
||||
- Adding/removing Grafana datasources, dashboards, plugins.
|
||||
- Tuning ingester/distributor sizing for Mimir/Loki/Tempo.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- **Bumping the chart version** of an observability tool — use [update-chart-version.md](update-chart-version.md).
|
||||
- **Onboarding a brand-new observability tool** to a cluster — use [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
||||
- **A blue-green migration** between sibling charts (`-latest`) — use [blue-green-chart-migration.md](blue-green-chart-migration.md).
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `victoria-metrics-cluster` |
|
||||
| Versioned-sibling target (if applicable) | `victoria-metrics-cluster-latest` |
|
||||
| Change kind | `retention bump`, `new scrape target`, `new alert rule`, `dashboard add` |
|
||||
| Promql expression(s) touched (if any) | `sum(rate(...)) by (job) > 0.05` |
|
||||
| Approval ticket | CMR-1234 (if BU policy requires) |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The cluster directory and chart override exist.
|
||||
- [ ] The cluster's Argo `Application` points at the chart you're editing (and not its sibling). Check `github.com/Meesho/devops-infra-argo-config`.
|
||||
- [ ] You have the previous PR diff for context (most observability edits are touching a known knob).
|
||||
- [ ] You have access to PromQL/promtool (or vmalert binary) locally for rule validation.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Confirm which sibling the cluster runs
|
||||
|
||||
```bash
|
||||
# In the sister repo
|
||||
gh search code --repo Meesho/devops-infra-argo-config "<chart>" -- path:**/<cluster>*
|
||||
```
|
||||
|
||||
Find the `Application` whose `spec.source.path` points at this repo. Note whether it's `victoria-metrics-cluster` or `victoria-metrics-cluster-latest`. **Editing the wrong one is silent** — the file diff merges, but no cluster picks it up.
|
||||
|
||||
### 2. Read the current values
|
||||
|
||||
```bash
|
||||
yq e '.' helm-overrides/<cluster>/<chart>/custom-values.yaml | less
|
||||
```
|
||||
|
||||
Note current retention, scrape targets, and any inline alert rules.
|
||||
|
||||
### 3. Author the change
|
||||
|
||||
Edit `helm-overrides/<cluster>/<chart>/custom-values.yaml`. Surgical — only the keys the task requires. Preserve YAML key order; preserve comments.
|
||||
|
||||
#### Cardinality discipline
|
||||
|
||||
If the change adds a label, scrape target, or `relabel_configs` rule, ask: can the new label exceed ~few-hundred distinct values? If yes, drop or aggregate. See [observability.md §Cardinality discipline](../../global/coding-guidelines/observability.md).
|
||||
|
||||
#### Retention changes
|
||||
|
||||
Increasing `vmstorage.retentionPeriod`, Loki `retention_period`, or Tempo `block_retention` grows the backing PVC. Confirm `persistence.size` (or `vmstorage.persistentVolume.size`) has been bumped to match — otherwise vmstorage runs out of disk silently.
|
||||
|
||||
#### Alert-rule edits
|
||||
|
||||
If the rule's `for:` window or `severity` label changes, the PagerDuty routing may swap. Cross-check the cluster's Alertmanager / vmalert notifier config before merging.
|
||||
|
||||
### 4. Render the chart locally
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
The render must succeed. If it errors, fix the values before continuing.
|
||||
|
||||
### 5. Validate any PromQL touched
|
||||
|
||||
For Prometheus/`kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
# Extract PrometheusRule objects from rendered output
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
|
||||
# Validate
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
# Extract VMRule (or ConfigMap with rules)
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
|
||||
# vmalert dry-run
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
If `promtool` / `vmalert` is unavailable locally, surface the rule expression in the PR description and request reviewer to validate.
|
||||
|
||||
### 6. (Grafana) Validate datasource URL is in-cluster
|
||||
|
||||
Datasources should point at in-cluster Service DNS (e.g. `http://victoria-metrics-cluster-vmselect:8481`), not external endpoints. **Never** pin a Grafana datasource at `*.meeshogcp.in`, `prd.meesho.int`, or any production hostname — that violates the NEVER-DO list and routes through external networking unnecessarily.
|
||||
|
||||
### 7. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b obs/<cluster>-<chart>-<short-desc>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "obs(<cluster>/<chart>): <short desc>"
|
||||
git push origin obs/<cluster>-<chart>-<short-desc>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<one-line: what changed and why>
|
||||
|
||||
## Cluster × chart
|
||||
- Cluster: `<cluster>`
|
||||
- Chart: `<chart>` (sibling: `<sibling-or-N/A>`)
|
||||
- File: `helm-overrides/<cluster>/<chart>/custom-values.yaml`
|
||||
|
||||
## Validation
|
||||
- [ ] `helm template` renders cleanly
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed (PromQL expression: `<expr>`)
|
||||
- [ ] Cardinality bounded (no unbounded label introduced)
|
||||
- [ ] Retention/PVC headroom confirmed (if retention changed)
|
||||
- [ ] PagerDuty routing unchanged (if alert rule changed)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- Cluster owner: <handle>
|
||||
|
||||
## Sister repo
|
||||
- N/A (no Application change required)
|
||||
```
|
||||
|
||||
### 8. After merge
|
||||
|
||||
Argo CD on the target cluster reconciles. Most observability charts use **manual sync** (see [observability.md](../../global/coding-guidelines/observability.md)) — open the cluster's Argo CD UI, find the Application, click **Sync**. Watch:
|
||||
|
||||
```bash
|
||||
kubectl --context=<ctx> -n <observability-ns> get pods -w
|
||||
kubectl --context=<ctx> -n <observability-ns> logs <chart>-pod | tail -50
|
||||
```
|
||||
|
||||
If a metric goes missing post-sync → [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Editing the wrong sibling.** The Argo Application points at `*-latest`; you edit the stable chart's values. Diff merges, nothing applies.
|
||||
2. **Adding `pod_name` / `request_id` / `user_id` as a label** without aggregation — explodes cardinality.
|
||||
3. **Bumping retention without bumping PVC.** vmstorage runs out of disk; ingest fails silently.
|
||||
4. **Silent PagerDuty re-route.** Changing `severity:` from `warning` to `critical` (or vice versa) without coordinating on-call.
|
||||
5. **Inlining production hostnames** as Grafana datasource URLs — see [SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3.
|
||||
6. **Cross-cluster normalising** — touching every cluster's `custom-values.yaml` in one PR. Surgical only; one cluster per PR.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the values PR. Argo CD will re-render with the previous values; click Sync.
|
||||
- For retention shrinks: data older than the new retention is dropped on next compaction. Reverting restores the *config* but not the *data*.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md).
|
||||
- Procedure: [modify-alert-rules.md](modify-alert-rules.md) — alert-only edits.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- ADR: [../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md](../../../wiki/analyses/ADR-A2-blue-green-sibling-pattern.md).
|
||||
- Wiki: [../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md](../../../wiki/entities/DevOps%20Infra%20Helm%20Charts.md).
|
||||
@@ -0,0 +1,204 @@
|
||||
# Procedure — Onboard a new app to an existing cluster
|
||||
|
||||
> **Layer:** Layer 1 — Agent-Writable.
|
||||
> **Blast radius:** one new Helm release on one cluster.
|
||||
> **Approval:** app owner + cluster owner.
|
||||
|
||||
This procedure adds a new infrastructure-tooling Helm release to a cluster that already exists in `helm-overrides/`. It covers the slice owned by `devops-infra-helm-charts`. The matching Argo `Application` lives in `github.com/Meesho/devops-infra-argo-config` and must be paired.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Chart name (must exist in `helm-templates/`) | `kube-state-metrics` |
|
||||
| Target cluster directory | `k8s-supply-prd-ase1` |
|
||||
| Release name | `kube-state-metrics` (matches chart name; may be different for variants) |
|
||||
| Workload namespace | `monitoring` |
|
||||
| Image tag | `v2.10.1` |
|
||||
| Sized resources (CPU/memory requests + limits) | `250m / 512Mi` |
|
||||
| Node-pool key (per cluster) | `dedicated: monitoring` |
|
||||
| Whether this needs a sidecar `external-dns` Service | yes / no |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The chart exists in `helm-templates/<chart>/` with a current `Chart.yaml`.
|
||||
- [ ] The cluster directory exists in `helm-overrides/<cluster>/`.
|
||||
- [ ] The chart is appropriate for an *infra* release (services don't go here — they live in `devops-argo-config`).
|
||||
- [ ] The matching sister-repo `Application` PR is drafted (or will be drafted in parallel).
|
||||
- [ ] CMR ticket open if required by BU policy.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Verify the chart renders with a sibling cluster's values
|
||||
|
||||
Pick a cluster that already runs this chart and use its values as a starting point:
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -60
|
||||
```
|
||||
|
||||
If the render errors → the chart's dependencies may be unresolved. Run `helm dependency update helm-templates/<chart>` first.
|
||||
|
||||
### 2. Identify the cluster's scheduling profile
|
||||
|
||||
```bash
|
||||
# Standard GKE: uses 'dedicated:' keys
|
||||
grep -rh 'dedicated:' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
|
||||
|
||||
# GKE Autopilot: uses 'cloud.google.com/compute-class' keys
|
||||
grep -rh 'cloud.google.com/compute-class' helm-overrides/<cluster>/*/custom-values.yaml | sort -u
|
||||
```
|
||||
|
||||
For Contour, cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). For others, copy from a sibling app on the **same** cluster — never from the same app on a different cluster ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md)).
|
||||
|
||||
### 3. Create the directory and `custom-values.yaml`
|
||||
|
||||
```bash
|
||||
mkdir -p helm-overrides/<cluster>/<app>
|
||||
$EDITOR helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
```
|
||||
|
||||
Author from scratch using [custom-values-schema.md](../schemas/custom-values-schema.md) and the cluster's scheduling profile from step 2. Do **not** copy a sibling cluster's values verbatim.
|
||||
|
||||
Skeleton:
|
||||
|
||||
```yaml
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<image>
|
||||
tag: <tag>
|
||||
|
||||
replicaCount: <N>
|
||||
|
||||
resources:
|
||||
requests: {cpu: <c>, memory: <m>}
|
||||
limits: {cpu: <c>, memory: <m>}
|
||||
|
||||
nodeSelector:
|
||||
<pool-key>: <pool-value>
|
||||
tolerations:
|
||||
- {key: <pool-key>, value: <pool-value>, effect: NoSchedule}
|
||||
```
|
||||
|
||||
### 4. (If needed) Add sidecar raw manifests
|
||||
|
||||
If the app needs sidecar resources (e.g. `external-dns` `Service`, `ComputeClass`, `ExternalSecret`), drop them in the same directory under a subfolder:
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<app>/
|
||||
custom-values.yaml
|
||||
external-dns-services/<svc>.yaml
|
||||
computeclass/<name>-cc.yaml
|
||||
```
|
||||
|
||||
See [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
|
||||
### 5. Validate locally
|
||||
|
||||
```bash
|
||||
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
|
||||
# Render
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -60
|
||||
|
||||
# Optional: dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
|
||||
--kube-context=<context>
|
||||
```
|
||||
|
||||
### 6. Open the sister-repo PR
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`, draft an `Application` (or add to an existing `ApplicationSet`):
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: <release>-<cluster>
|
||||
spec:
|
||||
destination:
|
||||
name: <cluster>
|
||||
namespace: <workload-namespace>
|
||||
source:
|
||||
repoURL: https://github.com/Meesho/devops-infra-helm-charts.git
|
||||
targetRevision: main
|
||||
path: helm-overrides/<cluster>/<app>
|
||||
helm:
|
||||
valueFiles: [custom-values.yaml]
|
||||
syncPolicy:
|
||||
syncOptions: [CreateNamespace=true]
|
||||
# Most infra apps DO NOT use automated sync — see ADR-A5
|
||||
```
|
||||
|
||||
### 7. Open the values-side PR (this repo)
|
||||
|
||||
```bash
|
||||
git checkout -b onboard/<chart>-on-<cluster>
|
||||
git add helm-overrides/<cluster>/<app>/
|
||||
git commit
|
||||
# pre-commit hook runs TruffleHog
|
||||
git push origin onboard/<chart>-on-<cluster>
|
||||
gh pr create --base main --title "Onboard <chart> to <cluster>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- App owner approver tag.
|
||||
- Cluster owner approver tag.
|
||||
- Sister-repo PR link (`devops-infra-argo-config#<n>`).
|
||||
- CMR ticket reference (if applicable).
|
||||
- Confirmation that the chart renders cleanly and `helm diff` (if run) showed only additions.
|
||||
|
||||
### 8. After both merge — sync in Argo CD
|
||||
|
||||
`devops-infra-argo-config`'s reconciler will create the `Application` resource on the cluster's Argo CD. Most infra apps are **manual sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)), so the workload deploy is a separate step:
|
||||
|
||||
1. Open the cluster's Argo CD UI.
|
||||
2. Search for the new `Application`.
|
||||
3. Verify the manifest renders (Diff view shows the chart's resources).
|
||||
4. Click **Sync**.
|
||||
5. Watch the rollout: `kubectl --context=<context> get pods -n <ns> -w`.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Copying the entire `helm-overrides/<sibling-cluster>/<app>/` directory** verbatim. Per-cluster scheduling differs.
|
||||
2. **Bundling onboarding with a chart-version bump.** Two separate PRs.
|
||||
3. **Skipping the sister-repo PR.** Without an `Application`, the values do nothing.
|
||||
4. **Setting `automated.{prune,selfHeal}: true`** in the sister-repo `Application` "to make life easier." Manual sync is the default safety property.
|
||||
5. **Inlining secrets** in `custom-values.yaml`. Use `ExternalSecret`.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the merge causes a problem before Sync:
|
||||
|
||||
- Revert the values-side PR (and the sister-repo PR).
|
||||
- Argo CD will prune the `Application` resource on next reconcile of the sister repo.
|
||||
|
||||
If Sync was clicked and the workload broke:
|
||||
|
||||
- Click **Rollback** in Argo CD UI to the previous synced revision (if there is one).
|
||||
- Or revert both PRs and re-Sync — the previous state had no `Application`, so the workload is removed.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Procedure: [update-chart-version.md](update-chart-version.md) for bumping after onboarding.
|
||||
- Procedure: [deboard-app.md](deboard-app.md) for retirement.
|
||||
- Skill: [skills/infra/onboard-app.md](../../../skills/infra/onboard-app.md) — agent-callable wrapper.
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,140 @@
|
||||
# Procedure — Onboard a new cluster's overrides
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK.
|
||||
> **Blast radius:** establishing a new deployment target. Mistakes here are repeated for every app subsequently onboarded.
|
||||
> **Approval:** platform team + cluster owner. CMR mandatory.
|
||||
|
||||
When a new GKE cluster is provisioned (typically by `terraform-gcp-infra` or its successor), this repo gains a new `helm-overrides/<cluster>/` directory and the sister repo gains the `ApplicationSet` (or per-cluster Applications) that route to it. This procedure covers the slice owned by `devops-infra-helm-charts`.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The cluster exists in GCP — confirmed by the platform team.
|
||||
- [ ] The cluster is registered in the relevant Argo CD instance(s).
|
||||
- [ ] The cluster's `nodeSelector` / `tolerations` / `computeClass` topology is documented (node pool names + taints).
|
||||
- [ ] The cluster has a `SecretStore` / `ClusterSecretStore` for `external-secrets` — or onboarding `external-secrets` is part of this PR.
|
||||
- [ ] CMR open.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Confirm the naming convention
|
||||
|
||||
| Pattern | Use |
|
||||
|---------|-----|
|
||||
| `k8s-<bu>-prd-ase1[c]` | Standard BU prod cluster (GCP zone-a or zone-c) |
|
||||
| `k8s-shared-int-ase1` | Shared int |
|
||||
| `k8s-aurva-prd-ase1` | Aurva |
|
||||
| `k8s-supply-dev-ase1` | Dev/sandbox |
|
||||
| `db-<numeric-id>-...` | Auto-named dataplane / data-tier |
|
||||
|
||||
The directory name is a **contract** — it must match the cluster name as registered in Argo CD. Deviating by a hyphen or case is a silent bind failure.
|
||||
|
||||
### 2. Determine the cluster's scheduling profile
|
||||
|
||||
| Cluster type | Scheduling key |
|
||||
|--------------|----------------|
|
||||
| GKE Autopilot | `cloud.google.com/compute-class` |
|
||||
| Standard GKE | `dedicated:` |
|
||||
|
||||
Get the actual node-pool / compute-class names from the cluster owner. If GKE Autopilot, also collect the list of `ComputeClass` resources that need to land in `helm-overrides/<cluster>/<app>/computeclass/`.
|
||||
|
||||
### 3. Decide the minimal app set
|
||||
|
||||
Most clusters need at least:
|
||||
|
||||
- `external-secrets` — to materialise secrets from GCP Secret Manager
|
||||
- `kube-state-metrics` — for fleet observability
|
||||
- `victoria-metrics-agent` — to ship metrics to the central VM
|
||||
- `fluentd` (or equivalent log shipper)
|
||||
|
||||
Plus per-cluster role:
|
||||
|
||||
- BU clusters: `contour-internal-0`, `contour-internal-1`, sometimes `contour-external` — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) for the convention
|
||||
- DSGPU / Spark clusters: GPU operators, `keda` for queue autoscaling
|
||||
- Dataplane (`db-*`) clusters: minimal — `kube-state-metrics` + `victoria-metrics-agent` only
|
||||
|
||||
### 4. Create the directory and minimal apps
|
||||
|
||||
```bash
|
||||
mkdir -p helm-overrides/<cluster>
|
||||
cd helm-overrides/<cluster>
|
||||
```
|
||||
|
||||
For each app in the minimal set, follow [onboard-app-to-cluster.md](onboard-app-to-cluster.md) (one app per sub-PR after this base PR lands, OR all in one base PR — see step 7).
|
||||
|
||||
### 5. Update the per-cluster Contour matrix
|
||||
|
||||
If the cluster runs Contour, add a section to [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md) with the per-instance scheduling. Without this, future Contour edits on the cluster have no reference.
|
||||
|
||||
### 6. (If GKE Autopilot) Add the `ComputeClass` resources
|
||||
|
||||
Each Autopilot Contour / pool needs a corresponding `ComputeClass`. Drop them in:
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<app>/computeclass/<name>-cc.yaml
|
||||
```
|
||||
|
||||
Schema: [raw-manifest-sidecar-schema.md §ComputeClass](../schemas/raw-manifest-sidecar-schema.md). The `metadata.name` of each `ComputeClass` must equal the value the corresponding app's `nodeSelector: {cloud.google.com/compute-class: <name>}` references.
|
||||
|
||||
### 7. Open the base PR
|
||||
|
||||
```bash
|
||||
git checkout -b cluster/<cluster>-base
|
||||
git add helm-overrides/<cluster>/
|
||||
git add contour-nodeselector-tolerations-summary.md # if updated
|
||||
git commit
|
||||
git push origin cluster/<cluster>-base
|
||||
gh pr create --base main --title "Onboard <cluster> — base override set"
|
||||
```
|
||||
|
||||
PR description must include:
|
||||
|
||||
- Cluster owner approver tag.
|
||||
- Platform team approver tag.
|
||||
- CMR ticket.
|
||||
- Sister-repo PR for the `ApplicationSet` / per-cluster `Application` set.
|
||||
- The minimal app set being shipped, with per-app rationale.
|
||||
- The cluster's scheduling profile (key style + sample `nodeSelector`).
|
||||
|
||||
### 8. Pair the sister-repo PR
|
||||
|
||||
In `github.com/Meesho/devops-infra-argo-config`, draft the `ApplicationSet` (or per-cluster Application set) that walks `helm-overrides/<cluster>/`. The sister-repo PR depends on this PR being merged first (otherwise the paths it references don't exist).
|
||||
|
||||
### 9. Stage subsequent app additions
|
||||
|
||||
After the base PR merges:
|
||||
|
||||
- Each additional app (`contour-internal-0`, `flagger`, etc.) is its own PR via [onboard-app-to-cluster.md](onboard-app-to-cluster.md).
|
||||
- Don't try to land the whole cluster in one PR — review surface explodes; rollback is all-or-nothing.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Naming the directory `<cluster-name>` slightly differently** from how Argo CD registered it. Silent bind failure.
|
||||
2. **Cloning another cluster's directory verbatim.** Per-cluster scheduling differs; `external-secrets` `SecretStore` references differ; image-tag pinning differs.
|
||||
3. **Onboarding 30 apps in the base PR.** Stage them — base + per-app.
|
||||
4. **Forgetting the Contour matrix update.** Future Contour edits then have no reference and silently mis-schedule.
|
||||
5. **Skipping the `ComputeClass` resources** on a GKE Autopilot cluster. `nodeSelector` references a class that doesn't exist; pods Pending.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If the cluster onboarding turns out to be premature:
|
||||
|
||||
- Revert the values-side PR.
|
||||
- Revert the sister-repo `ApplicationSet` PR.
|
||||
- The cluster reverts to "no Argo Applications" — workloads that were synced manually before revert remain (Argo CD doesn't `prune` what's not in its scope after the `Application` is removed); for a clean wipe, manually `kubectl delete ns` the affected namespaces.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [onboard-app-to-cluster.md](onboard-app-to-cluster.md) — for each app after the base PR.
|
||||
- Schema: [raw-manifest-sidecar-schema.md](../schemas/raw-manifest-sidecar-schema.md) — for `ComputeClass` and `ExternalSecret` sidecars.
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,176 @@
|
||||
# Procedure — Update a chart's pinned dependency version
|
||||
|
||||
> **Layer:** Layer 1 — HIGH RISK (charts in `helm-templates/` are consumed by every cluster that runs them).
|
||||
> **Blast radius:** every cluster that has an Argo `Application` referencing this chart will pick up the new version on next sync.
|
||||
> **Approval:** platform team. CMR mandatory for prod-fleet charts.
|
||||
|
||||
In this repo, charts in `helm-templates/<chart>/` are typically thin wrappers — `Chart.yaml` declares an upstream chart as a dependency, and `Chart.lock` pins the resolved subchart. "Bumping the chart version" means:
|
||||
|
||||
1. Update `dependencies[].version` in `Chart.yaml`.
|
||||
2. Run `helm dependency update` to refresh `Chart.lock` (and re-pull the subchart).
|
||||
3. Test render with representative cluster overrides.
|
||||
|
||||
---
|
||||
|
||||
## When to use this procedure
|
||||
|
||||
- Upgrading a wrapper chart's pinned subchart version (e.g. `argo-cd 7.7.23` → `7.8.0`).
|
||||
- Following a security advisory in an upstream chart.
|
||||
|
||||
This is **not** the right procedure for:
|
||||
|
||||
- A blue-green migration to a new major version → use [blue-green-chart-migration.md](blue-green-chart-migration.md).
|
||||
- Editing chart templates → use [fork-upstream-chart.md](fork-upstream-chart.md).
|
||||
- Changing values without a chart bump → that's per-cluster `custom-values.yaml` work, not this.
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] The new upstream version exists and has a published changelog.
|
||||
- [ ] You have read the changelog for breaking template / values changes.
|
||||
- [ ] The bump is the **explicit headline** of the PR (not a side-effect of another change).
|
||||
- [ ] CMR open if this is a prod-fleet chart (Argo CD, Contour, VictoriaMetrics, ingress, cert-manager).
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Read the changelog
|
||||
|
||||
```bash
|
||||
# For most upstream charts:
|
||||
helm repo update
|
||||
helm search repo <repo>/<chart> --versions | head -20
|
||||
# Read the chart's CHANGELOG.md or release notes on GitHub.
|
||||
```
|
||||
|
||||
Look specifically for:
|
||||
|
||||
- **Removed values keys** — would silently no-op overrides.
|
||||
- **Renamed values keys** — same.
|
||||
- **CRD changes** — might require manual `kubectl apply` of new CRDs.
|
||||
- **Breaking template changes** (e.g. label selector immutability on Deployments).
|
||||
- **Required Kubernetes version bumps**.
|
||||
|
||||
If any of these apply, this is not a simple version bump — escalate to a blue-green migration ([blue-green-chart-migration.md](blue-green-chart-migration.md)).
|
||||
|
||||
### 2. Update `Chart.yaml`
|
||||
|
||||
```bash
|
||||
$EDITOR helm-templates/<chart>/Chart.yaml
|
||||
```
|
||||
|
||||
Find the `dependencies[]` entry and bump `version:`. Example:
|
||||
|
||||
```diff
|
||||
dependencies:
|
||||
- name: argo-cd
|
||||
- version: 7.7.23
|
||||
+ version: 7.8.0
|
||||
repository: https://argoproj.github.io/argo-helm
|
||||
```
|
||||
|
||||
### 3. Refresh `Chart.lock`
|
||||
|
||||
```bash
|
||||
helm dependency update helm-templates/<chart>
|
||||
```
|
||||
|
||||
This:
|
||||
|
||||
- Verifies the new version resolves.
|
||||
- Updates `Chart.lock` with the new digest.
|
||||
- Re-pulls the subchart `.tgz` into `helm-templates/<chart>/charts/`.
|
||||
|
||||
The lockfile must be committed alongside `Chart.yaml`. A bump without a refreshed lockfile is incomplete.
|
||||
|
||||
### 4. Render against a representative cluster's overrides
|
||||
|
||||
Pick a cluster that runs this chart with a non-trivial override set:
|
||||
|
||||
```bash
|
||||
sibling=$(find helm-overrides -maxdepth 2 -type d -name '<chart>' \
|
||||
| grep -v '^helm-overrides/db-' | head -1)
|
||||
helm template <chart> helm-templates/<chart> -f "$sibling/custom-values.yaml" | head -120
|
||||
```
|
||||
|
||||
If the render errors → the chart bump introduced a values incompatibility. **Stop.** Either:
|
||||
|
||||
- Find the new values shape and update affected `custom-values.yaml` files in this PR, OR
|
||||
- Defer the bump until those updates are scoped.
|
||||
|
||||
### 5. Spot-check `helm diff` against a live cluster (recommended)
|
||||
|
||||
```bash
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml \
|
||||
--kube-context=<cluster-context>
|
||||
```
|
||||
|
||||
What you want to see:
|
||||
|
||||
- Image tag bumps.
|
||||
- Label updates (often `helm.sh/chart`).
|
||||
- Possibly new CRDs or RBAC.
|
||||
|
||||
What's a red flag:
|
||||
|
||||
- `Deployment` selector changes (immutable; will fail to apply).
|
||||
- `StatefulSet` `volumeClaimTemplates` changes.
|
||||
- Resource removals you didn't expect.
|
||||
|
||||
### 6. Commit and open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b chart-bump/<chart>-<new-version>
|
||||
git add helm-templates/<chart>/Chart.yaml helm-templates/<chart>/Chart.lock
|
||||
git add helm-templates/<chart>/charts/ # if the .tgz was re-pulled
|
||||
git commit
|
||||
git push origin chart-bump/<chart>-<new-version>
|
||||
gh pr create --base main --title "chart bump: <chart> <old> → <new>"
|
||||
```
|
||||
|
||||
PR description:
|
||||
|
||||
- Procedure followed: this file.
|
||||
- Old → new with a link to the upstream changelog.
|
||||
- The list of clusters that consume this chart (`grep -rl '<chart>' helm-overrides | head`).
|
||||
- The render(s) and diff(s) you ran.
|
||||
- Whether any breaking-change fallout is bundled (preferably not — if so, scope to a separate PR).
|
||||
|
||||
### 7. After merge — Argo CD picks up the new chart on next sync
|
||||
|
||||
For each cluster running this chart:
|
||||
|
||||
- The cluster's Argo CD detects the chart hash change.
|
||||
- The `Application` becomes `OutOfSync`.
|
||||
- A human clicks **Sync** per [ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md).
|
||||
- Watch rollout per cluster — don't sync 20 clusters simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Bumping `Chart.yaml` without refreshing `Chart.lock`.** Argo CD uses the lockfile; the bump silently no-ops.
|
||||
2. **Bundling a chart bump with values changes.** Two separate PRs — one for the bump, one for any values that need to change because of the bump.
|
||||
3. **Bumping a major version (e.g. 7.x → 8.x) in place.** Use [blue-green-chart-migration.md](blue-green-chart-migration.md) — give yourself a sibling chart and migrate cluster-by-cluster.
|
||||
4. **Syncing every cluster simultaneously after merge.** Stagger; one cluster at a time, with a soak between.
|
||||
5. **Skipping the changelog read.** Surprises you'll regret.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
Open a follow-up PR that reverts `Chart.yaml` and `Chart.lock` to the previous versions. After merge, each cluster's Argo CD shows `OutOfSync` against the old chart; click **Sync** to roll back per cluster.
|
||||
|
||||
If the chart bump landed CRDs that are now incompatible with the old version, the rollback may require manual CRD cleanup — coordinate with the platform team.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [blue-green-chart-migration.md](blue-green-chart-migration.md) — for major-version bumps.
|
||||
- Procedure: [fork-upstream-chart.md](fork-upstream-chart.md) — when the bump requires template edits.
|
||||
- Skill: [skills/infra/bump-chart-version.md](../../../skills/infra/bump-chart-version.md).
|
||||
- ADR: [ADR-A1-cache-vs-upstream-charts.md](../../../wiki/analyses/ADR-A1-cache-vs-upstream-charts.md).
|
||||
@@ -0,0 +1,231 @@
|
||||
# Runbook — Argo CD Sync Failure (infra release)
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** an Argo CD `Application` for an infra release is `OutOfSync`, errored, or stuck `Progressing`.
|
||||
> **Layer:** mostly Layer 1 (read state, propose YAML diff). Some branches are Layer 2 (advisory).
|
||||
|
||||
This runbook handles sync failures for infra Applications routed by `github.com/Meesho/devops-infra-argo-config` and rendering against this repo's `helm-overrides/<cluster>/<app>/`.
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
APP=<release> # e.g. argocd, contour-internal-0
|
||||
CLUSTER=<cluster-name> # e.g. k8s-supply-prd-ase1
|
||||
NS=$(argocd app get $APP -o json | jq -r '.spec.destination.namespace')
|
||||
PROJECT=$(argocd app get $APP -o json | jq -r '.spec.project')
|
||||
|
||||
argocd app get $APP # headline
|
||||
argocd app get $APP -o json | jq -r '.status.conditions[]?'
|
||||
argocd app get $APP -o json | jq -r '.status.operationState.message // empty'
|
||||
```
|
||||
|
||||
Note which Argo CD instance you're hitting — most infra Applications live in a per-cluster Argo CD install.
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── Is `argocd app get $APP` known to this Argo at all?
|
||||
│
|
||||
├── NO → §1 — Application not found
|
||||
│
|
||||
└── YES → What's the symptom?
|
||||
│
|
||||
├── Sync failed with an error message → §2 — Errored sync
|
||||
├── Sync stuck `Progressing` for >5 min → §3 — Stuck progressing
|
||||
├── App is `OutOfSync` but Sync hasn't run → §4 — OutOfSync only
|
||||
└── `Synced`+`Healthy` but workload bad → §5 — Wrong workload (leave runbook)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Application not found
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Are you on the right Argo CD instance? | Most v2 infra apps live in per-cluster Argo CDs. |
|
||||
| Was the app deboarded recently? | `git -C <argo-config> log --diff-filter=D -- 'apps/<cluster>/<app>*'` and `git log --diff-filter=D -- 'helm-overrides/<cluster>/<app>/'`. |
|
||||
| Did the values directory land on `main`? | `git log --all -- 'helm-overrides/<cluster>/<app>/'`. |
|
||||
|
||||
If the file *should* exist on `main` but the `Application` resource isn't created → **Layer 2** — escalate to the platform team. The cluster's Argo CD bootstrap (`ApplicationSet`) may not be picking up the path.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Errored sync (read the error message)
|
||||
|
||||
### §2a — `repository not accessible / authentication required`
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Check `spec.source.repoURL` is `github.com/Meesho/devops-infra-helm-charts.git`. |
|
||||
| If yes, the credentials in Argo CD's repo-list need refreshing. **Layer 2** — recommend platform team rotates credentials. |
|
||||
|
||||
### §2b — `path 'X' does not exist in repo Y`
|
||||
|
||||
```text
|
||||
path 'helm-overrides/k8s-supply-prd-ase1/argocd' does not exist
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| `ls helm-overrides/<cluster>/<app>/` on `main`. |
|
||||
| If absent → either the values-side PR wasn't merged, or the path was typo'd in the sister-repo `Application`. **Layer 1** — open a fix PR (sister repo). |
|
||||
| If a blue-green migration just landed: the `Application` may be pointing at the **old** chart path that was retired. **Layer 1** — repoint the `Application` to the new sibling path. |
|
||||
|
||||
### §2c — `Helm template error` / `values file not found`
|
||||
|
||||
```text
|
||||
open helm-overrides/<cluster>/<app>/custom-values.yaml: no such file or directory
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Verify the file exists on `main`: `git ls-tree origin/main -- helm-overrides/<cluster>/<app>/custom-values.yaml`. |
|
||||
| If absent → onboarding is incomplete. **Layer 1** — open the missing values-side PR. |
|
||||
|
||||
### §2d — `unable to render manifests` / `template error`
|
||||
|
||||
Helm template error inside the chart (missing required value, type mismatch).
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Reproduce locally: `helm template <app> helm-templates/<chart> -f helm-overrides/<cluster>/<app>/custom-values.yaml`. |
|
||||
| Determine whether the fix is in this repo (rare — usually values shape changed) or `helm-templates/<chart>` (more common after a chart bump). **Layer 1**. |
|
||||
| If the error is `Cannot use existing release: ...` — see §2h. |
|
||||
|
||||
### §2e — `cluster not found / dial tcp ... no route to host`
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| **Layer 2** — escalate to platform team. Cluster API server unreachable, or cluster-secret stale in Argo CD. |
|
||||
| Do **not** edit `spec.destination.{server,name}` to redirect; that masks the underlying cluster issue. |
|
||||
|
||||
### §2f — `forbidden: ...` / admission webhook deny
|
||||
|
||||
```text
|
||||
admission webhook "validate.kyverno.svc-fail" denied the request
|
||||
```
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Look at the rule that denied (Kyverno? PSP? OPA? GKE Autopilot policy?). |
|
||||
| Often the chart's manifest violates a cluster policy (e.g. `runAsUser: 0`, missing `securityContext`). |
|
||||
| **Layer 1** — fix in chart values; pair with the policy team if the policy is wrong. |
|
||||
| GKE Autopilot specifically denies many privileged settings — read the deny message carefully. |
|
||||
|
||||
### §2g — `webhook errored: ... cert-manager / external-secrets / kyverno`
|
||||
|
||||
A webhook that should validate the new resource is itself unhealthy.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| `kubectl get pods -n cert-manager` (or the relevant operator's namespace) — is it running? |
|
||||
| **Layer 2** — recommend recovering the webhook before re-syncing this app. |
|
||||
|
||||
### §2h — `cannot patch ... immutable field`
|
||||
|
||||
Most often: `Deployment.spec.selector` or `StatefulSet.volumeClaimTemplates`. A chart bump that changes labels.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Read the upstream changelog to confirm. |
|
||||
| **Layer 2** — recommend deleting the old `Deployment` / `StatefulSet` (with the workload owner) so the chart can recreate it. **Do not delete blindly** — for `StatefulSet`, the PVCs survive but the rollout is disruptive. |
|
||||
| For systemic immutable-field changes across a chart bump, this is a sign the bump should have been a [blue-green migration](../procedures/blue-green-chart-migration.md). Roll back, plan the migration. |
|
||||
|
||||
### §2i — `dependent CRD ... not installed`
|
||||
|
||||
The chart needs a CRD that doesn't exist yet on the cluster.
|
||||
|
||||
| Action |
|
||||
|--------|
|
||||
| Check whether the chart includes the CRD in `templates/crds/` (most upstream charts ship CRDs). |
|
||||
| If yes: the chart's `helm template` may not include CRDs by default — Argo CD has `IncludeCRDs` semantics; check the `Application`'s `helm.skipCrds` setting. |
|
||||
| If the CRD is supposed to come from a different chart (`cert-manager`, `kube-prometheus-stack`): **Layer 2** — sync that chart first. |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Stuck `Progressing` for > 5 minutes
|
||||
|
||||
The sync started but resources aren't reconciling.
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `argocd app get $APP --refresh` shows resource-level status. | Look for `Progressing` resources. |
|
||||
| Is a Deployment failing to roll out? | `kubectl rollout status deploy/<name> -n $NS`. If yes → see [pod-pending-scheduling.md](pod-pending-scheduling.md) or [ingress-down.md](ingress-down.md). |
|
||||
| Is a Job hung? | `kubectl describe job/<name> -n $NS`. Old `Job`s sometimes block syncs (Helm pre-/post-install hooks). |
|
||||
| Is a `PreSync`/`PostSync` hook hanging? | `kubectl get pods -n $NS -l argocd.argoproj.io/hook=PostSync`. |
|
||||
|
||||
If the workload itself is the problem, leave this runbook.
|
||||
|
||||
---
|
||||
|
||||
## §4 — `OutOfSync` only (no error, sync hasn't run)
|
||||
|
||||
Argo CD sees a diff between git and the cluster. **Most infra apps are intentionally manual-sync** ([ADR-A5](../../../wiki/analyses/ADR-A5-manual-sync-default-for-infra.md)).
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is this expected? (e.g. you just merged a PR.) | Click Sync. |
|
||||
| Diff suspicious? (e.g. someone `kubectl edit`-ed.) | `argocd app diff $APP`. If out-of-band edit happened, the GitOps contract was violated; recommend reverting the manual change or capturing it in a PR. |
|
||||
| Diff has sat for > 1 day? | Notify the app owner — manual-sync apps rot if no one clicks. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — `Synced` and `Healthy` but workload misbehaving
|
||||
|
||||
Argo thinks all is fine; the workload is broken. Not a sync failure. **Leave this runbook.**
|
||||
|
||||
| Symptom | Where to go |
|
||||
|---------|-------------|
|
||||
| Pods crashlooping | [pod-pending-scheduling.md](pod-pending-scheduling.md) §3 |
|
||||
| Ingress 5xx | [ingress-down.md](ingress-down.md) |
|
||||
| Specific feature broken | App-team playbook |
|
||||
|
||||
---
|
||||
|
||||
## §6 — Special: blue-green migration in flight
|
||||
|
||||
If this app is in a `<chart>` ↔ `<chart>-<variant>` migration:
|
||||
|
||||
- Confirm which variant the `Application` points at (check `spec.source.path`).
|
||||
- The chart name may have changed in the new variant; release-name pinning via `fullnameOverride` may be required to keep the same Service DNS during cutover.
|
||||
- A failed sync mid-migration is the trigger to roll back (`spec.source.path` ← old) and Sync, not to push forward.
|
||||
- Read [blue-green-chart-migration.md](../procedures/blue-green-chart-migration.md) before deciding.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | Action | Escalate to |
|
||||
|---------|--------|-------------|
|
||||
| §1 + bootstrap looks healthy | Investigate further | App owner |
|
||||
| §2a (repo auth) | Confirm allowed repoURL; rotate creds | Platform team |
|
||||
| §2b (path missing) | Fix in this repo or sister repo | App owner |
|
||||
| §2c, §2d (helm render) | Reproduce; fix values or chart | App owner / platform team |
|
||||
| §2e (cluster unreachable) | Don't edit destination | Platform team |
|
||||
| §2f (admission webhook) | Fix in chart values | App + policy team |
|
||||
| §2g (webhook unhealthy) | Recover the webhook first | Platform team |
|
||||
| §2h (immutable field) | Probably needs blue-green | Platform team |
|
||||
| §3 (stuck > 30 min) | Check pod events; consider workload rollback | App owner |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `argocd app get $APP` shows `Synced` + `Healthy`.
|
||||
- The PR or manual fix that resolved it is on `main`.
|
||||
- If the failure was caused by a regression, a postmortem / RCA is scheduled.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
|
||||
- Runbook: [ingress-down.md](ingress-down.md).
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- Boundaries: [AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md).
|
||||
@@ -0,0 +1,197 @@
|
||||
# Runbook — Ingress (Contour) is down on a cluster
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** services on a cluster are returning 5xx, not reachable, or DNS doesn't resolve to working endpoints.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl actions). Layer 1 only when the fix is a values diff in this repo.
|
||||
|
||||
Contour is the ingress for almost every BU cluster, and most clusters run **multiple Contour releases** — `contour-external`, `contour-external-1`, `contour-internal-0`, `contour-internal-1`, `contour-internal-intra-{0,1}`. Each maps to a different node pool / dedicated taint or compute class. The matrix is in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Read it before editing any Contour values.**
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
|
||||
# Which Contour instances does this cluster run?
|
||||
ls helm-overrides/$CLUSTER | grep '^contour'
|
||||
|
||||
# Pod state for each Contour
|
||||
for c in $(ls helm-overrides/$CLUSTER | grep '^contour'); do
|
||||
echo "=== $c ==="
|
||||
kubectl --context=$CTX get pods -n projectcontour -l app.kubernetes.io/instance=$c -o wide 2>/dev/null \
|
||||
|| kubectl --context=$CTX get pods --all-namespaces -l app.kubernetes.io/instance=$c -o wide
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── Which Contour instance is affected?
|
||||
│
|
||||
├── External (`contour-external*`) → §A — North-south traffic
|
||||
├── Internal (`contour-internal-*`) → §B — East-west traffic
|
||||
└── Both / unsure → §C — Cluster-wide (worst case)
|
||||
```
|
||||
|
||||
For each branch:
|
||||
|
||||
```text
|
||||
└── What's the failure shape?
|
||||
│
|
||||
├── Pods Pending → §1 — Scheduling failure
|
||||
├── Pods CrashLooping → §2 — Contour boot failure
|
||||
├── Pods Running but no endpoints → §3 — Service / load-balancer unhealthy
|
||||
├── HTTPS responses are 5xx → §4 — Backend / cert / config error
|
||||
└── DNS doesn't resolve → §5 — external-dns / DNS plumbing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Contour pods Pending
|
||||
|
||||
This is almost always a scheduling-key mismatch. **The single most common Contour incident.**
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <contour-pod>
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common causes:
|
||||
|
||||
| Reason | Fix in |
|
||||
|--------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint <key>=<value>` | Tolerations in `helm-overrides/<cluster>/<contour-instance>/custom-values.yaml`. Cross-reference [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | `nodeSelector` in the values. Same fix. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) had no available compute class` (GKE Autopilot) | Either the `ComputeClass` resource is missing, or `nodeSelector: cloud.google.com/compute-class: <X>` references a class that doesn't exist. Check `helm-overrides/<cluster>/<contour-instance>/computeclass/*-cc.yaml`. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu/memory` | Node-pool autoscaler is at max. **Layer 2** — escalate to cluster owner / platform team. |
|
||||
|
||||
The fix is always to bring the values' `nodeSelector` / `tolerations` into agreement with the cluster's actual node-pool topology. Never guess — read the matrix and the cluster's other apps.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Contour pods CrashLooping
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <contour-pod> -c contour --previous
|
||||
kubectl --context=$CTX logs <contour-pod> -c envoy --previous
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `failed to load TLS certificates` | Missing `Secret`, expired cert, wrong key | `cert-manager` / `external-secrets`. **Layer 2.** |
|
||||
| `error parsing config: invalid HTTPProxy` | A user `HTTPProxy` is malformed and Contour refuses to load | The user's namespace. **Layer 2** — recommend `kubectl get httpproxy -A` to find the offender, then fix in the consuming team's repo. |
|
||||
| `bind: address already in use` | Two Contour pods on the same node fighting for the host port | Pod anti-affinity in values. **Layer 1.** |
|
||||
| `error: ratelimit service ... unavailable` | `contour-rate-limit` sidecar / external service is down | Operator action. **Layer 2.** |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Contour Running but no endpoints / LB unhealthy
|
||||
|
||||
The Contour pods are healthy, but downstream LB / DNS / Service routing is broken.
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get svc -n projectcontour
|
||||
kubectl --context=$CTX get endpoints -n projectcontour
|
||||
kubectl --context=$CTX describe svc <contour-svc> -n projectcontour
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Service` of type `LoadBalancer` has no `EXTERNAL-IP` | GCP LB provisioning failed. Check the `Service` annotations match the cluster's LB pattern. **Layer 2** — escalate. |
|
||||
| `Endpoints` empty | The Contour pods aren't matching the `Service` selector. Often a label drift between values and the chart's defaults. **Layer 1** — fix selectors. |
|
||||
| `Service` annotations mention `cloud.google.com/load-balancer-type: Internal` but external traffic is expected | Wrong Contour instance / Service annotation. **Layer 1.** |
|
||||
| `Health checks failing` on the GCP LB | Backend pods aren't ready. See §4. |
|
||||
|
||||
---
|
||||
|
||||
## §4 — HTTPS responses are 5xx
|
||||
|
||||
Don't curl the production endpoint yourself — that violates [SANCTITY_RULES R3](../../global/SANCTITY_RULES.md). Instead, recommend the user check from a controlled vantage:
|
||||
|
||||
```bash
|
||||
# From inside the cluster
|
||||
kubectl --context=$CTX run -it --rm curl-test --image=curlimages/curl --restart=Never -- \
|
||||
curl -v https://<service>.<ns>.svc.cluster.local
|
||||
|
||||
# Contour access logs
|
||||
kubectl --context=$CTX logs <envoy-pod> -c envoy | tail -50
|
||||
```
|
||||
|
||||
| Pattern | Cause | Fix in |
|
||||
|---------|-------|--------|
|
||||
| `503 no_healthy_upstream` | Backend pods all unhealthy | App's pod readiness — see [pod-pending-scheduling.md](pod-pending-scheduling.md). **Layer 2.** |
|
||||
| `502 upstream connect error` | Backend connection refused / TLS mismatch | App config / mTLS. **Layer 2.** |
|
||||
| `404 route not found` | No matching `HTTPProxy`/`Ingress` | The user's `HTTPProxy` is missing or has the wrong host/path. **Layer 2.** |
|
||||
| `500` from the app | Application error | App-team playbook. **Out of scope for this runbook.** |
|
||||
|
||||
---
|
||||
|
||||
## §5 — DNS doesn't resolve
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is `external-dns` running on this cluster? | `kubectl --context=$CTX get pods -n external-dns`. |
|
||||
| Are there `Service` resources with `external-dns.alpha.kubernetes.io/hostname` annotations? | `kubectl --context=$CTX get svc -A -o json \| jq '.items[] \| select(.metadata.annotations."external-dns.alpha.kubernetes.io/hostname")'`. |
|
||||
| Did `external-dns` reconcile recently? | `kubectl --context=$CTX logs deploy/external-dns -n external-dns \| tail -30`. |
|
||||
| Is the Cloud DNS zone wired up? | **Layer 2** — out of scope; escalate to platform team. |
|
||||
|
||||
---
|
||||
|
||||
## §C — Cluster-wide ingress outage
|
||||
|
||||
If both internal and external Contour are degraded simultaneously:
|
||||
|
||||
1. **Stop. Don't iterate values fixes.** This is incident-grade.
|
||||
2. **Layer 2 — escalate to platform team immediately.**
|
||||
3. Check whether a recent merge in this repo or the sister repo correlates: `git log --since='2 hours ago' -- helm-overrides/$CLUSTER/contour*` and the same in `devops-infra-argo-config`.
|
||||
4. If a recent merge is implicated: revert it, click Sync to roll back to the previous state.
|
||||
5. If no recent merge: cluster-level issue (node pool, network policy, GCP LB) — out of scope for this repo.
|
||||
|
||||
---
|
||||
|
||||
## When this repo *is* the right place to fix
|
||||
|
||||
A Contour outage traces back to `devops-infra-helm-charts` only when:
|
||||
|
||||
1. **`nodeSelector` / `tolerations` / `computeClass`** were copied from the wrong cluster.
|
||||
2. **A chart bump** introduced an immutable-selector change or removed a values key.
|
||||
3. **A blue-green migration** was cutover prematurely (Application points at `<contour>-vX.Y.Z` but values are still on the old shape).
|
||||
4. **`fullnameOverride`** was changed (very rare, but catastrophic).
|
||||
|
||||
For 80%+ of ingress incidents, the fix is **outside** this repo (cluster issues, app-side issues, GCP LB, DNS).
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 (pending — scheduling) | Yourself with the values fix | Cluster owner if node pool full |
|
||||
| §2 (CrashLoop — TLS) | cert-manager team | Security if cert source unknown |
|
||||
| §2 (CrashLoop — invalid HTTPProxy) | Consuming team | Platform if Contour itself is broken |
|
||||
| §3 (no endpoints / LB) | Cluster owner | Platform team |
|
||||
| §4 (5xx) | App team | Platform team if Contour-side |
|
||||
| §5 (DNS) | Platform team | — |
|
||||
| §C (cluster-wide outage) | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl get pods -n projectcontour` shows N/N Ready for the affected instances.
|
||||
- Synthetic probes from the app team return expected status codes.
|
||||
- If the root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- ADR: [ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md).
|
||||
@@ -0,0 +1,233 @@
|
||||
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Runbook — Metrics gap on a cluster / namespace
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** "Grafana panels are blank for `<cluster>` / `<namespace>` / `<service>`," or an alert that should be firing isn't, or VictoriaMetrics shows `no data`.
|
||||
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
|
||||
|
||||
The observability path on Meesho's GKE fleet is:
|
||||
|
||||
```
|
||||
workload Pod (exposes /metrics)
|
||||
└→ victoria-metrics-agent (vmagent) scrapes
|
||||
└→ remote_write to victoria-metrics-cluster (vmstorage)
|
||||
└→ vmselect ← Grafana / vmalert query
|
||||
```
|
||||
|
||||
A metrics gap can be at any hop. Walk this tree top-down — the most common root cause is hop 1 (scrape config).
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<workload-namespace-where-metric-is-missing>
|
||||
METRIC=<metric-name-or-job-label>
|
||||
|
||||
# Confirm the cluster runs VM agent + cluster
|
||||
ls helm-overrides/$CLUSTER | grep -E '^victoria-metrics-(agent|cluster)'
|
||||
|
||||
# Confirm pods are healthy
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=victoria-metrics-agent -o wide
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
|
||||
```
|
||||
|
||||
If pods are missing/CrashLooping → that's the gap. Skip to §5.
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
├── Is the metric known to be emitted by the workload?
|
||||
│ ├── No → §0 — Workload not emitting; out of scope (app team)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §1 — Is vmagent scraping the workload?
|
||||
│ ├── No → fix scrape config (Layer 1)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §2 — Are scrape targets healthy (status=up)?
|
||||
│ ├── Down → fix endpoint reachability (Layer 2 / Layer 1)
|
||||
│ └── Up →
|
||||
│
|
||||
├── §3 — Are relabel rules dropping the metric?
|
||||
│ ├── Yes → adjust relabel_configs (Layer 1)
|
||||
│ └── No →
|
||||
│
|
||||
├── §4 — Is remote_write succeeding?
|
||||
│ ├── No → vmagent → vmstorage path broken (Layer 2 / Layer 1)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §5 — Is vmstorage healthy and ingesting?
|
||||
│ ├── No → vmstorage outage (Layer 2)
|
||||
│ └── Yes →
|
||||
│
|
||||
└── §6 — Is the Grafana datasource / tenant correct?
|
||||
└── Misrouted query → fix datasource URL or tenant header (Layer 1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §0 — Workload not emitting
|
||||
|
||||
Out of scope for this repo. Confirm with:
|
||||
|
||||
```bash
|
||||
# Port-forward and curl /metrics directly
|
||||
kubectl --context=$CTX -n $NS port-forward <pod> 9090:<metrics-port> &
|
||||
curl -s localhost:9090/metrics | grep -i "$METRIC"
|
||||
```
|
||||
|
||||
If `/metrics` is empty or doesn't contain `$METRIC` → app team. Stop.
|
||||
|
||||
---
|
||||
|
||||
## §1 — Is vmagent scraping the workload?
|
||||
|
||||
```bash
|
||||
# vmagent UI exposes /api/v1/targets
|
||||
kubectl --context=$CTX -n monitoring port-forward svc/victoria-metrics-agent 8429:8429 &
|
||||
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.labels.namespace=="'$NS'")'
|
||||
```
|
||||
|
||||
If no targets for `$NS`:
|
||||
|
||||
- Look at `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml`.
|
||||
- Check the `additionalScrapeConfigs` (or `config.scrape_configs`) for a job that matches the workload's labels / namespace selector.
|
||||
- Common cause: a `kubernetes_sd_configs` `namespaces.names` filter excludes `$NS`.
|
||||
- Common cause: a missing `Pod`/`Service`/`PodMonitor` annotation `prometheus.io/scrape: "true"` on the workload.
|
||||
|
||||
**Layer 1 fix** (if the gap is in the scrape config): edit `helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml` per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
|
||||
**Layer 2 fix** (if the gap is on the workload — missing annotation): hand off to app team.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Targets healthy?
|
||||
|
||||
```bash
|
||||
# In the same vmagent /targets output
|
||||
curl -s localhost:8429/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up") | {labels, lastError}'
|
||||
```
|
||||
|
||||
If targets are `down`:
|
||||
|
||||
| `lastError` | Cause | Fix |
|
||||
|-------------|-------|-----|
|
||||
| `connection refused` | Workload not listening on declared port | App team — Layer 2. |
|
||||
| `i/o timeout` | NetworkPolicy / firewall blocking vmagent → workload | Check `NetworkPolicy` in `$NS`. Often a NetworkPolicy allowing only intra-namespace traffic and not vmagent's namespace. **Layer 2** — recommend the workload team allow vmagent. |
|
||||
| `x509: certificate signed by unknown authority` | mTLS misconfigured | App team — Layer 2. |
|
||||
| `404 Not Found` | Wrong path (default `/metrics` vs custom) | Add `metrics_path:` in scrape config. **Layer 1.** |
|
||||
|
||||
---
|
||||
|
||||
## §3 — Relabel rules dropping the metric?
|
||||
|
||||
```bash
|
||||
yq e '.config.scrape_configs[].metric_relabel_configs, .config.scrape_configs[].relabel_configs' \
|
||||
helm-overrides/$CLUSTER/victoria-metrics-agent/custom-values.yaml
|
||||
```
|
||||
|
||||
Look for:
|
||||
|
||||
- `action: drop` rules with regexes matching `$METRIC`.
|
||||
- `action: keep` rules whose regex *excludes* `$METRIC`.
|
||||
- `action: labeldrop` removing a label the query uses.
|
||||
|
||||
Test in vmagent's UI under the *Targets* tab — it shows the labels post-relabel.
|
||||
|
||||
**Layer 1 fix:** loosen the relabel rule. Re-PR per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
|
||||
---
|
||||
|
||||
## §4 — Remote_write succeeding?
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n monitoring logs deploy/victoria-metrics-agent | grep -i 'remote_write\|error\|failed' | tail -20
|
||||
```
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `429 Too Many Requests` from vmstorage | vmstorage ingest saturated | Scale vmstorage / vminsert (Layer 1 — see [observability.md](../../global/coding-guidelines/observability.md)). |
|
||||
| `connection refused` to vmstorage URL | vmstorage Service down or wrong URL | Verify `remoteWrite.url` in vmagent values matches the live `vmstorage` Service DNS. Layer 1. |
|
||||
| `out of bounds timestamp` | Clock skew on vmagent's node | Layer 2 — node time sync. |
|
||||
| `series limit exceeded` | Cardinality bomb on vmstorage tenant | Layer 1 — drop the offending label. See [observability.md §Cardinality](../../global/coding-guidelines/observability.md). |
|
||||
|
||||
---
|
||||
|
||||
## §5 — vmstorage healthy?
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n monitoring get pods -l app.kubernetes.io/name=vmstorage -o wide
|
||||
kubectl --context=$CTX -n monitoring describe pod vmstorage-0 | tail -30
|
||||
kubectl --context=$CTX -n monitoring exec vmstorage-0 -- df -h /storage
|
||||
```
|
||||
|
||||
Failure modes:
|
||||
|
||||
| Symptom | Layer | Fix |
|
||||
|---------|-------|-----|
|
||||
| Pod CrashLooping with `out of disk` | Layer 1 | Bump `persistence.size`. Note: PVC growth requires the StorageClass to support `allowVolumeExpansion: true`. See [../schemas/storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md). |
|
||||
| Pod CrashLooping with retention/index errors | Layer 2 | Escalate — may need data-side intervention. |
|
||||
| Pod Pending | Layer 1 | Scheduling issue. See [pod-pending-scheduling.md](pod-pending-scheduling.md) and [../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md](../../../wiki/analyses/ADR-A3-per-cluster-scheduling.md). |
|
||||
| Pod Running but vmselect can't reach it | Layer 1 / 2 | Verify the headless Service and StatefulSet pod-DNS records. |
|
||||
|
||||
---
|
||||
|
||||
## §6 — Grafana datasource / tenant correct?
|
||||
|
||||
```bash
|
||||
yq e '.datasources.datasources.yaml.datasources[] | select(.name == "*VictoriaMetrics*" or .type == "prometheus")' \
|
||||
helm-overrides/$CLUSTER/grafana/custom-values.yaml
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `url:` points at the correct in-cluster vmselect Service DNS | If wrong, **Layer 1** — fix the values per [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md). |
|
||||
| `httpHeaderName1: X-Scope-OrgID` (multi-tenant clusters only) | If the cluster runs multi-tenant VM, the tenant header must be set. Layer 1 fix. |
|
||||
| Datasource `url:` points at an external `*.meeshogcp.in` host | Forbidden — see [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md) R3. Repoint at in-cluster Service DNS. |
|
||||
|
||||
---
|
||||
|
||||
## Remediation summary
|
||||
|
||||
| Hop | Likely fix | Layer | Procedure |
|
||||
|-----|-----------|-------|-----------|
|
||||
| §1 scrape | Add scrape config / fix selector | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §2 endpoint | Workload-side / NetworkPolicy | 2 | Hand off to app team |
|
||||
| §3 relabel | Loosen drop rule | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §4 remote_write | Scale vmstorage / fix URL | 1/2 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
| §5 vmstorage | PVC grow / scheduling fix | 1/2 | [pod-pending-scheduling.md](pod-pending-scheduling.md) |
|
||||
| §6 datasource | Fix Grafana datasource | 1 | [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md) |
|
||||
|
||||
---
|
||||
|
||||
## Escalation triggers
|
||||
|
||||
- §5 vmstorage outage with no obvious values-side fix → platform team (observability owner).
|
||||
- Multi-cluster simultaneous metrics gap → platform team — cluster-level / control-plane issue.
|
||||
- Cardinality explosion impacting vmstorage stability → platform team + workload owner — joint fix.
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- The query that was returning `no data` returns the expected series in Grafana Explore.
|
||||
- No remote_write errors in vmagent logs for at least 5 minutes post-fix.
|
||||
- Alerts that depend on the metric have transitioned from `pending` / silent back to expected state.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Procedure: [../procedures/modify-observability-config.md](../procedures/modify-observability-config.md).
|
||||
- Procedure: [../procedures/modify-alert-rules.md](../procedures/modify-alert-rules.md) — if the gap is "alert silent" not "metric missing."
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — for §5 vmstorage scheduling failures.
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if a sync didn't take.
|
||||
@@ -0,0 +1,192 @@
|
||||
# Runbook — Pod Pending / wrong-node scheduling
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** an infra workload's pods are `Pending` indefinitely, or scheduling onto the wrong node pool.
|
||||
> **Layer:** mostly Layer 2 (advisory — recommend kubectl). Layer 1 when the fix is a values edit here.
|
||||
|
||||
The single most common values-side bug in this repo is **`nodeSelector` / `tolerations` / `computeClass` copied from the wrong cluster**. ([SANCTITY_RULES R5](../../global/SANCTITY_RULES.md))
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
APP=<release>
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<namespace>
|
||||
|
||||
kubectl --context=$CTX get pods -n $NS -o wide
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS | tail -40 # Events: section
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
└── What state are the pods in?
|
||||
│
|
||||
├── Pending — never scheduled → §1 — Pending pods
|
||||
├── ContainerCreating long → §1 — Pending pods
|
||||
├── Running but on the WRONG node pool → §2 — Wrong-pool scheduling
|
||||
├── ImagePullBackOff / ErrImagePull → §3 — Image pull
|
||||
├── CrashLoopBackOff → §4 — Crash loop
|
||||
├── Running but PVC unbound → §5 — Storage
|
||||
└── Running fine → leave runbook
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — Pending pods
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
Read the `Events:` section. Common patterns:
|
||||
|
||||
| Reason | Diagnosis | Fix in |
|
||||
|--------|-----------|--------|
|
||||
| `0/N nodes available: 1 node(s) had untolerated taint {key: dedicated, value: <X>, effect: NoSchedule}` | Pod has wrong toleration or no toleration. | `helm-overrides/<cluster>/<app>/custom-values.yaml` `tolerations:`. **Layer 1.** |
|
||||
| `0/N nodes available: 1 node(s) didn't match Pod's node affinity/selector` | Pod's `nodeSelector` doesn't match any node label. | Values `nodeSelector:`. **Layer 1.** |
|
||||
| `0/N nodes available: ... had no available compute class` (Autopilot) | `cloud.google.com/compute-class: <X>` references a `ComputeClass` that doesn't exist on the cluster. | (a) Add the `ComputeClass` resource under `helm-overrides/<cluster>/<app>/computeclass/`, or (b) use the right class name. **Layer 1.** |
|
||||
| `0/N nodes available: insufficient cpu` / `insufficient memory` | Node-pool autoscaler at max, or pod's `requests` too high. | Either reduce `resources.requests`, or **Layer 2** — escalate to cluster owner to raise node-pool max. |
|
||||
| `0/N nodes available: pod has unbound immediate PersistentVolumeClaims` | PVC is `Pending`. | See §5. |
|
||||
| `0/N nodes available: didn't tolerate node-pressure taint` | Node has `node.kubernetes.io/disk-pressure` etc. | **Layer 2** — cluster-level issue. |
|
||||
| `volume "X" not found` | PVC bound to a non-existent PV. | See §5. |
|
||||
|
||||
**The diagnostic for "wrong cluster's scheduling values":**
|
||||
|
||||
```bash
|
||||
# What does the pod's nodeSelector say?
|
||||
kubectl --context=$CTX get pod <pod> -n $NS -o yaml \
|
||||
| yq e '.spec.nodeSelector'
|
||||
|
||||
# What labels do the cluster's nodes actually have?
|
||||
kubectl --context=$CTX get nodes --show-labels | head -3
|
||||
|
||||
# Cross-reference: is the cluster's key style 'dedicated:' or 'cloud.google.com/compute-class'?
|
||||
grep -h 'dedicated:\|cloud.google.com/compute-class' \
|
||||
helm-overrides/$CLUSTER/*/custom-values.yaml | sort -u | head
|
||||
```
|
||||
|
||||
If the values use `dedicated:` but the cluster only has `cloud.google.com/compute-class:` keys (or vice versa), the values were copied from a sibling cluster. **Author the values from scratch** using the cluster's own key style.
|
||||
|
||||
---
|
||||
|
||||
## §2 — Running but on the wrong node pool
|
||||
|
||||
The pod scheduled, but on a node it shouldn't be on (e.g. a Contour-internal pod landed on the Contour-external pool).
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| What `nodeSelector` does the pod actually have? | `kubectl --context=$CTX get pod <pod> -o yaml \| yq e '.spec.nodeSelector'`. |
|
||||
| Where is it running? | `kubectl --context=$CTX get pod <pod> -o wide` — note the `NODE`. Check that node's labels. |
|
||||
| Is the values-side `nodeSelector` too permissive? | If the chart's default merges with your override, you may have inherited an unintended key. Render with `helm template` and inspect. |
|
||||
|
||||
The fix is to make the `nodeSelector` selective enough that only the intended pool matches. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §3 — Image pull failing
|
||||
|
||||
```text
|
||||
ErrImagePull / ImagePullBackOff
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| Is the image pinned to Meesho's GAR mirror? | `kubectl get deploy <d> -n $NS -o jsonpath='{.spec.template.spec.containers[].image}'`. If it's Docker Hub / Quay / GCR upstream, that's [SANCTITY_RULES R11](../../global/SANCTITY_RULES.md) violation. **Layer 1** — fix the image reference. |
|
||||
| Does the tag exist in the registry? | Out-of-band check (registry UI). |
|
||||
| Is the registry-pull credential present on the cluster? | `kubectl get secret -n $NS \| grep gcr-pull`. **Layer 2** if missing. |
|
||||
|
||||
The image tag is set in `image.repository` / `image.tag` of `custom-values.yaml`. **Layer 1.**
|
||||
|
||||
---
|
||||
|
||||
## §4 — CrashLoopBackOff
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX logs <pod> -n $NS --previous
|
||||
kubectl --context=$CTX describe pod <pod> -n $NS
|
||||
```
|
||||
|
||||
| Pattern | Likely cause | Fix in |
|
||||
|---------|--------------|--------|
|
||||
| Application stack trace, missing config | App expects an env var / file that isn't there | `custom-values.yaml` config section. **Layer 1.** |
|
||||
| `connection refused` to a dependency | Dependency not up; or wrong DNS | Dependency team. **Layer 2.** |
|
||||
| `exit code 137` | OOMKilled — `kubectl describe` confirms | Bump `resources.limits.memory` in values. **Layer 1.** |
|
||||
| `permission denied` on a file | Volume mount / `securityContext` | Values. **Layer 1.** |
|
||||
| `existing Secret <X> not found` | `existingSecret:` references a secret that doesn't exist | Either fix the name, or add the `ExternalSecret` to `helm-overrides/<cluster>/external-secrets/`. **Layer 1.** |
|
||||
| Init container failed | Init logs explain | `kubectl --context=$CTX logs <pod> -c <init-container> -n $NS`. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — Running but PVC unbound
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get pvc -n $NS
|
||||
kubectl --context=$CTX describe pvc <pvc> -n $NS
|
||||
```
|
||||
|
||||
| Sub-check | Action |
|
||||
|-----------|--------|
|
||||
| `Events: Failed to provision volume with StorageClass "<X>"` | The StorageClass doesn't exist on this cluster. | `ls manifests/storageclass/<X>.yaml`. If absent, fix `persistence.storageClass:` in values to a real class. **Layer 1.** |
|
||||
| `Events: ProvisioningFailed: googleapi: Error 403` | CSI driver lacks IAM permission. **Layer 2.** | Platform / IAM team. |
|
||||
| PVC `Pending` with no events | StorageClass has `volumeBindingMode: WaitForFirstConsumer` and the consuming pod hasn't been scheduled. | Schedule the pod (resolve §1 first). |
|
||||
| PVC bound but pod can't mount | Often the access mode mismatch (`ReadWriteOnce` PVC referenced by a multi-replica `Deployment`). | Either set replicas to 1, or use a `StatefulSet` chart variant, or use a Filestore-backed `ReadWriteMany` class. **Layer 1.** |
|
||||
|
||||
---
|
||||
|
||||
## §6 — When this repo *is* the right place to fix
|
||||
|
||||
For these cases, the fix is a values diff in `helm-overrides/<cluster>/<app>/custom-values.yaml`:
|
||||
|
||||
1. Wrong-cluster `nodeSelector` / `tolerations` / `computeClass`.
|
||||
2. Image tag pointing outside the GAR mirror.
|
||||
3. `resources.requests` / `limits` mis-sized (OOMKill, throttling).
|
||||
4. `persistence.storageClass` referencing a non-existent class.
|
||||
5. `existingSecret:` referencing a non-existent secret.
|
||||
6. `replicaCount` set on an HPA-managed release.
|
||||
|
||||
For these cases, the fix is **outside** this repo:
|
||||
|
||||
- Node pool full → cluster-owner / Terraform.
|
||||
- Image not in GAR → image-mirror automation / build pipeline.
|
||||
- CSI provisioning failures → platform / IAM team.
|
||||
- App-internal crashes (config, dependencies) → app team.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 — wrong scheduling values | Yourself with values fix | Cluster owner if topology is unclear |
|
||||
| §1 — node pool full | Cluster owner | Platform team if quota raise needed |
|
||||
| §3 — image pull (mirror miss) | Yourself with values fix | Platform if mirror push is missing |
|
||||
| §4 — OOMKilled | Yourself with `resources.limits` bump | App team if root cause is leak |
|
||||
| §4 — secret missing | Yourself with `ExternalSecret` add | Security if cluster-level `SecretStore` is missing |
|
||||
| §5 — CSI provision failure | Platform team | — |
|
||||
| §C — cluster-wide scheduling failure | Platform team — pager | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `kubectl rollout status deploy/<name> -n $NS` returns "successfully rolled out".
|
||||
- `kubectl get pods -n $NS` shows N/N Ready for the expected replica count.
|
||||
- Pods scheduled onto the **intended** node pool (verify `kubectl get pods -o wide` shows the right node names).
|
||||
- If root cause was in this repo, the fix is on `main` and synced.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Reference: [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md).
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md).
|
||||
- Runbook: [ingress-down.md](ingress-down.md).
|
||||
- Schema: [custom-values-schema.md](../schemas/custom-values-schema.md), [storageclass-priorityclass-schema.md](../schemas/storageclass-priorityclass-schema.md).
|
||||
@@ -0,0 +1,251 @@
|
||||
> Per AI Blitz Plan §platform.runbooks. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Runbook — Vault unavailable / External Secrets failing to render
|
||||
|
||||
> **Type:** Decision tree.
|
||||
> **Entry symptom:** Pods are CrashLooping referencing a missing `Secret`, or `kubectl describe externalsecret` shows `SecretSyncedError`, or `kubectl get secrets <name>` returns NotFound for a name an `ExternalSecret` should be creating.
|
||||
> **Layer:** mostly Layer 2 (advisory). Layer 1 only when the fix is a values diff in this repo.
|
||||
>
|
||||
> **Important Layer 3 boundary:** **Vault HA itself is Layer 3.** Vault runs in production-only with no lower environment, so the agent must NOT attempt server-side fixes (unsealing, leader-election toggling, raft config changes, restoring from snapshot). Those are platform-team / security-team operations. The agent's role here is *diagnose, narrow root cause, and escalate*.
|
||||
|
||||
---
|
||||
|
||||
## Architecture refresher
|
||||
|
||||
```
|
||||
workload Pod (mounts Secret <name>)
|
||||
↑ created by
|
||||
External Secrets Operator (ESO) controller
|
||||
│
|
||||
└─ reads SecretStore / ClusterSecretStore
|
||||
│
|
||||
├─ kind: gcpsm → GCP Secret Manager
|
||||
│ └─ auth via Workload Identity (KSA → GSA binding)
|
||||
└─ kind: vault → Vault HA cluster
|
||||
└─ auth via Kubernetes auth (KSA token review)
|
||||
```
|
||||
|
||||
Most clusters in the fleet use **GCP Secret Manager** as the primary backend (via Workload Identity), with Vault HA as the secondary for legacy services. Some clusters use Vault as the primary. Confirm before troubleshooting:
|
||||
|
||||
```bash
|
||||
ls helm-overrides/<cluster>/external-secrets/
|
||||
yq e '.spec.provider' helm-overrides/<cluster>/external-secrets/*.yaml 2>/dev/null
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entry — gather context
|
||||
|
||||
```bash
|
||||
CLUSTER=<cluster>
|
||||
CTX=<kubectl-context>
|
||||
NS=<namespace-of-the-failing-workload>
|
||||
ES=<external-secret-name>
|
||||
|
||||
# ESO controller status
|
||||
kubectl --context=$CTX -n external-secrets get pods
|
||||
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets | tail -50
|
||||
|
||||
# The failing ExternalSecret
|
||||
kubectl --context=$CTX -n $NS describe externalsecret $ES
|
||||
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.status'
|
||||
|
||||
# The SecretStore / ClusterSecretStore it references
|
||||
STORE=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.name}')
|
||||
KIND=$(kubectl --context=$CTX -n $NS get externalsecret $ES -o jsonpath='{.spec.secretStoreRef.kind}')
|
||||
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status, .spec.provider'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Decision tree
|
||||
|
||||
```text
|
||||
START
|
||||
│
|
||||
├── §1 — Is ESO controller running and healthy?
|
||||
│ ├── No → §1a — ESO outage
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §2 — Is the SecretStore / ClusterSecretStore Ready?
|
||||
│ ├── No → §2a — Store config / auth broken
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §3 — Does the ExternalSecret reference a real remote key?
|
||||
│ ├── No → §3a — Bad spec.data[].remoteRef.key (Layer 1 likely)
|
||||
│ └── Yes →
|
||||
│
|
||||
├── §4 — Is the auth path working? (WI binding or Vault K8s auth)
|
||||
│ ├── No → §4a — Identity binding (Layer 2/3)
|
||||
│ └── Yes →
|
||||
│
|
||||
└── §5 — Is the upstream backend healthy?
|
||||
├── GCP Secret Manager 5xx → §5a — escalate to platform
|
||||
└── Vault unsealed / leader OK? → §5b — Vault outage (Layer 3)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §1 — ESO controller health
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n external-secrets get pods
|
||||
kubectl --context=$CTX -n external-secrets logs deploy/external-secrets --tail=100 | grep -iE 'error|failed|panic'
|
||||
```
|
||||
|
||||
| Symptom | Cause | Layer | Action |
|
||||
|---------|-------|-------|--------|
|
||||
| Pods CrashLooping | Bad chart upgrade or RBAC misconfig | 1 | Check `helm-overrides/<cluster>/external-secrets/custom-values.yaml`. Last bump? Revert. |
|
||||
| Pods Pending | Scheduling — wrong nodeSelector | 1 | See [pod-pending-scheduling.md](pod-pending-scheduling.md). |
|
||||
| Pods Running, no logs about reconcile | Cluster-watch RBAC missing | 2 | Escalate. |
|
||||
| `Forbidden` errors on CRD list | RBAC on the CRDs | 1 / 2 | Verify chart values' `rbac.create: true`. |
|
||||
|
||||
---
|
||||
|
||||
## §2 — SecretStore / ClusterSecretStore status
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX get $KIND $STORE -o yaml | yq e '.status'
|
||||
```
|
||||
|
||||
Look for `conditions[].status` and `conditions[].message`.
|
||||
|
||||
| Status / message | Cause | Layer | Action |
|
||||
|------------------|-------|-------|--------|
|
||||
| `Ready: False, ValidationFailed` | Store spec invalid | 1 | Fix the `SecretStore` YAML in `helm-overrides/<cluster>/external-secrets/`. |
|
||||
| `Ready: False, InvalidProviderConfig` | Provider block malformed | 1 | Validate `spec.provider.gcpsm.projectID` / `spec.provider.vault.server`. |
|
||||
| `Ready: False, AuthFailed` (gcpsm) | Workload Identity binding broken | 2/3 | §4 below. |
|
||||
| `Ready: False, AuthFailed` (vault) | KSA token review fails | 2/3 | §4 below. |
|
||||
| `Ready: True` | Store OK; problem is elsewhere | — | Continue to §3. |
|
||||
|
||||
---
|
||||
|
||||
## §3 — ExternalSecret spec validity
|
||||
|
||||
```bash
|
||||
kubectl --context=$CTX -n $NS get externalsecret $ES -o yaml | yq e '.spec.data, .spec.dataFrom'
|
||||
```
|
||||
|
||||
For each `remoteRef.key`:
|
||||
|
||||
- **GCP SM:** the key is the secret name in the project. Verify it exists:
|
||||
```bash
|
||||
gcloud secrets list --project=<gcp-project> --filter="name:<key>"
|
||||
```
|
||||
(Read-only — no write to GCP SM from the agent.)
|
||||
|
||||
- **Vault:** the key is the path under the engine. Cannot directly verify without Vault access; rely on the ESO controller's reconcile error message.
|
||||
|
||||
If the controller logs say `secret not found in backend` → the `remoteRef.key` is wrong. **Layer 1 fix:** correct the key in the `ExternalSecret` YAML.
|
||||
|
||||
---
|
||||
|
||||
## §4 — Auth path
|
||||
|
||||
### §4a — GCP Secret Manager (Workload Identity)
|
||||
|
||||
```bash
|
||||
# The ServiceAccount the ESO controller (or this ExternalSecret's pod) runs as
|
||||
kubectl --context=$CTX -n external-secrets get sa external-secrets -o yaml | yq e '.metadata.annotations'
|
||||
|
||||
# Should have:
|
||||
# iam.gke.io/gcp-service-account: <gsa>@<project>.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
| Sub-check | Layer | Action |
|
||||
|-----------|-------|--------|
|
||||
| KSA missing the `iam.gke.io/gcp-service-account` annotation | 1 | Add via `helm-overrides/<cluster>/external-secrets/custom-values.yaml` `serviceAccount.annotations`. |
|
||||
| GSA exists but no IAM binding to KSA | 2 | Escalate to platform — IAM is out of repo. |
|
||||
| GSA lacks `roles/secretmanager.secretAccessor` on the project | 2 | Escalate to platform. |
|
||||
|
||||
### §4b — Vault Kubernetes auth
|
||||
|
||||
| Sub-check | Layer | Action |
|
||||
|-----------|-------|--------|
|
||||
| `SecretStore.spec.provider.vault.auth.kubernetes.role` references a Vault role | — | Read-only; verify against existing working stores on the same cluster. |
|
||||
| ESO logs say `permission denied` from Vault | 3 | The role/policy on the Vault server is wrong. **Cannot fix from this repo.** Escalate to security / Vault platform team. |
|
||||
| ESO logs say `Vault is sealed` | 3 | **Vault HA outage. Do not attempt to unseal.** Escalate immediately. |
|
||||
|
||||
---
|
||||
|
||||
## §5 — Backend health
|
||||
|
||||
### §5a — GCP Secret Manager
|
||||
|
||||
GCP SM is a managed service. 5xx from it is rare and is a GCP-side incident. Action: escalate to platform; check GCP status dashboard. **No fix in this repo.**
|
||||
|
||||
### §5b — Vault HA
|
||||
|
||||
Vault HA on Meesho's fleet runs in production-only, no lower env. It is **Layer 3** — agent does not write to or operate Vault. Diagnostic *read* of pod state is OK; mutation is not.
|
||||
|
||||
```bash
|
||||
# Diagnostic only
|
||||
kubectl --context=$CTX -n vault get pods
|
||||
kubectl --context=$CTX -n vault logs <vault-pod> | tail -50
|
||||
# Look for: "core: Vault is sealed", "leader election", "raft", panic stacks
|
||||
```
|
||||
|
||||
| Observed | Action |
|
||||
|----------|--------|
|
||||
| Some Vault pods sealed (HA quorum still up) | Escalate to security team. **Do not unseal.** |
|
||||
| All Vault pods sealed (full outage) | Page security team. ESO will surface stale data only as long as the in-memory cache holds. |
|
||||
| Leader-election thrashing | Escalate; could be a network/raft issue. |
|
||||
| Vault pod Pending | Scheduling — see [pod-pending-scheduling.md](pod-pending-scheduling.md). Even here, restart of a Vault pod requires a security-team-led unseal afterwards. |
|
||||
|
||||
---
|
||||
|
||||
## Mitigations while Vault is down
|
||||
|
||||
If a workload's `ExternalSecret` is failing because Vault is unavailable, options are limited:
|
||||
|
||||
1. **Wait** — ESO caches the last-rendered Secret value. Pods that already mounted continue. New pod scheduling fails until Vault returns.
|
||||
2. **Switch the `ExternalSecret` to GCP SM** if the same secret exists there (most do, with Vault as legacy). Layer 1 PR — change `secretStoreRef.name` to the GCP SM store. **Coordinate with security** before doing this in an outage.
|
||||
3. **Hand-create the Secret as a temporary `kubectl apply`** — out of scope for the agent. This is incident response by a human operator.
|
||||
|
||||
The agent should **not** auto-cut option 2 without explicit human approval — switching the source of truth for a secret has security implications.
|
||||
|
||||
---
|
||||
|
||||
## Escalation matrix
|
||||
|
||||
| Symptom | First responder | Escalate to |
|
||||
|---------|-----------------|-------------|
|
||||
| §1 (ESO down) | DevOps on-call | Platform team |
|
||||
| §2 (Store config) | DevOps on-call | Layer 1 PR + reviewer |
|
||||
| §3 (Bad remoteRef) | DevOps on-call | Layer 1 PR + reviewer |
|
||||
| §4a (WI binding) | Platform team | — |
|
||||
| §4b (Vault role/policy) | Security team | — |
|
||||
| §5a (GCP SM 5xx) | Platform team | GCP support |
|
||||
| §5b (Vault outage) | **Security team — page** | — |
|
||||
|
||||
---
|
||||
|
||||
## Done conditions
|
||||
|
||||
- `ExternalSecret` `status.conditions[type=Ready].status == True`.
|
||||
- The downstream `Secret` exists with expected keys.
|
||||
- Workload pods consuming the Secret are Running.
|
||||
- ESO logs are quiet for at least 5 minutes after the fix.
|
||||
|
||||
---
|
||||
|
||||
## What this repo can and cannot fix
|
||||
|
||||
| Fix kind | Layer | This repo? |
|
||||
|----------|-------|-----------|
|
||||
| `ExternalSecret` / `SecretStore` YAML edits | 1 | Yes — values PR. |
|
||||
| ESO chart values (sizing, RBAC, metrics) | 1 | Yes. |
|
||||
| Workload-Identity KSA annotation | 1 | Yes (in chart values' `serviceAccount.annotations`). |
|
||||
| GSA IAM bindings on GCP | 2 | No — platform team. |
|
||||
| Vault server-side role/policy | 3 | **Never** — security team. |
|
||||
| Vault unseal / raft / leader | 3 | **Never** — security team. |
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Schema: [../schemas/raw-manifest-sidecar-schema.md §ExternalSecret](../schemas/raw-manifest-sidecar-schema.md).
|
||||
- Runbook: [pod-pending-scheduling.md](pod-pending-scheduling.md) — if any of the above pods are Pending.
|
||||
- Runbook: [argocd-sync-failure.md](argocd-sync-failure.md) — if the `ExternalSecret` itself didn't sync.
|
||||
- Boundaries: [../../global/AGENT_BOUNDARIES.md](../../global/AGENT_BOUNDARIES.md), [../../global/SANCTITY_RULES.md](../../global/SANCTITY_RULES.md).
|
||||
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
@@ -0,0 +1,379 @@
|
||||
# Schema — `custom-values.yaml` (Helm override)
|
||||
|
||||
> Field-by-field annotation of the values file Argo CD's `valueFiles` references.
|
||||
> Authoritative for `helm-overrides/<cluster>/<app>/custom-values.yaml`.
|
||||
>
|
||||
> Cross-reference: [coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md), [../../architecture.md](../../architecture.md).
|
||||
|
||||
---
|
||||
|
||||
## What this file is — and isn't
|
||||
|
||||
A `custom-values.yaml` is **not** a Helm chart. It is the *override* layer Argo CD merges over the chart's own `values.yaml` (or, for thin-wrapper charts, the upstream subchart's defaults).
|
||||
|
||||
- **Schema is owned by the chart, not by us.** Every key here must be a key the chart understands. Adding a key the chart doesn't read does nothing.
|
||||
- **No multi-doc YAML.** One document per file.
|
||||
- **No `apiVersion` / `kind`.** This file is values, not a Kubernetes manifest.
|
||||
|
||||
---
|
||||
|
||||
## Top-level shape (typical)
|
||||
|
||||
```yaml
|
||||
# image registry / repository / tag
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/<image>
|
||||
tag: <semver-or-sha>
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
# release-name pinning (rarely changed once set)
|
||||
fullnameOverride: <release>-<bu>-prd # e.g. kube-state-metrics-dbc-dsci-prd
|
||||
|
||||
# scaling
|
||||
replicaCount: 3 # static, OR
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 3
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 70
|
||||
|
||||
# scheduling — bespoke per cluster
|
||||
nodeSelector:
|
||||
dedicated: <pool-key> # standard GKE
|
||||
# cloud.google.com/compute-class: <cc> # GKE Autopilot
|
||||
tolerations:
|
||||
- key: dedicated
|
||||
value: <pool-key>
|
||||
effect: NoSchedule
|
||||
|
||||
# resources
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 1
|
||||
memory: 2Gi
|
||||
|
||||
# persistence (stateful charts only)
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: sc-pd-ssd # MUST exist in manifests/storageclass/
|
||||
size: 100Gi
|
||||
accessModes: [ReadWriteOnce]
|
||||
|
||||
# secrets — by reference, never inline
|
||||
existingSecret: <name-managed-by-external-secrets>
|
||||
|
||||
# probes
|
||||
livenessProbe:
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
|
||||
# ingress / Contour HTTPProxy (if applicable)
|
||||
ingress:
|
||||
enabled: false # most internal services
|
||||
|
||||
# chart-specific top-level keys
|
||||
# (varies — read the chart's values.yaml for the full schema)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §image
|
||||
|
||||
| Field | Required | Convention | Notes |
|
||||
|-------|----------|------------|-------|
|
||||
| `image.registry` | yes (production) | `asia-southeast1-docker.pkg.dev` | Meesho's Artifact Registry mirror. ([SANCTITY_RULES R11](../../global/SANCTITY_RULES.md)) |
|
||||
| `image.repository` | yes | `meesho-devops-admin-0622/admin/sre/<image>` | Mirror path. |
|
||||
| `image.tag` | yes (production) | semver, datestamp, or git SHA | Never `latest`. Never unpinned. |
|
||||
| `image.pullPolicy` | no | `IfNotPresent` | `Always` only for development; pulls slow rollout. |
|
||||
| `image.pullSecrets` | no | omit | Mirror is public to fleet; pull secrets are a lock-in trap. |
|
||||
|
||||
Note: chart authors disagree on the shape — some use `image: {repository, tag}` (no `registry`), some use a single `image: <fully-qualified>`, some split into `image.repository: <registry>/<repo>`. **Read the chart's own `values.yaml`** before structuring this block.
|
||||
|
||||
---
|
||||
|
||||
## §release identity
|
||||
|
||||
| Field | Required | Convention | Notes |
|
||||
|-------|----------|------------|-------|
|
||||
| `fullnameOverride` | varies | Set once at release creation; stable forever | Service DNS, PVC binding, ConfigMap refs depend on it. ([SANCTITY_RULES R9](../../global/SANCTITY_RULES.md)) |
|
||||
| `nameOverride` | rare | Omit unless explicitly needed | |
|
||||
| `commonLabels` / `commonAnnotations` | rare | Omit unless the chart documents the pattern | |
|
||||
|
||||
For dataplane (`db-*`) clusters: `fullnameOverride: <kind>-dbc-<bu>-prd` is the convention (e.g. `kube-state-metrics-dbc-dsci-prd`). For BU clusters, omit unless the chart's default name collides.
|
||||
|
||||
---
|
||||
|
||||
## §scaling
|
||||
|
||||
Charts diverge sharply here. Common shapes:
|
||||
|
||||
```yaml
|
||||
# Pattern A — static replicas
|
||||
replicaCount: 3
|
||||
|
||||
# Pattern B — chart-level autoscaling block
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 3
|
||||
maxReplicas: 10
|
||||
targetCPUUtilizationPercentage: 70
|
||||
|
||||
# Pattern C — separate HPA resource (KEDA / VictoriaMetrics)
|
||||
hpa:
|
||||
enabled: true
|
||||
min: 3
|
||||
max: 20
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Don't set `replicaCount`** when `autoscaling.enabled: true`. | HPA and the chart's static deployment fight; replica count flaps. |
|
||||
| **Set both `minReplicas` and `maxReplicas`** when autoscaling. | Without `min`, the HPA can scale to zero on a quiet hour. |
|
||||
| **`minReplicas` ≥ 2** for any production-traffic-path workload. | Single-replica releases die on node drain. |
|
||||
|
||||
---
|
||||
|
||||
## §scheduling
|
||||
|
||||
The single biggest source of silent mis-deploys. Per [SANCTITY_RULES R5](../../global/SANCTITY_RULES.md):
|
||||
|
||||
| Cluster | Key style |
|
||||
|---------|-----------|
|
||||
| GKE Autopilot (`k8s-central-prd-ase1`, `k8s-dsgpu-prd-ase1`, `k8s-shared-int-ase1`) | `cloud.google.com/compute-class:` |
|
||||
| Standard GKE | `dedicated:` |
|
||||
|
||||
```yaml
|
||||
# Standard GKE — Contour internal-0 on most clusters
|
||||
nodeSelector:
|
||||
dedicated: contour-internal-0
|
||||
tolerations:
|
||||
- key: dedicated
|
||||
value: contour-internal-0
|
||||
effect: NoSchedule
|
||||
# often a second toleration to allow scheduling onto shared nodes
|
||||
- key: dedicated
|
||||
value: contour-shared
|
||||
effect: NoSchedule
|
||||
|
||||
# GKE Autopilot — same Contour internal-0 on k8s-central-prd-ase1
|
||||
nodeSelector:
|
||||
cloud.google.com/compute-class: contour-internal-0-cc
|
||||
tolerations:
|
||||
- key: cloud.google.com/compute-class
|
||||
value: contour-internal-0-cc
|
||||
effect: NoSchedule
|
||||
- key: cloud.google.com/compute-class
|
||||
value: contour-shared-cc
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
The full per-cluster Contour matrix lives in [`contour-nodeselector-tolerations-summary.md`](../../../contour-nodeselector-tolerations-summary.md). For non-Contour apps, copy from a sibling app on the **same** cluster.
|
||||
|
||||
| Field | Notes |
|
||||
|-------|-------|
|
||||
| `nodeSelector` | Hard requirement — no scheduling onto non-matching nodes. Use exactly one key (the cluster's pool key). |
|
||||
| `tolerations` | Allow scheduling onto tainted nodes. May list multiple to allow shared nodes alongside dedicated. |
|
||||
| `affinity.nodeAffinity` | Use sparingly; `nodeSelector` is sufficient for most cases here. |
|
||||
| `topologySpreadConstraints` | Use for multi-zone resilience on chatty workloads (Contour, ingress-nginx). |
|
||||
|
||||
---
|
||||
|
||||
## §resources
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 1
|
||||
memory: 2Gi
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Always set `requests`** in production. | Without requests, pods are best-effort; first to evict on memory pressure. |
|
||||
| **Set `limits` unless the chart explicitly recommends omitting them.** | Some sidecars (cert-manager, external-dns) deliberately skip `limits`. |
|
||||
| **Don't copy resources from another cluster.** | Workload sizing is per-traffic-tier. Supply prd is not demand prd. |
|
||||
| **CPU limits cause throttling**, not OOMKill. Memory limits cause OOMKill. Tune accordingly. |
|
||||
|
||||
---
|
||||
|
||||
## §persistence
|
||||
|
||||
```yaml
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: sc-pd-ssd
|
||||
size: 100Gi
|
||||
accessModes: [ReadWriteOnce]
|
||||
```
|
||||
|
||||
Allowed `storageClass` values (must match a file under `manifests/storageclass/`):
|
||||
|
||||
| Class | Backend | Use case |
|
||||
|-------|---------|----------|
|
||||
| `pd-standard-retain-dr` | GCP Persistent Disk standard, retention-on-delete | DR-critical state |
|
||||
| `sc-pd-ssd` | GCP PD SSD | Default for write-heavy workloads (etcd, ClickHouse) |
|
||||
| `sc-pd-standard` | GCP PD standard | Default for read-heavy / archival |
|
||||
| `sc-filestore-standard` | GCP Filestore | Shared volumes (Jenkins build cache, JFrog filestore) |
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Never reference a `storageClass`** that doesn't exist in `manifests/storageclass/`. | PVC stays Pending forever. |
|
||||
| **`size` must be set explicitly.** | Chart defaults are often wrong (8Gi for everything). |
|
||||
| **`accessModes`** for `ReadWriteMany` requires Filestore. PD-backed classes are RWO. |
|
||||
|
||||
---
|
||||
|
||||
## §secrets
|
||||
|
||||
```yaml
|
||||
# Chart-specific — read the chart's values.yaml
|
||||
existingSecret: <name>
|
||||
# OR
|
||||
auth:
|
||||
existingSecret: <name>
|
||||
# OR
|
||||
envFrom:
|
||||
- secretRef: {name: <name>}
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Never inline `password:`, `apiKey:`, etc.** | Pre-commit TruffleHog will catch many; it won't catch all. ([SANCTITY_RULES R4](../../global/SANCTITY_RULES.md)) |
|
||||
| **The `Secret` resource is materialised** by the cluster's `external-secrets` app. If the secret name doesn't appear in `helm-overrides/<cluster>/external-secrets/`, it doesn't exist. |
|
||||
| **Reference, don't author.** Adding the secret material to a Helm-rendered `Secret` template defeats the External Secrets pattern. |
|
||||
|
||||
---
|
||||
|
||||
## §probes
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet: {path: /healthz, port: http}
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet: {path: /ready, port: http}
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
|
||||
# For slow-warm workloads (Jenkins, JFrog, ClickHouse)
|
||||
startupProbe:
|
||||
httpGet: {path: /healthz, port: http}
|
||||
initialDelaySeconds: 60
|
||||
periodSeconds: 10
|
||||
failureThreshold: 60 # 10 minutes total runway
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Always set `liveness` + `readiness`** for any long-running container. |
|
||||
| **Use `startupProbe` for slow-warm workloads** instead of inflating `livenessProbe.initialDelaySeconds` to 600 s. |
|
||||
| **Don't set the same probe to both `liveness` and `readiness`** — they have different purposes (kill vs gate). |
|
||||
|
||||
---
|
||||
|
||||
## §ingress (Contour `HTTPProxy` / `Ingress`)
|
||||
|
||||
For external-facing apps, ingress is via Contour `HTTPProxy`. Specifics depend on the cluster's Contour topology (multiple Contour instances per cluster — see [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md)).
|
||||
|
||||
- The `external-dns` annotation is the DNS-binding step — match the cluster's DNS pattern.
|
||||
- TLS terminates at Contour with `cert-manager`-issued certs; reference the issuer by name.
|
||||
- Internal services use `contour-internal-0` / `contour-internal-1`; external services use `contour-external` / `contour-external-1`. The class is set via the `Service` annotation, not the values file directly.
|
||||
|
||||
---
|
||||
|
||||
## End-to-end example — a typical observability-side override
|
||||
|
||||
`helm-overrides/k8s-supply-prd-ase1/kube-state-metrics/custom-values.yaml`:
|
||||
|
||||
```yaml
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/kube-state-metrics
|
||||
tag: v2.10.1
|
||||
|
||||
replicaCount: 2
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
|
||||
nodeSelector:
|
||||
dedicated: monitoring
|
||||
tolerations:
|
||||
- key: dedicated
|
||||
value: monitoring
|
||||
effect: NoSchedule
|
||||
|
||||
prometheus:
|
||||
monitor:
|
||||
enabled: true
|
||||
honorLabels: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## End-to-end example — a dataplane override
|
||||
|
||||
`helm-overrides/db-2516183257845181-c-1204-195038-428/victoria-metrics-agent/custom-values.yaml`:
|
||||
|
||||
```yaml
|
||||
fullnameOverride: vmagent-dbc-supply-prd
|
||||
|
||||
image:
|
||||
registry: asia-southeast1-docker.pkg.dev
|
||||
repository: meesho-devops-admin-0622/admin/sre/victoria-metrics-agent
|
||||
tag: v1.96.0
|
||||
|
||||
replicaCount: 1
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
```
|
||||
|
||||
(Dataplane clusters intentionally have minimal overrides.)
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
# Render the chart locally with this values file
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml | head -80
|
||||
|
||||
# Dry-run diff against the live cluster (requires kubectl context + helm-diff plugin)
|
||||
helm diff upgrade <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
|
||||
# Lint
|
||||
yamllint helm-overrides/<cluster>/<app>/custom-values.yaml
|
||||
helm lint helm-templates/<chart>
|
||||
|
||||
# Sanity: storageClass referenced exists
|
||||
yq e '.persistence.storageClass' helm-overrides/<cluster>/<app>/custom-values.yaml \
|
||||
| xargs -I{} ls manifests/storageclass/{}.yaml 2>/dev/null \
|
||||
|| echo "WARN: storageClass not in manifests/storageclass/"
|
||||
```
|
||||
@@ -0,0 +1,195 @@
|
||||
> Per AI Blitz Plan §platform.schemas. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Schema — Incubator-tool values + sidecar contract
|
||||
|
||||
> **Scope:** the values-side contract for incubator (newly-vendored) infrastructure tools that live under `helm-overrides/<cluster>/<incubator-tool>/`.
|
||||
>
|
||||
> **Out of scope (explicit):** the Argo CD `Application` / `ApplicationSet` manifest that registers the incubator tool with a cluster's Argo. Those manifests live in the **sister repo** `github.com/Meesho/devops-infra-argo-config`, NOT here. This file documents only what `devops-infra-helm-charts` is responsible for: the values + raw sidecar manifests Argo CD reads from this repo.
|
||||
|
||||
This schema complements [custom-values-schema.md](custom-values-schema.md) (general values shape) and [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) (sidecar manifest shapes). Read both first.
|
||||
|
||||
---
|
||||
|
||||
## What "incubator" means here
|
||||
|
||||
A tool is **incubator** while it is being trialled on one or two clusters before fleet-wide rollout. In this repo that maps to:
|
||||
|
||||
- A new chart directory under `helm-templates/<incubator-tool>/` (often a thin wrapper `Chart.yaml` with an upstream dep).
|
||||
- An override under `helm-overrides/<cluster>/<incubator-tool>/` for the trial cluster(s) only — typically `k8s-shared-int-ase1` (integration / pre-prod) first, sometimes one BU prod cluster as a canary.
|
||||
- Possibly raw sidecar manifests alongside the values file.
|
||||
|
||||
Once the tool graduates from incubator status, the override pattern is identical to other infra tools — the "incubator" label is operational, not structural. This schema codifies the conventions that keep early-stage adoption sane.
|
||||
|
||||
---
|
||||
|
||||
## Directory layout
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<incubator-tool>/
|
||||
├── custom-values.yaml # Helm values (required if chart is Helm-based)
|
||||
├── computeclass/ # Optional — GKE Autopilot ComputeClass
|
||||
│ └── <name>-cc.yaml
|
||||
├── external-dns-services/ # Optional — Service for external-dns annotation
|
||||
│ └── <fqdn>.yaml
|
||||
├── external-secrets/ # Optional — ExternalSecret resources
|
||||
│ └── <name>.yaml
|
||||
└── <incubator-specific>.yaml # Tool-specific raw manifest (CRD, ConfigMap)
|
||||
```
|
||||
|
||||
The Argo `Application` for this directory (sister repo) determines whether files are Helm-rendered, raw-applied, or layered (see [raw-manifest-sidecar-schema.md §Layered-with-Helm vs standalone](raw-manifest-sidecar-schema.md)).
|
||||
|
||||
---
|
||||
|
||||
## `custom-values.yaml` keys an incubator chart should expose
|
||||
|
||||
A chart promoted to incubator status MUST surface (i.e. let the override file set without forking templates) at minimum:
|
||||
|
||||
| Key | Why required |
|
||||
|-----|--------------|
|
||||
| `image.registry`, `image.repository`, `image.tag` | Production overrides pin to `asia-southeast1-docker.pkg.dev/meesho-devops-admin-0622/admin/sre/<image>` (Meesho Artifact Registry mirror, not Docker Hub). |
|
||||
| `image.pullPolicy` | Default `IfNotPresent`. |
|
||||
| `resources.requests.cpu`, `.memory` | No defaults assumed; Autopilot scheduling requires explicit requests. |
|
||||
| `resources.limits.cpu`, `.memory` | Same. |
|
||||
| `nodeSelector` | Per-cluster scheduling (see [custom-values-schema.md](custom-values-schema.md)). |
|
||||
| `tolerations` | Per-cluster taints. |
|
||||
| `affinity` (optional) | Pod anti-affinity for replicated tools. |
|
||||
| `serviceAccount.create`, `.name`, `.annotations` | Workload Identity — `iam.gke.io/gcp-service-account` annotation goes here. |
|
||||
| `replicaCount` (or `autoscaling.{enabled,minReplicas,maxReplicas}`) | Sizing. |
|
||||
| `persistence.enabled`, `.storageClass`, `.size` | If stateful. `storageClass` MUST exist in `manifests/storageclass/`. See [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md). |
|
||||
| `priorityClassName` | If the tool needs preemption priority — must reference a class in `manifests/priorityclass/<cluster>/`. |
|
||||
| `podLabels`, `podAnnotations` | For prometheus.io scrape annotations and team-attribution labels. |
|
||||
| `extraEnv` (or `env`) | For environment-specific knobs the chart's templates don't already accept. |
|
||||
|
||||
If the upstream chart doesn't expose these — that's a chart-bug. **Do not** fork the chart in `helm-templates/` to add them ([NEVER-DO list](../../../CLAUDE.md)). Either upstream-PR the chart or wrap with a small Meesho-owned chart that re-exposes the keys.
|
||||
|
||||
---
|
||||
|
||||
## Values-file conventions for incubator tools
|
||||
|
||||
1. **Pin every image tag.** Never `latest`, never an unpinned SHA. This is the same rule as production overrides; incubator status does not relax it.
|
||||
2. **Set explicit `nodeSelector` and `tolerations`** authored from scratch using sibling apps on the same cluster. Cross-cluster copying is forbidden — see [custom-values-schema.md](custom-values-schema.md).
|
||||
3. **Use `fullnameOverride` deliberately or not at all.** Once set, do not change ([NEVER-DO list](../../../CLAUDE.md)). Incubator graduations to other clusters should reuse the same `fullnameOverride` string for portability.
|
||||
4. **Don't enable `autoscaling`** in the first incubator deploy. Pin `replicaCount: 1` (or 2 for HA-mandatory) until you have a load profile.
|
||||
5. **Don't expose the tool externally** in the incubator phase. No `external-dns-services/` until it's promoted to a cluster's stable inventory.
|
||||
6. **Annotate the values file** with a top-of-file YAML comment: `# Incubator: cluster=<...>, owner=<...>, graduation-target=<date>`. Comment is plain text; not parsed; serves as reviewer signal.
|
||||
7. **Confine secrets to `ExternalSecret`** under `external-secrets/`. No inline `existingSecret:` referencing a hand-applied Secret.
|
||||
|
||||
---
|
||||
|
||||
## Raw sidecar manifests in incubator directories
|
||||
|
||||
Concrete shapes already documented elsewhere; this section calls out which ones routinely appear with incubators.
|
||||
|
||||
### `computeclass/*-cc.yaml` (GKE Autopilot only)
|
||||
|
||||
Required when the override's `nodeSelector` uses `cloud.google.com/compute-class: <name>` and that class doesn't already exist on the cluster. The `metadata.name` MUST equal the value referenced. See [raw-manifest-sidecar-schema.md §ComputeClass](raw-manifest-sidecar-schema.md).
|
||||
|
||||
Example incubator pattern:
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling.gke.io/v1
|
||||
kind: ComputeClass
|
||||
metadata:
|
||||
name: <incubator-tool>-cc
|
||||
namespace: <ns>
|
||||
spec:
|
||||
priorities:
|
||||
- machineFamily: c4d
|
||||
minCores: 2
|
||||
minMemory: 8Gi
|
||||
nodePoolAutoCreation:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
Per-cluster only — never copy between clusters.
|
||||
|
||||
### `external-dns-services/*.yaml`
|
||||
|
||||
Skip in the first incubator deploy. Only add once the tool has a stable internal-only DNS need. See [raw-manifest-sidecar-schema.md §Service for external-dns binding](raw-manifest-sidecar-schema.md).
|
||||
|
||||
### `external-secrets/*.yaml`
|
||||
|
||||
Always present if the tool needs secrets. Use the existing per-cluster `SecretStore` / `ClusterSecretStore`; do not author new stores in an incubator PR. See [raw-manifest-sidecar-schema.md §ExternalSecret](raw-manifest-sidecar-schema.md).
|
||||
|
||||
### `elastic-cluster/argo-launch.yaml`-style operator-managed CRD launches
|
||||
|
||||
If the incubator tool is an operator (e.g. ECK, Pyroscope), a sidecar CRD instance often lives alongside the operator's chart values:
|
||||
|
||||
```
|
||||
helm-overrides/<cluster>/<incubator-operator>/
|
||||
├── custom-values.yaml # operator chart values
|
||||
└── <crd-instance>/
|
||||
└── argo-launch.yaml # the actual workload CRD instance
|
||||
```
|
||||
|
||||
Argo CD applies both in one Application (directory loader). The CRD instance must:
|
||||
|
||||
- Reference an `apiVersion` whose CRD the operator has already installed.
|
||||
- Pin its own image / version explicitly.
|
||||
- Avoid hand-rolling values that the operator templates would otherwise compute.
|
||||
|
||||
---
|
||||
|
||||
## Argo CD — sister repo contract (for cross-reference only)
|
||||
|
||||
The Application that points at this directory lives in `github.com/Meesho/devops-infra-argo-config`. For incubator tools the Application typically:
|
||||
|
||||
- Has `syncPolicy.automated` **disabled** (manual sync — incubator-grade safety; see [argocd.md](../../global/coding-guidelines/argocd.md)).
|
||||
- Is named `<incubator-tool>-<cluster>` to be unambiguous.
|
||||
- Lives in the cluster's namespace under `apps/<cluster>/`.
|
||||
|
||||
This file does NOT instruct on authoring the Application — that PR is in the sister repo. Cross-link the sister-repo PR in the values-side PR description.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
Same as any Helm override. Before opening a PR:
|
||||
|
||||
```bash
|
||||
yamllint helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml
|
||||
|
||||
helm template <release> helm-templates/<incubator-tool> \
|
||||
-f helm-overrides/<cluster>/<incubator-tool>/custom-values.yaml > /tmp/render.yaml
|
||||
|
||||
# Validate any sidecar manifests against the cluster's CRDs
|
||||
kubectl --context=<ctx> --dry-run=server \
|
||||
-f helm-overrides/<cluster>/<incubator-tool>/<sidecar>.yaml apply
|
||||
```
|
||||
|
||||
For tool-specific CRDs that aren't in vanilla Kubernetes, prefer `--dry-run=server` (the cluster validates against the registered CRD) over `--dry-run=client` (only client-side schema, often misses required fields).
|
||||
|
||||
---
|
||||
|
||||
## Graduation: incubator → stable
|
||||
|
||||
When the tool has been stable for some period (usually 2–4 weeks) and is ready to fleet-roll:
|
||||
|
||||
1. Remove the top-of-file `# Incubator:` comment.
|
||||
2. Open new override directories in target clusters — author from scratch each time, do not copy.
|
||||
3. Open the matching sister-repo `Application` PRs (one per new cluster) or extend the `ApplicationSet`.
|
||||
4. If the chart was a thin wrapper, audit `Chart.yaml` `dependencies[].version` is current.
|
||||
5. Promote the chart's documentation in `helm-templates/<incubator-tool>/README.md` (if a fork was needed) or note in PR description that no fork was needed.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Forking the chart in `helm-templates/<incubator-tool>/templates/`** to add a missing values key. Wrap or upstream-PR; don't fork. See NEVER-DO.
|
||||
2. **Copying the entire incubator directory across clusters** for graduation. Per-cluster scheduling is bespoke.
|
||||
3. **Enabling autoscaling on day one.** No load profile = thrashy autoscaler.
|
||||
4. **Promoting from `k8s-shared-int-ase1` straight to a critical BU cluster.** Insert a low-traffic prod canary first.
|
||||
5. **Inlining secrets** "just for the trial." TruffleHog will block; even if it didn't, the leak is real.
|
||||
6. **Setting `fullnameOverride`** without a graduation plan. The string travels — bad ones travel forever.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Schema: [custom-values-schema.md](custom-values-schema.md) — general values shape.
|
||||
- Schema: [raw-manifest-sidecar-schema.md](raw-manifest-sidecar-schema.md) — sidecar shapes.
|
||||
- Schema: [storageclass-priorityclass-schema.md](storageclass-priorityclass-schema.md) — cluster-singleton resources.
|
||||
- Procedure: [../procedures/onboard-app-to-cluster.md](../procedures/onboard-app-to-cluster.md) — the procedure used to land an incubator override.
|
||||
- Procedure: [../procedures/fork-upstream-chart.md](../procedures/fork-upstream-chart.md) — only when a fork is genuinely required.
|
||||
- Coding guideline: [../../global/coding-guidelines/helm-values.md](../../global/coding-guidelines/helm-values.md).
|
||||
- Coding guideline: [../../global/coding-guidelines/argocd.md](../../global/coding-guidelines/argocd.md) — sister-repo Application conventions.
|
||||
@@ -0,0 +1,191 @@
|
||||
# Schema — Raw-manifest sidecars in `helm-overrides/`
|
||||
|
||||
> Field-by-field guidance for `helm-overrides/<cluster>/<app>/<extra>.yaml` — Kubernetes manifests applied **alongside** a Helm release, not through it.
|
||||
|
||||
---
|
||||
|
||||
## What this is
|
||||
|
||||
Some Argo CD Applications point at a directory containing both a Helm `custom-values.yaml` *and* one or more raw Kubernetes manifests. The raw manifests are not Helm-rendered; Argo CD applies them as-is.
|
||||
|
||||
Common shapes seen in this repo:
|
||||
|
||||
| Path pattern | What it is |
|
||||
|--------------|------------|
|
||||
| `helm-overrides/<cluster>/<app>/computeclass/*-cc.yaml` | GKE Autopilot `ComputeClass` resource — declares a node-pool/compute-class profile referenced by `nodeSelector` in the Helm values. |
|
||||
| `helm-overrides/<cluster>/<app>/external-dns-services/<svc>.yaml` | A `Service` that exists purely to carry an `external-dns.alpha.kubernetes.io/hostname` annotation, binding a DNS name to a workload. |
|
||||
| `helm-overrides/<cluster>/elastic-cluster/argo-launch.yaml` | An `ElasticCluster` (ECK CRD) launched alongside the operator. |
|
||||
| `helm-overrides/<cluster>/<app>/mimir-distributed/alertmanager_config.yaml` | Inlined Alertmanager config materialised as a `ConfigMap`. |
|
||||
|
||||
If the sister-repo `Application` for this directory has `path: helm-overrides/<cluster>/<app>/`, then **every YAML file in the directory is applied** — Argo CD's directory loader treats them as a single deployment unit.
|
||||
|
||||
---
|
||||
|
||||
## File-level conventions
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **One Kubernetes resource per file** unless they're tightly coupled (e.g. a `Service` + a `ServiceAccount` referenced by it). | Reviewer cognition; rollback granularity. |
|
||||
| **Pin `apiVersion` explicitly.** | `extensions/v1beta1` and `networking.k8s.io/v1beta1` are real footguns — both are gone in current Kubernetes. |
|
||||
| **Set `metadata.namespace` explicitly.** | Don't rely on the Argo `Application.spec.destination.namespace` carrying through — different cluster Argos behave differently here. |
|
||||
| **Trailing newline at EOF.** |
|
||||
| **No multi-doc (`---`) within one file** unless genuinely required. |
|
||||
|
||||
---
|
||||
|
||||
## Top-level shape
|
||||
|
||||
```yaml
|
||||
apiVersion: <api-version> # e.g. v1, networking.k8s.io/v1, autoscaling.gke.io/v1
|
||||
kind: <kind> # e.g. Service, ComputeClass, ConfigMap, ExternalSecret
|
||||
metadata:
|
||||
name: <name>
|
||||
namespace: <namespace>
|
||||
labels: {...} # optional
|
||||
annotations: {...} # often the load-bearing field (DNS, LB)
|
||||
spec:
|
||||
... # kind-specific
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common kinds in this repo
|
||||
|
||||
### `ComputeClass` (`autoscaling.gke.io/v1`)
|
||||
|
||||
GKE Autopilot uses `ComputeClass` resources to declare named node profiles. The Helm values reference them via `cloud.google.com/compute-class:` keys.
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling.gke.io/v1
|
||||
kind: ComputeClass
|
||||
metadata:
|
||||
name: contour-internal-0-cc
|
||||
namespace: projectcontour
|
||||
spec:
|
||||
priorities:
|
||||
- machineFamily: c4d
|
||||
minCores: 4
|
||||
minMemory: 16Gi
|
||||
nodePoolAutoCreation:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **`metadata.name` MUST match the value referenced by `nodeSelector` in the Helm values.** | Otherwise pods stay `Pending`. |
|
||||
| **Per-cluster.** A `ComputeClass` for `k8s-central-prd-ase1` does not exist on `k8s-supply-prd-ase1`. Don't share files. |
|
||||
| **The matrix of which `ComputeClass` exists where** is recorded in [contour-nodeselector-tolerations-summary.md](../../../contour-nodeselector-tolerations-summary.md). |
|
||||
|
||||
### `Service` for `external-dns` binding
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: <hostname>-dns
|
||||
namespace: <ns>
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: <fqdn>.meeshogcp.in
|
||||
cloud.google.com/load-balancer-type: Internal
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector: {app.kubernetes.io/name: <app>}
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 8080
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **`metadata.annotations.external-dns.alpha.kubernetes.io/hostname`** is the contract. Check the cluster's existing DNS pattern before authoring. |
|
||||
| **`selector` must match labels of an actual workload Pod** in the same namespace. |
|
||||
| **Cloud-LB annotations** (`cloud.google.com/load-balancer-type`, etc.) — copy from a sibling on the same cluster. |
|
||||
|
||||
### `ExternalSecret` (per-cluster `external-secrets/`)
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: <secret-name>
|
||||
namespace: <ns>
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: gcp-sm-store # cluster-level SecretStore, defined elsewhere
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: <secret-name>
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
- secretKey: <env-key>
|
||||
remoteRef:
|
||||
key: <gcp-secret-name>
|
||||
version: latest
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **`spec.target.name` is the `Secret` name** the workload's `existingSecret:` references. |
|
||||
| **`spec.secretStoreRef.name`** must reference a `SecretStore` / `ClusterSecretStore` that already exists on the cluster. |
|
||||
| **`creationPolicy: Owner`** is the default; `creationPolicy: Merge` is for the rare case where another tool also manages the same `Secret`. |
|
||||
|
||||
### `ConfigMap` for inlined config
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: <name>
|
||||
namespace: <ns>
|
||||
data:
|
||||
<key>: |
|
||||
<multi-line content>
|
||||
```
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Use `|` (literal block scalar)** for multi-line content with significant whitespace. |
|
||||
| **Don't inline secrets** (TruffleHog will catch obvious ones; subtle ones slip through). |
|
||||
| **Reference from the workload's chart values** via the `ConfigMap` name; don't duplicate config across files. |
|
||||
|
||||
### `ElasticCluster` / other CRDs
|
||||
|
||||
For ECK, Pyroscope, etc. — the schema is the operator's, not Kubernetes's. **Read the operator's CRD docs** before authoring; copy a sibling cluster's existing `argo-launch.yaml` first.
|
||||
|
||||
---
|
||||
|
||||
## Layered-with-Helm vs standalone
|
||||
|
||||
| Pattern | Argo Application points at |
|
||||
|---------|----------------------------|
|
||||
| Pure Helm | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` |
|
||||
| Pure raw manifests | `path: helm-overrides/<cluster>/<app>/` with `directory.recurse: true`, no `helm:` block |
|
||||
| Layered | `path: helm-overrides/<cluster>/<app>/` with `helm.valueFiles: ['custom-values.yaml']` AND extra files alongside |
|
||||
|
||||
Argo CD's behaviour for layered directories depends on its `directory.include` / `directory.exclude` settings — when in doubt, read the sister-repo `Application` to see exactly what gets picked up.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
# Validate the manifest against its API
|
||||
kubectl --dry-run=client -f helm-overrides/<cluster>/<app>/<extra>.yaml apply
|
||||
|
||||
# Lint
|
||||
yamllint helm-overrides/<cluster>/<app>/<extra>.yaml
|
||||
|
||||
# Confirm the Argo Application loads this path
|
||||
# (read the matching file in github.com/Meesho/devops-infra-argo-config)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Inlining secrets** in a `ConfigMap` "to ship a hotfix." It's still a secret. Use `ExternalSecret`.
|
||||
2. **Using `apiVersion: v1beta1`** of a CRD whose stable version exists. Pin to the highest stable.
|
||||
3. **A `Service` whose `selector` doesn't match any Pod** — silent: the Service exists, DNS resolves, no endpoints.
|
||||
4. **Cross-cluster cloning** of a `ComputeClass` or DNS `Service` without rewriting cluster-specific fields.
|
||||
5. **Hand-rendered Helm output** dropped into a sidecar file (a release "freeze"). The chart bumps and your snapshot rots.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Schema — `manifests/storageclass/` and `manifests/priorityclass/`
|
||||
|
||||
> Cluster-wide singletons. **High blast radius — every PVC / scheduling decision in the cluster is affected.**
|
||||
>
|
||||
> Cross-reference: [SANCTITY_RULES R10](../../global/SANCTITY_RULES.md), [coding-guidelines/helm-values.md §persistence](../../global/coding-guidelines/helm-values.md).
|
||||
|
||||
---
|
||||
|
||||
## §StorageClass (`manifests/storageclass/*.yaml`)
|
||||
|
||||
These are **repo-global** — one StorageClass file is applied to every cluster that consumes it. A wrong reclaim policy or volume-binding mode breaks every new PVC.
|
||||
|
||||
### Inventory
|
||||
|
||||
| File | Backend | Reclaim | Binding | Use case |
|
||||
|------|---------|---------|---------|----------|
|
||||
| `pd-standard-retain-dr.yaml` | GCP PD standard | `Retain` | `WaitForFirstConsumer` | DR-critical state — do not lose data on PVC delete |
|
||||
| `sc-pd-ssd.yaml` | GCP PD SSD | `Delete` | `WaitForFirstConsumer` | Default for write-heavy (etcd, ClickHouse) |
|
||||
| `sc-pd-standard.yaml` | GCP PD standard | `Delete` | `WaitForFirstConsumer` | Default for read-heavy / archival |
|
||||
| `sc-filestore-standard.yaml` | GCP Filestore | varies | varies | Shared `ReadWriteMany` volumes (Jenkins build cache, JFrog filestore) |
|
||||
|
||||
### Schema
|
||||
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: sc-pd-ssd # MUST match filename
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "false" # exactly one default per cluster
|
||||
provisioner: pd.csi.storage.gke.io
|
||||
parameters:
|
||||
type: pd-ssd
|
||||
reclaimPolicy: Delete # Delete | Retain
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
allowVolumeExpansion: true
|
||||
```
|
||||
|
||||
| Field | Convention | Notes |
|
||||
|-------|------------|-------|
|
||||
| `metadata.name` | matches filename (without `.yaml`) | Helm values reference by name. |
|
||||
| `metadata.annotations."storageclass.kubernetes.io/is-default-class"` | `"false"` for all of these | Exactly one StorageClass should be `"true"` per cluster (declared elsewhere, often by Terraform/cluster-bootstrap). |
|
||||
| `provisioner` | `pd.csi.storage.gke.io` (PD) or `filestore.csi.storage.gke.io` (Filestore) | GCP CSI drivers. |
|
||||
| `parameters.type` | `pd-ssd`, `pd-standard`, `pd-balanced` | Cost/perf trade-off. |
|
||||
| `reclaimPolicy` | `Delete` (default) or `Retain` (DR) | **Changing this on an existing StorageClass does not retroactively change PVCs.** |
|
||||
| `volumeBindingMode` | `WaitForFirstConsumer` | Avoids zone-mismatch on multi-zone clusters. `Immediate` only for `ReadWriteMany`. |
|
||||
| `allowVolumeExpansion` | `true` | Required for online resizing. Default false on some classes — set explicitly. |
|
||||
|
||||
### Hard rules
|
||||
|
||||
1. **Never delete a StorageClass referenced by an existing PVC.** New PVCs against the deleted class fail; existing bound PVCs survive but lose the ability to expand.
|
||||
2. **Never change `reclaimPolicy` from `Delete` → `Retain`** as a "safety improvement" without auditing every PVC. Existing bound PVs keep their old policy; new PVs get the new policy. Drift.
|
||||
3. **Never change `provisioner`** — that's a delete-and-recreate, not an edit. Existing PVs become orphaned.
|
||||
4. **Never make a new StorageClass the cluster default** without coordinating with the platform team. A wrong default class hijacks every PVC that doesn't pin a class explicitly.
|
||||
5. **Validation:** every StorageClass referenced by any `helm-overrides/*/persistence.storageClass:` must have a corresponding file here.
|
||||
|
||||
```bash
|
||||
# Find every PVC reference
|
||||
grep -rE 'storageClass(Name)?:' helm-overrides | sort -u
|
||||
|
||||
# Find every StorageClass file
|
||||
ls manifests/storageclass/*.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §PriorityClass (`manifests/priorityclass/<cluster>/*.yaml`)
|
||||
|
||||
**Per-cluster** but **cluster-wide** — affects scheduling priority for every pod that references the class.
|
||||
|
||||
### Inventory pattern
|
||||
|
||||
```
|
||||
manifests/priorityclass/
|
||||
k8s-central-prd-ase1/
|
||||
priorityclass-high.yaml
|
||||
priorityclass-low.yaml
|
||||
k8s-supply-prd-ase1/
|
||||
priorityclass-high.yaml
|
||||
priorityclass-low.yaml
|
||||
…
|
||||
```
|
||||
|
||||
Most BU clusters get a `high` and a `low` class. Specialty clusters (`mqkafka`, `dsgpu`, dataplane `db-*`) sometimes have additional tiers.
|
||||
|
||||
### Schema
|
||||
|
||||
```yaml
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: high-priority
|
||||
value: 1000000 # higher = more important
|
||||
globalDefault: false # NEVER true on these — would clash with system defaults
|
||||
description: "High-priority workloads — promoted ahead of general pods on resource pressure"
|
||||
preemptionPolicy: PreemptLowerPriority # default; alternative is Never
|
||||
```
|
||||
|
||||
| Field | Convention | Notes |
|
||||
|-------|------------|-------|
|
||||
| `metadata.name` | `high-priority`, `low-priority`, etc. | Workload pods reference by name. |
|
||||
| `value` | High: 1,000,000 / Low: 100 / Critical (rare): >1,000,000,000 | Kubernetes system pods reserve `> 2,000,000,000`; don't conflict. |
|
||||
| `globalDefault` | **always `false`** | A `true` here applies to every pod that doesn't pin a class — chaos. |
|
||||
| `description` | one-line | For audit log. |
|
||||
| `preemptionPolicy` | `PreemptLowerPriority` (default) or `Never` | `Never` for batch workloads that shouldn't kick others out. |
|
||||
|
||||
### Hard rules
|
||||
|
||||
1. **Never set `globalDefault: true`** on any PriorityClass here. The cluster's implicit default is what we want.
|
||||
2. **Never raise `value`** of an existing class without auditing what gets preempted. A bump from 1,000,000 → 10,000,000 changes which pods get evicted under pressure.
|
||||
3. **Never delete a PriorityClass** referenced by any workload — pods that referenced it become invalid.
|
||||
4. **The same `name` must mean the same thing across clusters.** `high-priority` on supply ≠ `high-priority` on demand at the *value* level is an audit nightmare. Keep values consistent.
|
||||
5. **Validation:** find every pod-spec reference:
|
||||
```bash
|
||||
grep -rE 'priorityClassName:' helm-overrides
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## §The "PV/PVC singletons" — Jenkins / JFrog filestore
|
||||
|
||||
Beyond StorageClass and PriorityClass, `manifests/` also holds **per-env, one-shot** PV/PVC pairs:
|
||||
|
||||
```
|
||||
manifests/jenkins-filestore-caching/
|
||||
dev/{pv,pvc}.yaml
|
||||
prd/{pv,pvc}.yaml
|
||||
manifests/jenkins-gcs-caching/{pv,pvc,sc-gcs}.yaml
|
||||
manifests/jfrog-filestore-data/
|
||||
dev/...
|
||||
prd/...
|
||||
```
|
||||
|
||||
These are **already-created PVs being adopted** — typically because the underlying disk (Filestore mount, GCS bucket) was provisioned by Terraform or by hand. The PV file binds the existing disk to the cluster; the PVC file binds a workload to the PV.
|
||||
|
||||
| Rule | Why |
|
||||
|------|-----|
|
||||
| **Don't change `spec.csi.volumeHandle` / `spec.gcePersistentDisk.pdName`** without coordinating with Terraform. The handle is the disk identity. |
|
||||
| **Don't change `spec.persistentVolumeReclaimPolicy`** from `Retain` to `Delete` on these. The disk has data on it. |
|
||||
| **Don't move dev→prd or vice versa** — they reference different physical disks. |
|
||||
| **Treat as platform-team review.** Any change here is a 1:1 disk operation. |
|
||||
|
||||
---
|
||||
|
||||
## Validation script
|
||||
|
||||
```bash
|
||||
# StorageClass: every name in helm-overrides has a file here
|
||||
for sc in $(grep -rhE 'storageClass(Name)?:' helm-overrides \
|
||||
| sed -E 's/.*storageClass(Name)?:\s*//' \
|
||||
| tr -d '"' | sort -u); do
|
||||
[ -f "manifests/storageclass/${sc}.yaml" ] || echo "MISSING: $sc"
|
||||
done
|
||||
|
||||
# PriorityClass: every name referenced has a file in the matching cluster
|
||||
grep -rE 'priorityClassName:' helm-overrides
|
||||
# (cross-reference manually against manifests/priorityclass/<cluster>/)
|
||||
```
|
||||
Reference in New Issue
Block a user