added repo
This commit is contained in:
Your Name
@@ -0,0 +1,227 @@
|
||||
> Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.
|
||||
|
||||
# Procedure — Modify Prometheus / VictoriaMetrics alert rules
|
||||
|
||||
> **Layer:** Layer 1 — values diff + PR.
|
||||
> **Blast radius:** PagerDuty / on-call paging across the cluster's tenants. A wrong threshold pages everyone; a missing rule masks an outage.
|
||||
> **Approval:** observability owner + on-call lead for the affected severity.
|
||||
|
||||
This procedure is the narrow alert-rule slice of [modify-observability-config.md](modify-observability-config.md). Use this when the entire intent of the PR is alert-rule editing — adding, removing, retuning thresholds, changing `for:` windows, or rewriting `expr:` PromQL.
|
||||
|
||||
---
|
||||
|
||||
## Where alert rules live
|
||||
|
||||
| File | Engine | Notes |
|
||||
|------|--------|-------|
|
||||
| `helm-overrides/<cluster>/vmalert/custom-values.yaml` | vmalert | Most clusters use vmalert against VM cluster as the primary alerting engine. Rules typically under `config.rules` or rendered as a `VMRule` CRD. |
|
||||
| `helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml` | vmalert (bundled) / VM ruler | Some clusters carry rules in the cluster chart's `vmalert.config` block. |
|
||||
| `helm-overrides/<cluster>/prometheus/custom-values.yaml` | Prometheus / `kube-prometheus-stack` | Rules under `additionalPrometheusRulesMap` or `serverFiles.alerting_rules.yml`. |
|
||||
| `helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml` | Prometheus | Same as above; older clusters use this chart name. |
|
||||
| `helm-overrides/<cluster>/mimir/custom-values.yaml` | Mimir ruler | Rules go through the ruler API; YAML in `runtimeConfig` or via a sidecar `ConfigMap`. |
|
||||
|
||||
Confirm which cluster runs which engine before editing — sample existing rules in the override.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
- Adding/removing one or more alert rules in the cluster's primary alerting engine.
|
||||
- Tuning `expr:` PromQL or `for:` windows on existing rules.
|
||||
- Re-routing alerts (changing `severity`, `team`, or other routing labels).
|
||||
- Adding/removing recording rules for downstream alert efficiency.
|
||||
|
||||
Do **not** use this procedure for:
|
||||
|
||||
- Touching scrape configs, retention, datasources, or sizing in the same PR — split it. See [modify-observability-config.md](modify-observability-config.md).
|
||||
- Bumping the alerting chart's version — see [update-chart-version.md](update-chart-version.md).
|
||||
- Writing app-specific alert rules that belong in the consuming service's own values — those go in the service's repo.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Example |
|
||||
|-------|---------|
|
||||
| Target cluster | `k8s-supply-prd-ase1` |
|
||||
| Target chart | `vmalert` / `prometheus` / `kube-prometheus-stack` / `mimir` |
|
||||
| Rule name(s) added/changed | `HighErrorRate`, `KafkaConsumerLag` |
|
||||
| `expr:` PromQL | `sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 0.05` |
|
||||
| `for:` window | `5m` |
|
||||
| Severity / routing labels | `severity: critical`, `team: supply-platform` |
|
||||
| Approval ticket | CMR-… |
|
||||
|
||||
---
|
||||
|
||||
## Pre-conditions
|
||||
|
||||
- [ ] You have `promtool` and/or `vmalert` available locally.
|
||||
- [ ] You know the cluster's Alertmanager / vmalert notifier routing (which `severity` / `team` label routes where).
|
||||
- [ ] You have run the new PromQL expression against the cluster's VM/Prometheus to confirm it returns sensible values *before* writing the rule — open Grafana Explore.
|
||||
- [ ] You have on-call sign-off if this rule pages on-call.
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Locate the rules block
|
||||
|
||||
```bash
|
||||
yq e '.config.rules // .additionalPrometheusRulesMap // .serverFiles' \
|
||||
helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
Identify the YAML path the engine expects:
|
||||
|
||||
- **vmalert**: `config.groups[].rules[]`.
|
||||
- **kube-prometheus-stack**: `additionalPrometheusRulesMap.<group-name>.groups[].rules[]`.
|
||||
- **prometheus**: `serverFiles.alerting_rules.yml.groups[].rules[]`.
|
||||
- **mimir ruler**: per-tenant config — confirm the cluster's tenancy setup.
|
||||
|
||||
### 2. Author the rule
|
||||
|
||||
```yaml
|
||||
- alert: <RuleName> # PascalCase, no spaces; surfaces in pages and dashboards
|
||||
expr: |
|
||||
<PromQL expression>
|
||||
for: <duration> # e.g. 5m, 10m
|
||||
labels:
|
||||
severity: <warning|critical|info>
|
||||
team: <owning-team> # routes to the right PagerDuty service
|
||||
annotations:
|
||||
summary: "<one-line>"
|
||||
description: "<longer text, can reference $labels.* and $value>"
|
||||
runbook_url: <link to runbook>
|
||||
```
|
||||
|
||||
#### Threshold discipline
|
||||
|
||||
- Express thresholds as `rate(...) > 0.05` rather than `> 5%`. Ensure units in the expression match the units in `summary` / `description`.
|
||||
- Avoid `for:` windows shorter than the scrape interval × 2. Sub-`1m` windows are flap factories.
|
||||
- Avoid alerting on absolute counts (`sum(http_requests_total) > 1000`); prefer rates / ratios.
|
||||
- For SLO-style alerts, use multi-window multi-burn-rate (Google SRE workbook). Single-window threshold alerts are a known anti-pattern.
|
||||
|
||||
#### Routing implications
|
||||
|
||||
Changing `severity` from `warning` to `critical` (or vice versa) **silently re-routes the page** — the rule may now wake on-call where it previously emailed (or vice versa). Same for `team:` — a typo here drops alerts into the wrong PagerDuty service. Cross-check the cluster's Alertmanager / vmalert routing config:
|
||||
|
||||
```bash
|
||||
yq e '.config.route // .alertmanagerConfig' \
|
||||
helm-overrides/<cluster>/<alertmanager-chart>/custom-values.yaml
|
||||
```
|
||||
|
||||
### 3. Render the chart
|
||||
|
||||
```bash
|
||||
helm template <release> helm-templates/<chart> \
|
||||
-f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml
|
||||
```
|
||||
|
||||
### 4. Validate PromQL
|
||||
|
||||
For Prometheus / `kube-prometheus-stack`:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
|
||||
promtool check rules /tmp/rules.yaml
|
||||
```
|
||||
|
||||
For vmalert:
|
||||
|
||||
```bash
|
||||
yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
|
||||
vmalert -dryRun -rule=/tmp/vmrules.yaml
|
||||
```
|
||||
|
||||
For Mimir ruler — `mimirtool rules check` against the rendered ruler config.
|
||||
|
||||
If validation fails: fix the PromQL. **Do not** merge a rule that fails parse.
|
||||
|
||||
### 5. Sanity-check expressions in Grafana Explore
|
||||
|
||||
In a non-production manner: open Grafana on the cluster, paste the `expr:` into Explore, run over the last 6h. Confirm:
|
||||
|
||||
- The series returned makes sense for the rule.
|
||||
- The threshold isn't always-firing or never-firing in normal operation.
|
||||
- Empty result is allowed (`absent_over_time(...)` style).
|
||||
|
||||
### 6. Open the PR
|
||||
|
||||
```bash
|
||||
git checkout -b alerts/<cluster>-<rule-name>
|
||||
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
|
||||
git commit -m "alerts(<cluster>/<chart>): <add|tune|remove> <RuleName>"
|
||||
git push origin alerts/<cluster>-<rule-name>
|
||||
gh pr create --base main
|
||||
```
|
||||
|
||||
### PR description template
|
||||
|
||||
```markdown
|
||||
## Summary
|
||||
<add|tune|remove> alert rule(s) on `<cluster>` × `<chart>`.
|
||||
|
||||
## Rules touched
|
||||
| Rule | Change | New expr | New `for:` | Severity | Team |
|
||||
|------|--------|----------|-----------|----------|------|
|
||||
| `<RuleName>` | added | `<expr>` | `5m` | `critical` | `supply-platform` |
|
||||
|
||||
## Validation
|
||||
- [ ] `promtool check rules` / `vmalert -dryRun` passed
|
||||
- [ ] PromQL sanity-checked in Grafana Explore over last 6h
|
||||
- [ ] Threshold appropriate (no flap risk, no always-firing)
|
||||
- [ ] Routing labels (`severity`, `team`) verified against Alertmanager/vmalert routes
|
||||
- [ ] Runbook URL added (`annotations.runbook_url`)
|
||||
|
||||
## Approvers
|
||||
- Observability owner: <handle>
|
||||
- On-call lead (severity=<x>): <handle>
|
||||
```
|
||||
|
||||
### 7. After merge — Sync and observe
|
||||
|
||||
Sync the Argo Application. Watch:
|
||||
|
||||
```bash
|
||||
# vmalert
|
||||
kubectl --context=<ctx> -n monitoring logs deploy/vmalert | grep -i "<RuleName>\|error"
|
||||
|
||||
# kube-prometheus-stack
|
||||
kubectl --context=<ctx> -n monitoring logs prometheus-k8s-0 | grep -i "<RuleName>\|error"
|
||||
```
|
||||
|
||||
Open the alerting engine's UI (vmalert UI / Prometheus `/alerts`) to confirm the rule loaded.
|
||||
|
||||
### Watch for false positives
|
||||
|
||||
Stay on the alert for at least one full `for:` window after sync. If the rule pages immediately and was not expected to → revert.
|
||||
|
||||
---
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
1. **Sub-1m `for:` windows** — flap-prone.
|
||||
2. **Alerting on absolute counts** instead of rates/ratios.
|
||||
3. **`severity` flips without on-call sign-off** — silent re-route.
|
||||
4. **Missing `runbook_url`** — pages without remediation steps waste on-call cycles.
|
||||
5. **PromQL that uses `{job=~"foo.*"}` regex without anchors** — slow query, sometimes false matches.
|
||||
6. **Inlining the rule body in the chart `templates/`** to "make it sticky" — that's a chart fork; see NEVER-DO.
|
||||
7. **Adding the rule on every cluster in one PR** — surgical only.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert the PR. Sync.
|
||||
- vmalert/Prometheus reload picks up the previous ruleset within seconds.
|
||||
- Any in-flight alerts resolve on next evaluation.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- Coding guideline: [../../global/coding-guidelines/observability.md](../../global/coding-guidelines/observability.md).
|
||||
- Procedure: [modify-observability-config.md](modify-observability-config.md) — broader observability edits.
|
||||
- Runbook: [../runbooks/metrics-gap.md](../runbooks/metrics-gap.md) — when an alert is silent because the metric is gone.
|
||||
- Schema: [../schemas/custom-values-schema.md](../schemas/custom-values-schema.md).
|
||||
- Escalation: [../../global/escalation-matrix.md](../../global/escalation-matrix.md).
|
||||
Reference in New Issue
Block a user