devops-infra-helm-charts-gcp/docs/platform/procedures/modify-alert-rules.md

Per AI Blitz Plan §platform.procedures. Layer: 1. Repo: devops-infra-helm-charts.

Procedure — Modify Prometheus / VictoriaMetrics alert rules

Layer: Layer 1 — values diff + PR. Blast radius: PagerDuty / on-call paging across the cluster's tenants. A wrong threshold pages everyone; a missing rule masks an outage. Approval: observability owner + on-call lead for the affected severity.

This procedure is the narrow alert-rule slice of modify-observability-config.md. Use this when the entire intent of the PR is alert-rule editing — adding, removing, retuning thresholds, changing for: windows, or rewriting expr: PromQL.

---

Where alert rules live

File	Engine	Notes
helm-overrides/<cluster>/vmalert/custom-values.yaml	vmalert	Most clusters use vmalert against VM cluster as the primary alerting engine. Rules typically under config.rules or rendered as a VMRule CRD.
helm-overrides/<cluster>/victoria-metrics-cluster/custom-values.yaml	vmalert (bundled) / VM ruler	Some clusters carry rules in the cluster chart's vmalert.config block.
helm-overrides/<cluster>/prometheus/custom-values.yaml	Prometheus / kube-prometheus-stack	Rules under additionalPrometheusRulesMap or serverFiles.alerting_rules.yml.
helm-overrides/<cluster>/kube-prometheus-stack/custom-values.yaml	Prometheus	Same as above; older clusters use this chart name.
helm-overrides/<cluster>/mimir/custom-values.yaml	Mimir ruler	Rules go through the ruler API; YAML in runtimeConfig or via a sidecar ConfigMap.

Confirm which cluster runs which engine before editing — sample existing rules in the override.

---

When to use

• Adding/removing one or more alert rules in the cluster's primary alerting engine.
• Tuning expr: PromQL or for: windows on existing rules.
• Re-routing alerts (changing severity, team, or other routing labels).
• Adding/removing recording rules for downstream alert efficiency.

Do not use this procedure for:

• Touching scrape configs, retention, datasources, or sizing in the same PR — split it. See modify-observability-config.md.
• Bumping the alerting chart's version — see update-chart-version.md.
• Writing app-specific alert rules that belong in the consuming service's own values — those go in the service's repo.

---

Inputs

Input	Example
Target cluster	k8s-supply-prd-ase1
Target chart	vmalert / prometheus / kube-prometheus-stack / mimir
Rule name(s) added/changed	HighErrorRate, KafkaConsumerLag
expr: PromQL	sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 0.05
for: window	5m
Severity / routing labels	severity: critical, team: supply-platform
Approval ticket	CMR-…

---

Pre-conditions

• You have promtool and/or vmalert available locally.
• You know the cluster's Alertmanager / vmalert notifier routing (which severity / team label routes where).
• You have run the new PromQL expression against the cluster's VM/Prometheus to confirm it returns sensible values before writing the rule — open Grafana Explore.
• You have on-call sign-off if this rule pages on-call.

---

Steps

1. Locate the rules block

yq e '.config.rules // .additionalPrometheusRulesMap // .serverFiles' \
  helm-overrides/<cluster>/<chart>/custom-values.yaml

Identify the YAML path the engine expects:

• vmalert: config.groups[].rules[].
• kube-prometheus-stack: additionalPrometheusRulesMap.<group-name>.groups[].rules[].
• prometheus: serverFiles.alerting_rules.yml.groups[].rules[].
• mimir ruler: per-tenant config — confirm the cluster's tenancy setup.

2. Author the rule

- alert: <RuleName>             # PascalCase, no spaces; surfaces in pages and dashboards
  expr: |
    <PromQL expression>
  for: <duration>               # e.g. 5m, 10m
  labels:
    severity: <warning|critical|info>
    team: <owning-team>         # routes to the right PagerDuty service
  annotations:
    summary: "<one-line>"
    description: "<longer text, can reference $labels.* and $value>"
    runbook_url: <link to runbook>

Threshold discipline

• Express thresholds as rate(...) > 0.05 rather than > 5%. Ensure units in the expression match the units in summary / description.
• Avoid for: windows shorter than the scrape interval × 2. Sub-1m windows are flap factories.
• Avoid alerting on absolute counts (sum(http_requests_total) > 1000); prefer rates / ratios.
• For SLO-style alerts, use multi-window multi-burn-rate (Google SRE workbook). Single-window threshold alerts are a known anti-pattern.

Routing implications

Changing severity from warning to critical (or vice versa) silently re-routes the page — the rule may now wake on-call where it previously emailed (or vice versa). Same for team: — a typo here drops alerts into the wrong PagerDuty service. Cross-check the cluster's Alertmanager / vmalert routing config:

yq e '.config.route // .alertmanagerConfig' \
  helm-overrides/<cluster>/<alertmanager-chart>/custom-values.yaml

3. Render the chart

helm template <release> helm-templates/<chart> \
  -f helm-overrides/<cluster>/<chart>/custom-values.yaml > /tmp/rendered.yaml

4. Validate PromQL

For Prometheus / kube-prometheus-stack:

yq e 'select(.kind == "PrometheusRule")' /tmp/rendered.yaml > /tmp/rules.yaml
promtool check rules /tmp/rules.yaml

For vmalert:

yq e 'select(.kind == "VMRule" or .kind == "ConfigMap")' /tmp/rendered.yaml > /tmp/vmrules.yaml
vmalert -dryRun -rule=/tmp/vmrules.yaml

For Mimir ruler — mimirtool rules check against the rendered ruler config.

If validation fails: fix the PromQL. Do not merge a rule that fails parse.

5. Sanity-check expressions in Grafana Explore

In a non-production manner: open Grafana on the cluster, paste the expr: into Explore, run over the last 6h. Confirm:

• The series returned makes sense for the rule.
• The threshold isn't always-firing or never-firing in normal operation.
• Empty result is allowed (absent_over_time(...) style).

6. Open the PR

git checkout -b alerts/<cluster>-<rule-name>
git add helm-overrides/<cluster>/<chart>/custom-values.yaml
git commit -m "alerts(<cluster>/<chart>): <add|tune|remove> <RuleName>"
git push origin alerts/<cluster>-<rule-name>
gh pr create --base main

PR description template

## Summary
<add|tune|remove> alert rule(s) on `<cluster>` × `<chart>`.

## Rules touched
| Rule | Change | New expr | New `for:` | Severity | Team |
|------|--------|----------|-----------|----------|------|
| `<RuleName>` | added | `<expr>` | `5m` | `critical` | `supply-platform` |

## Validation
- [ ] `promtool check rules` / `vmalert -dryRun` passed
- [ ] PromQL sanity-checked in Grafana Explore over last 6h
- [ ] Threshold appropriate (no flap risk, no always-firing)
- [ ] Routing labels (`severity`, `team`) verified against Alertmanager/vmalert routes
- [ ] Runbook URL added (`annotations.runbook_url`)

## Approvers
- Observability owner: <handle>
- On-call lead (severity=<x>): <handle>

7. After merge — Sync and observe

Sync the Argo Application. Watch:

# vmalert
kubectl --context=<ctx> -n monitoring logs deploy/vmalert | grep -i "<RuleName>\|error"

# kube-prometheus-stack
kubectl --context=<ctx> -n monitoring logs prometheus-k8s-0 | grep -i "<RuleName>\|error"

Open the alerting engine's UI (vmalert UI / Prometheus /alerts) to confirm the rule loaded.

Watch for false positives

Stay on the alert for at least one full for: window after sync. If the rule pages immediately and was not expected to → revert.

---

Anti-patterns

1. Sub-1m for: windows — flap-prone.
2. Alerting on absolute counts instead of rates/ratios.
3. severity flips without on-call sign-off — silent re-route.
4. Missing runbook_url — pages without remediation steps waste on-call cycles.
5. PromQL that uses {job=~"foo.*"} regex without anchors — slow query, sometimes false matches.
6. Inlining the rule body in the chart templates/ to "make it sticky" — that's a chart fork; see NEVER-DO.
7. Adding the rule on every cluster in one PR — surgical only.

---

Rollback

• Revert the PR. Sync.
• vmalert/Prometheus reload picks up the previous ruleset within seconds.
• Any in-flight alerts resolve on next evaluation.

---

Related

• Coding guideline: ../../global/coding-guidelines/observability.md.
• Procedure: modify-observability-config.md — broader observability edits.
• Runbook: ../runbooks/metrics-gap.md — when an alert is silent because the metric is gone.
• Schema: ../schemas/custom-values-schema.md.
• Escalation: ../../global/escalation-matrix.md.