Files
devops-infra-helm-charts-gcp/helm-overrides/gke-toolshed-prd-usc1
Mukul SharmaandClaude Opus 5 df3e0a41af Keep three days of metrics rather than seven
Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every
extra day is charged against the project's shared 250GB SSD quota. Three
days still spans a weekend, which is the window that matters for working
out what happened to something overnight.

Worth being explicit about what this does not do, since it followed an
OOM-kill and could easily be mistaken for the fix: retention has no
bearing on the memory this needs. That tracks active time series and cache
size, neither of which depends on how long data is kept — which is why the
limit had to be raised rather than this lowered when it started
OOM-killing on restart. Lowering this now would not let that be undone.

The lever for memory, if it is ever needed, is scrape cardinality. Several
targets here emit more than forty labels per series, which VictoriaMetrics
warns about in its own log; dropping labels there reduces series count
directly, and with it the memory.

Old partitions are removed on VictoriaMetrics' own cycle rather than at
startup, so the disk figure falls over the following hours, not
immediately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
2026-09-17 08:40:19 +05:30
..