Keep three days of metrics rather than seven

Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every
extra day is charged against the project's shared 250GB SSD quota. Three
days still spans a weekend, which is the window that matters for working
out what happened to something overnight.

Worth being explicit about what this does not do, since it followed an
OOM-kill and could easily be mistaken for the fix: retention has no
bearing on the memory this needs. That tracks active time series and cache
size, neither of which depends on how long data is kept — which is why the
limit had to be raised rather than this lowered when it started
OOM-killing on restart. Lowering this now would not let that be undone.

The lever for memory, if it is ever needed, is scrape cardinality. Several
targets here emit more than forty labels per series, which VictoriaMetrics
warns about in its own log; dropping labels there reduces series count
directly, and with it the memory.

Old partitions are removed on VictoriaMetrics' own cycle rather than at
startup, so the disk figure falls over the following hours, not
immediately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
This commit is contained in:
Mukul Sharma
2026-09-17 08:40:19 +05:30
co-authored by Claude Opus 5
parent f6c659dd4b
commit df3e0a41af
@@ -1,9 +1,21 @@
victoria-metrics-single:
server:
# 7 days rather than the chart's month: this cluster exists to prove a
# 3 days rather than the chart's month: this cluster exists to prove a
# pipeline, and every extra day is disk against the project's 250GB SSD
# quota, which every standard-rwo volume in the cluster shares.
retentionPeriod: "7d"
# quota, which every standard-rwo volume in the cluster shares. Three
# days still spans a weekend, which is the window that matters for
# working out what happened to something overnight.
#
# Down from 7d. Note what this does and does not do: it reclaims disk as
# old partitions are dropped, and shrinks the index a little, but it
# does NOT reduce the memory this needs to run. That tracks active time
# series and cache size, neither of which depends on how long data is
# kept — which is why the limit above had to be raised rather than this
# lowered when it started OOM-killing on restart. The lever for memory
# is scrape cardinality: several targets here carry more than forty
# labels per series (see the maxLabelsPerTimeseries warnings in its
# log), and dropping labels there would cut series count directly.
retentionPeriod: "3d"
persistentVolume:
storageClassName: standard-rwo