Keep three days of metrics rather than seven
Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every extra day is charged against the project's shared 250GB SSD quota. Three days still spans a weekend, which is the window that matters for working out what happened to something overnight. Worth being explicit about what this does not do, since it followed an OOM-kill and could easily be mistaken for the fix: retention has no bearing on the memory this needs. That tracks active time series and cache size, neither of which depends on how long data is kept — which is why the limit had to be raised rather than this lowered when it started OOM-killing on restart. Lowering this now would not let that be undone. The lever for memory, if it is ever needed, is scrape cardinality. Several targets here emit more than forty labels per series, which VictoriaMetrics warns about in its own log; dropping labels there reduces series count directly, and with it the memory. Old partitions are removed on VictoriaMetrics' own cycle rather than at startup, so the disk figure falls over the following hours, not immediately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
This commit is contained in:
co-authored by
Claude Opus 5
parent
f6c659dd4b
commit
df3e0a41af
@@ -1,9 +1,21 @@
|
|||||||
victoria-metrics-single:
|
victoria-metrics-single:
|
||||||
server:
|
server:
|
||||||
# 7 days rather than the chart's month: this cluster exists to prove a
|
# 3 days rather than the chart's month: this cluster exists to prove a
|
||||||
# pipeline, and every extra day is disk against the project's 250GB SSD
|
# pipeline, and every extra day is disk against the project's 250GB SSD
|
||||||
# quota, which every standard-rwo volume in the cluster shares.
|
# quota, which every standard-rwo volume in the cluster shares. Three
|
||||||
retentionPeriod: "7d"
|
# days still spans a weekend, which is the window that matters for
|
||||||
|
# working out what happened to something overnight.
|
||||||
|
#
|
||||||
|
# Down from 7d. Note what this does and does not do: it reclaims disk as
|
||||||
|
# old partitions are dropped, and shrinks the index a little, but it
|
||||||
|
# does NOT reduce the memory this needs to run. That tracks active time
|
||||||
|
# series and cache size, neither of which depends on how long data is
|
||||||
|
# kept — which is why the limit above had to be raised rather than this
|
||||||
|
# lowered when it started OOM-killing on restart. The lever for memory
|
||||||
|
# is scrape cardinality: several targets here carry more than forty
|
||||||
|
# labels per series (see the maxLabelsPerTimeseries warnings in its
|
||||||
|
# log), and dropping labels there would cut series count directly.
|
||||||
|
retentionPeriod: "3d"
|
||||||
|
|
||||||
persistentVolume:
|
persistentVolume:
|
||||||
storageClassName: standard-rwo
|
storageClassName: standard-rwo
|
||||||
|
|||||||
Reference in New Issue
Block a user