From df3e0a41afcb745a2ed87401cca7a114e491c272 Mon Sep 17 00:00:00 2001 From: Mukul Sharma Date: Thu, 17 Sep 2026 08:40:19 +0530 Subject: [PATCH] Keep three days of metrics rather than seven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every extra day is charged against the project's shared 250GB SSD quota. Three days still spans a weekend, which is the window that matters for working out what happened to something overnight. Worth being explicit about what this does not do, since it followed an OOM-kill and could easily be mistaken for the fix: retention has no bearing on the memory this needs. That tracks active time series and cache size, neither of which depends on how long data is kept — which is why the limit had to be raised rather than this lowered when it started OOM-killing on restart. Lowering this now would not let that be undone. The lever for memory, if it is ever needed, is scrape cardinality. Several targets here emit more than forty labels per series, which VictoriaMetrics warns about in its own log; dropping labels there reduces series count directly, and with it the memory. Old partitions are removed on VictoriaMetrics' own cycle rather than at startup, so the disk figure falls over the following hours, not immediately. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N --- .../victoria-metrics-single/custom-values.yaml | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/helm-overrides/gke-toolshed-prd-usc1/victoria-metrics-single/custom-values.yaml b/helm-overrides/gke-toolshed-prd-usc1/victoria-metrics-single/custom-values.yaml index 2875db8..7b389d9 100644 --- a/helm-overrides/gke-toolshed-prd-usc1/victoria-metrics-single/custom-values.yaml +++ b/helm-overrides/gke-toolshed-prd-usc1/victoria-metrics-single/custom-values.yaml @@ -1,9 +1,21 @@ victoria-metrics-single: server: - # 7 days rather than the chart's month: this cluster exists to prove a + # 3 days rather than the chart's month: this cluster exists to prove a # pipeline, and every extra day is disk against the project's 250GB SSD - # quota, which every standard-rwo volume in the cluster shares. - retentionPeriod: "7d" + # quota, which every standard-rwo volume in the cluster shares. Three + # days still spans a weekend, which is the window that matters for + # working out what happened to something overnight. + # + # Down from 7d. Note what this does and does not do: it reclaims disk as + # old partitions are dropped, and shrinks the index a little, but it + # does NOT reduce the memory this needs to run. That tracks active time + # series and cache size, neither of which depends on how long data is + # kept — which is why the limit above had to be raised rather than this + # lowered when it started OOM-killing on restart. The lever for memory + # is scrape cardinality: several targets here carry more than forty + # labels per series (see the maxLabelsPerTimeseries warnings in its + # log), and dropping labels there would cut series count directly. + retentionPeriod: "3d" persistentVolume: storageClassName: standard-rwo