Mukul SharmaandClaude Opus 5
df3e0a41af
Disk, not memory. The volume sits at 1.4G of 2.9G at seven days, and every extra day is charged against the project's shared 250GB SSD quota. Three days still spans a weekend, which is the window that matters for working out what happened to something overnight. Worth being explicit about what this does not do, since it followed an OOM-kill and could easily be mistaken for the fix: retention has no bearing on the memory this needs. That tracks active time series and cache size, neither of which depends on how long data is kept — which is why the limit had to be raised rather than this lowered when it started OOM-killing on restart. Lowering this now would not let that be undone. The lever for memory, if it is ever needed, is scrape cardinality. Several targets here emit more than forty labels per series, which VictoriaMetrics warns about in its own log; dropping labels there reduces series count directly, and with it the memory. Old partitions are removed on VictoriaMetrics' own cycle rather than at startup, so the disk figure falls over the following hours, not immediately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
72 lines
3.2 KiB
YAML
72 lines
3.2 KiB
YAML
victoria-metrics-single:
|
|
server:
|
|
# 3 days rather than the chart's month: this cluster exists to prove a
|
|
# pipeline, and every extra day is disk against the project's 250GB SSD
|
|
# quota, which every standard-rwo volume in the cluster shares. Three
|
|
# days still spans a weekend, which is the window that matters for
|
|
# working out what happened to something overnight.
|
|
#
|
|
# Down from 7d. Note what this does and does not do: it reclaims disk as
|
|
# old partitions are dropped, and shrinks the index a little, but it
|
|
# does NOT reduce the memory this needs to run. That tracks active time
|
|
# series and cache size, neither of which depends on how long data is
|
|
# kept — which is why the limit above had to be raised rather than this
|
|
# lowered when it started OOM-killing on restart. The lever for memory
|
|
# is scrape cardinality: several targets here carry more than forty
|
|
# labels per series (see the maxLabelsPerTimeseries warnings in its
|
|
# log), and dropping labels there would cut series count directly.
|
|
retentionPeriod: "3d"
|
|
|
|
persistentVolume:
|
|
storageClassName: standard-rwo
|
|
# VictoriaMetrics' compression is why it replaced Prometheus here;
|
|
# this cluster's metric volume at 7 days fits well inside 3Gi. Unlike
|
|
# the homelab's local-path, this class can be expanded later.
|
|
size: 3Gi
|
|
|
|
# Raised from 128Mi/512Mi, which this no longer fits inside.
|
|
#
|
|
# The pod ran for weeks at the old limit and then OOM-killed in a loop
|
|
# the first time it was restarted — exit 137 roughly fifty seconds after
|
|
# a clean start, every time. Nothing had changed about its configuration;
|
|
# the dataset had simply grown into 3.14 billion rows, and the memory a
|
|
# restart needs to resume ingestion no longer fit. A long-lived pod can
|
|
# sit well past the limit it would need to start again, so the failure
|
|
# only appears the next time something restarts it.
|
|
#
|
|
# Memory here tracks active time series rather than disk, which is why
|
|
# shortening retentionPeriod above would not have helped: the scrape
|
|
# targets are the same either way, and several of them carry 40+ labels
|
|
# (see the maxLabelsPerTimeseries warnings in its log).
|
|
#
|
|
# Affordable: memory requests across the three nodes sit at 62%, 18% and
|
|
# 47%, so there is room. CPU is the constrained resource on this cluster,
|
|
# and this costs none.
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 1Gi
|
|
|
|
# vmui, VictoriaMetrics' built-in query UI, on the same pod and port —
|
|
# ad-hoc PromQL only, no saved dashboards; Grafana is the real UI. Free
|
|
# to expose, since it is not a separate component.
|
|
ingress:
|
|
enabled: true
|
|
ingressClassName: contour
|
|
annotations:
|
|
# Certificate for the real domain only; nip.io cannot have one.
|
|
cert-manager.io/cluster-issuer: letsencrypt-prod
|
|
hosts:
|
|
- name: vm.35.238.248.203.nip.io
|
|
path: ["/"]
|
|
port: http
|
|
- name: vm.infra.deployshed.com
|
|
path: ["/"]
|
|
port: http
|
|
tls:
|
|
- secretName: vm-tls
|
|
hosts:
|
|
- vm.infra.deployshed.com
|