It had run for weeks at 128Mi/512Mi and then OOM-killed in a loop the
first time anything restarted it — exit 137 about fifty seconds after a
clean start, repeatedly. Nothing about its configuration had changed. The
dataset had grown to 3.14 billion rows, and the memory needed to resume
ingestion no longer fit in the limit.
That failure mode is worth naming: a long-lived pod can sit comfortably
past the limit it would need in order to start again, so the problem stays
invisible until something restarts it — here, an unrelated sync adding a
hostname. The limit was not wrong when it was written; it was outgrown.
Memory tracks active time series rather than disk, so shortening
retentionPeriod would not have helped — the same targets are scraped
either way, and several carry more than forty labels.
Affordable: memory requests across the three nodes sit at 62%, 18% and
47%. CPU is what is scarce here, and this costs none.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEsTefWWifp4ikvhHF5s6N
