ADR-0009: JVM Heap Auto-Derived from Pod memory_request

Status: Accepted Category: RELIABILITY Date decided: Mid-project Date documented: 2026-05-12

Context

Java services running in Kubernetes pods are subject to two memory limits: the pod's memory_limit (enforced by the kubelet — exceed it and the pod is OOM-killed) and the JVM's heap size (-Xmx). By default, the JVM sets heap to 1/4 of the physical RAM it detects — but inside a container, it detects the node's physical RAM, not the pod's memory limit. A Java service in a pod with memory_limit: 2Gi running on a 64Gi node would default to a 16Gi heap, far exceeding its limit and triggering immediate OOM kill.

Meesho had multiple incidents where Java services were OOM-killed because:

The JVM was using the wrong default (node RAM, not pod limit)
Teams were setting -Xmx manually but forgetting to update it when memory_request changed
Teams were setting -Xmx too high, causing heap to exceed the pod limit

Decision

deployArgoCD.groovy automatically computes xms and xmx from the service's memory_request value in deployment.yaml. The derived values are injected into the Helm chart at deploy time. Services do not need to set -Xmx in JAVA_OPTS manually. The escape hatch jvm_memory_override: true in deployment.yaml allows a service to opt out and set its own JVM flags.

Alternatives Considered

Require teams to set -Xmx manually: Rejected — teams consistently forgot to update -Xmx when changing memory_request, causing OOM kills after pod resource changes. This happened across multiple services.
JVM container awareness flag (-XX:+UseContainerSupport): This flag (available in JDK 11+) allows the JVM to read the cgroup limit instead of physical RAM. Not adopted as the primary solution because it requires all services to use JDK 11+ and the flag needs to be explicitly set in each service's startup config — still a per-service manual step.
Fixed default heap values: Rejected — services have wildly different memory requirements; a fixed default would be wrong for most.

Consequences

Positive:

Java services cannot be OOM-killed due to JVM heap misconfiguration — the heap is always proportional to the pod's actual memory allocation.
Teams never need to update -Xmx manually when changing memory_request — the library keeps them in sync automatically.
Eliminates a whole class of incident: "service OOM-killed because someone bumped memory_request but forgot to update -Xmx."

Negative:

The auto-derived heap may not be optimal for services with unusual heap vs non-heap memory ratios (e.g., services with large off-heap caches). These services must use jvm_memory_override: true.
The calculation logic is not immediately transparent to service teams — they may not know why their -Xmx is what it is.

Neutral:

If deployment_args already contains an -Xmx or -Xms flag, the auto-derive reads and preserves those values rather than overwriting them. jvm_memory_override is the clean opt-out for services that need full control.

Constraints

Repeated OOM incidents across multiple Java services drove this decision. The library-level fix was preferred over per-service remediation because the root cause was systemic (wrong JVM defaults in containers) and would recur as long as teams configured heap manually.

Current Assessment

Still appropriate — no changes needed.

Notes

Key file: src/com/meesho/stages/deployArgoCD.groovy:205-268 — xms/xmx calculation
Opt-out: set jvm_memory_override: true in deployment.yaml to manage JVM flags manually
Documented in CLAUDE.md: "JVM memory: deployArgoCD.groovy auto-calculates xmx/xms from pod memory_request."

3.7 KiB Raw Blame History