54 lines
3.7 KiB
Markdown
54 lines
3.7 KiB
Markdown
# ADR-0009: JVM Heap Auto-Derived from Pod memory_request
|
|
|
|
**Status:** Accepted
|
|
**Category:** RELIABILITY
|
|
**Date decided:** Mid-project
|
|
**Date documented:** 2026-05-12
|
|
|
|
## Context
|
|
|
|
Java services running in Kubernetes pods are subject to two memory limits: the pod's `memory_limit` (enforced by the kubelet — exceed it and the pod is OOM-killed) and the JVM's heap size (`-Xmx`). By default, the JVM sets heap to 1/4 of the physical RAM it detects — but inside a container, it detects the node's physical RAM, not the pod's memory limit. A Java service in a pod with `memory_limit: 2Gi` running on a 64Gi node would default to a 16Gi heap, far exceeding its limit and triggering immediate OOM kill.
|
|
|
|
Meesho had multiple incidents where Java services were OOM-killed because:
|
|
1. The JVM was using the wrong default (node RAM, not pod limit)
|
|
2. Teams were setting `-Xmx` manually but forgetting to update it when `memory_request` changed
|
|
3. Teams were setting `-Xmx` too high, causing heap to exceed the pod limit
|
|
|
|
## Decision
|
|
|
|
`deployArgoCD.groovy` automatically computes `xms` and `xmx` from the service's `memory_request` value in `deployment.yaml`. The derived values are injected into the Helm chart at deploy time. Services do not need to set `-Xmx` in `JAVA_OPTS` manually. The escape hatch `jvm_memory_override: true` in `deployment.yaml` allows a service to opt out and set its own JVM flags.
|
|
|
|
## Alternatives Considered
|
|
|
|
- **Require teams to set -Xmx manually**: Rejected — teams consistently forgot to update `-Xmx` when changing `memory_request`, causing OOM kills after pod resource changes. This happened across multiple services.
|
|
- **JVM container awareness flag (-XX:+UseContainerSupport)**: This flag (available in JDK 11+) allows the JVM to read the cgroup limit instead of physical RAM. Not adopted as the primary solution because it requires all services to use JDK 11+ and the flag needs to be explicitly set in each service's startup config — still a per-service manual step.
|
|
- **Fixed default heap values**: Rejected — services have wildly different memory requirements; a fixed default would be wrong for most.
|
|
|
|
## Consequences
|
|
|
|
**Positive:**
|
|
- Java services cannot be OOM-killed due to JVM heap misconfiguration — the heap is always proportional to the pod's actual memory allocation.
|
|
- Teams never need to update `-Xmx` manually when changing `memory_request` — the library keeps them in sync automatically.
|
|
- Eliminates a whole class of incident: "service OOM-killed because someone bumped memory_request but forgot to update -Xmx."
|
|
|
|
**Negative:**
|
|
- The auto-derived heap may not be optimal for services with unusual heap vs non-heap memory ratios (e.g., services with large off-heap caches). These services must use `jvm_memory_override: true`.
|
|
- The calculation logic is not immediately transparent to service teams — they may not know why their `-Xmx` is what it is.
|
|
|
|
**Neutral:**
|
|
- If `deployment_args` already contains an `-Xmx` or `-Xms` flag, the auto-derive reads and preserves those values rather than overwriting them. `jvm_memory_override` is the clean opt-out for services that need full control.
|
|
|
|
## Constraints
|
|
|
|
Repeated OOM incidents across multiple Java services drove this decision. The library-level fix was preferred over per-service remediation because the root cause was systemic (wrong JVM defaults in containers) and would recur as long as teams configured heap manually.
|
|
|
|
## Current Assessment
|
|
|
|
**Still appropriate** — no changes needed.
|
|
|
|
## Notes
|
|
|
|
- Key file: `src/com/meesho/stages/deployArgoCD.groovy:205-268` — `xms`/`xmx` calculation
|
|
- Opt-out: set `jvm_memory_override: true` in `deployment.yaml` to manage JVM flags manually
|
|
- Documented in CLAUDE.md: "JVM memory: `deployArgoCD.groovy` auto-calculates `xmx`/`xms` from pod `memory_request`."
|