added files
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
# ADR-0009: JVM Heap Auto-Derived from Pod memory_request
|
||||
|
||||
**Status:** Accepted
|
||||
**Category:** RELIABILITY
|
||||
**Date decided:** Mid-project
|
||||
**Date documented:** 2026-05-12
|
||||
|
||||
## Context
|
||||
|
||||
Java services running in Kubernetes pods are subject to two memory limits: the pod's `memory_limit` (enforced by the kubelet — exceed it and the pod is OOM-killed) and the JVM's heap size (`-Xmx`). By default, the JVM sets heap to 1/4 of the physical RAM it detects — but inside a container, it detects the node's physical RAM, not the pod's memory limit. A Java service in a pod with `memory_limit: 2Gi` running on a 64Gi node would default to a 16Gi heap, far exceeding its limit and triggering immediate OOM kill.
|
||||
|
||||
Meesho had multiple incidents where Java services were OOM-killed because:
|
||||
1. The JVM was using the wrong default (node RAM, not pod limit)
|
||||
2. Teams were setting `-Xmx` manually but forgetting to update it when `memory_request` changed
|
||||
3. Teams were setting `-Xmx` too high, causing heap to exceed the pod limit
|
||||
|
||||
## Decision
|
||||
|
||||
`deployArgoCD.groovy` automatically computes `xms` and `xmx` from the service's `memory_request` value in `deployment.yaml`. The derived values are injected into the Helm chart at deploy time. Services do not need to set `-Xmx` in `JAVA_OPTS` manually. The escape hatch `jvm_memory_override: true` in `deployment.yaml` allows a service to opt out and set its own JVM flags.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- **Require teams to set -Xmx manually**: Rejected — teams consistently forgot to update `-Xmx` when changing `memory_request`, causing OOM kills after pod resource changes. This happened across multiple services.
|
||||
- **JVM container awareness flag (-XX:+UseContainerSupport)**: This flag (available in JDK 11+) allows the JVM to read the cgroup limit instead of physical RAM. Not adopted as the primary solution because it requires all services to use JDK 11+ and the flag needs to be explicitly set in each service's startup config — still a per-service manual step.
|
||||
- **Fixed default heap values**: Rejected — services have wildly different memory requirements; a fixed default would be wrong for most.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive:**
|
||||
- Java services cannot be OOM-killed due to JVM heap misconfiguration — the heap is always proportional to the pod's actual memory allocation.
|
||||
- Teams never need to update `-Xmx` manually when changing `memory_request` — the library keeps them in sync automatically.
|
||||
- Eliminates a whole class of incident: "service OOM-killed because someone bumped memory_request but forgot to update -Xmx."
|
||||
|
||||
**Negative:**
|
||||
- The auto-derived heap may not be optimal for services with unusual heap vs non-heap memory ratios (e.g., services with large off-heap caches). These services must use `jvm_memory_override: true`.
|
||||
- The calculation logic is not immediately transparent to service teams — they may not know why their `-Xmx` is what it is.
|
||||
|
||||
**Neutral:**
|
||||
- If `deployment_args` already contains an `-Xmx` or `-Xms` flag, the auto-derive reads and preserves those values rather than overwriting them. `jvm_memory_override` is the clean opt-out for services that need full control.
|
||||
|
||||
## Constraints
|
||||
|
||||
Repeated OOM incidents across multiple Java services drove this decision. The library-level fix was preferred over per-service remediation because the root cause was systemic (wrong JVM defaults in containers) and would recur as long as teams configured heap manually.
|
||||
|
||||
## Current Assessment
|
||||
|
||||
**Still appropriate** — no changes needed.
|
||||
|
||||
## Notes
|
||||
|
||||
- Key file: `src/com/meesho/stages/deployArgoCD.groovy:205-268` — `xms`/`xmx` calculation
|
||||
- Opt-out: set `jvm_memory_override: true` in `deployment.yaml` to manage JVM flags manually
|
||||
- Documented in CLAUDE.md: "JVM memory: `deployArgoCD.groovy` auto-calculates `xmx`/`xms` from pod `memory_request`."
|
||||
Reference in New Issue
Block a user