The Memory Leak Nobody Noticed

Step 1 — Understand the symptoms

Flat traffic ruling out load as the cause is the key detail. If more requests were driving more memory use, the graph would track traffic — rising and falling with it, not climbing in a straight line regardless of demand. A steady, traffic-independent climb that resets to baseline after every restart is close to the textbook shape of a memory leak: something is allocating memory during normal operation and never releasing it, so usage grows with time and workload volume (total requests processed), not with concurrent load.

Step 2 — Identify the likely bottleneck

CPU staying flat while memory climbs actually narrows things usefully — it rules out a computation-heavy cause and points at something more passive: objects being retained in memory that should have been garbage collected, a cache with no eviction policy quietly growing forever, event listeners or timers accumulating without being cleaned up, or a connection/resource pool that's not releasing what it opens. None of these show up as extra CPU work; they just sit there, taking up space.

Step 3 — Investigation

• Confirm the reset-on-restart pattern across multiple cycles — if memory reliably returns to the same low baseline after every restart and climbs at a similar rate each time, that's strong confirmation of a leak rather than a one-off anomaly.
• Take a heap snapshot (or the runtime's equivalent — Node's --inspect and Chrome DevTools, a JVM heap dump, etc.) at two points hours apart and diff them — this shows exactly what object types are growing in count, which is usually the fastest way to point at the actual leaking code path.
• Check for unbounded in-memory caches — a Map or object used as a cache with no TTL or size limit is one of the most common leak sources in long-running services.
• Check for event listener or timer accumulation — a setInterval or event subscription created per-request but never cleaned up will grow linearly with request volume.
• Correlate the climb rate against request volume, not traffic level — if a leak grows per-request rather than per-second, its rate will roughly track cumulative requests, not current load, which is a very telling signal.

Step 4 — Recommended action

Fix the actual retention point the heap diff surfaces — add an eviction policy to an unbounded cache, ensure listeners/timers are cleaned up on completion, close pooled resources properly. Don't just raise the memory limit or increase restart frequency as a workaround; that hides the symptom (fewer visible restarts) while leaving the underlying leak to keep consuming more memory per hour of uptime, and eventually the limit increase runs out of headroom too.

In the meantime, if restarts are currently silent and unalerted, add monitoring on restart count and memory trend specifically — a slow, one-directional climb over hours is a distinctive enough pattern that it's worth its own alert, separate from a generic memory-threshold alert that might not catch a leak until it's already causing restarts.

Step 5 — Engineering lesson

CPU and memory tell different stories, and a dashboard that only draws attention to CPU spikes will miss this kind of incident entirely — the graph was there, climbing in plain sight, for six hours before anyone noticed. A memory leak is exactly the kind of failure that's invisible until it isn't: quiet, gradual, and easy to dismiss as normal until the restarts start.