Skip to content

Day 24 of 100

The Database Connection Pool Problem

DatabaseSREIntermediate15–20 min

🔥 Problem Statement

Application replicas were recently scaled from 4 to 12 to handle growing traffic. Since then, users occasionally hit connection timeout errors — not constantly, but often enough to notice, and specifically during normal, not even peak, load.

The database itself looks fine: CPU usage is moderate, nowhere near its limit. Whatever's timing out, it isn't because the database is overloaded in any obvious way.

🏗️ Environment

  • Database: managed relational database, moderate CPU headroom
  • Application: recently scaled from 4 to 12 replicas
  • Connection pool: fixed size per-instance, unchanged since before scaling
  • Symptom: intermittent connection timeout errors under normal load

🤔 Your Challenge

If the database has CPU headroom to spare, why would application instances still be getting connection timeouts?

  • If each application instance has its own connection pool, what happens to the total number of database connections when you triple the instance count?
  • Does a database have a maximum connection limit separate from its CPU capacity?
  • What would happen to new connection attempts once that maximum is reached, even with CPU to spare?
  • Why would this problem appear specifically after scaling replicas, rather than before?

Solution Hidden

Think through the problem yourself before looking at the answer.

💡Solution

Step 1 — Understand the symptoms

CPU headroom on the database rules out compute as the bottleneck, but a database's connection limit is a completely separate ceiling — most databases cap the total number of concurrent connections regardless of how much CPU or memory is available, because each connection itself carries overhead (memory, file descriptors, internal bookkeeping) independent of query load. A database can be nearly idle on CPU and still refuse new connections once that separate limit is hit.

Step 2 — Identify the likely bottleneck

Each application instance almost certainly maintains its own connection pool, sized for a certain number of concurrent connections per instance. That per-instance pool size was presumably reasonable when there were 4 replicas — but going to 12 replicas, with the same per-instance pool size unchanged, means total potential connections to the database roughly tripled, with nobody having revisited whether the database's maximum connection limit could actually accommodate that. If the new total exceeds the limit, connection attempts beyond it fail or time out waiting for one to free up — which shows up exactly as intermittent timeouts under normal (now higher-concurrency) load, with the database's CPU never breaking a sweat.

Step 3 — Investigation

  • Calculate total possible connections: per-instance pool size × current replica count, and compare that directly against the database's configured maximum connection limit.
  • Check the database's current active/idle connection count in real time — if it's sitting near the maximum during normal operation, that's close to confirmation on its own.
  • Check whether the timeouts specifically coincide with periods where connection count is near the limit, versus being spread evenly across all conditions — a tight correlation points squarely at pool exhaustion.
  • Check for connection leaks — connections opened and not properly returned to the pool would make this worse than the raw math suggests, and are worth ruling out separately from the scaling-driven increase.

The direct fix is bringing total possible connections back under the database's actual limit — either by reducing per-instance pool size now that there are more instances sharing the load, or by raising the database's connection limit if it has room to support genuinely higher total concurrency. A connection pooler/proxy sitting between the application fleet and the database (e.g., PgBouncer for Postgres) is often the more durable fix at this scale — it lets many application-side connections share a smaller number of actual database-side connections, decoupling "how many app instances we run" from "how many real database connections that requires."

Whatever the fix, this is exactly the kind of dependency that should be part of any future scaling decision — replica count and per-instance resource assumptions (including connection pool size) need to be considered together, not independently.

Step 5 — Engineering lesson

Scaling one part of a system changes the load on every dependency it talks to, even when that dependency's own primary resource (CPU, in this case) has plenty of headroom. A database's connection limit is a capacity boundary that's easy to forget about specifically because it rarely shows up on a standard CPU/memory dashboard — until a scaling change on the other end of the connection quietly pushes past it.

🧠 Today’s Takeaway

Database capacity isn't only about CPU and memory.

🔔 Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.