Connection Pooling: Why Your API Dies at 200 Users (But the DB Is at 4% CPU)

FitLog is a small workout-tracking app: one API, one Postgres database behind it. At 20 users it's instant — every request feels like a local function call. Then a launch happens, 200 people sign up at once, and the API falls over. Requests time out, error rates spike, pagers go off.

So you open the database dashboard, bracing for a pegged CPU and a disk on fire — and Postgres is at 4% CPU. Barely awake. The database was never the bottleneck. It was never even reached.

That paradox is the whole reason connection pooling exists.

---

A connection is not a variable — it's a small server

The instinct is to think of "connecting to the database" as cheap: assign a variable, you're connected. It isn't. Opening a fresh Postgres connection is a small negotiation:

1. A TCP handshake (round trips).
2. A TLS handshake (more round trips, crypto).
3. Password authentication.
4. Then Postgres forks a whole backend process dedicated to that one connection — its own memory, its own everything.

Add it up and you're looking at roughly ~40ms and megabytes of RAM — to set up a connection that will then run a 3ms query. You're paying 90%+ of the cost on setup, before any real work happens.

# The trap: a brand-new connection per request
def handle_request(query):
    conn = psycopg2.connect(DATABASE_URL)  # ~40ms: TCP + TLS + auth + forked backend
    cur = conn.cursor()
    cur.execute(query)                     # 3ms of actual work
    result = cur.fetchall()
    conn.close()                           # ...and throw the expensive thing away
    return result

Do that once per request and every request drags a 40ms anchor behind a 3ms task.

The wall: max_connections = 100

It gets worse than "slow." Because every connection is a forked backend process, Postgres refuses to spawn an unbounded number of them. It ships with:

max_connections = 100

Connection #101 does not queue, and it does not slow down. It is refused, immediately:

FATAL: sorry, too many clients already

So when 200 users arrive at once and your app opens a connection per request, roughly 100 are turned away at the door, and the survivors each burn 40ms of setup for their 3ms of work. From the app it looks like the database is melting. From the database's point of view, it did almost nothing — it spent its time forking and rejecting, not querying. Hence 4% CPU while everything is on fire.

---

The fix: stop opening connections

A connection pool flips the model. Instead of creating a connection per request, you open a small, fixed set once, at boot, and keep them open for the life of the process. A request no longer creates a connection — it borrows one from the pool, runs its query, and hands it straight back, still open, still authenticated.

from psycopg2.pool import ThreadedConnectionPool

# Opened ONCE at startup — the expensive setup is paid a single time
pool = ThreadedConnectionPool(minconn=5, maxconn=20, dsn=DATABASE_URL)

def handle_request(query):
    conn = pool.getconn()          # borrow a warm, authenticated slot (microseconds)
    try:
        cur = conn.cursor()
        cur.execute(query)         # 3ms of actual work
        return cur.fetchall()
    finally:
        pool.putconn(conn)         # return the slot, still open, for the next request

The 40ms setup happens maxconn times over the whole life of the app, not once per request. That's the entire trick — and it's what PgBouncer, HikariCP, and SQLAlchemy's pool are all doing under the hood. One job.

Three things people get wrong

1. The pool is a queue, not a multiplier. A pool of 20 does not let you serve infinite users. When all 20 slots are busy, request #21 waits for a slot to free up — it doesn't fail with "too many clients." You've traded hard refusals for bounded waiting, which is almost always the trade you want.

2. A smaller pool is often faster. It's tempting to crank the pool to 100 "to be safe." But 100 connections fighting over the same CPU cores, locks, and memory bandwidth run slower than 20 connections running at full speed. Past the point where the pool size matches what the database can actually do in parallel, more connections is pure contention. 20 focused workers beat 100 elbowing each other.

3. Size it to the database, not to your users. The right pool size tracks the database's cores and disk, not how many users you have. And the ceiling is shared: the pool is per process. Run 10 app instances with a pool of 20 each and you've opened 200 connections to a database that allows 100. Do the math across your whole fleet, not per box.

	Connection per request	With a pool
Setup cost	~40ms on every request	Paid once, at boot
Under a burst	Connection #101 refused	Request #21 waits in a queue
DB processes	One fork per request	Fixed, reused
Failure mode	"too many clients already"	Bounded latency
Right size	—	DB cores, shared across all processes

The one that bites in production

A pool hides a burst, not a bug. When you see pool exhausted / QueuePool limit reached, the reflex is to raise the pool size. Resist it. Nine times out of ten the pool isn't too small — a slow query is holding slots. One query that used to take 3ms now takes 3 seconds (a missing index, a lock, a table scan), so each slot is occupied 1000× longer, and 20 slots drain in an instant. Bumping the pool to 40 just means you exhaust 40 slots a moment later while the real culprit — the slow query — sails on. Fix the query, not the number.

---

The same idea, one layer up: serving LLMs

If you build AI features, you've already met this exact shape twice — and pooling is the answer both times.

RAG retrieval hammers your database. A retrieval-augmented app runs a vector similarity search (pgvector, or a dedicated store) on every question, often several per user turn. That's a connection-per-request firehose pointed straight at Postgres. Without a pool you rebuild the 40ms handshake on every retrieval; with one, the embedding lookups borrow warm slots. The RAG path is one of the most connection-hungry workloads you can ship, and it's the first place a missing pool shows up as mysterious latency.

LLM inference servers are connection pools for the GPU. Look at how vLLM or TGI serve a model and you'll see the identical mental model. The GPU can only decode so many sequences at once — that's max_num_seqs, the exact analogue of max_connections. Incoming requests don't each spin up their own model; they borrow a decode slot via continuous batching, stream their tokens, and release it. Request #N over the limit waits in a queue — it isn't refused. And the ceiling is set by KV-cache memory (GPU cores and VRAM), not by your user count — you size the batch to the hardware, exactly like sizing a pool to the database's cores. Even "pool exhausted = slow query" carries over: when a serving queue backs up, the cause is usually a few very long generations holding slots, not a batch size that's too small. Same disease, same cure, one layer up the stack.

---

The takeaway

Connection pooling isn't an optimization you bolt on later — for anything with real concurrency it's the difference between "works in the demo" and "survives the launch." Open your connections once, keep them warm, borrow and return instead of create and destroy. Treat the pool as a queue you size to the database (and count across every process), and when it exhausts, go hunt the slow query before you touch the dial.

No more 3 a.m. too many clients already pages.

Want the 90-second visual version? Watch the reel.