Concurrent LLM Calls in Python: asyncio.gather + a Semaphore

Prefer to watch? ▶ Concurrent LLM calls — the full 6-minute walkthrough ✈ Telegram

You have 500 product reviews. You want a sentiment tag on each one, so you call a language model once per review. Each call takes about two seconds.

You write the obvious loop. You hit run. Seventeen minutes later, it finishes.

Here is the strange part: your computer did almost nothing for those seventeen minutes. It wasn't calculating. It was waiting.

This post is about the three lines that turn those seventeen minutes into about a hundred seconds — and the one line that stops you from getting blocked while you do it.

Where the time actually goes

Every one of those 500 calls does the same thing.

You send a request out over the network. Then you sit there. The model thinks. About two seconds later, the answer comes back.

Almost all of that is travelling and waiting. Almost none of it is work your processor is doing. Your CPU is pinned at roughly zero the entire time.

Now do that 500 times, one after another, and you get seventeen minutes of mostly nothing.

The waiting is the program. That's the whole diagnosis, and everything below follows from it. When a program is slow because it is computing, you need a faster machine. When it's slow because it's waiting, you don't need anything faster — you need to wait for more things at the same time.

The code everyone writes first

async def tag_all(reviews):
    out = []
    for r in reviews:
        out.append(await tag(r))   # start ONE call...
        # ...then stand here until it comes back
    return out

# 500 reviews x 2s = 1000s = ~17 minutes

Look at it. It has async in the signature. It has await in front of the call. It reads like modern, concurrent Python.

It is not concurrent. Watch what it actually does:

It starts one call.
It stops, and waits for that call to come back.
Only then does it start the next one.

Five hundred times, in a row. The word await is doing exactly what it says on the tin. It waits.

The trap, stated plainly

await does not mean concurrent.

await means: pause me here, and let the event loop go run something else in the meantime.

That's a real, useful thing. But read the loop again — in that loop, there is nothing else. You never gave the event loop another job to do. So it pauses, it sits idle, and it resumes. 500 times in a row.

You wrote asynchronous code and you got sequential timing. Async syntax is not a performance feature by itself. It's the ability to have multiple things in flight. You still have to actually put multiple things in flight.

asyncio.gather: fire them all, wait once

import asyncio

async def tag_all(reviews):
    return await asyncio.gather(
        *(tag(r) for r in reviews)
    )

# all 500 requests in flight together
# wall clock = the SLOWEST call, not the sum

That's the fix. Instead of awaiting each call one at a time, you build all 500 coroutines first and hand them to asyncio.gather in one go.

gather takes them, schedules every single one of them on the event loop, and then waits once — for all of them together.

Note the * and the generator: tag(r) doesn't run anything. Calling an async function just creates a coroutine object — an inert to-do item. Nothing executes until something schedules it. gather is that something, and it schedules all 500 at once.

Now the requests are out on the network at the same time. The wall clock stops being the sum of 500 calls and becomes roughly the length of the slowest single one.

Before, your timeline was 500 bars in a row. Now it's 500 bars stacked on top of each other. The work did not get faster. One call still takes its two seconds. You simply stopped doing them one at a time.

A detour: why is it called "gather"?

This trips people up, so it's worth thirty seconds.

gather is not named for the speed. It's named for what it does with the results.

It hands them back in the order you passed them in — not the order they finished. If review number three was your third argument, its answer is the third item in the returned list, even if that call came back dead last.

That ordering is the entire product. It's what lets you write this without a second thought:

tags = await asyncio.gather(*(tag(r) for r in reviews))

for review, tag_result in zip(reviews, tags):   # index i lines up with index i
    save(review, tag_result)

No correlation IDs. No dictionary keyed by request. The index you put in is the index you get out.

Its sibling, as_completed, is the one that gives you results in finish order — useful when you want to start processing the fast ones immediately and don't care which review they belonged to:

for coro in asyncio.as_completed(tasks):
    result = await coro       # whichever finishes first, arrives first

Two different jobs. gather = I need my array back, aligned. as_completed = give me things as soon as they're ready.

The word itself is borrowed from parallel computing, where MPI_Gather collects one value from every process into a single array, indexed by rank. Fan out, then gather back in order. Python kept the name and the guarantee.

So we're done. Right?

No. Run that gather against a real provider and it falls apart.

At time zero, you just opened 500 connections. Your provider allows, say, 60 requests a minute. It is not being difficult — it is obliged to refuse most of what you just threw at it.

You get a wall of 429 Too Many Requests.

Unbounded fan-out is a self-inflicted outage. You traded seventeen minutes of waiting for an instant denial, and you did it to yourself. In the worst case you don't just fail this batch — you burn through your quota and take down the other services sharing that API key.

asyncio.Semaphore: bound the fan-out

The fix is one object.

sem = asyncio.Semaphore(10)        # 10 in flight, ever

async def tag_limited(review):
    async with sem:                # take a permit...
        return await tag(review)   # ...release on exit

async def tag_all(reviews):
    return await asyncio.gather(
        *(tag_limited(r) for r in reviews)
    )

A semaphore is just a counter of permits. We make one with ten.

Now every call has to take a permit before it's allowed to run — that's the async with sem line. Ten coroutines hold a permit and are in flight. The other 490 are parked at that line, waiting their turn. When one finishes, the async with block exits, its permit is released, and the next coroutine walks through.

We still call gather. We're just gathering bounded work now.

And the arithmetic is easy:

	Sequential `await`	Naive `gather`	`gather` + `Semaphore(10)`
In flight at once	1	500	10
Wall clock	~1000s (17 min)	~2s… in theory	~100s
What the provider sees	A polite trickle	A burst it must refuse	A steady 10
Actual outcome	Works, painfully	A wall of 429s	Works

500 reviews, 10 at a time, 2 seconds each. That's 50 rounds of 2 seconds. About a hundred seconds — and nobody refused you.

Notice what the semaphore actually controls. It does not make anything faster. It decides how much of your work is allowed to be in flight at once. That number is yours to choose, and it should come from the provider's published limit — not from optimism.

The 429 that still gets through

Ten in flight does not make you immune.

Limits are counted per minute, not per instant. Other jobs share your key. Providers have bad days and tighten limits without telling you.

So when a 429 does slip through, you catch it, you wait, and you try again — and you make the wait grow a little longer each time.

async def tag_with_retry(review, tries=5):
    for i in range(tries):
        try:
            return await tag_limited(review)
        except RateLimitError:
            await asyncio.sleep(2 ** i)   # 2s, 4s, 8s...
    raise

Two seconds, then four, then eight. That's exponential backoff. The doubling matters: if everyone retried after a flat one second, all your parked coroutines would slam the provider again in lockstep and you'd re-create the same burst you were recovering from.

(In production, add a little random jitter to each sleep so retries from different workers don't re-synchronise. And respect the Retry-After header if the provider sends one — it's telling you the answer.)

The distinction worth keeping

Hold these two apart, because they are not the same job:

The semaphore is how you AVOID the rate limit. Backoff is how you SURVIVE the one that gets through anyway.

One is prevention. The other is recovery.

If you only have backoff, you spend your life apologising for a burst you should never have sent. Your throughput collapses into a retry storm.
If you only have the semaphore, one bad minute on the provider's side takes down the whole batch — because nothing catches the failure.

Ship both. They're four lines together.

The whole thing

await inside a for-loop is sequential. That's not concurrency; that's a queue with extra syntax.
gather puts everything in flight, and hands the results back in argument order (as_completed is the finish-order sibling).
A semaphore bounds how much is in flight — and that bound comes from the provider's limit, not from hope.
Backoff catches whatever still slips through.

Seventeen minutes, down to about a hundred seconds. Same machine. Same model. Same 500 calls.

Three lines of code — and one of them is the one that keeps you from being your own DDoS.

Want the full walkthrough, line by line? Watch the 6-minute video.