---
title: "Concurrent LLM Calls in Python: asyncio.gather + a Semaphore"
description: "500 LLM calls in a for-loop takes 17 minutes — and your CPU does nothing for all of it. await is not concurrency. Here's how asyncio.gather puts every request in flight at once, why it's named for the results and not the speed, and why asyncio.Semaphore is the one line that stops you from DDoS-ing yourself into a wall of 429s."
keywords: "asyncio.gather, asyncio Semaphore, concurrent LLM calls, python async await, rate limit 429, exponential backoff, async python, event loop, as_completed, MPI_Gather, LLM batch processing, aiohttp, AI engineering"
created_at: "2026-07-13T20:30:00"
post_type: "anonym_post"
content_type: "technical_article"
---
You have 500 product reviews. You want a sentiment tag on each one, so you call a language model once per review. Each call takes about two seconds.
You write the obvious loop. You hit run. Seventeen minutes later, it finishes.
Here is the strange part: **your computer did almost nothing for those seventeen minutes.** It wasn't calculating. It was waiting.
This post is about the three lines that turn those seventeen minutes into about a hundred seconds — and the one line that stops you from getting blocked while you do it.
---
## Where the time actually goes
Every one of those 500 calls does the same thing.
You send a request out over the network. Then you sit there. The model thinks. About two seconds later, the answer comes back.
Almost all of that is travelling and waiting. Almost none of it is work your processor is doing. Your CPU is pinned at roughly zero the entire time.
Now do that 500 times, one after another, and you get seventeen minutes of mostly nothing.
**The waiting *is* the program.** That's the whole diagnosis, and everything below follows from it. When a program is slow because it is *computing*, you need a faster machine. When it's slow because it's *waiting*, you don't need anything faster — you need to wait for more things at the same time.
---
## The code everyone writes first
```python
async def tag_all(reviews):
out = []
for r in reviews:
out.append(await tag(r)) # start ONE call...
# ...then stand here until it comes back
return out
# 500 reviews x 2s = 1000s = ~17 minutes
```
Look at it. It has `async` in the signature. It has `await` in front of the call. It reads like modern, concurrent Python.
It is not concurrent. Watch what it actually does:
1. It starts one call.
2. It **stops**, and waits for that call to come back.
3. Only then does it start the next one.
Five hundred times, in a row. The word `await` is doing exactly what it says on the tin. It waits.
### The trap, stated plainly
> **`await` does not mean concurrent.**
`await` means: *pause me here, and let the event loop go run something else in the meantime.*
That's a real, useful thing. But read the loop again — **in that loop, there is nothing else.** You never gave the event loop another job to do. So it pauses, it sits idle, and it resumes. 500 times in a row.
You wrote asynchronous code and you got sequential timing. Async syntax is not a performance feature by itself. It's the *ability* to have multiple things in flight. You still have to actually put multiple things in flight.
---
## asyncio.gather: fire them all, wait once
```python
import asyncio
async def tag_all(reviews):
return await asyncio.gather(
*(tag(r) for r in reviews)
)
# all 500 requests in flight together
# wall clock = the SLOWEST call, not the sum
```
That's the fix. Instead of awaiting each call one at a time, you build all 500 coroutines first and hand them to `asyncio.gather` in one go.
`gather` takes them, schedules **every single one** of them on the event loop, and then waits **once** — for all of them together.
Note the `*` and the generator: `tag(r)` doesn't *run* anything. Calling an async function just creates a coroutine object — an inert to-do item. Nothing executes until something schedules it. `gather` is that something, and it schedules all 500 at once.
Now the requests are out on the network at the same time. The wall clock stops being the *sum* of 500 calls and becomes roughly the length of **the slowest single one**.
Before, your timeline was 500 bars in a row. Now it's 500 bars stacked on top of each other. **The work did not get faster.** One call still takes its two seconds. You simply stopped doing them one at a time.
---
## A detour: why is it called "gather"?
This trips people up, so it's worth thirty seconds.
`gather` is **not named for the speed.** It's named for what it does with the *results*.
It hands them back **in the order you passed them in — not the order they finished.** If review number three was your third argument, its answer is the third item in the returned list, even if that call came back dead last.
That ordering is the entire product. It's what lets you write this without a second thought:
```python
tags = await asyncio.gather(*(tag(r) for r in reviews))
for review, tag_result in zip(reviews, tags): # index i lines up with index i
save(review, tag_result)
```
No correlation IDs. No dictionary keyed by request. The index you put in is the index you get out.
Its sibling, **`as_completed`**, is the one that gives you results in *finish* order — useful when you want to start processing the fast ones immediately and don't care which review they belonged to:
```python
for coro in asyncio.as_completed(tasks):
result = await coro # whichever finishes first, arrives first
```
Two different jobs. `gather` = *I need my array back, aligned.* `as_completed` = *give me things as soon as they're ready.*
The word itself is borrowed from parallel computing, where **`MPI_Gather`** collects one value from every process into a single array, **indexed by rank**. Fan out, then gather back in order. Python kept the name and the guarantee.
---
## So we're done. Right?
No. Run that `gather` against a real provider and it falls apart.
At time zero, you just opened **500 connections**. Your provider allows, say, 60 requests a minute. It is not being difficult — it is obliged to refuse most of what you just threw at it.
You get a wall of **429 Too Many Requests.**
**Unbounded fan-out is a self-inflicted outage.** You traded seventeen minutes of waiting for an instant denial, and you did it to yourself. In the worst case you don't just fail this batch — you burn through your quota and take down the other services sharing that API key.
---
## asyncio.Semaphore: bound the fan-out
The fix is one object.
```python
sem = asyncio.Semaphore(10) # 10 in flight, ever
async def tag_limited(review):
async with sem: # take a permit...
return await tag(review) # ...release on exit
async def tag_all(reviews):
return await asyncio.gather(
*(tag_limited(r) for r in reviews)
)
```
A **semaphore** is just a counter of permits. We make one with ten.
Now every call has to take a permit before it's allowed to run — that's the `async with sem` line. Ten coroutines hold a permit and are in flight. The other 490 are **parked at that line**, waiting their turn. When one finishes, the `async with` block exits, its permit is released, and the next coroutine walks through.
We still call `gather`. We're just gathering **bounded** work now.
And the arithmetic is easy:
Sequential await
Naive gather
gather + Semaphore(10)
In flight at once
1
500
10
Wall clock
~1000s (17 min)
~2s… in theory
~100s
What the provider sees
A polite trickle
A burst it must refuse
A steady 10
Actual outcome
Works, painfully
A wall of 429s
Works
500 reviews, 10 at a time, 2 seconds each. That's 50 rounds of 2 seconds. **About a hundred seconds** — and nobody refused you.
Notice what the semaphore actually controls. **It does not make anything faster.** It decides how much of your work is allowed to be in flight at once. That number is yours to choose, and it should come from the provider's published limit — not from optimism.
---
## The 429 that still gets through
Ten in flight does not make you immune.
Limits are counted per minute, not per instant. Other jobs share your key. Providers have bad days and tighten limits without telling you.
So when a 429 does slip through, you catch it, you wait, and you try again — and you make the wait grow a little longer each time.
```python
async def tag_with_retry(review, tries=5):
for i in range(tries):
try:
return await tag_limited(review)
except RateLimitError:
await asyncio.sleep(2 ** i) # 2s, 4s, 8s...
raise
```
Two seconds, then four, then eight. That's **exponential backoff**. The doubling matters: if everyone retried after a flat one second, all your parked coroutines would slam the provider again in lockstep and you'd re-create the same burst you were recovering from.
(In production, add a little random jitter to each sleep so retries from different workers don't re-synchronise. And respect the `Retry-After` header if the provider sends one — it's telling you the answer.)
---
## The distinction worth keeping
Hold these two apart, because they are **not the same job**:
> **The semaphore is how you AVOID the rate limit.**
> **Backoff is how you SURVIVE the one that gets through anyway.**
One is **prevention**. The other is **recovery**.
- If you only have **backoff**, you spend your life apologising for a burst you should never have sent. Your throughput collapses into a retry storm.
- If you only have the **semaphore**, one bad minute on the provider's side takes down the whole batch — because nothing catches the failure.
**Ship both.** They're four lines together.
---
## The whole thing
- **`await` inside a for-loop is sequential.** That's not concurrency; that's a queue with extra syntax.
- **`gather` puts everything in flight**, and hands the results back in **argument order** (`as_completed` is the finish-order sibling).
- **A semaphore bounds how much is in flight** — and that bound comes from the provider's limit, not from hope.
- **Backoff catches whatever still slips through.**
Seventeen minutes, down to about a hundred seconds. Same machine. Same model. Same 500 calls.
Three lines of code — and one of them is the one that keeps you from being your own DDoS.
> Want the full walkthrough, line by line? [Watch the 6-minute video.](https://youtu.be/nBzDJJR-rUQ)