Design a Web Crawler: The Four-Point Answer, and the "Polite" Trap
"Design a web crawler" sounds like a scripting exercise — fetch a page, pull the links, repeat. It is really a distributed systems question, and the interviewer is marking two things: do you treat the internet as someone else's server, and does your job survive a crash? Here is the whole answer in four points, in the order one URL travels — and the one word everybody forgets.
"Design a web crawler" is one of the most common system-design interview questions, and one of the easiest to answer badly — because it sounds like a scripting exercise. Fetch a page, pull the links out, fetch those. You could write it in an afternoon.
You could. It would also fall over within the hour, and get your IP banned by lunchtime.
The interviewer is marking two things, and neither of them is speed: do you treat the internet as someone else's server, and does your job survive a crash? Everything below follows from those two.
Here is the whole answer, in the order one URL travels.
1. A frontier queue, not a loop
The naive answer is a loop with a list inside it:
to_visit = [seed]
while to_visit:
url = to_visit.pop()
html = fetch(url)
to_visit.extend(extract_links(html))
This works on one machine, for about ten minutes. Then the process dies — OOM, deploy, spot instance reclaimed — and to_visit dies with it. You start again from the seed.
The real answer inverts it. The list becomes a frontier queue that lives outside the process:
┌──────────── frontier queue ────────────┐
│ url url url url url url url │
└───┬────────────────────────────▲───────┘
│ workers pull │ new links go back in
▼ │
┌────────────────────────────────────┐
│ fetch → parse → extract links │
└────────────────────────────────────┘
Workers pull URLs out. Newly discovered links go back in. Nothing about the crawl's progress is held in a worker's memory.
That single change is what buys you everything else people expect from a crawler. It is resumable, because the queue is the state — kill every worker and the crawl continues when they come back. It is horizontally scalable, because adding a worker requires no coordination with the others. And it gives you a place to put priority: a frontier is usually not one queue but many, so a news site can be re-crawled hourly while a static archive waits a month.
Say this part explicitly in an interview. "The state lives in the queue, so the crawler can crash and resume where it stopped" is the sentence being listened for.
2. robots.txt, and a per-domain delay
Before you touch a site, fetch its robots.txt — and cache it, because re-fetching it on every request is its own small act of rudeness.
User-agent: *
Disallow: /admin
Crawl-delay: 2
The thing to understand is what that file is. It is not your configuration. It is the site's instruction to you, and honouring it is the price of being allowed to keep crawling. Ignore it and you are not running a crawler, you are running an unauthorised load test against a stranger's infrastructure.
Then the harder half: a per-domain delay.
Your workers are domain-agnostic by default, and that is the bug. Ten thousand workers pulling from one shared frontier will, sooner or later, all pull URLs from the same popular domain in the same second — and you have built a distributed denial-of-service against a site that did nothing to you.
The fix is to shard the frontier by domain, not to shuffle it:
| Lane | Who serves it | Pace |
|---|---|---|
a.com | one worker at a time | its own delay — say 2s apart |
b.com | one worker at a time | its own delay, read from its own robots.txt |
c.com | one worker at a time | its own delay |
One lane per domain, one worker per lane, each lane holding its own delay. Now you can run ten thousand workers flat out across ten thousand domains and never hit a single one harder than once every two seconds. Throughput comes from breadth, not from hammering.
3. Dedupe with a bloom filter
The web is a graph with cycles, and a very small number of pages link to a very large number of others. You will meet the same URL millions of times.
So before a URL enters the frontier: have I seen this?
The obvious structure is a hash set, and the obvious problem is that it does not fit. A billion URLs at ~60 bytes each is tens of gigabytes of RAM, per crawler — and a billion URLs is a small crawl.
A bloom filter answers the same question in a few bits per URL. (I've written a whole piece on how one works — the point here is why it fits this job.) The reason is that its error is asymmetric:
- a false positive — it says "seen it" about a page you have not crawled — means you skip one page. On a web of billions, that is nothing.
- a false negative cannot happen. It can never let a duplicate through.
Line those up against the costs. Skipping a page costs you one page. Re-crawling duplicates forever costs you the crawl, because a cycle will happily consume every worker you own. The bloom filter is wrong in precisely the direction you can afford — which is the actual reason to reach for it here, and the reason worth saying out loud.
4. Store and index separately
Last point, and the one most often skipped.
The fetcher's job ends when the raw page is written to blob storage. Parsing, extracting text, building the search index — all of that belongs to a different pipeline, reading from that storage.
fetcher
│
▼
┌──────────────┐
│ blob storage │ ← raw HTML, exactly as fetched
└──────┬───────┘
│ a different pipeline
▼
┌──────────────┐
│ parse + index│
└──────────────┘
Two reasons, and they are independent.
They scale differently. Fetching is network-bound and latency-dominated — mostly waiting. Indexing is CPU- and memory-bound. Weld them into one process and you size your fleet for the worse of the two and waste the other. Split them and each scales on its own signal.
Your parser will change. This is the one that bites. Ship a parser bug, or decide six months later that you also want to extract publication dates, and the question becomes: do you re-crawl the web? If the raw page is in blob storage, no — you re-run the parsing pipeline over data you already own, overnight, for free. If you only ever kept the parsed output, you re-crawl a billion pages to fix a regex.
Keep the raw bytes. Storage is the cheapest thing in this design, and it is the only reason a parser change is a re-run instead of a re-crawl.
The same four points, when the crawler feeds an LLM
This question stopped being purely academic the moment every company started building a RAG pipeline. Ingesting a customer's documentation site, keeping an internal knowledge base fresh, assembling a pre-training corpus — all of it is a crawler, and all four points transfer with the labels changed.
| Crawler point | What it becomes in an AI ingestion pipeline |
|---|---|
| Frontier queue | Ingestion is long-running and embedding calls fail. The queue is what lets a half-finished corpus resume instead of restarting — and it is where you put re-crawl priority, because a docs site that changed yesterday matters more than one that has not moved in a year. |
| robots.txt + delay | Now it is also a policy question. Sites publish rules for GPTBot, ClaudeBot and friends specifically, and "we ignored robots.txt" is the sentence that turns an ingestion job into a legal problem. Send an honest user-agent and honour the file. |
| Dedupe | Duplicates are worse here than in search. A page ingested twice becomes near-identical chunks that crowd out the top-k of every retrieval, and duplicated text in a training corpus is a documented way to make a model memorise it. URL dedup is the cheap first pass; content hashing catches the same page served on three URLs. |
| Store, then index | The strongest transfer of the four. Keep the raw page, because your chunking strategy and your embedding model will both change — and re-chunking 200k saved documents is an overnight batch job, while re-crawling to fix your chunk size is a week you did not need to spend. |
Same architecture, higher stakes: the crawler's storage layer is now the thing that decides whether swapping your embedding model is a re-index or a re-crawl.
The trap: they are waiting for the word "polite"
Everyone describes a fast crawler. More workers, async I/O, connection pooling, a bigger fleet.
The interviewer is waiting for one word: polite.
A crawler that gets your IP banned crawls nothing at all. Ignore robots.txt, skip the per-domain delay, and your beautifully scalable frontier drains into 403s from every site worth crawling — while your abuse inbox fills up. Every other decision in this design assumes you are still allowed to make requests.
That is why politeness is not a fifth point bolted onto the end. It is the precondition for the other four.
The verdict: the whole answer in four lines
| Point | What it buys you | What breaks without it |
|---|---|---|
| A frontier queue | The state lives outside the workers | Any crash restarts the crawl from the seed |
| robots.txt + per-domain delay | One lane per domain — breadth, not hammering | You have written a distributed DoS, and get banned |
| A bloom filter | "Seen it?" in a few bits, wrong only in the affordable direction | Cycles eat every worker you own |
| Store, then index | The raw page is the contract between two pipelines | A parser change means re-crawling the web |
Answer it in that order, name the two things being marked — resumability and treating the internet as someone else's server — and finish on polite. That is the full-marks answer.
▶ Watch the 100-second version: Design a Web Crawler — and the trap everyone falls into