Question 11 · Track 4 · Find it
Design a web crawler
Politeness is a design constraint, not good manners.
Find it. 4 clarifying questions to ask first, the answer in 4 moves, the follow-up that catches a memorised answer and 2 supporting topics to watch or read.
Short answer live
What it is really testing
Whether you can keep a queue moving while obeying a per-domain rate limit, and how you avoid crawling the same page forever.
Ask these first
Before a single box goes on the board. The answers change the design, which is the point of asking out loud.
- How many pages, and in what time?
- How fresh must a crawled page stay — do we recrawl?
- Do we render JavaScript, or fetch HTML only?
- Do we store the raw page, or just the extracted text?
The answer, in 4 moves
In this order. Each move earns the next one — say them out loud rather than drawing all four and narrating afterwards.
- 1A frontier queue of URLs, partitioned so one domain never monopolises the workers.
- 2Respect robots.txt and a per-domain delay — the delay is per domain, never global.
- 3Deduplicate URLs (and content hashes) with a bloom filter before you fetch.
- 4Store the raw page and build the index separately, so a parser change is a re-index, not a re-crawl.
The trap
The follow-up that separates a rehearsed answer from a real one.
"Polite" means per domain. A single global rate limit makes one slow site throttle the entire crawl, and a single shared queue makes one huge domain starve every other one.
Watch or read
The pieces of this answer, each covered on its own. Take them whichever way suits you, then give the whole answer without looking.
Design a web crawler, end to end
"Design a web crawler" sounds like a scripting exercise — fetch a page, pull the links, repeat. It is really a distributed systems question, and the interviewer is marking two things: do you treat the internet as someone else's server, and does your job survive a crash? Here is the whole answer in four points, in the order one URL travels — and the one word everybody forgets.
Inverted indexes (FTS5)
You have 10,000 markdown notes and you search for 'postgres backup'. grep takes 400 ms and returns 40 files in folder order. SQLite FTS5 takes 3 ms and puts the right note first. Same files, same query — the difference is an inverted index and BM25 ranking. Here's how both actually work, the negative-score gotcha everyone hits once, the staleness trap of a derived index, and where lexical search ends and vector search begins — including why RAG pipelines still run BM25 next to embeddings.
Part of System Design Interviews, Answered Out Loud. The fundamentals underneath it are the free CS course.