Design a web crawler
Hard · Whiteboard
Design a crawler that fetches 1 billion web pages a month for a search engine. It must be polite to websites, avoid fetching the same page twice, and keep pages fresh. Estimate throughput and storage.
Create a free account to answer it and get scored on what a strong answer covers.