1 · Crawl pages
Crawler Service pulls ranked links, avoids already-crawled pages, and sends follow-on work through two queues.
Interactive diagram · architecture
Design a web crawler
2 · Index words
The reverse-index queue feeds the service that maps words to the pages that contain them.
3 · Build documents
The document queue feeds the service that creates static titles and snippets for pages.
4 · Search pages
A client request reaches Query API, which uses the reverse index and document service to return relevant pages.
Client
What it is
The user enters a search term and sees relevant pages.
Its job here
It asks DNS where to connect, then sends the search request to Load Balancer. It receives titles and snippets in the result.
In the primer
DNS
What it is
DNS resolves a name so the client can find the service.
Its job here
It receives the lookup from Client and gives back the connection destination. Client then sends its request to Load Balancer.
Trade-off
DNS is an extra dependency before the request reaches the crawler system.
In the primer
Load Balancer
What it is
A load balancer distributes incoming requests across service instances.
Its job here
It receives the search request from Client and passes it to Web Server. The stacked copies show that it can have several instances.
Trade-off
It adds another component to operate before requests reach Web Server.
In the primer
Web Server
What it is
The public web layer accepts the client request.
Its job here
It receives traffic from Load Balancer and forwards the request to Query API. The response returns through this web layer to Client.
Trade-off
Using a reverse proxy adds another hop in the request path.
In the primer
Query API
What it is
The public REST API parses a search query.
Its job here
Web Server forwards the request here. Query API uses Reverse Index Service, then Document Service, to return matching pages with titles and snippets.
Trade-off
Query parsing adds work before results can be returned.
In the primer
Reverse Index Service
What it is
This service builds a reverse index of words to pages.
Its job here
Index Queue delivers crawl jobs to it. It returns documents that match a query to Query API.
Trade-off
Generating the index is separate work from crawling pages.
In the primer
Document Service
What it is
This service creates static titles and snippets for pages.
Its job here
Document Queue delivers crawl jobs to it. It passes page documents to Memory Cache and Query API uses it to return titles and snippets.
Trade-off
Titles and snippets are static and do not change for each query.
In the primer
Reverse index queue
What it is
A queue holds reverse-index jobs.
Its job here
Crawler Service adds a job here after it crawls a link. The queue hands that job to Reverse Index Service.
Trade-off
A queue separates crawling from indexing, so the work does not have to finish in the crawler loop.
In the primer
Document queue
What it is
A queue holds document-generation jobs.
Its job here
Crawler Service adds a job here after it crawls a link. The queue hands that job to Document Service.
Trade-off
A queue separates document work from the crawler loop.
In the primer
Crawler Service
What it is
The service processes the highest-ranked page link in a loop.
Its job here
It reads ranked links and signatures from NoSQL, then adds jobs to both queues. It writes the crawled link and page signature back to NoSQL.
Examples
- Jaccard index
- Cosine similarity
Trade-off
It must detect similar pages so a graph cycle does not become an infinite loop.
In the primer
Memory Cache
What it is
A memory cache keeps generated page documents ready to read.
Its job here
Document Service feeds it with titles and snippets. Query API can use those documents after Reverse Index Service finds matching pages.
Trade-off
Cached data can require an update strategy as pages are crawled again.
In the primer
NoSQL Database
What it is
A key-value NoSQL database stores links to crawl and crawled links with signatures.
Its job here
Crawler Service reads the next ranked link and checks similar page signatures here. It then removes the link to crawl and inserts the crawled link and signature.
Examples
- Redis
Trade-off
Choosing SQL or NoSQL depends on the use cases and trade-offs.
In the primer
←/→ step · Space play/pause · R replay · Home/End jump
Detail · follows playback, or click any box or step to pin it
Close · follow playbackClick any box for an explanation, or use Next or Play to walk through the four steps.
1 · Crawl pages
Crawler Service pulls ranked links, checks crawled signatures, and sends indexing and document jobs through queues.
It stores processed links and signatures in NoSQL.
2 · Index words
Reverse Index Service creates the map from words to the pages that contain them.
The queue lets indexing run separately from crawling.
3 · Build page documents
Document Service creates static titles and snippets for pages.
Generated documents are ready for a later search response.
4 · Search pages
Client reaches Query API through DNS, Load Balancer, and Web Server.
Query API uses the reverse index and page documents to return relevant pages.