1 · Post a tweet
The write path stores the tweet, fans it out to followers, indexes it, stores media, and notifies followers.
Interactive diagram · architecture
Design the Twitter timeline and search
2 · View home timeline
The read path gets timeline ids from cache and enriches them with tweet and user information.
3 · View user timeline
The read API retrieves the user timeline from the SQL database.
4 · Search keywords
The search API parses the query and asks the search service for ranked results.
Client
What it is
The person or program making a request.
Its job here
It resolves DNS, can fetch static content from CDN, and sends requests to Load Balancer.
In the primer
DNS
What it is
DNS translates a domain name to an IP address.
Its job here
It receives the client lookup and directs the client to the service address.
In the primer
CDN
What it is
A CDN serves content close to users.
Its job here
It receives static requests from Client and can serve media held in Object Store.
In the primer
Load Balancer
What it is
A load balancer distributes requests across computing resources.
Its job here
It receives client traffic and sends it to Web Server instances.
In the primer
Web Server
What it is
A reverse proxy forwards requests to application services.
Its job here
It receives traffic from Load Balancer and routes requests to Read API, Write API, or Search API.
In the primer
Read API
What it is
An application API that handles timeline reads.
Its job here
It receives requests from Web Server and asks Timeline Service or SQL Read Replicas for timeline data.
In the primer
Write API
What it is
An application API that handles tweet writes.
Its job here
It receives work from Web Server, stores the tweet in SQL Write Master–Slave, and contacts Fan Out Service.
In the primer
Search API
What it is
An application API for keyword search.
Its job here
It receives a Web Server request and contacts Search Service for results.
In the primer
Tweet Info Service
What it is
A service that supplies tweet details.
Its job here
Timeline Service calls it with a multiget to enrich timeline tweet ids.
In the primer
Timeline Service
What it is
A service that builds home timeline results.
Its job here
It receives Read API requests, reads ids from Memory Cache, and calls Tweet Info Service and User Info Service.
In the primer
Fan Out Service
What it is
A service that distributes a tweet to followers.
Its job here
It receives the new tweet from Write API, queries User Graph Service, writes follower timelines to Memory Cache, indexes search, and starts notifications.
Trade-off
Users with millions of followers can take minutes to finish fanout.
User Graph Service
What it is
A service that finds a user’s followers.
Its job here
Fan Out Service queries it, and it uses Memory Cache for follower information.
In the primer
User Info Service
What it is
A service that supplies user details.
Its job here
Timeline Service calls it with a multiget to enrich user ids.
Memory Cache
What it is
An in-memory store for timeline and follower data.
Its job here
Timeline Service reads it, while Fan Out Service and User Graph Service update or query it.
In the primer
Search Service
What it is
A service that parses and queries keyword search.
Its job here
It receives Search API queries and new tweets from Fan Out Service, then returns ranked search results.
Examples
- Lucene
Notification Service
What it is
A service that sends push notifications.
Its job here
Fan Out Service calls it to notify followers asynchronously through a queue not pictured.
In the primer
SQL Read Replicas
What it is
Replicas serve database reads after changes are copied from the writer.
Its job here
Read API uses them for cache misses and user timeline reads.
In the primer
SQL Write Master–Slave
What it is
A SQL writer stores tweets and replicates changes.
Its job here
Write API sends new tweets here; user timeline reads use the database data.
In the primer
Object Store
What it is
Storage for media such as photos and videos.
Its job here
Write API stores media here, and CDN can serve that media to Client.
←/→ step · Space play/pause · R replay · Home/End jump
Detail · follows playback, or click any box or step to pin it
Close · follow playbackClick any box for an explanation, or use Next or Play to walk through the four steps.
1 · Post a tweet
The Write API stores a tweet, then the Fan Out Service builds followers’ home timelines, indexes search, stores media, and sends notifications.
The post path favors fast writes and fanout.
2 · View home timeline
Timeline Service reads timeline ids from Memory Cache, then multigets tweet and user information.
The home timeline favors fast reads.
3 · View user timeline
Read API retrieves the user’s own timeline from the SQL database.
This path reads the user’s stored tweets.
4 · Search keywords
Search API forwards a parsed query to Search Service, which queries the Search Cluster and returns ranked results.
The search path is read-heavy.