1 · Create a paste
The client sends a create request through the web layer. Write API creates a unique URL, saves the record, stores the paste, and returns the URL.
Interactive diagram · architecture
Design Pastebin.com (or Bit.ly)
2 · View a paste
Read API checks the generated URL and fetches the paste contents. Cache and read replicas help scale the read-heavy path.
3 · Track analytics
Web server logs feed analytics. MapReduce can turn those logs into hit counts without requiring realtime processing.
4 · Delete expired pastes
The service scans for rows older than their expiration timestamp, then deletes them or marks them expired.
Client
What it is
The anonymous person using Pastebin.
Its job here
It sends a create or get request. DNS gives it an address, then the request reaches the Load Balancer.
Trade-off
Following a short link should be fast.
In the primer
Interview approach › Step 1: Outline use cases, constraints, and assumptions →
DNS
What it is
DNS points the Pastebin name to an address.
Its job here
It receives the client lookup. The client then reaches the CDN or Load Balancer.
Trade-off
DNS is a separate part of the request path.
In the primer
CDN
What it is
A CDN can serve content close to the client.
Its job here
It receives requests after DNS. It can serve popular content before a request reaches the Load Balancer.
Trade-off
CDN caching is one of several places to cache.
In the primer
Load Balancer
What it is
A load balancer distributes requests across servers.
Its job here
It receives client traffic and sends it to a Web Server. Multiple Web Servers improve availability.
Trade-off
A load balancer adds another component to operate.
In the primer
Web Server
What it is
A reverse proxy handles the public request.
Its job here
It receives traffic from the Load Balancer. It forwards create requests to Write API and get requests to Read API.
Trade-off
Horizontal scaling adds more servers to manage.
In the primer
Write API
What it is
The application endpoint that creates a paste.
Its job here
It receives a create request from Web Server. It checks SQL Write Master–Slave for a duplicate URL, saves the record, and saves contents to Object Store.
Trade-off
SQL or NoSQL is a design trade-off for this lookup.
In the primer
Read API
What it is
The application endpoint that retrieves a paste.
Its job here
It receives a get request from Web Server. It checks Memory Cache, then SQL Read Replicas for the URL and Object Store for contents.
Trade-off
Cache misses still depend on the database.
In the primer
Analytics
What it is
A process that turns web server logs into hit counts.
Its job here
It receives Web Server logs. It hands monthly page data to SQL Analytics.
Trade-off
Page view analytics do not need to be realtime.
In the primer
Memory Cache
What it is
An in-memory store for popular content.
Its job here
Read API checks it before the database. A cache miss goes to SQL Read Replicas.
Trade-off
The cache helps with uneven traffic and traffic spikes.
In the primer
SQL Write Master–Slave
What it is
The SQL database for paste records and unique shortlinks.
Its job here
Write API checks it for a duplicate and saves the pastes record. It replicates changes to SQL Read Replicas.
Trade-off
More write scaling may need federation, sharding, denormalization, or SQL tuning.
In the primer
SQL Read Replicas
What it is
Replicated SQL databases that serve reads.
Its job here
They receive replicated writes from SQL Write Master–Slave. Read API uses them after a cache miss.
Trade-off
Replicas must not be bogged down with replicating writes.
In the primer
SQL Analytics
What it is
A database for analytics results.
Its job here
Analytics writes page hit counts here. It stores the results of processing Web Server logs.
Examples
- Amazon Redshift
- Google BigQuery
Trade-off
Realtime analytics are not a requirement.
In the primer
Object Store
What it is
Managed storage for paste contents.
Its job here
Write API saves paste data here. Read API fetches contents from it after it finds the generated URL.
Examples
- Amazon S3
Trade-off
Pastebin stores paste contents instead of an original unshortened URL.
In the primer
←/→ step · Space play/pause · R replay · Home/End jump
Detail · follows playback, or click any box or step to pin it
Close · follow playbackClick any box for an explanation, or use Next or Play to walk through the four steps.
1 · Create a paste
Write API creates the generated URL and saves the record and paste contents.
The request moves from Client to Web Server to Write API, SQL Write Master–Slave, and Object Store.
2 · View a paste
Read API finds the generated URL and fetches paste contents.
The read path checks Memory Cache first, then SQL Read Replicas and Object Store.
3 · Track analytics
MapReduce turns Web Server logs into page hit counts.
Analytics writes the resulting counts to SQL Analytics.
4 · Delete expired pastes
Expired records are found by their expiration timestamp.
The service deletes the entries or marks them expired.