HLD Problem

Design Twitter/X

Design a social media platform with timelines, tweets, followers, likes, retweets, and real-time notifications at massive scale.

45 min readHard

1Requirements Gathering

Before diving into design, clarify requirements with the interviewer:

Functional Requirements
  • •Post tweets (text, images, videos - 280 chars)
  • •Follow/unfollow users
  • •View home timeline (tweets from followed users)
  • •Like, retweet, quote tweet, reply
  • •Search tweets and users
  • •Notifications (mentions, likes, retweets)
  • •Direct messages (optional)
  • •Trending topics/hashtags
Non-Functional Requirements
  • •High availability (99.99%)
  • •Low latency reads (< 200ms timeline)
  • •Eventual consistency acceptable
  • •Handle viral tweets (celebrity posts)
  • •Scale to 500M DAU, 1B+ users
  • •Handle 500M tweets/day
  • •Global distribution
Clarifying Questions to Ask
  • What is the character limit for tweets?
  • Do we need to support media (images/videos)?
  • How real-time should the timeline be?
  • Do we need DMs or just focus on public features?
  • What is the expected scale (users, tweets/day)?

2Capacity Estimation

Traffic Estimates

Users:
  • Total users: 1 billion
  • Daily active users: 500 million
  • Avg follows per user: 200
Tweets:
  • Tweets per day: 500 million
  • Tweets per second: ~6,000
  • Read requests/sec: ~600,000

Storage Estimates

Tweet metadata (500M/day × 500B)250 GB/day
Media storage (20% with media, avg 1MB)100 TB/day
5 year storage~180 PB

Bandwidth Estimates

Read-heavy system (100:1 read-to-write ratio)

  • Ingress: ~3 GB/s (tweets + media uploads)
  • Egress: ~300 GB/s (timeline reads, media serving)

3High-Level Architecture

System Architecture

Mobile
Web
API
↓
CDN
Media, Static
Load Balancer
L7 Routing
↓
API Gateway
Auth, Rate Limit, Routing
↓
Tweet Service
Create, read tweets
Timeline Service
Build user feeds
User Service
Profiles, auth
Follow Service
Follow graph
Search Service
Tweet/user search
Notification
Push, email, SMS
Media Service
Image/video process
Analytics
Trending, metrics
↓
Tweets DB
Sharded MySQL
Timeline Cache
Redis Cluster
Social Graph
Graph DB
Search Index
Elasticsearch

4Core Components Deep Dive

4.1 Timeline Generation (The Heart of Twitter)

Pull Model (Fan-out on Read)
When user opens app, fetch tweets from all followed users and merge.
  • ✅ Simple, real-time
  • ✅ Good for users with few followers
  • ❌ Slow for users following many accounts
  • ❌ High read latency
Push Model (Fan-out on Write)
When user tweets, push to all followers timelines immediately.
  • ✅ Fast reads (pre-computed timeline)
  • ✅ Great for most users
  • ❌ Celebrity problem (millions of followers)
  • ❌ Wasted work if followers are inactive
Twitter's Hybrid Approach

For regular users (<10K followers): Push model - fan-out tweets to followers' timelines in Redis.

For celebrities (>10K followers): Pull model - merge celebrity tweets at read time. This avoids pushing to millions of timelines.

4.2 Tweet Storage

Tweet Table Schema
tweets ├── tweet_id (bigint, PK, Snowflake ID) ├── user_id (bigint, FK) ├── content (varchar 280) ├── media_urls (json) ├── reply_to_tweet_id (bigint, nullable) ├── retweet_of_id (bigint, nullable) ├── like_count (int) ├── retweet_count (int) ├── reply_count (int) ├── created_at (timestamp) └── is_deleted (boolean)
Sharded by user_id for write locality. Use Snowflake IDs for globally unique, time-sortable IDs.

4.3 Social Graph (Follow Relationships)

Adjacency List (Simple)
follows ├── follower_id ├── followee_id ├── created_at PK: (follower_id, followee_id)
Graph Database (Complex queries)
Use Neo4j or similar for:
  • Friends of friends
  • Suggested follows
  • Mutual followers

5API Design

POST/api/v1/tweets
Create a new tweet
Request:
{
  "content": "Hello Twitter!",
  "media_ids": ["123", "456"],
  "reply_to": null
}
Response:
{
  "tweet_id": "1234567890",
  "created_at": "2024-01-15T10:30:00Z"
}
GET/api/v1/timeline/home?cursor=xxx&limit=20
Get home timeline (paginated)
Response:
{
  "tweets": [...],
  "next_cursor": "abc123",
  "has_more": true
}
POST/api/v1/users/:id/follow
Follow a user
POST/api/v1/tweets/:id/like
Like a tweet
GET/api/v1/search?q=keyword&type=tweets
Search tweets or users

6Scaling Strategies

Database Sharding
  • Shard tweets by user_id
  • Shard timeline cache by user_id
  • Use consistent hashing for distribution
  • Cross-shard queries for search
Caching Strategy
  • Cache timelines in Redis (last 800 tweets)
  • Cache user profiles and follow counts
  • Cache hot tweets (viral content)
  • Use CDN for media files
Async Processing
  • Fan-out via message queue (Kafka)
  • Async media processing
  • Async notification delivery
  • Eventual consistency for counts
Read Replicas
  • Multiple read replicas per shard
  • Route reads to nearest replica
  • Handle replication lag gracefully
  • Write to primary, read from replica

7Handling Edge Cases

Celebrity Tweet (Viral Content)
Problem: User with 50M followers posts - pushing to all timelines would take hours.
Solution: Don't fan-out for celebrities. Pull their tweets at read time and merge with pre-computed timeline. Cache celebrity tweets aggressively.
Hot Tweet (Going Viral)
Problem: A tweet suddenly gets millions of likes/retweets - database hotspot.
Solution: Use Redis for like/retweet counters. Batch update to DB every few seconds. Shard counters across multiple keys.
Spam/Abuse
Problem: Bots creating fake accounts and spam tweets.
Solution: Rate limiting per user/IP. ML-based spam detection. Phone verification for new accounts. Shadow banning.

8Key Takeaways

1Hybrid timeline: Push for regular users, pull for celebrities.
2Heavy caching: Redis for timelines, CDN for media.
3Async fan-out: Use message queues for timeline updates.
4Sharding: Shard by user_id for write locality.
5Use Snowflake IDs for globally unique, time-sortable IDs.
6Design for read-heavy workload (100:1 read-to-write).