HLD Problem

Design Instagram

Design a photo and video sharing social network with feeds, stories, reels, likes, comments, and direct messaging at massive scale.

45 min readHard

1Requirements Gathering

Functional Requirements
  • •Upload photos and videos
  • •Follow/unfollow users
  • •News feed (posts from followed users)
  • •Stories (24-hour ephemeral content)
  • •Like, comment, save posts
  • •Direct messaging
  • •Search users and hashtags
  • •Explore/discover page
  • •Notifications
Non-Functional Requirements
  • •High availability (99.99%)
  • •Low latency feeds (< 500ms)
  • •Support 2B+ users, 500M DAU
  • •Handle 100M+ photo uploads/day
  • •Fast image/video loading
  • •Global content delivery
  • •Eventual consistency acceptable

2Capacity Estimation

Scale Numbers

2B+
Total Users
500M
Daily Active Users
100M+
Photos/Day

Storage Estimates

Average photo size (after compression)200 KB
Daily photo storage (100M × 200KB)20 TB/day
With multiple resolutions (4x)80 TB/day
10-year storage requirement~300 PB

3High-Level Architecture

System Architecture

iOS App
Android App
Web App
↓
CDN (CloudFront/Akamai)
Images, Videos, Static Assets
↓
Load Balancer
API Gateway
↓
Post Service
CRUD posts
Feed Service
Generate feeds
User Service
Profiles, auth
Story Service
Ephemeral content
Media Service
Upload, process
Search Service
Users, hashtags
Message Service
DMs
Notification
Push notifications
↓
PostgreSQL
Users, Posts
Redis
Feeds, Sessions
Cassandra
Likes, Comments
S3
Media Storage
Elasticsearch
Search Index

4Core Components Deep Dive

4.1 Photo Upload Flow

1
Client uploads photo
App sends photo to Media Service via presigned S3 URL
2
S3 stores original
Photo stored in S3 bucket, triggers Lambda
3
Image processing
Lambda resizes to multiple sizes (thumbnail, medium, full), applies filters
4
CDN distribution
Processed images pushed to CDN edge locations
5
Metadata saved
Post metadata saved to PostgreSQL, feed updated
6
Fan-out
Push to followers' feed caches via async queue

4.2 News Feed Generation

Pull Model
Fetch posts from all followed users at request time. Slow for users following many accounts.
Push Model (Instagram uses)
Pre-compute and cache feeds in Redis. When user posts, fan-out to all followers feed caches.
Feed Cache Structure (Redis Sorted Set):
ZADD user:123:feed <timestamp> <post_id>
ZREVRANGE user:123:feed 0 19  # Get latest 20 posts

Key: user:{user_id}:feed
Score: post timestamp
Value: post_id

4.3 Stories (24-Hour Ephemeral Content)

  • Store in Redis with TTL of 24 hours
  • Group by user: stories:user_id:story_id
  • Track views: story_views:story_id → SET of viewer_ids
  • Archive to Cassandra for analytics after expiry
  • Use separate CDN cache with 24h max-age

5Database Design

Users Table
users ├── user_id (PK) ├── username (unique) ├── email ├── password_hash ├── profile_pic_url ├── bio ├── follower_count ├── following_count ├── post_count └── created_at
Posts Table
posts ├── post_id (PK) ├── user_id (FK) ├── media_urls (array) ├── caption ├── location ├── like_count ├── comment_count ├── created_at └── is_archived
Likes/Comments (Cassandra - High Write Volume)
likes (partition by post_id) ├── post_id (partition key) ├── user_id (clustering key) └── created_at comments ├── post_id (partition key) ├── comment_id (clustering key) ├── user_id ├── text └── created_at

6API Design

POST/api/v1/posts
Create a new post
GET/api/v1/feed?cursor=xxx&limit=20
Get user's news feed (paginated)
POST/api/v1/posts/:id/like
Like a post
POST/api/v1/stories
Create a story (auto-expires in 24h)
GET/api/v1/stories/feed
Get stories from followed users

7Scaling Strategies

Image Optimization
  • Generate multiple resolutions on upload
  • WebP format for smaller size
  • Lazy loading for off-screen images
  • Progressive JPEGs for faster perceived load
CDN Strategy
  • Edge caching for images/videos
  • Geographic distribution
  • Origin shield to reduce S3 costs
  • Cache invalidation on delete
Database Sharding
  • Shard users by user_id
  • Shard posts by user_id (locality)
  • Shard likes/comments by post_id
  • Cross-shard for explore/search
Async Processing
  • Kafka for feed fan-out
  • SQS for image processing
  • Background notification delivery
  • Async analytics updates

8Key Takeaways

1CDN is critical - images/videos are served from edge locations.
2Pre-compute feeds using push model with Redis caches.
3Multiple image sizes - generate on upload, serve appropriate size.
4Use Cassandra for high-write data (likes, comments).
5Stories use TTL - Redis with 24h expiration.
6Async processing - queue-based image processing and fan-out.