Module 6 - Distributed Systems

Distributed Locks

Ensuring only one process accesses a resource across multiple nodes.

1The Bathroom Key Analogy

Simple Analogy
A coffee shop has one bathroom with one key. Whoever has the key can use it. Others wait. When done, you return the key. But what if someone takes the key and leaves? You need a timeout to reclaim it. That's a distributed lock with TTL.

A distributed lock ensures mutual exclusion across multiple nodes. Only one process can hold the lock at a time. Critical for preventing race conditions.

2Use Cases

Cron Jobs

Only one instance should run scheduled task

Rate Limiting

Coordinate rate limits across instances

Leader Election

Only one node is the leader

Cache Updates

Prevent thundering herd on cache miss

Inventory

Prevent overselling the last item

Payment Processing

Process each payment exactly once

3Redis Lock Implementation

Simple Redis Lock with SETNX

1Acquire Lock
SET lock_key unique_value NX PX 30000

NX = only if not exists, PX = expire in 30 seconds

2Do Work
// Critical section

Process that requires exclusive access

3Release Lock
DEL lock_key (if value matches)

Only delete if you own the lock (check value first)

Warning: Check Before Delete

Always verify the lock value before deleting. Otherwise, you might delete someone else's lock if yours expired.

4Real-World Dry Run

Scenario: Two workers processing same job

1
Worker A: SET job:123 workerA NX PX 30000
Returns OK. Lock acquired.
2
Worker B: SET job:123 workerB NX PX 30000
Returns nil. Lock denied.
3
Worker A processes job
30 seconds to complete
4
Worker A: DEL job:123 (after checking value)
Lock released.
5
Worker B retries and acquires lock
Now B can process.

5Redlock Algorithm

Redlock by Redis author: acquire lock on majority of Redis instances (N/2+1). Safer for distributed deployments but more complex.

Steps
  • 1. Get current time
  • 2. Try to acquire lock on all N instances
  • 3. Calculate elapsed time
  • 4. Lock valid if majority acquired + time left
Requirements
  • • Odd number of Redis instances (3, 5, 7)
  • • Independent failure domains
  • • Clock drift consideration
  • • Controversial: some experts disagree

6Key Takeaways

1Distributed lock = mutual exclusion across nodes
2Always use TTL to prevent dead locks
3Redis SETNX with PX is simple and common
4Check lock value before deleting (fencing token)
5Redlock for higher availability but more complex

?Quiz

1. Worker A's lock expires while still processing. Worker B acquires. A finishes and deletes. What happens?

2. Redlock requires lock on how many Redis instances for N=5?