Module 6 - Distributed Systems

Leader Election

Choosing one node to coordinate when multiple can do the job.

1The Team Captain Analogy

Simple Analogy
A sports team needs one captain to call plays. All players could do it, but having multiple captains causes confusion. If the captain gets injured, the team quickly elects a new one. Leader election works the same way.

Leader election is a process where distributed nodes agree on exactly one node to act as leader. The leader coordinates actions that require single authority.

2Why Elect a Leader?

Single Writer

Only leader writes to DB. Avoids write conflicts.

Task Coordination

Leader assigns work to followers.

Cron Jobs

Only leader runs scheduled tasks.

Consensus

Leader proposes values in Raft/Paxos.

3Election Approaches

Bully Algorithm

Highest ID wins. Node detects leader failure, starts election, higher IDs take over.

+ Simple- Lots of messages, assumes IDs are ordered

Ring Algorithm

Nodes arranged in ring. Election message travels around, collecting votes.

+ Less messages than bully- Slow, ring must be maintained

Raft/Paxos

Consensus-based. Nodes vote, majority wins. Production-grade.

+ Battle-tested, handles partitions- Complex implementation

Zookeeper/etcd

Use coordination service. Nodes race to create ephemeral node.

+ Easy to use, reliable- External dependency

4Real-World Dry Run: Zookeeper Election

Scenario: 3 Kafka brokers electing controller

1
Broker 1, 2, 3 start
All try to create /controller ephemeral node in Zookeeper
2
Broker 2 wins
Creates /controller first. Becomes controller.
3
Brokers 1, 3 watch /controller
Set watch on the node to detect changes.
4
Broker 2 crashes
Ephemeral node deleted (session ends).
5
Watch triggers
Brokers 1, 3 notified. Race to create /controller.
6
Broker 3 wins
New controller elected. Cluster continues.

5Split-Brain Problem

Split-brain: Network partition causes two nodes to both think they're leader. Dangerous! Can cause data corruption.

Fencing Tokens

Each leader gets a monotonically increasing token. Storage rejects writes with older tokens.

Quorum

Require majority vote to be leader. Both sides of partition can't have majority.

Lease Timeout

Leader must renew lease. If partition, old leader's lease expires.

STONITH

Shoot The Other Node In The Head. Force-kill suspected failed leader.

6Key Takeaways

1Leader election chooses one coordinator from many nodes
2Use for single-writer, task assignment, cron jobs
3Zookeeper/etcd are easiest for production use
4Split-brain is dangerous-use fencing tokens + quorum
5Leaders must renew leases or step down

?Quiz

1. Network partition: old leader on one side, new leader elected on other. What's this called?

2. How do fencing tokens prevent split-brain damage?