Learn about the common techniques and challenges of distributed locking in distributed systems, and how to avoid concurrency issues and deadlocks.

Centralized locking results in a single point of failure ad might become a potential bottleneck at large scale. It's like having a building with many doors but only a single key.

The easiest way is to have a redis and use that as a storage, the redis itself has a Redlock algorithm which helps you if you have multiple instances of redis.

What are some common distributed locking patterns and anti-patterns that you have encountered or used?

Distributed locking is a technique to coordinate the access and modification of shared resources among multiple processes or nodes in a distributed system. It can help prevent concurrency issues, such as data inconsistency, race conditions, or deadlocks. However, distributed locking also introduces some challenges and trade-offs, such as network latency, availability, scalability, and fault tolerance. In this article, we will explore some common distributed locking patterns and anti-patterns that you have encountered or used, and discuss their benefits and drawbacks.

1 Centralized locking

One of the simplest and most intuitive distributed locking patterns is to use a centralized service or component to manage the locks for the shared resources. For example, you can use a database, a message queue, a file system, or a dedicated locking service, such as ZooKeeper or etcd, to store and grant the locks. The advantage of this approach is that it is easy to implement and reason about, and it can provide strong consistency and atomicity guarantees. The disadvantage is that it creates a single point of failure and a bottleneck for the system, and it can increase the network overhead and latency for acquiring and releasing the locks.

Add your perspective

Neo Kim

Writes System Design Newsletter • I Teach You System Design
Report contribution
Centralized locking results in a single point of failure ad might become a potential bottleneck at large scale. It's like having a building with many doors but only a single key.

Like
Diar S.

Senior Software Engineer at Nordvpn
Report contribution
The easiest way is to have a redis and use that as a storage, the redis itself has a Redlock algorithm which helps you if you have multiple instances of redis.

Like

2 Consensus-based locking

Another common distributed locking pattern is to use a consensus algorithm, such as Paxos or Raft, to elect a leader among a group of nodes, and delegate the locking responsibility to the leader. The leader node can then act as a centralized lock manager, or use its authority to assign locks to other nodes. The advantage of this approach is that it can tolerate the failure of some nodes, and it can provide fault-tolerant and consistent locking semantics. The disadvantage is that it can be complex to implement and maintain, and it can also incur network overhead and latency for reaching consensus and communicating with the leader.

Add your perspective

Dhawal Rank

Sr. Full Stack Developer
Report contribution
In highly contended scenarios, where many processes are competing for the same lock, consensus-based locking can lead to bottlenecks and decreased throughput as processes wait for access to the shared resource.

Like

3 Decentralized locking

A third distributed locking pattern is to use a decentralized scheme, such as a distributed hash table (DHT), a gossip protocol, or a logical clock, to distribute the locks among the nodes without relying on a central authority or a leader. For example, you can use a DHT to map the shared resources to the nodes that are responsible for locking them, or you can use a gossip protocol to propagate the lock status among the nodes, or you can use a logical clock to order the lock requests based on timestamps. The advantage of this approach is that it can improve the scalability and availability of the system, and it can reduce the network overhead and latency for acquiring and releasing the locks. The disadvantage is that it can introduce weaker consistency and synchronization guarantees, and it can be prone to conflicts and collisions.

Add your perspective

4 Optimistic locking

A distributed locking anti-pattern is to use optimistic locking, which is a technique that assumes that conflicts are rare, and does not acquire a lock before accessing or modifying a shared resource. Instead, it checks the version or state of the resource before and after the operation, and aborts or retries the operation if the resource has changed. The advantage of this approach is that it can avoid blocking and waiting for locks, and it can improve the performance and concurrency of the system. The disadvantage is that it can cause more aborts and retries, and it can increase the complexity and overhead of the conflict detection and resolution mechanisms.

Add your perspective

Jintao Zhang

Software Engineer, Machine Learning at Meta
(edited)
Report contribution
I believe this is what DynamoDB is using for resolving version conflicts. It guarantees scalability and low latency but increases complication and creates burden to client side to resolve version conflicts.

Like

5 Lock-free algorithms

Another distributed locking anti-pattern is to use lock-free algorithms, which are algorithms that do not use any locks or synchronization primitives, but rely on atomic operations, such as compare-and-swap (CAS), to coordinate the access and modification of shared resources. The advantage of this approach is that it can eliminate the problems of deadlock, livelock, and starvation, and it can achieve high performance and scalability. The disadvantage is that it can be very difficult to design and implement correctly, and it can have limited applicability and functionality.

Add your perspective

Yong Zhang

Senior Engineering Manager Flexport | Ex-Amazon | Ex-Intel
Report contribution
The pattern requires each of clients of the shared resources to implement their own algorithm to address potential concurrency issues, it's hard and not a good practice to build the distrbuted system. Whe you build a large distributed system, imaging each service within the system will be owned by different team, in such system, each service owner should follow the same standard/guiding principal: do not ask clients to take the responsibility which should be owned by the service provider. If different team follow different standard, the final system will be in mess. As the service provider, do not only provide lock free service for your client, you should provide the option for client to choose based on their own use cases.

Like

6 Lock coupling

A final distributed locking anti-pattern is to use lock coupling, which is a technique that acquires multiple locks at the same time, or holds one lock while acquiring another lock. This can lead to deadlock, which is a situation where two or more processes are waiting for each other to release a lock, and none of them can proceed. The solution to this problem is to use lock ordering, which is a technique that defines a global or partial order for acquiring the locks, and ensures that every process follows the same order. This can prevent circular dependencies and deadlocks among the locks.

Add your perspective

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

Add your perspective

What are some common distributed locking patterns and anti-patterns that you have encountered or used?

1

2

3

4

5

6

7

1 Centralized locking

2 Consensus-based locking

3 Decentralized locking

4 Optimistic locking

5 Lock-free algorithms

6 Lock coupling

7 Here’s what else to consider

Distributed Systems

Rate this article

Thanks for your feedback

More articles on Distributed Systems

More relevant reading