Source: https://dev.to/fauna/the-why-and-how-of-distributed-databases-4p1p
The design of distributed databases implies that the states of a system should be in sync across all nodes, although they can be physically separated over multiple networks. However, the networks are unreliable, and there are plenty of possible points of failure in modern cloud-based multi-node databases. The problem exists because any client-server architecture assumes that the single database instance can be accessed by all clients. At the same time, a distributed database system requires that each node maintains an up-to-date version of the state. The main challenges of distributed databases are the result of network partitioning and the necessity to make a trade-off between the CAP theorem principles of consistency, availability, and partition tolerance. The CAP theorem is the basis for understanding the problems of distributed database design. According to the CAP conjecture, it is impossible to design a distributed system that simultaneously satisfies the three principles of consistency, availability, and partition tolerance. It is evident that partition tolerance is the fundamental requirement for any distributed system since there is no guarantee of network reliability. The network partition creates a situation where a database has physically separated nodes that cannot communicate with each other. Therefore, any modern distributed database assumes that the network can be partitioned either symmetrically or asymmetrically, and the architecture should be able to handle both scenarios. For instance, AP systems will sacrifice consistency to maintain availability when the network partition occurs. On the other hand, CP systems will sacrifice availability by denying updates on the sides of physically separated nodes. The main challenge in the implementation of distributed databases is the ability to make the right choice when a network partition happens. In general, such databases utilize consensus algorithms in order to maintain consensus among nodes. There are multiple algorithms that allow the design of a consensus mechanism, but they are all based on a set of principles that should be taken into account. For example, the Raft consensus algorithm explicitly separates the problem of achieving consensus into several sub-problems, including leader election, log replication, and decision-making. At the same time, the Raft algorithm is the first one to clearly define the notions of cluster leadership, candidate, and follower. According to the Raft algorithm, the leader is elected, and all decisions are made by the leader through log entries that are then replicated to the other nodes in the cluster. The consensus is only committed once it is replicated to a quorum, meaning that the number of updates exceeds half of the total number of nodes. The quorum is the central concept in terms of partition tolerance, which implies that the system will be able to withstand a network partition due to the replication of information. In other words, if the network is physically separated into two clusters, the one that contains the quorum will continue to process updates in order to avoid conflicting writes. The challenge of AP systems, on the other hand, is related to the need to make sure that the majority of nodes in the cluster will be able to continue processing updates. If asymmetric partitioning occurs, when the messages from A to B are delivered, but B to A are not, there is a possibility of a split-brain scenario, when node A will be able to update the state while being physically separated from the rest of the cluster. In such a case, the updates on the side of node B will be denied because the node will realize that it does not have the quorum. Such principles are essential for consensus algorithms that use the concept of heartbeats, timeouts, and pre-votes in order to avoid split-brain scenarios. When a network partition occurs, the nodes that are affected will see that some of their updates are not committed to the master. Modern distributed databases utilize a set of solutions in order to be able to recover from a network partition. The first type of solution includes heartbeat timeouts, pre-votes, and leasing logic that allows the system to deny split-brain updates. For example, the master node will have to step down if it does not receive heartbeats from a majority of nodes in the cluster. Second, the nodes that were physically separated from the majority of the cluster will realize that the updates that they committed are now stale updates. At the same time, the recovery from network partition also implies that the majority quorum will need to update the lagging nodes in order for the system to restore consistency and update the state of the system. The leader of the majority quorum will be responsible for sending append-entries updates to update the lagging nodes. The updates will overwrite any conflicting entries that were written to the lagging nodes by the local leader during the timeline of the network partition. Alternatively, in the case of AP systems, the vector clock, conflict-free replicated data types, or last-write-wins strategy should be used to reconcile the conflicting updates.
Sources
https://www.mongodb.com/resources/basics/databases/cap-theorem https://www.pingcap.com/article/understanding-cap-theorem-basics-in-distributed-systems/ CAP Theorem for System Design Interviews explanation.