Document Open Access Logo

Making Democracy Work: Fixing and Simplifying Egalitarian Paxos

Authors Fedor Ryabinin, Alexey Gotsman , Pierre Sutra

PDF

File

Thumbnail PDF
PDF
LIPIcs.OPODIS.2025.22.pdf
  • Filesize: 1 MB
  • 19 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Distributed computing models
Keywords
  • Consensus
  • state-machine replication
  • fault tolerance

Metrics

Abstract

Classical state-machine replication protocols, such as Paxos, rely on a distinguished leader process to order commands. Unfortunately, this approach makes the leader a single point of failure and increases the latency for clients that are not co-located with it. As a response to these drawbacks, Egalitarian Paxos [Iulian Moraru et al., 2013] introduced an alternative, leaderless approach, that allows replicas to order commands collaboratively. Not relying on a single leader allows the protocol to maintain non-zero throughput with up to f crashes of any processes out of a total of n = 2f+1. The protocol furthermore allows any process to execute a command c fast, in 2 message delays, provided no more than e = ⌈(f+1)/2⌉ other processes fail, and all concurrently submitted commands commute with c; the latter condition is often satisfied in practical systems.
Egalitarian Paxos has served as a foundation for many other replication protocols. But unfortunately, the protocol is very complex, ambiguously specified and suffers from nontrivial bugs. In this paper, we present EPaxos* - a simpler and correct variant of Egalitarian Paxos. Our key technical contribution is a simpler failure-recovery algorithm, which we have rigorously proved correct. Our protocol also generalizes Egalitarian Paxos to cover the whole spectrum of failure thresholds f and e such that n ≥ max{2e+f-1, 2f+1} - the number of processes that we show to be optimal.

Cite As Get BibTex

Fedor Ryabinin, Alexey Gotsman, and Pierre Sutra. Making Democracy Work: Fixing and Simplifying Egalitarian Paxos. In 29th International Conference on Principles of Distributed Systems (OPODIS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 361, pp. 22:1-22:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.OPODIS.2025.22

Author Details

Fedor Ryabinin
  • IMDEA Software Institute, Madrid, Spain
Alexey Gotsman
  • IMDEA Software Institute, Madrid, Spain
Pierre Sutra
  • Institut Polytechnique de Paris, Palaiseau, France

Funding

This work was partially supported by the projects BYZANTIUM, funded by MCIN/AEI and NextGenerationEU/PRTR; DECO and María de Maeztu, funded by MICIU/AEI; and REDONDA, funded by the EU CHIST-ERA network.

Acknowledgements

We thank Benedict Elliott Smith, whose work on leaderless consensus in Apache Cassandra inspired this paper.

References

  1. URL: https://github.com/efficient/epaxos.
  2. Karolos Antoniadis, Julien Benhaim, Antoine Desjardins, Elias Poroma, Vincent Gramoli, Rachid Guerraoui, Gauthier Voron, and Igor Zablotchi. Leaderless consensus. J. Parallel Distributed Comput., 176, 2023. URL: https://doi.org/10.1016/J.JPDC.2023.01.009.
  3. Balaji Arun, Sebastiano Peluso, Roberto Palmieri, Giuliano Losa, and Binoy Ravindran. Speeding up consensus by chasing fast decisions. In Conference on Dependable Systems and Networks (DSN), 2017. Google Scholar
  4. Matthew Burke, Audrey Cheng, and Wyatt Lloyd. Gryff: Unifying consensus and shared registers. In Symposium on Networked Systems Design and Implementation (NSDI), 2020. Google Scholar
  5. Tushar Deepak Chandra and Sam Toueg. Unreliable failure detectors for reliable distributed systems. J. ACM, 43(2), 1996. URL: https://doi.org/10.1145/226643.226647.
  6. James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson C. Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. Spanner: Google’s globally-distributed database. In Symposium on Operating Systems Design and Implementation (OSDI), 2012. Google Scholar
  7. Cynthia Dwork, Nancy A. Lynch, and Larry J. Stockmeyer. Consensus in the presence of partial synchrony. J. ACM, 35(2), 1988. URL: https://doi.org/10.1145/42282.42283.
  8. Vitor Enes, Carlos Baquero, Alexey Gotsman, and Pierre Sutra. Efficient replication via timestamp stability. In European Conference on Computer Systems (EuroSys), 2021. Google Scholar
  9. Vitor Enes, Carlos Baquero, Tuanir França Rezende, Alexey Gotsman, Matthieu Perrin, and Pierre Sutra. State-machine replication for planet-scale systems. In European Conference on Computer Systems (EuroSys), 2020. Google Scholar
  10. Maurice Herlihy and Jeannette M. Wing. Linearizability: A correctness condition for concurrent objects. ACM Trans. Program. Lang. Syst., 12(3), 1990. URL: https://doi.org/10.1145/78969.78972.
  11. Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Commun. ACM, 21(7), 1978. URL: https://doi.org/10.1145/359545.359563.
  12. Leslie Lamport. The part-time parliament. ACM Trans. Comput. Syst., 16(2), 1998. URL: https://doi.org/10.1145/279227.279229.
  13. Leslie Lamport. Generalized consensus and Paxos. Technical Report MSR-TR-2005-33, Microsoft Research, 2005. Google Scholar
  14. Leslie Lamport. Fast Paxos. Distributed Comput., 19(2), 2006. URL: https://doi.org/10.1007/S00446-006-0005-X.
  15. Leslie Lamport. Lower bounds for asynchronous consensus. Distributed Comput., 19(2), 2006. URL: https://doi.org/10.1007/S00446-006-0155-X.
  16. Giuliano Losa, Sebastiano Peluso, and Binoy Ravindran. Brief announcement: A family of leaderless generalized-consensus algorithms. In Symposium on Principles of Distributed Computing (PODC), 2016. Google Scholar
  17. Iulian Moraru. Egalitarian distributed consensus. Technical report, CMU-CS-14-133, Carnegie Mellon University, PhD Thesis, 2014. Google Scholar
  18. Iulian Moraru, David G. Andersen, and Michael Kaminsky. A proof of correctness for Egalitarian Paxos. Technical Report CMU-PDL-13-111, Carnegie Mellon University, 2013. Google Scholar
  19. Iulian Moraru, David G. Andersen, and Michael Kaminsky. There is more consensus in egalitarian parliaments. In Symposium on Operating Systems Principles (SOSP), 2013. Google Scholar
  20. Shuai Mu, Yang Cui, Yang Zhang, Wyatt Lloyd, and Jinyang Li. Extracting more concurrency from distributed transactions. In Symposium on Operating Systems Design and Implementation (OSDI), 2014. Google Scholar
  21. Shuai Mu, Lamont Nelson, Wyatt Lloyd, and Jinyang Li. Consolidating concurrency control and consensus for commits under conflicts. In Symposium on Operating Systems Design and Implementation (OSDI), 2016. Google Scholar
  22. Diego Ongaro and John K. Ousterhout. In search of an understandable consensus algorithm. In USENIX Annual Technical Conference (USENIX ATC), 2014. Google Scholar
  23. Fernando Pedone and André Schiper. Generic broadcast. In International Symposium on Distributed Computing (DISC), 1999. Google Scholar
  24. Fernando Pedone and André Schiper. Brief announcement: On the inherent cost of generic broadcast. In Symposium on Principles of Distributed Computing (PODC), 2004. Google Scholar
  25. Tuanir França Rezende and Pierre Sutra. Leaderless state-machine replication: Specification, properties, limits. In Symposium on Distributed Computing (DISC), 2020. Google Scholar
  26. Fedor Ryabinin, Alexey Gotsman, and Pierre Sutra. Brief announcement: Revisiting lower bounds for two-step consensus. In Symposium on Principles of Distributed Computing (PODC), 2025. Google Scholar
  27. Fedor Ryabinin, Alexey Gotsman, and Pierre Sutra. Making democracy work: Fixing and simplifying Egalitarian Paxos (extended version). arXiv, 2511.02743, 2025. URL: http://arxiv.org/abs/2511.02743.
  28. Fred B. Schneider. Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Comput. Surv., 22, 1990. URL: https://doi.org/10.1145/98163.98167.
  29. Benedict Elliott Smith, Tony Zhang, Blake Eggleston, and Scott Andreas. CEP-15: Fast general purpose transactions. https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf, 2023. Google Scholar
  30. Pierre Sutra. On the correctness of Egalitarian Paxos. Inf. Process. Lett., 156, 2020. URL: https://doi.org/10.1016/J.IPL.2019.105901.
  31. Sarah Tollman, Seo Jin Park, and John K. Ousterhout. EPaxos revisited. In Symposium on Networked Systems Design and Implementation (NSDI), 2021. Google Scholar
  32. Michael J. Whittaker, Neil Giridharan, Adriana Szekeres, Joseph M. Hellerstein, and Ion Stoica. SoK: A generalized multi-leader state machine replication tutorial. J. Syst. Res., 1(1), 2021. URL: https://doi.org/10.5070/SR31154817.
  33. Piotr Zieliński. Optimistic generic broadcast. In Conference on Distributed Computing (DISC), 2005. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact