Document Open Access Logo

Distributed Recoverable Sketches

Authors Diana Cohen , Roy Friedman , Rana Shahout

PDF
Thumbnail PDF

File

LIPIcs.OPODIS.2024.23.pdf
  • Filesize: 1.31 MB
  • 16 pages

Document Identifiers

Author Details

Diana Cohen
  • Computer Science Department, Technion, Haifa, Israel
Roy Friedman
  • Computer Science Department, Technion, Haifa, Israel
Rana Shahout
  • Computer Science Department, Harvard University, Cambridge, MA, USA

Funding

This work was partially funded by the Israeli Science Foundation grant #3119/21.

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments.

Cite As Get BibTex

Diana Cohen, Roy Friedman, and Rana Shahout. Distributed Recoverable Sketches. In 28th International Conference on Principles of Distributed Systems (OPODIS 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 324, pp. 23:1-23:16, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024) https://doi.org/10.4230/LIPIcs.OPODIS.2024.23

Abstract

Sketches are commonly used in computer systems and network monitoring tools to provide efficient query executions while maintaining a compact data representation. Switches and routers maintain sketches to track statistical characteristics of the network traffic. The availability of such data is essential for the network analysis as a whole. Consequently, being able to recover sketches is critical following a switch crash.
In this paper, we explore how nodes in a network environment can cooperate to recover sketch data whenever any of them crashes. In particular, we focus on frequency estimation linear sketches, such as the Count-Min Sketch. We consider various approaches to ensure data reliability and explore the trade-offs between space consumption, runtime overheads, and traffic during recovery, which we point out as design guidelines. Besides different aspects of efficacy, we design a modular system for ease of maintenance and further scaling.
A key aspect we examine is how nodes update each other about their sketch content as it evolves over time. In particular, we compare between periodic full updates vs. incremental updates. We also examine several data structures to economically represent and encode a batch of latest changes. Our framework is generic, and other data structures can be plugged-in via an abstract API as long as they implement the corresponding API methods.

Subject Classification

ACM Subject Classification
  • Networks → Network monitoring
  • Computer systems organization → Dependable and fault-tolerant systems and networks
  • Computer systems organization → Reliability
Keywords
  • Sketches
  • Stream Processing
  • Distributed Recovery
  • Incremental Updates
  • Sketch Partitioning

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Nikolas Askitis. Fast and compact hash tables for integer keys. In Proceedings of the Thirty-Second Australasian Conference on Computer Science - Volume 91, ACSC '09, pages 113-122, Darlinghurst, Australia, January 2009. Australian Computer Society, Inc. URL: http://dl.acm.org/citation.cfm?id=1862675.
  2. Ran Ben Basat, Xiaoqi Chen, Gil Einziger, Roy Friedman, and Yaron Kassner. Randomized admission policy for efficient top-k, frequency, and volume estimation. IEEE/ACM Transactions on Networking, 27(4):1432-1445, August 2019. URL: https://doi.org/10.1109/TNET.2019.2918929.
  3. Theophilus Benson, Ashok Anand, Aditya Akella, and Ming Zhang. Microte: fine grained traffic engineering for data centers. In Proceedings of the Seventh COnference on Emerging Networking EXperiments and Technologies, CoNEXT '11, New York, NY, USA, 2011. Association for Computing Machinery. URL: https://doi.org/10.1145/2079296.2079304.
  4. Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theoretical Computer Science, 312(1):3-15, January 2004. Automata, Languages and Programming. URL: https://doi.org/10.1016/S0304-3975(03)00400-6.
  5. Peter M. Chen, Edward K. Lee, Garth A. Gibson, Randy H. Katz, and David A. Patterson. Raid: High-performance, reliable secondary storage. ACM Computing Surveys, 26(2):145-185, June 1994. URL: https://doi.org/10.1145/176979.176981.
  6. Diana Cohen. Measuring average frequency of batched items in Internet traces, November 2024. URL: https://github.com/DianaCohenCS/measure-traces.
  7. Saar Cohen and Yossi Matias. Spectral bloom filters. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, SIGMOD '03, pages 241-252, New York, NY, USA, 2003. Association for Computing Machinery. URL: https://doi.org/10.1145/872757.872787.
  8. Graham Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. Journal of Algorithms, 55(1):58-75, April 2005. URL: https://doi.org/10.1016/j.jalgor.2003.12.001.
  9. Martin Dietzfelbinger and Christoph Weidling. Balanced allocation and dictionaries with tightly packed constant size bins. Theoretical Computer Science, 380(1):47-68, June 2007. Automata, Languages and Programming. URL: https://doi.org/10.1016/j.tcs.2007.02.054.
  10. Gero Dittmann and Andreas Herkersdorf. Network processor load balancing for high-speed links. In Proceedings of the 2002 International Symposium on Performance Evaluation of Computer and Telecommunication Systems, July 2002. Google Scholar
  11. Java SE Documentation. Class hashmap. URL: https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html.
  12. RabbitMQ Documentation. Batch publishing. URL: https://www.rabbitmq.com/docs/publishers#batch-publishing.
  13. Shir Landau Feibish, Yehuda Afek, Anat Bremler-Barr, Edith Cohen, and Michal Shagam. Mitigating dns random subdomain ddos attacks by distinct heavy hitters sketches. In Proceedings of the Fifth ACM/IEEE Workshop on Hot Topics in Web Systems and Technologies, HotWeb '17, New York, NY, USA, 2017. Association for Computing Machinery. URL: https://doi.org/10.1145/3132465.3132474.
  14. B. Mukherjee, L. T. Heberlein, and K. N. Levitt. Network intrusion detection. Netwrk. Mag. of Global Internetwkg., 8(3):26-41, May 1994. URL: https://doi.org/10.1109/65.283931.
  15. Rasmus Pagh and Flemming Friche Rodler. Cuckoo hashing. Journal of Algorithms, 51(2):122-144, May 2004. URL: https://doi.org/10.1016/j.jalgor.2003.12.002.
  16. Nicolas Le Scouarnec. Cuckoo++ hash tables: High-performance hash tables for networking applications, 2017. https://arxiv.org/abs/1712.09624, URL: https://doi.org/10.48550/arXiv.1712.09624.
  17. W. Richard Stevens. TCP/IP illustrated (vol. 1): the protocols. Addison-Wesley Longman Publishing Co., Inc., USA, 1993. URL: http://dl.acm.org/citation.cfm?id=161724.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact