Document Open Access Logo

LMQ-Sketch: Lagom Multi-Query Sketch for High-Rate Online Analytics

Authors Martin Hilgendorf , Marina Papatriantafilou

PDF

File

Thumbnail PDF
PDF
LIPIcs.DISC.2025.36.pdf
  • Filesize: 1.26 MB
  • 24 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Computing methodologies → Shared memory algorithms
  • Theory of computation → Concurrent algorithms
  • Information systems → Data structures
Keywords
  • Concurrent Data Structures
  • Data Sketches
  • IVL
  • Freshness
  • Synchronization

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Data sketches balance resource efficiency with controllable approximations for extracting features in high-volume, high-rate data. Two important points of interest are highlighted separately in recent works; namely, to (1) answer multiple types of queries from a single data structure built in one pass over the data, and (2) perform both queries and updates concurrently. In this work, we now tackle the new challenges arising when combining these useful directions together.
We investigate the trade-offs around efficiency, consistency, and accuracy to be balanced and synthesize key ideas into LMQ-Sketch, a single, composite data sketch supporting concurrent updates and multiple queries (frequency point queries, frequency moments F₁, and F₂ as representative selection). Our method "Lagom" is a cornerstone of LMQ-Sketch for low-latency global querying (<100µs), combining freshness, timeliness, and accuracy with a low memory footprint and high throughput (>2B updates/s). We analyze and evaluate the accuracy of Lagom, which builds on a simple geometric argument and efficiently combines work distribution with synchronization for proper concurrency semantics - monotonicity of operations and intermediate value linearizability. Comparing with state-of-the-art methods, which, as mentioned, provide either mixed queries or concurrency separately, LMQ-Sketch shows highly competitive throughput, with additional accuracy guarantees and concurrency semantics, while also reducing the required memory budget by an order of magnitude. We expect the methodology to have broader impact on concurrent multi-query sketches.

Cite As Get BibTex

Martin Hilgendorf and Marina Papatriantafilou. LMQ-Sketch: Lagom Multi-Query Sketch for High-Rate Online Analytics. In 39th International Symposium on Distributed Computing (DISC 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 356, pp. 36:1-36:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.DISC.2025.36

Author Details

Martin Hilgendorf
  • Chalmers University of Technology, Gothenburg, Sweden
  • University of Gothenburg, Sweden
Marina Papatriantafilou
  • Chalmers University of Technology, Gothenburg, Sweden
  • University of Gothenburg, Sweden

Funding

Work supported by the Swedish Research Council project EPITOME 2021-05424, by the Marie Skłodowska-Curie Doctoral Network RELAX-DN, funded by the European Union under Horizon Europe 2021-2027 Framework Programme (grant nr. 101072456) and by Chalmers University AoA frameworks Energy and Production, WP. INDEED, and "Scalability, Big Data and AI".

Supplementary Materials

References

  1. Noga Alon, Phillip B. Gibbons, Yossi Matias, and Mario Szegedy. Tracking join and self-join sizes in limited storage. In Proceedings of the Eighteenth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS '99, pages 10-20, New York, NY, USA, May 1999. Association for Computing Machinery. URL: https://doi.org/10.1145/303976.303978.
  2. DataSketches | Apache® DataSketches. URL: https://datasketches.apache.org/.
  3. Apache Druid | Apache® Druid. URL: https://druid.apache.org/.
  4. Vladimir Braverman and Rafail Ostrovsky. Zero-one frequency laws. In Proceedings of the Forty-Second ACM Symposium on Theory of Computing, STOC '10, pages 281-290, New York, NY, USA, June 2010. Association for Computing Machinery. URL: https://doi.org/10.1145/1806689.1806729.
  5. Chiranjeeb Buragohain and Subhash Suri. Quantiles on Streams. In Encyclopedia of Database Systems, pages 2235-2240. Springer US, Boston, MA, 2009. URL: https://doi.org/10.1007/978-0-387-39940-9_290.
  6. The CAIDA UCSD Anonymized Internet Traces - 2018. URL: https://www.caida.org/catalog/datasets/passive_dataset.
  7. Lidia Ceriani and Paolo Verme. The origins of the Gini index: Extracts from Variabilità e Mutabilità (1912) by Corrado Gini. The Journal of Economic Inequality, 10(3):421-443, September 2012. URL: https://doi.org/10.1007/s10888-011-9188-x.
  8. Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding Frequent Items in Data Streams. In Automata, Languages and Programming, Lecture Notes in Computer Science, pages 693-703, Berlin, Heidelberg, 2002. Springer. URL: https://doi.org/10.1007/3-540-45465-9_59.
  9. Monica Chiosa, Thomas B. Preußer, and Gustavo Alonso. SKT: A one-pass multi-sketch data analytics accelerator. Proceedings of the VLDB Endowment, 14(11):2369-2382, July 2021. URL: https://doi.org/10.14778/3476249.3476287.
  10. Graham Cormode and Minos Garofalakis. Sketching streams through the net: Distributed approximate query tracking. In Proceedings of the 31st International Conference on Very Large Data Bases, VLDB '05, pages 13-24, Trondheim, Norway, August 2005. VLDB Endowment. URL: http://www.vldb.org/archives/website/2005/program/paper/tue/p13-cormode.pdf.
  11. Graham Cormode and Minos Garofalakis. Join sizes, frequency moments, and applications. In Data Stream Management: Processing High-Speed Data Streams, pages 87-102. Springer, 2016. URL: https://doi.org/10.1007/978-3-540-28608-0_4.
  12. Graham Cormode, Minos Garofalakis, and Dimitris Sacharidis. Fast Approximate Wavelet Tracking on Streams. In Advances in Database Technology - EDBT 2006, pages 4-22, Berlin, Heidelberg, 2006. Springer. URL: https://doi.org/10.1007/11687238_4.
  13. Graham Cormode and S. Muthukrishnan. An improved data stream summary: The count-min sketch and its applications. Journal of Algorithms, 55(1):58-75, April 2005. URL: https://doi.org/10.1016/j.jalgor.2003.12.001.
  14. Graham Cormode and S. Muthukrishnan. Summarizing and Mining Skewed Data Streams. In Proceedings of the 2005 SIAM International Conference on Data Mining, pages 44-55. Society for Industrial and Applied Mathematics, April 2005. URL: https://doi.org/10.1137/1.9781611972757.5.
  15. Graham Cormode and Ke Yi. Small Summaries for Big Data. Cambridge University Press, Cambridge, 2020. URL: https://doi.org/10.1017/9781108769938.
  16. Alin Dobra, Minos Garofalakis, Johannes Gehrke, and Rajeev Rastogi. Sketch-Based Multi-Query Processing over Data Streams. In Data Stream Management: Processing High-Speed Data Streams, Data-Centric Systems and Applications, pages 241-261. Springer, Berlin, Heidelberg, 2016. URL: https://doi.org/10.1007/978-3-540-28608-0_12.
  17. Cynthia Dwork, Maurice Herlihy, Serge Plotkin, and Orli Waarts. Time-Lapse Snapshots. SIAM Journal on Computing, 28(5):1848-1874, January 1999. URL: https://doi.org/10.1137/S0097539793243685.
  18. Shaked Elias Zada, Arik Rinberg, and Idit Keidar. Quancurrent: A Concurrent Quantiles Sketch. In Proceedings of the 35th ACM Symposium on Parallelism in Algorithms and Architectures, SPAA '23, pages 15-25, New York, NY, USA, June 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3558481.3591074.
  19. Minos Garofalakis. Discrete Wavelet Transform and Wavelet Synopses. In Encyclopedia of Database Systems, pages 857-863. Springer US, Boston, MA, 2009. URL: https://doi.org/10.1007/978-0-387-39940-9_539.
  20. Anna C. Gilbert, Yannis Kotidis, S. Muthukrishnan, and Martin J. Strauss. How to summarize the universe: Dynamic maintenance of quantiles. In Proceedings of the 28th International Conference on Very Large Data Bases, VLDB '02, pages 454-465, Hong Kong, China, August 2002. VLDB Endowment. URL: https://doi.org/10.1016/B978-155860869-6/50047-0.
  21. Anna C. Gilbert, Yannis Kotidis, S. Muthukrishnan, and Martin J. Strauss. One-pass wavelet decompositions of data streams. IEEE Transactions on Knowledge and Data Engineering, 15(3):541-554, May 2003. URL: https://doi.org/10.1109/TKDE.2003.1198389.
  22. Corrado Gini. Variabilità e Mutabilità. Reprinted in Memorie Di Metodologia Statistica (Ed. E. Pizetti and T. Salvemini.) 1955. Libreria Eredi Virgilio Veschi, Rome, 1912. Google Scholar
  23. Itamar Haber. Count-Min Sketch: The Art and Science of Estimating Stuff, March 2022. URL: https://redis.io/blog/count-min-sketch-the-art-and-science-of-estimating-stuff/.
  24. Thomas A. Henzinger, Christoph M. Kirsch, Hannes Payer, Ali Sezgin, and Ana Sokolova. Quantitative relaxation of concurrent data structures. In Proceedings of the 40th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '13, pages 317-328, New York, NY, USA, January 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2429069.2429109.
  25. Maurice Herlihy, Nir Shavit, Victor Luchangco, and Michael Spear. The Art of Multiprocessor Programming. Elsevier, Morgan Kaufmann Publishers, Cambridge, MA, United States, second edition, 2021. Google Scholar
  26. Martin Hilgendorf. LMQ-Sketch. URL: https://gitlab.com/marhilg/lmq-sketch.
  27. Victor Jarlow, Charalampos Stylianopoulos, and Marina Papatriantafilou. QPOPSS: Query and Parallelism Optimized Space-Saving for finding frequent stream elements. Journal of Parallel and Distributed Computing, 204:105134, October 2025. URL: https://doi.org/10.1016/j.jpdc.2025.105134.
  28. Lior Kogan. T-digest: A New Probabilistic Data Structure in Redis Stack, March 2023. URL: https://redis.io/blog/t-digest-in-redis-stack/.
  29. Balachander Krishnamurthy, Subhabrata Sen, Yin Zhang, and Yan Chen. Sketch-based change detection: Methods, evaluation, and applications. In Proceedings of the 3rd ACM SIGCOMM Conference on Internet Measurement, IMC '03, pages 234-247, New York, NY, USA, October 2003. Association for Computing Machinery. URL: https://doi.org/10.1145/948205.948236.
  30. Leslie Lamport. Concurrent reading and writing. Communications of the ACM, 20(11):806-811, November 1977. URL: https://doi.org/10.1145/359863.359878.
  31. Zaoxing Liu, Antonis Manousis, Gregory Vorsanger, Vyas Sekar, and Vladimir Braverman. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon. In Proceedings of the 2016 ACM SIGCOMM Conference, pages 101-114, Florianopolis Brazil, August 2016. ACM. URL: https://doi.org/10.1145/2934872.2934906.
  32. Antonis Manousis, Zhuo Cheng, Ran Ben Basat, Zaoxing Liu, and Vyas Sekar. Enabling efficient and general subpopulation analytics in multidimensional data streams. Proceedings of the VLDB Endowment, 15(11):3249-3262, July 2022. URL: https://doi.org/10.14778/3551793.3551867.
  33. Gonçalo Matos, Salvatore Signorello, and Fernando M. V. Ramos. Generic change detection (almost entirely) in the dataplane. In Proceedings of the Symposium on Architectures for Networking and Communications Systems, ANCS '21, pages 113-120, New York, NY, USA, January 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3493425.3502767.
  34. Vinh Quang Ngo and Marina Papatriantafilou. Cuckoo heavy keeper and the balancing act of maintaining heavy hitters in stream processing. Proceedings of the VLDB Endowment, 18(9):3149-3161, August 2025. URL: https://doi.org/10.14778/3746405.3746434.
  35. Yiannis Nikolakopoulos, Anders Gidenstam, Marina Papatriantafilou, and Philippas Tsigas. A Consistency Framework for Iteration Operations in Concurrent Data Structures. In 2015 IEEE International Parallel and Distributed Processing Symposium, pages 239-248, May 2015. URL: https://doi.org/10.1109/IPDPS.2015.84.
  36. Savannah Norem. Probabilistic Data Structures in Redis, August 2022. URL: https://redis.io/blog/streaming-analytics-with-probabilistic-data-structures/.
  37. Erez Petrank and Shahar Timnat. Lock-Free Data-Structure Iterators. In Proceedings of the 27th International Symposium on Distributed Computing - Volume 8205, DISC 2013, pages 224-238, Berlin, Heidelberg, October 2013. Springer-Verlag. URL: https://doi.org/10.1007/978-3-642-41527-2_16.
  38. Wieger R. Punter, Odysseas Papapetrou, and Minos Garofalakis. OmniSketch: Efficient Multi-Dimensional High-Velocity Stream Analytics with Arbitrary Predicates. Proc. VLDB Endow., 17(3):319-331, November 2023. URL: https://doi.org/10.14778/3632093.3632098.
  39. Arik Rinberg and Idit Keidar. Intermediate Value Linearizability: A Quantitative Correctness Criterion. Journal of the ACM, 70(2):17:1-17:21, April 2023. URL: https://doi.org/10.1145/3584699.
  40. Arik Rinberg, Alexander Spiegelman, Edward Bortnikov, Eshcar Hillel, Idit Keidar, Lee Rhodes, and Hadar Serviansky. Fast Concurrent Data Sketches. ACM Transactions on Parallel Computing, 9(2):6:1-6:35, April 2022. URL: https://doi.org/10.1145/3512758.
  41. Pratanu Roy, Arijit Khan, and Gustavo Alonso. Augmented Sketch: Faster and More Accurate Stream Processing. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD '16, pages 1449-1463, New York, NY, USA, June 2016. Association for Computing Machinery. URL: https://doi.org/10.1145/2882903.2882948.
  42. Charalampos Stylianopoulos, Ivan Walulya, Magnus Almgren, Olaf Landsiedel, and Marina Papatriantafilou. Delegation sketch: A parallel design with support for fast and accurate concurrent operations. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys '20, pages 1-16, New York, NY, USA, April 2020. Association for Computing Machinery. URL: https://doi.org/10.1145/3342195.3387542.
  43. Da Tong and Viktor K. Prasanna. Sketch Acceleration on FPGA and its Applications in Network Anomaly Detection. IEEE Transactions on Parallel and Distributed Systems, 29(4):929-942, April 2018. URL: https://doi.org/10.1109/TPDS.2017.2766633.
  44. R. Kent Treiber. Systems Programming: Coping with Parallelism. Number 5118 in Research Report. International Business Machines Incorporated Research Division, Thomas J. Watson Research Center, 1986. URL: https://dominoweb.draco.res.ibm.com/reports/rj5118.pdf.
  45. David P. Woodruff. Sketching as a Tool for Numerical Linear Algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, October 2014. URL: https://doi.org/10.1561/0400000060.
  46. David P. Woodruff. Frequency Moments. In Encyclopedia of Database Systems, pages 1518-1519. Springer, New York, NY, 2018. URL: https://doi.org/10.1007/978-1-4614-8265-9_167.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact