Document Open Access Logo

High Probability Streaming Lower Bounds for F₂ Estimation

Authors William Swartworth , David P. Woodruff , Samson Zhou

PDF

File

Thumbnail PDF
PDF
LIPIcs.APPROX-RANDOM.2026.76.pdf
  • Filesize: 0.91 MB
  • 23 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Streaming, sublinear and near linear time algorithms
Keywords
  • streaming algorithms
  • lower bounds
  • moment estimation

Metrics

Abstract

Estimating the second frequency moment (F₂) of an underlying frequency vector is a fundamental problem in the streaming model. While recent work by Braverman and Zamir [STOC 2025] resolved the space complexity for constant failure probability in the insertion-only model, the optimal dependence on the failure parameter δ remained open.
We close this gap by proving a tight high-probability lower bound of Ω(1/ε² log(1/δ) log(ε√n) / log(1/δ)) for (1±ε)-approximate F₂ estimation. The key challenge is the failure of prior multi-scale direct sum arguments under noise sensitivity. We introduce a noise-robust communication primitive, Exam Mostly Set Disjointness, and prove an Ω(m/t log(1/δ)) one-way lower bound. Embedding this into a multi-scale reduction yields the correct log(1/δ) dependence.
We also give two complementary algorithms under natural structure assumptions. For streams with frequency bound B, we design a subsampling method using continuous F₀ tracking that replaces a log n factor with log B. For k-sparse streams, we develop a two-stage sketch using approximate Morris counters, replacing log n with log k and achieving a further log log m dependence on stream length.

Cite As Get BibTex

William Swartworth, David P. Woodruff, and Samson Zhou. High Probability Streaming Lower Bounds for F₂ Estimation. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 392, pp. 76:1-76:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.APPROX/RANDOM.2026.76

Author Details

William Swartworth
  • Carnegie Mellon University, Pittsburgh, PA, USA
David P. Woodruff
  • Carnegie Mellon University, Pittsbrugh, PA, USA
Samson Zhou
  • Texas A&M University, College Station, TX, USA

Funding

  • Woodruff, David P.: David P. Woodruff is supported in part by Office of Naval Research award number N000142112647 and a Simons Investigator Award.
  • Zhou, Samson: Samson Zhou is supported in part by NSF CCF-2335411 and gratefully acknowledges funding provided by the Oak Ridge Associated Universities (ORAU) Ralph E. Powe Junior Faculty Enhancement Award.

References

  1. Noga Alon, Phillip B. Gibbons, Yossi Matias, and Mario Szegedy. Tracking join and self-join sizes in limited storage. J. Comput. Syst. Sci., 64(3):719-747, 2002. URL: https://doi.org/10.1006/JCSS.2001.1813.
  2. Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. J. Comput. Syst. Sci., 58(1):137-147, 1999. URL: https://doi.org/10.1006/JCSS.1997.1545.
  3. Ziv Bar-Yossef, T. S. Jayram, Ravi Kumar, and D. Sivakumar. An information statistics approach to data stream and communication complexity. J. Comput. Syst. Sci., 68(4):702-732, 2004. URL: https://doi.org/10.1016/J.JCSS.2003.11.006.
  4. Omri Ben-Eliezer, Talya Eden, and Krzysztof Onak. Adversarially robust streaming via dense-sparse trade-offs. In 5th Symposium on Simplicity in Algorithms, SOSA@SODA, pages 214-227, 2022. URL: https://doi.org/10.1137/1.9781611977066.15.
  5. Jaroslaw Blasiok. Optimal streaming and tracking distinct elements with high probability. ACM Trans. Algorithms, 16(1):3:1-3:28, 2020. URL: https://doi.org/10.1145/3309193.
  6. Jaroslaw Blasiok, Jian Ding, and Jelani Nelson. Continuous monitoring of 𝓁_p norms in data streams. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM, volume 81, pages 32:1-32:13, 2017. URL: https://doi.org/10.4230/LIPIcs.APPROX-RANDOM.2017.32.
  7. Mark Braverman and Or Zamir. Optimality of frequency moment estimation. In Michal Koucký and Nikhil Bansal, editors, Proceedings of the 57th Annual ACM Symposium on Theory of Computing, STOC, pages 360-370, 2025. URL: https://doi.org/10.1145/3717823.3718171.
  8. Vladimir Braverman, Jonathan Katzman, Charles Seidell, and Gregory Vorsanger. An optimal algorithm for large frequency moments using O(n^1-2/k) bits. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM, pages 531-544, 2014. URL: https://doi.org/10.4230/LIPIcs.APPROX-RANDOM.2014.531.
  9. Vladimir Braverman and Rafail Ostrovsky. Approximating large frequency moments with pick-and-drop sampling. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques - 16th International Workshop, APPROX, and 17th International Workshop, RANDOM. Proceedings, 2013. Google Scholar
  10. Vladimir Braverman, Emanuele Viola, David P. Woodruff, and Lin F. Yang. Revisiting frequency moment estimation in random order streams. In 45th International Colloquium on Automata, Languages, and Programming, ICALP, pages 25:1-25:14, 2018. URL: https://doi.org/10.4230/LIPIcs.ICALP.2018.25.
  11. Amit Chakrabarti, Subhash Khot, and Xiaodong Sun. Near-optimal lower bounds on the multi-party communication complexity of set disjointness. In 18th Annual IEEE Conference on Computational Complexity, pages 107-117, 2003. URL: https://doi.org/10.1109/CCC.2003.1214414.
  12. Sumit Ganguly. Polynomial estimators for high frequency moments. CoRR, abs/1104.4552, 2011. URL: https://arxiv.org/abs/1104.4552.
  13. Sumit Ganguly and David P. Woodruff. High probability frequency moment sketches. In 45th International Colloquium on Automata, Languages, and Programming, ICALP, 2018. URL: https://doi.org/10.4230/LIPIcs.ICALP.2018.58.
  14. Anna C. Gilbert, Sudipto Guha, Piotr Indyk, Yannis Kotidis, S. Muthukrishnan, and Martin Strauss. Fast, small-space algorithms for approximate histogram maintenance. In Proceedings on 34th Annual ACM Symposium on Theory of Computing, pages 389-398, 2002. URL: https://doi.org/10.1145/509907.509966.
  15. Elena Gribelyuk, Honghao Lin, David P. Woodruff, Huacheng Yu, and Samson Zhou. Lifting linear sketches: Optimal bounds and adversarial robustness. In Proceedings of the 57th Annual ACM Symposium on Theory of Computing, STOC, pages 395-406, 2025. URL: https://doi.org/10.1145/3717823.3718227.
  16. Elena Gribelyuk, Honghao Lin, David P. Woodruff, Huacheng Yu, and Samson Zhou. Adversarial robustness on insertion-deletion streams. In Proceedings of the 58th Annual ACM Symposium on Theory of Computing, STOC, pages 2278-2289, 2026. URL: https://doi.org/10.1145/3798129.3800929.
  17. Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM, 53(3):307-323, 2006. URL: https://doi.org/10.1145/1147954.1147955.
  18. Piotr Indyk and David P. Woodruff. Optimal approximations of the frequency moments of data streams. In Proceedings of the 37th Annual ACM Symposium on Theory of Computing (STOC), pages 202-208, 2005. URL: https://doi.org/10.1145/1060590.1060621.
  19. Rajesh Jayaram, David P. Woodruff, and Samson Zhou. Streaming algorithms with few state changes. Proc. ACM Manag. Data, 2(2):82, 2024. URL: https://doi.org/10.1145/3651145.
  20. Akshay Kamath, Eric Price, and David P. Woodruff. A simple proof of a new set disjointness with applications to data streams. In Valentine Kabanets, editor, 36th Computational Complexity Conference, CCC 2021, July 20-23, 2021, Toronto, Ontario, Canada (Virtual Conference), volume 200 of LIPIcs, pages 37:1-37:24. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021. URL: https://doi.org/10.4230/LIPIcs.CCC.2021.37.
  21. Daniel M. Kane, Jelani Nelson, Ely Porat, and David P. Woodruff. Fast moment estimation in data streams in optimal space. In Proceedings of the 43rd ACM Symposium on Theory of Computing, STOC, pages 745-754, 2011. URL: https://doi.org/10.1145/1993636.1993735.
  22. Daniel M. Kane, Jelani Nelson, and David P. Woodruff. On the exact space complexity of sketching and streaming small norms. In Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA, pages 1161-1178, 2010. URL: https://doi.org/10.1137/1.9781611973075.93.
  23. Daniel M Kane, Jelani Nelson, and David P Woodruff. An optimal algorithm for the distinct elements problem. In Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 41-52, 2010. URL: https://doi.org/10.1145/1807085.1807094.
  24. Balachander Krishnamurthy, Subhabrata Sen, Yin Zhang, and Yan Chen. Sketch-based change detection: methods, evaluation, and applications. In Proceedings of the 3rd ACM SIGCOMM Internet Measurement Conference, IMC, pages 234-247, 2003. URL: https://doi.org/10.1145/948205.948236.
  25. Ping Li. Estimators and tail bounds for dimension reduction in 𝓁_α (0 < α ≤ 2) using stable random projections. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA, pages 10-19, 2008. Google Scholar
  26. Honghao Lin, Zhao Song, David P. Woodruff, Shenghao Xie, and Samson Zhou. l_p sampling in distributed data streams with applications to adversarial robustness. In Proceedings of the 2026 Annual ACM-SIAM Symposium on Discrete Algorithms, SODA, pages 4342-4409, 2026. URL: https://doi.org/10.1137/1.9781611978971.160.
  27. Robert Morris. Counting large numbers of events in small registers. Communications of the ACM, 21(10):840-842, 1978. URL: https://doi.org/10.1145/359619.359627.
  28. Shanmugavelayutham Muthukrishnan. Data streams: Algorithms and applications. Foundations and Trends® in Theoretical Computer Science, 1(2):117-236, 2005. URL: https://doi.org/10.1561/0400000002.
  29. David P. Woodruff. Optimal space lower bounds for all frequency moments. In Proceedings of the Fifteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA, pages 167-175, 2004. URL: http://dl.acm.org/citation.cfm?id=982792.982817.
  30. David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, 2014. URL: https://doi.org/10.1561/0400000060.
  31. David P. Woodruff and Samson Zhou. Separations for estimating large frequency moments on data streams. In 48th International Colloquium on Automata, Languages, and Programming, ICALP, pages 112:1-112:21, 2021. URL: https://doi.org/10.4230/LIPIcs.ICALP.2021.112.
  32. David P. Woodruff and Samson Zhou. Tight bounds for adversarially robust streams and sliding windows via difference estimators. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS, pages 1183-1196, 2021. URL: https://doi.org/10.1109/FOCS52979.2021.00116.
  33. David P. Woodruff and Samson Zhou. Adversarially robust dense-sparse tradeoffs via heavy-hitters. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems, NeurIPS, 2024. URL: https://doi.org/10.52202/079017-0348.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact