Document Open Access Logo

Entropy Equivalence Testing

Authors Clément L. Canonne , Yash Pote , Jonathan Scarlett , Joy Qiping Yang

PDF

File

Thumbnail PDF
PDF
LIPIcs.APPROX-RANDOM.2026.50.pdf
  • Filesize: 0.95 MB
  • 23 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Lower bounds and information complexity
  • Mathematics of computing → Probabilistic algorithms
Keywords
  • Entropy
  • distribution testing
  • sublinear algorithm
  • Bayesian network

Metrics

Abstract

We introduce the problem of entropy equivalence testing for probability distributions, a relaxation of the well-studied closeness testing problem, where the distribution testing algorithm is now only required to distinguish, given samples from two unknown distributions p,q and a parameter ε ∈ (0,1/2], between p = q and |H(p)-H(q)| ⩾ ε (where H denotes the Shannon entropy). We provide a time- and sample-efficient algorithm for this task, showing that the optimal sample complexity for this task can be significantly lower than that of closeness testing. As an application, we leverage this result to provide the first non-trivial testing algorithm for (standard) closeness of low-degree Bayesian networks, which significantly improves on either the sample or time complexity of a baseline based on full learning.

Cite As Get BibTex

Clément L. Canonne, Yash Pote, Jonathan Scarlett, and Joy Qiping Yang. Entropy Equivalence Testing. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 392, pp. 50:1-50:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.APPROX/RANDOM.2026.50

Author Details

Clément L. Canonne
  • University of Sydney, Australia
Yash Pote
  • National University of Singapore, Singapore
Jonathan Scarlett
  • National University of Singapore, Singapore
Joy Qiping Yang
  • University of Sydney, Australia

Funding

  • Scarlett, Jonathan: Supported by the National University of Singapore under the Presidential Young Professorship scheme.
  • Yang, Joy Qiping: Supported by a JD Technology Research Scholarship in Artificial intelligence.

Acknowledgements

We would like to thank Sayantan Sen, Jan Seyfried and Marco Tomamichel for the discussion on their work of testing Conditional Mutual Information. JY would also like to thank Davin Choo for the early discussions predating this project. This work is conducted while JY was interning at National Institute of Informatics, hosted by Yuichi Yoshida.

References

  1. Jayadev Acharya, Constantinos Daskalakis, and Gautam Kamath. Optimal testing for properties of distributions. In NeurIPS, pages 3591-3599, 2015. URL: https://proceedings.neurips.cc/paper/2015/hash/1f36c15d6a3d18d52e8d493bc8187cb9-Abstract.html.
  2. Jayadev Acharya, Alon Orlitsky, Ananda Theertha Suresh, and Himanshu Tyagi. The complexity of estimating rényi entropy. In SODA, pages 1855-1869. SIAM, 2015. URL: https://doi.org/10.1137/1.9781611973730.124.
  3. Tugkan Batu, Sanjoy Dasgupta, Ravi Kumar, and Ronitt Rubinfeld. The complexity of approximating the entropy. SIAM J. Comput., 35(1):132-150, 2005. URL: https://doi.org/10.1137/S0097539702403645.
  4. Tugkan Batu, Lance Fortnow, Eldar Fischer, Ravi Kumar, Ronitt Rubinfeld, and Patrick White. Testing random variables for independence and identity. In FOCS, pages 442-451. IEEE Computer Society, 2001. URL: https://doi.org/10.1109/SFCS.2001.959920.
  5. Tugkan Batu, Lance Fortnow, Ronitt Rubinfeld, Warren D. Smith, and Patrick White. Testing that distributions are close. In FOCS, pages 259-269. IEEE Computer Society, 2000. URL: https://doi.org/10.1109/SFCS.2000.892113.
  6. Arnab Bhattacharyya, Clément L. Canonne, and Joy Qiping Yang. Independence testing for bounded degree bayesian networks. In NeurIPS, 2022. Google Scholar
  7. Arnab Bhattacharyya, Davin Choo, Sutanu Gayen, and Dimitrios Myrisiotis. Learnability of parameter-bounded bayes nets. In AAAI, pages 15559-15566. AAAI Press, 2025. URL: https://doi.org/10.1609/AAAI.V39I15.33708.
  8. Arnab Bhattacharyya, Sutanu Gayen, Philips George John, Sayantan Sen, and N. V. Vinodchandran. Distribution learning meets graph structure sampling. In Advances in Neural Information Processing Systems, volume 38, pages 21411-21432, 2025. Google Scholar
  9. Arnab Bhattacharyya, Sutanu Gayen, Eric Price, Vincent Y. F. Tan, and N. V. Vinodchandran. Near-optimal learning of tree-structured distributions by chow and liu. SIAM J. Comput., 52(3):761-793, 2023. URL: https://doi.org/10.1137/22M1489678.
  10. Clément L. Canonne. A Survey on Distribution Testing: Your Data is Big. But is it Blue? Number 9 in Graduate Surveys. Theory of Computing Library, 2020. URL: https://doi.org/10.4086/toc.gs.2020.009.
  11. Clément L. Canonne. Topics and techniques in distribution testing: A biased but representative sample. Found. Trends Commun. Inf. Theory, 19(6):1032-1198, 2022. URL: https://doi.org/10.1561/0100000114.
  12. Clément L Canonne, Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. Testing Bayesian networks. In Conference on Learning Theory, pages 370-448. PMLR, 2017. URL: http://proceedings.mlr.press/v65/canonne17a.html.
  13. Clément L. Canonne, Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Testing conditional independence of discrete distributions. In STOC, pages 735-748. ACM, 2018. URL: https://doi.org/10.1145/3188745.3188756.
  14. Clément L. Canonne, Yash Pote, Jonathan Scarlett, and Joy Qiping Yang. Entropy equivalence testing. CoRR, abs/2605.23225, 2026. URL: https://doi.org/10.48550/arXiv.2605.23225.
  15. Clément L Canonne and Joy Q Yang. Entropy testing and its application to testing Bayesian networks. Advances in Neural Information Processing Systems, 2024. Google Scholar
  16. Clément L. Canonne. A short note on learning discrete distributions, 2020. URL: https://arxiv.org/abs/2002.11457.
  17. Siu On Chan, Ilias Diakonikolas, Paul Valiant, and Gregory Valiant. Optimal algorithms for testing closeness of discrete distributions. In SODA, pages 1193-1203. SIAM, 2014. URL: https://doi.org/10.1137/1.9781611973402.88.
  18. David Maxwell Chickering, David Heckerman, and Christopher Meek. Large-sample learning of bayesian networks is np-hard. J. Mach. Learn. Res., 5:1287-1330, 2004. URL: https://jmlr.org/papers/volume5/chickering04a/chickering04a.pdf.
  19. Davin Choo, Joy Qiping Yang, Arnab Bhattacharyya, and Clément L. Canonne. Learning bounded-degree polytrees with known skeleton. In ALT, Proceedings of Machine Learning Research, pages 402-443. PMLR, 2024. URL: https://proceedings.mlr.press/v237/choo24a.html.
  20. C. K. Chow and C. N. Liu. Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theory, 14(3):462-467, 1968. URL: https://doi.org/10.1109/TIT.1968.1054142.
  21. Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press, 2011. Google Scholar
  22. Sanjoy Dasgupta. Learning polytrees. In UAI, pages 134-141. Morgan Kaufmann, 1999. URL: https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=162&proceeding_id=15.
  23. Constantinos Daskalakis, Nishanth Dikkala, and Gautam Kamath. Testing ising models. In SODA, pages 1989-2007. SIAM, 2018. URL: https://doi.org/10.1137/1.9781611975031.130.
  24. Constantinos Daskalakis, Gautam Kamath, and John Wright. Which distribution distances are sublinearly testable? In SODA, pages 2747-2764. SIAM, 2018. URL: https://doi.org/10.1137/1.9781611975031.175.
  25. Constantinos Daskalakis and Qinxuan Pan. Square hellinger subadditivity for bayesian networks and its applications to identity testing. In COLT, volume 65 of Proceedings of Machine Learning Research, pages 697-703. PMLR, 2017. URL: http://proceedings.mlr.press/v65/daskalakis17a.html.
  26. Ilias Diakonikolas, Themis Gouleakis, Daniel M. Kane, John Peebles, and Eric Price. Optimal testing of discrete distributions with high probability. In STOC, pages 542-555. ACM, 2021. URL: https://doi.org/10.1145/3406325.3450997.
  27. Ilias Diakonikolas, Themis Gouleakis, John Peebles, and Eric Price. Sample-optimal identity testing with high probability. In ICALP, LIPIcs, pages 41:1-41:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018. URL: https://doi.org/10.4230/LIPIcs.ICALP.2018.41.
  28. Ilias Diakonikolas and Daniel M. Kane. A new approach for testing properties of discrete distributions. In FOCS, pages 685-694. IEEE Computer Society, 2016. URL: https://doi.org/10.1109/FOCS.2016.78.
  29. Oded Goldreich. Introduction to Property Testing. Cambridge University Press, 2017. URL: https://doi.org/10.1017/9781108135252.
  30. Oded Goldreich, Shafi Goldwasser, and Dana Ron. Property testing and its connection to learning and approximation. J. ACM, 45(4):653-750, 1998. URL: https://doi.org/10.1145/285055.285060.
  31. Jiantao Jiao, Kartik Venkat, Yanjun Han, and Tsachy Weissman. Minimax estimation of functionals of discrete distributions. IEEE Trans. Inf. Theory, 61(5):2835-2885, 2015. URL: https://doi.org/10.1109/TIT.2015.2412945.
  32. Liam Paninski. A coincidence-based test for uniformity given very sparsely sampled discrete data. IEEE Trans. Inf. Theory, 54(10):4750-4755, 2008. URL: https://doi.org/10.1109/TIT.2008.928987.
  33. Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2025. Google Scholar
  34. Ronitt Rubinfeld. Taming big probability distributions. XRDS, 19(1):24-28, 2012. URL: https://doi.org/10.1145/2331042.2331052.
  35. Ronitt Rubinfeld and Madhu Sudan. Robust characterizations of polynomials with applications to program testing. SIAM J. Comput., 25(2):252-271, 1996. URL: https://doi.org/10.1137/S0097539793255151.
  36. Igal Sason and Sergio Verdú. f-divergence inequalities. IEEE Trans. Inf. Theory, 62(11):5973-6006, 2016. URL: https://doi.org/10.1109/TIT.2016.2603151.
  37. Jan Seyfried, Sayantan Sen, and Marco Tomamichel. Testing (conditional) mutual information. Electron. Colloquium Comput. Complex., TR25, 2025. URL: https://eccc.weizmann.ac.il/report/2025/076.
  38. Flemming Topsøe. Some inequalities for information divergence and related measures of discrimination. IEEE Trans. Inf. Theory, 46(4):1602-1609, 2000. URL: https://doi.org/10.1109/18.850703.
  39. Gregory Valiant and Paul Valiant. Estimating the unseen: An n/log(n)-sample estimator for entropy and support size, shown optimal via new clts. In STOC, pages 685-694. ACM, 2011. URL: https://doi.org/10.1145/1993636.1993727.
  40. Gregory Valiant and Paul Valiant. An automatic inequality prover and instance optimal identity testing. In FOCS, pages 51-60. IEEE Computer Society, 2014. URL: https://doi.org/10.1109/FOCS.2014.14.
  41. Yihong Wu and Pengkun Yang. Optimal entropy estimation on large alphabets via best polynomial approximation. In ISIT, pages 824-828. IEEE, 2015. URL: https://doi.org/10.1109/ISIT.2015.7282570.
  42. Yihong Wu and Pengkun Yang. Minimax rates of entropy estimation on large alphabets via best polynomial approximation. IEEE Trans. Inf. Theory, 2016. URL: https://doi.org/10.1109/TIT.2016.2548468.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact