Document Open Access Logo

Limitations of Membership Queries in Testable Learning

Authors Jane Lange , Mingda Qiao

PDF

File

Thumbnail PDF
PDF
LIPIcs.ITCS.2026.91.pdf
  • Filesize: 0.81 MB
  • 23 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Query learning
Keywords
  • Testable learning
  • PAC learning

Metrics

Abstract

Membership queries (MQ) often yield speedups for learning tasks, particularly in the distribution-specific setting. We show that in the testable learning model of Rubinfeld and Vasilyan [Rubinfeld and Vasilyan, 2023], membership queries cannot decrease the time complexity of testable learning algorithms beyond the complexity of sample-only distribution-specific learning. In the testable learning model, the learner must output a hypothesis whenever the data distribution satisfies a desired property, and if it outputs a hypothesis, the hypothesis must be near-optimal.
We give a general reduction from sample-based refutation of boolean concept classes, as presented in [Vadhan, 2017; Kothari and Livni, 2018], to testable learning with queries (TL-Q). This yields lower bounds for TL-Q via the reduction from learning to refutation given in [Kothari and Livni, 2018]. The result is that, relative to a concept class and a distribution family, no m-sample TL-Q algorithm can be super-polynomially more time-efficient than the best m-sample PAC learner.
Finally, we define a class of "statistical" MQ algorithms that encompasses many known distribution-specific MQ learners, such as those based on influence estimation or subcube-conditional statistical queries. We show that TL-Q algorithms in this class imply efficient statistical-query refutation and learning algorithms. Thus, combined with known SQ dimension lower bounds, our results imply that these efficient membership query learners cannot be made testable.

Cite As Get BibTex

Jane Lange and Mingda Qiao. Limitations of Membership Queries in Testable Learning. In 17th Innovations in Theoretical Computer Science Conference (ITCS 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 362, pp. 91:1-91:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ITCS.2026.91

Author Details

Jane Lange
  • Massachusetts Institute of Technology, Cambridge, MA, USA
Mingda Qiao
  • University of Massachusetts Amherst, MA, USA

Funding

  • Lange, Jane: Supported in part by NSF Awards CCF-2006664, DMS-2022448, CCF-2310818, and Big George Fellowship

References

  1. Guy Blanc, Jane Lange, Mingda Qiao, and Li-Yang Tan. Properly learning decision trees in almost polynomial time. Journal of the ACM, 69(6):1-19, 2022. URL: https://doi.org/10.1145/3561047.
  2. Avrim Blum, Merrick Furst, Jeffrey Jackson, Michael Kearns, Yishay Mansour, and Steven Rudich. Weakly learning dnf and characterizing statistical query learning using fourier analysis. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pages 253-262, 1994. URL: https://doi.org/10.1145/195058.195147.
  3. Avrim L Blum and Pat Langley. Selection of relevant features and examples in machine learning. Artificial intelligence, 97(1-2):245-271, 1997. URL: https://doi.org/10.1016/S0004-3702(97)00063-5.
  4. Nader H Bshouty and Vitaly Feldman. On using extended statistical queries to avoid membership queries. Journal of Machine Learning Research, 2(Feb):359-395, 2002. URL: https://jmlr.org/papers/v2/bshouty02a.html.
  5. Amit Daniely. Complexity theoretic limitations on learning halfspaces. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 105-117, 2016. URL: https://doi.org/10.1145/2897518.2897520.
  6. Amit Daniely, Nati Linial, and Shai Shalev-Shwartz. From average case complexity to improper learning complexity. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 441-448, 2014. URL: https://doi.org/10.1145/2591796.2591820.
  7. Amit Daniely and Shai Shalev-Shwartz. Complexity theoretic limitations on learning dnf’s. In Conference on Learning Theory, pages 815-830. PMLR, 2016. URL: http://proceedings.mlr.press/v49/daniely16.html.
  8. Ilias Diakonikolas, Daniel Kane, Vasilis Kontonis, Sihan Liu, and Nikos Zarifis. Efficient testable learning of halfspaces with adversarial label noise. Advances in Neural Information Processing Systems, 36:39470-39490, 2023. Google Scholar
  9. Ariel Elbaz, Homin K Lee, Rocco A Servedio, and Andrew Wan. Separating models of learning from correlated and uncorrelated data. The Journal of Machine Learning Research, 8:277-290, 2007. URL: https://jmlr.org/papers/v8/elbaz07a.html.
  10. Vitaly Feldman. On the power of membership queries in agnostic learning. The Journal of Machine Learning Research, 10:163-182, 2009. URL: https://doi.org/10.5555/1577069.1577076.
  11. Vitaly Feldman and Shrenik Shah. Separating models of learning with faulty teachers. Theoretical computer science, 410(19):1903-1912, 2009. URL: https://doi.org/10.1016/J.TCS.2009.01.017.
  12. Aravind Gollakota, Adam Klivans, Konstantinos Stavropoulos, and Arsen Vasilyan. Tester-learners for halfspaces: Universal algorithms. Advances in Neural Information Processing Systems, 36:10145-10169, 2023. Google Scholar
  13. Aravind Gollakota, Adam R. Klivans, and Pravesh K. Kothari. A moment-matching approach to testable learning and a new characterization of rademacher complexity. In Symposium on Theory of Computing (STOC), pages 1657-1670, 2023. URL: https://doi.org/10.1145/3564246.3585206.
  14. Parikshit Gopalan, Adam Tauman Kalai, and Adam R Klivans. Agnostically learning decision trees. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 527-536, 2008. URL: https://doi.org/10.1145/1374376.1374451.
  15. Adam Klivans, Konstantinos Stavropoulos, and Arsen Vasilyan. Learning intersections of halfspaces with distribution shift: Improved algorithms and sq lower bounds. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pages 2944-2978. PMLR, 30 June-03 July 2024. URL: https://proceedings.mlr.press/v247/klivans24b.html.
  16. Adam Klivans, Konstantinos Stavropoulos, and Arsen Vasilyan. Testable learning with distribution shift. In The Thirty Seventh Annual Conference on Learning Theory, pages 2887-2943. PMLR, 2024. URL: https://proceedings.mlr.press/v247/klivans24a.html.
  17. Pravesh K. Kothari and Roi Livni. Improper Learning by Refuting. In Innovations in Theoretical Computer Science (ITCS), pages 55:1-55:10, 2018. URL: https://doi.org/10.4230/LIPIcs.ITCS.2018.55.
  18. Eyal Kushilevitz and Yishay Mansour. Learning decision trees using the fourier spectrum. In Symposium on Theory of Computing (STOC), pages 455-464, 1991. Google Scholar
  19. Cassandra Marcussen, Ronitt Rubinfeld, and Madhu Sudan. Quality control in sublinear time: a case study via random graphs. arXiv preprint arXiv:2508.16531, 2025. URL: https://doi.org/10.48550/arXiv.2508.16531.
  20. Elchanan Mossel, Ryan O'Donnell, and Rocco A Servedio. Learning functions of k relevant variables. Journal of Computer and System Sciences, 69(3):421-434, 2004. URL: https://doi.org/10.1016/J.JCSS.2004.04.002.
  21. Ronitt Rubinfeld and Arsen Vasilyan. Testing distributional assumptions of learning algorithms. In Symposium on Theory of Computing (STOC), pages 1643-1656, 2023. URL: https://doi.org/10.1145/3564246.3585117.
  22. Lucas Slot, Stefan Tiegel, and Manuel Wiedmer. Testably learning polynomial threshold functions. Advances in Neural Information Processing Systems, 37:3781-3831, 2024. Google Scholar
  23. Salil Vadhan. On learning vs. refutation. In Conference on Learning Theory (COLT), pages 1835-1848, 2017. URL: http://proceedings.mlr.press/v65/vadhan17a.html.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact