Document Open Access Logo

Nearest Neighbor Complexity and Boolean Circuits

Authors Mason DiCicco , Vladimir Podolskii , Daniel Reichman

PDF
Thumbnail PDF

File

LIPIcs.ITCS.2025.42.pdf
  • Filesize: 0.87 MB
  • 23 pages

Document Identifiers

Author Details

Mason DiCicco
  • Worcester Polytechnic Institute, MA, USA
Vladimir Podolskii
  • Tufts University, Medford, MA, USA
Daniel Reichman
  • Worcester Polytechnic Institute, MA, USA

Acknowledgements

We would like to thank Bill Martin for several insightful comments. The second and third author thank the Simons Institute for the Theory of Computing where collaboration on this project has began.

Cite As Get BibTex

Mason DiCicco, Vladimir Podolskii, and Daniel Reichman. Nearest Neighbor Complexity and Boolean Circuits. In 16th Innovations in Theoretical Computer Science Conference (ITCS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 325, pp. 42:1-42:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ITCS.2025.42

Abstract

A nearest neighbor representation of a Boolean function f is a set of vectors (anchors) labeled by 0 or 1 such that f(x) = 1 if and only if the closest anchor to x is labeled by 1. This model was introduced by Hajnal, Liu and Turán [2022], who studied bounds on the minimum number of anchors required to represent Boolean functions under different choices of anchors (real vs. Boolean vectors) as well as the analogous model of k-nearest neighbors representations.
We initiate a systematic study of the representational power of nearest and k-nearest neighbors through Boolean circuit complexity. To this end, we establish a close connection between Boolean functions with polynomial nearest neighbor complexity and those that can be efficiently represented by classes based on linear inequalities - min-plus polynomial threshold functions - previously studied in relation to threshold circuits. This extends an observation of Hajnal et al. [2022]. Next, we further extend the connection between nearest neighbor representations and circuits to the k-nearest neighbors case. 
As an outcome of these connections we obtain exponential lower bounds on the k-nearest neighbors complexity of explicit n-variate functions, assuming k ≤ n^{1-ε}. Previously, no superlinear lower bound was known for any k > 1. At the same time, we show that proving superpolynomial lower bounds for the k-nearest neighbors complexity of an explicit function for arbitrary k would require a breakthrough in circuit complexity. In addition, we prove an exponential separation between the nearest neighbor and k-nearest neighbors complexity (for unrestricted k) of an explicit function. These results address questions raised by [Hajnal et al., 2022] of proving strong lower bounds for k-nearest neighbors and understanding the role of the parameter k. Finally, we devise new bounds on the nearest neighbor complexity for several families of Boolean functions.

Subject Classification

ACM Subject Classification
  • Theory of computation → Complexity classes
Keywords
  • Complexity
  • Nearest Neighbors
  • Circuits

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

References

  1. Josh Alman and Ryan Williams. Probabilistic polynomials and hamming nearest neighbors. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 136-150. IEEE, 2015. URL: https://doi.org/10.1109/FOCS.2015.18.
  2. Alexandr Andoni. Nearest neighbor search: the old, the new, and the impossible. PhD thesis, Massachusetts Institute of Technology, 2009. Google Scholar
  3. Richard Beigel and Jun Tarui. On ACC. Comput. Complex., 4:350-366, 1994. URL: https://doi.org/10.1007/BF01263423.
  4. Harry Buhrman, Nikolay Vereshchagin, and Ronald de Wolf. On computation and communication with small bias. In Twenty-Second Annual IEEE Conference on Computational Complexity (CCC'07), pages 24-32. IEEE, 2007. URL: https://doi.org/10.1109/CCC.2007.18.
  5. Arkadev Chattopadhyay, Meena Mahajan, Nikhil S. Mande, and Nitin Saurabh. Lower bounds for linear decision lists. Chic. J. Theor. Comput. Sci., 2020, 2020. URL: http://cjtcs.cs.uchicago.edu/articles/2020/1/contents.html.
  6. Yan Qiu Chen, Mark S Nixon, and Robert I Damper. Implementing the k-nearest neighbour rule via a neural network. In Proceedings of ICNN'95-International Conference on Neural Networks, volume 1, pages 136-140. IEEE, 1995. URL: https://doi.org/10.1109/ICNN.1995.488081.
  7. Kenneth L Clarkson. Nearest neighbor queries in metric spaces. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 609-617, 1997. URL: https://doi.org/10.1145/258533.258655.
  8. Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967. URL: https://doi.org/10.1109/TIT.1967.1053964.
  9. Yogesh Dahiya, K. Vignesh, Meena Mahajan, and Karteek Sreenivasaiah. Linear threshold functions in decision lists, decision trees, and depth-2 circuits. Inf. Process. Lett., 183:106418, 2024. URL: https://doi.org/10.1016/J.IPL.2023.106418.
  10. Luc Devroye. On the asymptotic probability of error in nonparametric discrimination. The Annals of Statistics, 9(6):1320-1327, 1981. Google Scholar
  11. Luc Devroye, László Györfi, and Gábor Lugosi. A Probabilistic Theory of Pattern Recognition, volume 31. Springer Science & Business Media, 2013. Google Scholar
  12. Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pages 907-940. PMLR, 2016. URL: http://proceedings.mlr.press/v49/eldan16.html.
  13. Jürgen Forster. A linear lower bound on the unbounded error probabilistic communication complexity. Journal of Computer and System Sciences, 65(4):612-625, 2002. URL: https://doi.org/10.1016/S0022-0000(02)00019-3.
  14. Mikael Goldmann, Johan Håstad, and Alexander Razborov. Majority gates vs. general weighted threshold gates. Computational Complexity, 2:277-300, 1992. URL: https://doi.org/10.1007/BF01200426.
  15. Mikael Goldmann and Marek Karpinski. Simulating threshold circuits by majority circuits. SIAM Journal on Computing, 27(1):230-246, 1998. URL: https://doi.org/10.1137/S0097539794274519.
  16. András Hajnal, Wolfgang Maass, Pavel Pudlák, Mario Szegedy, and György Turán. Threshold circuits of bounded depth. Journal of Computer and System Sciences, 46(2):129-154, 1993. URL: https://doi.org/10.1016/0022-0000(93)90001-D.
  17. Péter Hajnal, Zhihao Liu, and György Turán. Nearest neighbor representations of boolean functions. Information and Computation, 285:104879, 2022. URL: https://doi.org/10.1016/J.IC.2022.104879.
  18. Kristoffer Arnsfelt Hansen and Vladimir V Podolskii. Polynomial threshold functions and boolean threshold circuits. Information and Computation, 240:56-73, 2015. URL: https://doi.org/10.1016/J.IC.2014.09.008.
  19. Lisa Hellerstein and Rocco A Servedio. On PAC learning algorithms for rich boolean function classes. Theoretical Computer Science, 384(1):66-76, 2007. URL: https://doi.org/10.1016/J.TCS.2007.05.018.
  20. Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604-613, 1998. URL: https://doi.org/10.1145/276698.276876.
  21. Piotr Indyk and Tal Wagner. Approximate nearest neighbors in limited space. In Conference On Learning Theory, pages 2012-2036. PMLR, 2018. URL: http://proceedings.mlr.press/v75/indyk18a.html.
  22. Jeffrey C Jackson, Adam R Klivans, and Rocco A Servedio. Learnability beyond AC⁰. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 776-784, 2002. Google Scholar
  23. Stasys Jukna. Boolean function complexity: advances and frontiers, volume 5. Springer, 2012. URL: https://doi.org/10.1007/978-3-642-24508-4.
  24. Kordag Mehmet Kilic, Jin Sima, and Jehoshua Bruck. On the information capacity of nearest neighbor representations. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 1663-1668, 2023. URL: https://doi.org/10.1109/ISIT54713.2023.10206832.
  25. Kordag Mehmet Kilic, Jin Sima, and Jehoshua Bruck. Nearest neighbor representations of neurons. In 2024 IEEE International Symposium on Information Theory (ISIT), 2024. Google Scholar
  26. Adam R Klivans and Rocco A Servedio. Learning DNF in time 2^O(n^1/3). Journal of Computer and System Sciences, 2(68):303-318, 2004. Google Scholar
  27. Adam R Klivans and Rocco A Servedio. Toward attribute efficient learning of decision lists and parities. Journal of Machine Learning Research, 7(4), 2006. URL: https://jmlr.org/papers/v7/klivans06a.html.
  28. Eyal Kushilevitz, Rafail Ostrovsky, and Yuval Rabani. Efficient search for approximate nearest neighbor in high dimensional spaces. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 614-623, 1998. URL: https://doi.org/10.1145/276698.276877.
  29. Nathan Linial, Yishay Mansour, and Noam Nisan. Constant depth circuits, fourier transform, and learnability. Journal of the ACM (JACM), 40(3):607-620, 1993. URL: https://doi.org/10.1145/174130.174138.
  30. James Martens, Arkadev Chattopadhya, Toni Pitassi, and Richard Zemel. On the representational efficiency of restricted boltzmann machines. Advances in Neural Information Processing Systems, 26, 2013. Google Scholar
  31. O Murphy. Nearest neighbor pattern classification perceptrons. Neural Networks: Theoretical Foundations and Analysis, pages 263-266, 1992. Google Scholar
  32. Edward A Patrick and Frederic P Fischer III. A generalized k-nearest neighbor rule. Information and control, 16(2):128-152, 1970. URL: https://doi.org/10.1016/S0019-9958(70)90081-1.
  33. Alexander A Razborov. On the distributional complexity of disjointness. In International Colloquium on Automata, Languages, and Programming, pages 249-253. Springer, 1990. URL: https://doi.org/10.1007/BFB0032036.
  34. Alexander A Razborov. On small depth threshold circuits. In Scandinavian Workshop on Algorithm Theory, pages 42-52. Springer, 1992. URL: https://doi.org/10.1007/3-540-55706-7_4.
  35. Alexander A Razborov and Alexander A Sherstov. The sign-rank of AC⁰. SIAM Journal on Computing, 39(5):1833-1855, 2010. Google Scholar
  36. Ronald L Rivest. Learning decision lists. Machine learning, 2:229-246, 1987. URL: https://doi.org/10.1007/BF00058680.
  37. Michael E. Saks. Slicing the hypercube, pages 211-256. London Mathematical Society Lecture Note Series. Cambridge University Press, 1993. Google Scholar
  38. Matus Telgarsky. Benefits of depth in neural networks. In Conference on learning theory, pages 1517-1539. PMLR, 2016. URL: http://proceedings.mlr.press/v49/telgarsky16.html.
  39. Gal Vardi, Daniel Reichman, Toniann Pitassi, and Ohad Shamir. Size and depth separation in approximating benign functions with neural networks. In Conference on Learning Theory, pages 4195-4223. PMLR, 2021. URL: http://proceedings.mlr.press/v134/vardi21a.html.
  40. Nikhil Vyas and R. Ryan Williams. Lower bounds against sparse symmetric functions of ACC circuits: Expanding the reach of #SAT algorithms. Theory Comput. Syst., 67(1):149-177, 2023. URL: https://doi.org/10.1007/S00224-022-10106-8.
  41. R. Ryan Williams. Limits on representing boolean functions by linear combinations of simple functions: Thresholds, relus, and low-degree polynomials. In 33rd Computational Complexity Conference (CCC 2018). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact