Document Open Access Logo

Exponential Lower Bounds for Threshold Circuits of Sub-Linear Depth and Energy

Authors Kei Uchizawa, Haruki Abe

PDF

File

Thumbnail PDF
PDF
LIPIcs.MFCS.2023.85.pdf
  • Filesize: 0.65 MB
  • 15 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Models of computation
Keywords
  • Circuit complexity
  • disjointness function
  • equality function
  • neural networks
  • threshold circuits
  • ReLU cicuits
  • sigmoid circuits
  • sparse activity

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

In this paper, we investigate computational power of threshold circuits and other theoretical models of neural networks in terms of the following four complexity measures: size (the number of gates), depth, weight and energy. Here, the energy of a circuit measures sparsity of their computation, and is defined as the maximum number of gates outputting non-zero values taken over all the input assignments.
As our main result, we prove that any threshold circuit C of size s, depth d, energy e and weight w satisfies log(rk(M_C)) ≤ ed (log s + log w + log n), where rk(M_C) is the rank of the communication matrix M_C of a 2n-variable Boolean function that C computes. Thus, such a threshold circuit C is able to compute only a Boolean function of which communication matrix has rank bounded by a product of logarithmic factors of s, w and linear factors of d, e. This implies an exponential lower bound on the size of even sublinear-depth and sublinear-energy threshold circuit. For example, we can obtain an exponential lower bound s = 2^Ω(n^{1/3}) for threshold circuits of depth n^{1/3}, energy n^{1/3} and weight 2^o(n^{1/3}). We also show that the inequality is tight up to a constant factor when the depth d and energy e satisfies ed = o(n/log n).
For other models of neural networks such as a discretized ReLU circuits and descretized sigmoid circuits, we define energy as the maximum number of gates outputting non-zero values. We then prove that a similar inequality also holds for a discretized circuit C: rk(M_C) = O(ed(log s + log w + log n)³). Thus, if we consider the number gates outputting non-zero values as a measure for sparse activity of a neural network, our results suggest that larger depth linearly helps neural networks to acquire sparse activity.

Cite As Get BibTex

Kei Uchizawa and Haruki Abe. Exponential Lower Bounds for Threshold Circuits of Sub-Linear Depth and Energy. In 48th International Symposium on Mathematical Foundations of Computer Science (MFCS 2023). Leibniz International Proceedings in Informatics (LIPIcs), Volume 272, pp. 85:1-85:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023) https://doi.org/10.4230/LIPIcs.MFCS.2023.85

Author Details

Kei Uchizawa
  • Graduate School of Science and Engineering, Yamagata University, Yonezawa-shi Yamagata, Japan
Haruki Abe
  • Graduate School of Science and Engineering, Yamagata University, Yonezawa-shi Yamagata, Japan

Funding

  • Uchizawa, Kei: This work was supported by JSPS KAKENHI Grant Number JP22K11897.

Acknowledgements

We thank the anonymous reviewers for their careful reading and helpful comments.

References

  1. Kazuyuki Amano. On the size of depth-two threshold circuits for the Inner Product mod 2 function. In Alberto Leporati, Carlos Martín-Vide, Dana Shapira, and Claudio Zandron, editors, Language and Automata Theory and Applications, pages 235-247, Cham, 2020. Springer International Publishing. Google Scholar
  2. Kazuyuki Amano and Akira Maruoka. On the complexity of depth-2 circuits with threshold gates. In Proc. of the 30th international conference on Mathematical Foundations of Computer Science, pages 107-118, 2005. Google Scholar
  3. Ruiwen Chen, Rahul Santhanam, and Srikanth Srinivasan. Average-case lower bounds and satisfiability algorithms for small threshold circuits. Theory of Computing, 14(9):1-55, 2018. Google Scholar
  4. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 3123-3131, Cambridge, MA, USA, 2015. MIT Press. Google Scholar
  5. James J. DiCarlo and David D. Cox. Untangling invariant object recognition. Trends in Cognitive Sciences, 11(8):333-341, 2007. URL: https://doi.org/10.1016/j.tics.2007.06.010.
  6. Krishnamoorthy Dinesh, Samir Otiv, and Jayalal Sarma. New bounds for energy complexity of boolean functions. Theoretical Computer Science, 845:59-75, 2020. Google Scholar
  7. Peter Földiák. Sparse coding in the primate cortex. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 1064-1068. MIT Press, 2003. Google Scholar
  8. Jürgen Forster, Matthias Krause, Satyanarayana V. Lokam, Rustam Mubarakzjanov, Niels Schmitt, and Hans Ulrich Simon. Relations between communication complexity, linear arrangements, and computational complexity. In Proc. of the 21st International Conference on Foundations of Software Technology and Theoretical Computer Science, pages 171-182, 2001. Google Scholar
  9. András Hajnal, Wolfgang Maass, Pavel Pudlák, Márió Szegedy, and György Turán. Threshold circuits of bounded depth. Journal of Computer and System Sciences, 46:129-154, 1993. Google Scholar
  10. Johan Håstad and Mikael Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1(2):113-129, 1991. Google Scholar
  11. Yunlong He, Koray Kavukcuoglu, Yun Wang, Arthur Szlam, and Yanjun Qi. Unsupervised feature learning by deep sparse coding. In Proc. of SIAM International Conference on Data Mining, pages 902-910, 2014. Google Scholar
  12. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1-30, 2018. URL: http://jmlr.org/papers/v18/16-456.html.
  13. Russell Impagliazzo, Ramamohan Paturi, and Michael E. Saks. Size-depth tradeoffs for threshold circuits. SIAM Journal on Computing, 26(3):693-707, 1997. Google Scholar
  14. Stasys Jukna. Extremal Combinatorics with Applications in Computer Science. Springer-Verlag Berlin Heidelberg, 2011. Google Scholar
  15. D. M. Kane and R. Williams. Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits. In Proc. of the 48th Annual ACM Symposium on Theory of Computing, pages 633-643, 2016. Google Scholar
  16. O. M. Kasim-zade. On a measure of active circuits of functional elements (russian). Mathematical problems in cybernetics "Nauka", 4:218?-228, 1992. Google Scholar
  17. Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y. Ng. Efficient sparse coding algorithms. In Proc. of the 19th Advances in Neural Information Processing Systems, pages 801-808, 2006. Google Scholar
  18. Peter Lennie. The cost of cortical computation. Current Biology, 13:493-497, 2003. Google Scholar
  19. Nancy Lynch and Cameron Musco. A Basic Compositional Model for Spiking Neural Networks, pages 403-449. Springer Nature Switzerland, 2022. Google Scholar
  20. Wolffanf Maass, Georg. Schnitger, and Eduardo D. Sontag. On the computational power of sigmoid versus boolean threshold circuits. In Proc. of 32nd Annual Symposium of Foundations of Computer Science, pages 767-776, 1991. URL: https://doi.org/10.1109/SFCS.1991.185447.
  21. Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9):1659-1671, 1997. Google Scholar
  22. Hiroki Maniwa, Takayuki Oki, Akira Suzuki, Kei, and Xiao Zhou. Computational power of threshold circuits of energy at most two. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E101.A(9):1431-1439, 2018. Google Scholar
  23. Warren S. McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-133, 1943. Google Scholar
  24. Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. MIT Press, 1988. Google Scholar
  25. Andrew Ng. Sparse autoencoder. CS294A Lecture notes, 2011. Google Scholar
  26. Noam Nisan. The communication complexity of threshold gates. Proceeding of "Combinatorics, Paul Erd̈os is Eighty", pages 301-315, 1993. Google Scholar
  27. Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481-487, 2004. Google Scholar
  28. Ian Parberry. Circuit Complexity and Neural Networks. MIT Press, 1994. Google Scholar
  29. Thomas Pfeil, Tobias Potjans, Sven Schrader, Wiebke Potjans, Johannes Schemmel, Markus Diesmann, and Karlheinz Meier. Is a 4-bit synaptic weight resolution enough? - constraints on enabling spike-timing dependent plasticity in neuromorphic hardware. Frontiers in Neuroscience, 6:90, 2012. URL: https://doi.org/10.3389/fnins.2012.00090.
  30. Alexander A. Razborov and Alexander A. Sherstov. The sign-rank of AC⁰. SIAM Journal on Computing, 39(5):1833-1855, 2010. Google Scholar
  31. Frank Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386-408, 1958. Google Scholar
  32. Alexander A. Sherstov. Separating AC⁰ from depth-2 majority circuits. In Proceedings of the Thirty-Ninth Annual ACM Symposium on Theory of Computing, STOC '07, page 294?301, New York, NY, USA, 2007. Association for Computing Machinery. URL: https://doi.org/10.1145/1250790.1250834.
  33. Janio Carlos Nascimento Silva and Uéverton S. Souza. Computing the best-case energy complexity of satisfying assignments in monotone circuits. Theoretical Computer Science, 932:41-55, 2022. Google Scholar
  34. K. Y. Siu and J. Bruck. On the power of threshold circuits with small weights. SIAM Journal on Discrete Mathematics, 4(3):423-435, 1991. Google Scholar
  35. Kai-Yeung. Siu, Vwani Roychowdhury, and Thomas Kailath. Discrete Neural Computation: A Theoretical Foundation. Prentice Hall, 1995. Google Scholar
  36. Kai-Yeung Siu and Vwani P. Roychowdhury. On optimal depth threshold circuits for multiplication and related problems. SIAM Journal on Discrete Mathematics, 7(2):284-292, 1994. Google Scholar
  37. Xiaoming Sun, Yuan Sun, Kewen Wu, and Zhiyu Xia. On the relationship between energy complexity and other boolean function measures. In Proc. of the 25th International Computing and Combinatorics Conference, pages 516-528, 2019. Google Scholar
  38. Akira Suzuki, Kei Uchizawa, and Xiao Zhou. Energy-efficient threshold circuits computing MOD functions. In Proc. of the 17th Computing: the Australasian Theory Symposium, pages 105-110, 2011. Google Scholar
  39. Akira Suzuki, Kei Uchizawa, and Xiao Zhou. Energy-efficient threshold circuits computing MOD functions. International Journal of Foundations of Computer Science, 24(1):15-29, 2013. Google Scholar
  40. Kei Uchizawa. Lower bounds for threshold circuits of bounded energy. Interdisciplinary Information Sciences, 20(1):27-50, 2014. Google Scholar
  41. Kei Uchizawa. Size, Depth and Energy of Threshold Circuits Computing Parity Function. In Yixin Cao, Siu-Wing Cheng, and Minming Li, editors, 31st International Symposium on Algorithms and Computation (ISAAC 2020), volume 181 of Leibniz International Proceedings in Informatics (LIPIcs), pages 54:1-54:13, Dagstuhl, Germany, 2020. Schloss Dagstuhl-Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.ISAAC.2020.54.
  42. Kei Uchizawa, Rodney Douglas, and Wolfgang Maass. On the computational power of threshold circuits with sparse activity. Neural Computation, 18(12):2994-3008, 2008. Google Scholar
  43. Kei Uchizawa and Eiji Takimoto. Exponential lower bounds on the size of constant-depth threshold circuits with small energy complexity. Theoretical Computer Science, 407(1-3):474-487, 2008. Google Scholar
  44. Kei Uchizawa, Eiji Takimoto, and Takao Nishizeki. Size-energy tradeoffs of unate circuits computing symmetric Boolean functions. Theoretical Computer Science, 412:773-782, 2011. Google Scholar
  45. M. N. Vaintsvaig. On the power of networks of functional elements (Russian). Doklady Akademii Nauk, 139(2):320?323, 1961. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact