Document

# Exponential Lower Bounds for Threshold Circuits of Sub-Linear Depth and Energy

## File

LIPIcs.MFCS.2023.85.pdf
• Filesize: 0.65 MB
• 15 pages

## Acknowledgements

We thank the anonymous reviewers for their careful reading and helpful comments.

## Cite As

Kei Uchizawa and Haruki Abe. Exponential Lower Bounds for Threshold Circuits of Sub-Linear Depth and Energy. In 48th International Symposium on Mathematical Foundations of Computer Science (MFCS 2023). Leibniz International Proceedings in Informatics (LIPIcs), Volume 272, pp. 85:1-85:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023)
https://doi.org/10.4230/LIPIcs.MFCS.2023.85

## Abstract

In this paper, we investigate computational power of threshold circuits and other theoretical models of neural networks in terms of the following four complexity measures: size (the number of gates), depth, weight and energy. Here, the energy of a circuit measures sparsity of their computation, and is defined as the maximum number of gates outputting non-zero values taken over all the input assignments. As our main result, we prove that any threshold circuit C of size s, depth d, energy e and weight w satisfies log(rk(M_C)) ≤ ed (log s + log w + log n), where rk(M_C) is the rank of the communication matrix M_C of a 2n-variable Boolean function that C computes. Thus, such a threshold circuit C is able to compute only a Boolean function of which communication matrix has rank bounded by a product of logarithmic factors of s, w and linear factors of d, e. This implies an exponential lower bound on the size of even sublinear-depth and sublinear-energy threshold circuit. For example, we can obtain an exponential lower bound s = 2^Ω(n^{1/3}) for threshold circuits of depth n^{1/3}, energy n^{1/3} and weight 2^o(n^{1/3}). We also show that the inequality is tight up to a constant factor when the depth d and energy e satisfies ed = o(n/log n). For other models of neural networks such as a discretized ReLU circuits and descretized sigmoid circuits, we define energy as the maximum number of gates outputting non-zero values. We then prove that a similar inequality also holds for a discretized circuit C: rk(M_C) = O(ed(log s + log w + log n)³). Thus, if we consider the number gates outputting non-zero values as a measure for sparse activity of a neural network, our results suggest that larger depth linearly helps neural networks to acquire sparse activity.

## Subject Classification

##### ACM Subject Classification
• Theory of computation → Models of computation
##### Keywords
• Circuit complexity
• disjointness function
• equality function
• neural networks
• threshold circuits
• ReLU cicuits
• sigmoid circuits
• sparse activity

## Metrics

• Access Statistics
• Total Accesses (updated on a weekly basis)
0

## References

1. Kazuyuki Amano. On the size of depth-two threshold circuits for the Inner Product mod 2 function. In Alberto Leporati, Carlos Martín-Vide, Dana Shapira, and Claudio Zandron, editors, Language and Automata Theory and Applications, pages 235-247, Cham, 2020. Springer International Publishing.
2. Kazuyuki Amano and Akira Maruoka. On the complexity of depth-2 circuits with threshold gates. In Proc. of the 30th international conference on Mathematical Foundations of Computer Science, pages 107-118, 2005.
3. Ruiwen Chen, Rahul Santhanam, and Srikanth Srinivasan. Average-case lower bounds and satisfiability algorithms for small threshold circuits. Theory of Computing, 14(9):1-55, 2018.
4. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 3123-3131, Cambridge, MA, USA, 2015. MIT Press.
5. James J. DiCarlo and David D. Cox. Untangling invariant object recognition. Trends in Cognitive Sciences, 11(8):333-341, 2007. URL: https://doi.org/10.1016/j.tics.2007.06.010.
6. Krishnamoorthy Dinesh, Samir Otiv, and Jayalal Sarma. New bounds for energy complexity of boolean functions. Theoretical Computer Science, 845:59-75, 2020.
7. Peter Földiák. Sparse coding in the primate cortex. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks, pages 1064-1068. MIT Press, 2003.
8. Jürgen Forster, Matthias Krause, Satyanarayana V. Lokam, Rustam Mubarakzjanov, Niels Schmitt, and Hans Ulrich Simon. Relations between communication complexity, linear arrangements, and computational complexity. In Proc. of the 21st International Conference on Foundations of Software Technology and Theoretical Computer Science, pages 171-182, 2001.
9. András Hajnal, Wolfgang Maass, Pavel Pudlák, Márió Szegedy, and György Turán. Threshold circuits of bounded depth. Journal of Computer and System Sciences, 46:129-154, 1993.
10. Johan Håstad and Mikael Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1(2):113-129, 1991.
11. Yunlong He, Koray Kavukcuoglu, Yun Wang, Arthur Szlam, and Yanjun Qi. Unsupervised feature learning by deep sparse coding. In Proc. of SIAM International Conference on Data Mining, pages 902-910, 2014.
12. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1-30, 2018. URL: http://jmlr.org/papers/v18/16-456.html.
13. Russell Impagliazzo, Ramamohan Paturi, and Michael E. Saks. Size-depth tradeoffs for threshold circuits. SIAM Journal on Computing, 26(3):693-707, 1997.
14. Stasys Jukna. Extremal Combinatorics with Applications in Computer Science. Springer-Verlag Berlin Heidelberg, 2011.
15. D. M. Kane and R. Williams. Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits. In Proc. of the 48th Annual ACM Symposium on Theory of Computing, pages 633-643, 2016.
16. O. M. Kasim-zade. On a measure of active circuits of functional elements (russian). Mathematical problems in cybernetics "Nauka", 4:218?-228, 1992.
17. Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y. Ng. Efficient sparse coding algorithms. In Proc. of the 19th Advances in Neural Information Processing Systems, pages 801-808, 2006.
18. Peter Lennie. The cost of cortical computation. Current Biology, 13:493-497, 2003.
19. Nancy Lynch and Cameron Musco. A Basic Compositional Model for Spiking Neural Networks, pages 403-449. Springer Nature Switzerland, 2022.
20. Wolffanf Maass, Georg. Schnitger, and Eduardo D. Sontag. On the computational power of sigmoid versus boolean threshold circuits. In Proc. of 32nd Annual Symposium of Foundations of Computer Science, pages 767-776, 1991. URL: https://doi.org/10.1109/SFCS.1991.185447.
21. Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9):1659-1671, 1997.
22. Hiroki Maniwa, Takayuki Oki, Akira Suzuki, Kei, and Xiao Zhou. Computational power of threshold circuits of energy at most two. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E101.A(9):1431-1439, 2018.
23. Warren S. McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-133, 1943.
24. Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. MIT Press, 1988.
25. Andrew Ng. Sparse autoencoder. CS294A Lecture notes, 2011.
26. Noam Nisan. The communication complexity of threshold gates. Proceeding of "Combinatorics, Paul Erd̈os is Eighty", pages 301-315, 1993.
27. Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481-487, 2004.
28. Ian Parberry. Circuit Complexity and Neural Networks. MIT Press, 1994.
29. Thomas Pfeil, Tobias Potjans, Sven Schrader, Wiebke Potjans, Johannes Schemmel, Markus Diesmann, and Karlheinz Meier. Is a 4-bit synaptic weight resolution enough? - constraints on enabling spike-timing dependent plasticity in neuromorphic hardware. Frontiers in Neuroscience, 6:90, 2012. URL: https://doi.org/10.3389/fnins.2012.00090.
30. Alexander A. Razborov and Alexander A. Sherstov. The sign-rank of AC⁰. SIAM Journal on Computing, 39(5):1833-1855, 2010.
31. Frank Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386-408, 1958.
32. Alexander A. Sherstov. Separating AC⁰ from depth-2 majority circuits. In Proceedings of the Thirty-Ninth Annual ACM Symposium on Theory of Computing, STOC '07, page 294?301, New York, NY, USA, 2007. Association for Computing Machinery. URL: https://doi.org/10.1145/1250790.1250834.
33. Janio Carlos Nascimento Silva and Uéverton S. Souza. Computing the best-case energy complexity of satisfying assignments in monotone circuits. Theoretical Computer Science, 932:41-55, 2022.
34. K. Y. Siu and J. Bruck. On the power of threshold circuits with small weights. SIAM Journal on Discrete Mathematics, 4(3):423-435, 1991.
35. Kai-Yeung. Siu, Vwani Roychowdhury, and Thomas Kailath. Discrete Neural Computation: A Theoretical Foundation. Prentice Hall, 1995.
36. Kai-Yeung Siu and Vwani P. Roychowdhury. On optimal depth threshold circuits for multiplication and related problems. SIAM Journal on Discrete Mathematics, 7(2):284-292, 1994.
37. Xiaoming Sun, Yuan Sun, Kewen Wu, and Zhiyu Xia. On the relationship between energy complexity and other boolean function measures. In Proc. of the 25th International Computing and Combinatorics Conference, pages 516-528, 2019.
38. Akira Suzuki, Kei Uchizawa, and Xiao Zhou. Energy-efficient threshold circuits computing MOD functions. In Proc. of the 17th Computing: the Australasian Theory Symposium, pages 105-110, 2011.
39. Akira Suzuki, Kei Uchizawa, and Xiao Zhou. Energy-efficient threshold circuits computing MOD functions. International Journal of Foundations of Computer Science, 24(1):15-29, 2013.
40. Kei Uchizawa. Lower bounds for threshold circuits of bounded energy. Interdisciplinary Information Sciences, 20(1):27-50, 2014.
41. Kei Uchizawa. Size, Depth and Energy of Threshold Circuits Computing Parity Function. In Yixin Cao, Siu-Wing Cheng, and Minming Li, editors, 31st International Symposium on Algorithms and Computation (ISAAC 2020), volume 181 of Leibniz International Proceedings in Informatics (LIPIcs), pages 54:1-54:13, Dagstuhl, Germany, 2020. Schloss Dagstuhl-Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.ISAAC.2020.54.
42. Kei Uchizawa, Rodney Douglas, and Wolfgang Maass. On the computational power of threshold circuits with sparse activity. Neural Computation, 18(12):2994-3008, 2008.
43. Kei Uchizawa and Eiji Takimoto. Exponential lower bounds on the size of constant-depth threshold circuits with small energy complexity. Theoretical Computer Science, 407(1-3):474-487, 2008.
44. Kei Uchizawa, Eiji Takimoto, and Takao Nishizeki. Size-energy tradeoffs of unate circuits computing symmetric Boolean functions. Theoretical Computer Science, 412:773-782, 2011.
45. M. N. Vaintsvaig. On the power of networks of functional elements (Russian). Doklady Akademii Nauk, 139(2):320?323, 1961.
X

Feedback for Dagstuhl Publishing

### Thanks for your feedback!

Feedback submitted

### Could not send message

Please try again later or send an E-mail