Document

# Limits on Representing Boolean Functions by Linear Combinations of Simple Functions: Thresholds, ReLUs, and Low-Degree Polynomials

## File

LIPIcs.CCC.2018.6.pdf
• Filesize: 0.56 MB
• 24 pages

## Cite As

Richard Ryan Williams. Limits on Representing Boolean Functions by Linear Combinations of Simple Functions: Thresholds, ReLUs, and Low-Degree Polynomials. In 33rd Computational Complexity Conference (CCC 2018). Leibniz International Proceedings in Informatics (LIPIcs), Volume 102, pp. 6:1-6:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2018)
https://doi.org/10.4230/LIPIcs.CCC.2018.6

## Abstract

We consider the problem of representing Boolean functions exactly by "sparse" linear combinations (over R) of functions from some "simple" class C. In particular, given C we are interested in finding low-complexity functions lacking sparse representations. When C forms a basis for the space of Boolean functions (e.g., the set of PARITY functions or the set of conjunctions) this sort of problem has a well-understood answer; the problem becomes interesting when C is "overcomplete" and the set of functions is not linearly independent. We focus on the cases where C is the set of linear threshold functions, the set of rectified linear units (ReLUs), and the set of low-degree polynomials over a finite field, all of which are well-studied in different contexts. We provide generic tools for proving lower bounds on representations of this kind. Applying these, we give several new lower bounds for "semi-explicit" Boolean functions. Let alpha(n) be an unbounded function such that n^{alpha(n)} is time constructible (e.g. alpha(n) = log^*(n)). We show: - Functions in NTIME[n^{alpha(n)}] that require super-polynomially many linear threshold functions to represent (depth-two neural networks with sign activation function, a special case of depth-two threshold circuit lower bounds). - Functions in NTIME[n^{alpha(n)}] that require super-polynomially many ReLU gates to represent (depth-two neural networks with ReLU activation function). - Functions in NTIME[n^{alpha(n)}] that require super-polynomially many O(1)-degree F_p-polynomials to represent exactly, for every prime p (related to problems regarding Higher-Order "Uncertainty Principles"). We also obtain a function in E^{NP} requiring 2^{Omega(n)} linear combinations. - Functions in NTIME[n^{poly(log n)}] that require super-polynomially many ACC ° THR circuits to represent exactly (further generalizing the recent lower bounds of Murray and the author). We also obtain "fixed-polynomial" lower bounds for functions in NP, for the first three representation classes. All our lower bounds are obtained via algorithms for analyzing linear combinations of simple functions in the above scenarios, in ways which substantially beat exhaustive search.

## Subject Classification

##### ACM Subject Classification
• Theory of computation → Circuit complexity
• Computer systems organization → Neural networks
##### Keywords
• linear threshold functions
• lower bounds
• neural networks
• low-degree polynomials

## Metrics

• Access Statistics
• Total Accesses (updated on a weekly basis)
0

## References

1. Amir Abboud, Aviad Rubinstein, and R. Ryan Williams. Distributed PCP theorems for hardness of approximation in P. In FOCS, pages 25-36, 2017.
2. Josh Alman, Timothy M. Chan, and R. Ryan Williams. Polynomial representations of threshold functions and algorithmic applications. In FOCS, pages 467-476, 2016.
3. Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491, 2016.
4. László Babai, Kristoffer Arnsfelt Hansen, Vladimir V. Podolskii, and Xiaoming Sun. Weights of exact threshold functions. In Mathematical Foundations of Computer Science, pages 66-77, 2010.
5. David A. Mix Barrington, Howard Straubing, and Denis Thérien. Non-uniform automata over groups. Inf. Comput., 89(2):109-132, 1990.
6. Richard Beigel and Jun Tarui. On ACC. Computational Complexity, pages 350-366, 1994.
7. Eli Ben-Sasson and Emanuele Viola. Short PCPs with projection queries. In ICALP, pages 163-173, 2014.
8. Andreas Björklund, Thore Husfeldt, and Mikko Koivisto. Set partitioning via inclusion-exclusion. SIAM J. Comput., 39(2):546-563, 2009.
9. Jean Bourgain. Estimation of certain exponential sums arising in complexity theory. C.R. Acad. Sci. Paris Ser. I, 340:627-631, 2005.
10. Jin-yi Cai, Frederic Green, and Thomas Thierauf. On the correlation of symmetric functions. Mathematical Systems Theory, 29(3):245-258, 1996.
11. Chris Calabro. A lower bound on the size of series-parallel graphs dense in long paths. Electronic Colloquium on Computational Complexity (ECCC), 15(110), 2008.
12. Timothy M. Chan and Ryan Williams. Deterministic APSP, Orthogonal Vectors, and more: Quickly derandomizing Razborov-Smolensky. In SODA, pages 1246-1255, 2016.
13. Arkadev Chattopadhyay and Nikhil S. Mande. Weights at the bottom matter when the top is heavy. Electronic Colloquium on Computational Complexity (ECCC), 24:83, 2017.
14. Ruiwen Chen, Rahul Santhanam, and Srikanth Srinivasan. Average-case lower bounds and satisfiability algorithms for small threshold circuits. In CCC, pages 1:1-1:35, 2016.
15. Amit Daniely. Depth separation for neural networks. In Proceedings of COLT, pages 690-696, 2017.
16. Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Proceedings of COLT, pages 907-940, 2016.
17. Yuval Filmus, Hamed Hatami, Steven Heilman, Elchanan Mossel, Ryan O'Donnell, Sushant Sachdeva, Andrew Wan, and Karl Wimmer. Real Analysis in Computer Science: A collection of open problems, Simons Institute, 2014. URL: https://simons.berkeley.edu/sites/default/files/openprobsmerged.pdf.
18. Fedor V. Fomin and Dieter Kratsch. Exact Exponential Algorithms. Springer, 2010.
19. Anna Gál and Vladimir Trifonov. On the correlation between parity and modular polynomials. Theory Comput. Syst., 50(3):516-536, 2012.
20. Frederic Green. The correlation between parity and quadratic polynomials mod 3. Journal of Computer and System Sciences, 69(1):28-44, 2004.
21. Jacques Hadamard. Résolution d'une question relative aux déterminants. Bull. Sci. Math., 17:30-31, 1893.
22. András Hajnal, Wolfgang Maass, Pavel Pudlák, Mario Szegedy, and György Turán. Threshold circuits of bounded depth. J. Comput. Syst. Sci., 46(2):129-154, 1993.
23. Kristoffer Arnsfelt Hansen and Vladimir V Podolskii. Exact threshold circuits. In CCC, pages 270-279, 2010.
24. Johan Håstad and Mikael Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1:113-129, 1991.
25. Hamed Hatami, Pooya Hatami, and Shachar Lovett. Higher-order Fourier analysis and applications. Manuscript, 2016. URL: https://cseweb.ucsd.edu/~slovett/files/survey-higher_order_fourier.pdf.
26. Ellis Horowitz and Sartaj Sahni. Computing partitions with applications to the knapsack problem. JACM, 21(2):277-292, 1974.
27. Hamid Jahanjou, Eric Miles, and Emanuele Viola. Local reductions. In Proceedings of ICALP, pages 749-760, 2015.
28. Daniel M. Kane and Ryan Williams. Super-linear gate and super-quadratic wire lower bounds for depth-two and depth-three threshold circuits. In STOC, pages 633-643, 2016.
29. Daniel Lokshtanov, Ramamohan Paturi, Suguru Tamaki, R. Ryan Williams, and Huacheng Yu. Beating brute force for systems of polynomial equations over finite fields. In SODA, pages 2190-2202, 2017.
30. Shachar Lovett. Personal communication, 2017.
31. Wolfgang Maass. Bounds for the computational power and learning complexity of analog neural nets. SIAM Journal on Computing, 26(3):708-732, 1997.
32. Peter Bro Miltersen, N. V. Vinodchandran, and Osamu Watanabe. Super-polynomial versus half-exponential circuit size in the exponential hierarchy. In COCOON, Springer LNCS 1627, pages 210-220, 1999.
33. Anirbit Mukherjee and Amitabh Basu. Lower bounds over Boolean inputs for deep neural networks with ReLU gates. ArXiv e-prints, 2017. URL: http://arxiv.org/abs/1711.03073.
34. S. Muroga, I. Toda, and S. Takasu. Theory of majority decision elements. Journal of the Franklin Institute, 271:376-418, 1961.
35. Cody Murray and Ryan Williams. Circuit lower bounds for nondeterministic quasi-polytime: An easy witness lemma for NP and NQP. Electronic Colloquium on Computational Complexity (ECCC), TR17-188, 2017.
36. Noam Nisan. The communication complexity of threshold gates. In Proceedings of "Combinatorics, Paul Erdos is Eighty", pages 301-315, 1994.
37. Mihai Pǎtraşcu and Ryan Williams. On the possibility of faster sat algorithms. In SODA, pages 1065-1075, 2010.
38. Vwani P. Roychowdhury, Alon Orlitsky, and Kai-Yeung Siu. Lower bounds on threshold and related circuits via communication complexity. IEEE Transactions on Information Theory, 40(2):467-474, 1994.
39. Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In International Conference on Machine Learning, pages 2979-2987, 2017.
40. Rahul Santhanam. Circuit lower bounds for Merlin-Arthur classes. SIAM J. Comput., 39(3):1038-1061, 2009.
41. Rahul Santhanam and Ryan Williams. On medium-uniformity and circuit lower bounds. In IEEE Conf. Computational Complexity, pages 15-23, 2013.
42. Joel Seiferas, Michael Fischer, and Albert Meyer. Separating nondeterministic time complexity classes. Journal of the ACM, 25(1):146-167, jan 1978.
43. Suguru Tamaki. A satisfiability algorithm for depth two circuits with a sub-quadratic number of symmetric and threshold gates. Electronic Colloquium on Computational Complexity (ECCC), 23:100, 2016.
44. Matus Telgarsky. benefits of depth in neural networks. In Proceedings of COLT, pages 1517-1539, 2016.
45. Roei Tell. Proving that prBPP=prP is as hard as "almost" proving that P ≠ NP. Electronic Colloquium on Computational Complexity (ECCC), 18(3), 2018.
46. S. Toda. PP is as hard as the polynomial-time hierarchy. sicomp, 20(5):865-877, 1991.
47. L. G. Valiant. Graph-theoretic arguments in low-level complexity. In J. Gruska, editor, MFCS, volume 53 of LNCS, pages 162-176, Tatranská Lomnica, Czechoslovakia, sep 1977. Springer.
48. Virginia Vassilevska Williams and Ryan Williams. Finding, minimizing, and counting weighted subgraphs. SIAM Journal on Computing, 42(3):831-854, 2013.
49. Emanuele Viola. Guest column: correlation bounds for polynomials over 0, 1. SIGACT News, 40(1):27-44, 2009.
50. Ryan Williams. A casual tour around a circuit complexity bound. SIGACT News, 42(3):54-76, 2011.
51. Ryan Williams. Improving exhaustive search implies superpolynomial lower bounds. sicomp, 42(3):1218-1244, 2013.
52. Ryan Williams. New algorithms and lower bounds for circuits with linear threshold gates. In STOC, pages 194-202, 2014.
53. Ryan Williams. Nonuniform ACC circuit lower bounds. JACM, 61(1):2, 2014.
54. Ryan Williams. Counting solutions to polynomial systems via reductions. In Raimund Seidel, editor, 1st Symposium on Simplicity in Algorithms (SOSA 2018), pages 6:1-6:15, 2018.
55. R. O. Winder. Threshold Logic. PhD thesis, Princeton University, 1962. Preliminary version in FOCS'60.
56. Stanislav Žák. A Turing machine time hierarchy. Theoretical Computer Science, 26(3):327-333, 1983.
X

Feedback for Dagstuhl Publishing