Document

# Weighted Polynomial Approximations: Limits for Learning and Pseudorandomness

## File

LIPIcs.APPROX-RANDOM.2015.625.pdf
• Filesize: 0.52 MB
• 20 pages

## Cite As

Mark Bun and Thomas Steinke. Weighted Polynomial Approximations: Limits for Learning and Pseudorandomness. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2015). Leibniz International Proceedings in Informatics (LIPIcs), Volume 40, pp. 625-644, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2015)
https://doi.org/10.4230/LIPIcs.APPROX-RANDOM.2015.625

## Abstract

Low-degree polynomial approximations to the sign function underlie pseudorandom generators for halfspaces, as well as algorithms for agnostically learning halfspaces. We study the limits of these constructions by proving inapproximability results for the sign function. First, we investigate the derandomization of Chernoff-type concentration inequalities. Schmidt et al. (SIAM J. Discrete Math. 1995) showed that a tail bound of delta can be established for sums of Bernoulli random variables with only O(log(1/delta))-wise independence. We show that their results are tight up to constant factors. Secondly, the “polynomial regression” algorithm of Kalai et al. (SIAM J. Comput. 2008) shows that halfspaces can be efficiently learned with respect to log-concave distributions on R^n in the challenging agnostic learning model. The power of this algorithm relies on the fact that under log-concave distributions, halfspaces can be approximated arbitrarily well by low-degree polynomials. In contrast, we exhibit a large class of non-log-concave distributions under which polynomials of any degree cannot approximate the sign function to within arbitrarily low error.
##### Keywords
• Polynomial Approximations
• Pseudorandomness
• Concentration
• Learning Theory
• Halfspaces

## Metrics

• Access Statistics
• Total Accesses (updated on a weekly basis)
0

## References

1. Scott Aaronson and Yaoyun Shi. Quantum lower bounds for the collision and the element distinctness problems. J. ACM, 51(4):595-605, 2004.
2. Noga Alon, László Babai, and Alon Itai. A fast and simple randomized parallel algorithm for the maximal independent set problem. Journal of Algorithms, 7(4):567-583, 1986.
3. Noga Alon and Asaf Nussboim. K-wise independent random graphs. In Foundations of Computer Science, 2008. FOCS'08. IEEE 49th Annual IEEE Symposium on, pages 813-822. IEEE, 2008.
4. Louay M. J. Bazzi. Polylogarithmic independence can fool DNF formulas. SIAM J. Comput., 38(6):2220-2272, March 2009.
5. Robert Beals, Harry Buhrman, Richard Cleve, Michele Mosca, and Ronald de Wolf. Quantum lower bounds by polynomials. J. ACM, 48(4):778-797, 2001.
6. Richard Beigel. The polynomial method in circuit complexity. In Structure in Complexity Theory Conference, pages 82-95, 1993.
7. Richard Beigel. Perceptrons, PP, and the polynomial hierarchy. Computational Complexity, 4:339-349, 1994.
8. M. Bellare and J. Rompel. Randomness-efficient oblivious sampling. In FOCS, pages 276-287, Nov 1994.
9. Itai Benjamini, Ori Gurel-Gurevich, and Ron Peled. On k-wise independent distributions and boolean functions. arXiv preprint arXiv:1201.3261, 2012.
10. S. N. Bernstein. Le problème de l'approximation des fonctions continues sur tout l'axe réel et l'une de ses applications. Bull. Math. Soc. France, 52:399-410, 1924.
11. Eric Blais, Ryan O'Donnell, and Karl Wimmer. Polynomial regression under arbitrary product distributions. Machine Learning, 80(2-3):273-294, 2010.
12. Aline Bonami. Étude des coefficients de fourier des fonctions de l^p(g). Annales de l'institut Fourier, 20(2):335-402, 1970.
13. Mark Braverman. Polylogarithmic independence fools AC0 circuits. J. ACM, 57(5):28:1-28:10, June 2008.
14. Mark Braverman, Anup Rao, Ran Raz, and Amir Yehudayoff. Pseudorandom generators for regular branching programs. FOCS, pages 40-47, 2010.
15. Joshua Brody and Elad Verbin. The coin problem and pseudorandomness for branching programs. In FOCS, pages 30-39, 2010.
16. Lennart Carleson. Bernstein’s approximation problem. Proc. Amer. Math. Soc., 2:953-961, 1951.
17. E.W. Cheney. Introduction to Approximation Theory. AMS Chelsea Publishing Series. AMS Chelsea Pub., 1982.
18. Dana Dachman-Soled, Vitaly Feldman, Li-Yang Tan, Andrew Wan, and Karl Wimmer. Approximate resilience, monotonicity, and the complexity of agnostic learning. CoRR, abs/1405.5268, 2014. To appear in SODA 2015.
19. Amit Daniely, Nati Linial, and Shai Shalev-Shwartz. The complexity of learning halfspaces using generalized linear methods. CoRR, abs/1211.0616, 2014.
20. Anindya De, Omid Etesami, Luca Trevisan, and Madhur Tulsiani. Improved pseudorandom generators for depth 2 circuits. In Maria Serna, Ronen Shaltiel, Klaus Jansen, and Jose Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, volume 6302 of Lecture Notes in Computer Science, pages 504-517, 2010.
21. Ilias Diakonikolas, Parikshit Gopalan, Ragesh Jaiswal, Rocco A. Servedio, and Emanuele Viola. Bounded independence fools halfspaces. In In Proc. 50th Annual Symposium on Foundations of Computer Science (FOCS), pages 171-180, 2009.
22. Ilias Diakonikolas, Rocco A. Servedio, Li-Yang Tan, and Andrew Wan. A regularity lemma, and low-weight approximators, for low-degree polynomial threshold functions. In Proceedings of the 2010 IEEE 25th Annual Conference on Computational Complexity, CCC'10, pages 211-222, Washington, DC, USA, 2010. IEEE Computer Society.
23. H. Ehlich and K. Zeller. Schwankung von polynomen zwischen gitterpunkten. Mathematische Zeitschrift, 86:41-44, 1964.
24. Vitaly Feldman, Parikshit Gopalan, Subhash Khot, and Ashok Kumar Ponnuswami. New results for learning noisy parities and halfspaces. In Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science, FOCS'06, pages 563-574, Washington, DC, USA, 2006. IEEE Computer Society.
25. Vitaly Feldman and Pravesh Kothari. Agnostic learning of disjunctions on symmetric distributions. CoRR, abs/1405.6791, 2014.
26. Parikshit Gopalan, Adam Tauman Kalai, and Adam R. Klivans. Agnostically learning decision trees. In STOC, pages 527-536, 2008.
27. Parikshit Gopalan, Daniel Kane, and Raghu Meka. Pseudorandomness for concentration bounds and signed majorities. CoRR, abs/1411.4584, 2014.
28. Parikshit Gopalan, Ryan O'Donnell, Yi Wu, and David Zuckerman. Fooling functions of halfspaces under product distributions. In Proceedings of the 2010 IEEE 25th Annual Conference on Computational Complexity, CCC'10, pages 223-234, Washington, DC, USA, 2010. IEEE Computer Society.
29. Parikshit Gopalan and Jaikumar Radhakrishnan. Finding duplicates in a data stream. In Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 402-411. Society for Industrial and Applied Mathematics, 2009.
30. V. Guruswami and P. Raghavendra. Hardness of learning halfspaces with noise. In Proceedings of FOCS'06, pages 543-552, 2006.
31. Nicholas J. A. Harvey, Jelani Nelson, and Krzysztof Onak. Sketching and streaming entropy via approximation theory. In FOCS, pages 489-498, 2008.
32. Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):pp. 13-30, 1963.
33. Russell Impagliazzo, Noam Nisan, and Avi Wigderson. Pseudorandomness for network algorithms. In STOC, pages 356-364, 1994.
34. S. Izumi and T. Kawata. Quasi-analytic class and closure of tⁿ in the interval (-∞, ∞). Tohoku Math. J., 43:267-273, 1937.
35. Adam Tauman Kalai, Adam R. Klivans, Yishay Mansour, and Rocco A. Servedio. Agnostically learning halfspaces. SIAM J. Comput., 37(6):1777-1805, 2008.
36. Daniel M. Kane, Adam Klivans, and Raghu Meka. Learning halfspaces under log-concave densities: Polynomial approximations and moment matching. In COLT, pages 522-545, 2013.
37. Daniel M Kane and Jelani Nelson. A derandomized sparse Johnson-Lindenstrauss transform. arXiv preprint arXiv:1006.3585, 2010.
38. Michael Kearns, Robert E. Schapire, and Linda M. Sellie. Toward efficient agnostic learning. In Machine Learning, pages 341-352. ACM Press, 1994.
39. Adam R. Klivans, Philip M. Long, and Alex K. Tang. Baum’s algorithm learns intersections of halfspaces with respect to log-concave distributions. In Irit Dinur, Klaus Jansen, Joseph Naor, and José Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, volume 5687 of Lecture Notes in Computer Science, pages 588-600. Springer Berlin Heidelberg, 2009.
40. Adam R. Klivans, Ryan O'Donnell, and Rocco A. Servedio. Learning geometric concepts via gaussian surface area. In FOCS, pages 541-550, 2008.
41. Adam R. Klivans and Rocco A. Servedio. Learning DNF in time 2^õ(n^1/3). J. Comput. Syst. Sci., 68(2):303-318, 2004.
42. Adam R. Klivans and Alexander A. Sherstov. Lower bounds for agnostic learning via approximate rank. Computational Complexity, 19(4):581-604, 2010.
43. Michal Koucký, Prajakta Nimbhorkar, and Pavel Pudlák. Pseudorandom generators for group products. In STOC, pages 263-272, 2011.
44. Wee Sun Lee, Peter L. Bartlett, and Robert C. Williamson. On efficient agnostic learning of linear combinations of basis functions. In Proceedings of the Eighth Annual Conference on Computational Learning Theory, COLT'95, pages 369-376, New York, NY, USA, 1995. ACM.
45. Doron Lubinsky. A survey of weighted polynomial approximation with exponential weights. Surveys in Approximation Theory, 3:1-105, 2007.
46. Michael Luby and Avi Wigderson. Pairwise independence and derandomization. Citeseer, 1995.
47. A. A. Markov. On a question of D. I. Mendeleev. Zapiski Imperatorskoi Akademii Nauk,, 62:1-24, 1890.
48. Raghu Meka and David Zuckerman. Pseudorandom generators for polynomial threshold functions. In Proceedings of the Forty-second ACM Symposium on Theory of Computing, STOC'10, pages 427-436, New York, NY, USA, 2010. ACM.
49. Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. MIT Press, Cambridge MA, 1972.
50. Michael Mitzenmacher and Salil Vadhan. Why simple hash functions work: Exploiting the entropy in a data stream. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'08, pages 746-755, Philadelphia, PA, USA, 2008. Society for Industrial and Applied Mathematics.
51. Elchanan Mossel and Mesrob I Ohannessian. On the impossibility of learning the missing mass. arXiv preprint arXiv:1503.03613, 2015.
52. Joseph Naor and Moni Naor. Small-bias probability spaces: Efficient constructions and applications. SIAM J. Computing, 22:838-856, 1993.
53. P. Nevai and V. Totik. Sharp Nikolskii inequalities with exponential weights. Analysis Mathematica, 13(4):261-267, 1987.
54. Paul Nevai. Géza Freud, orthogonal polynomials and Christoffel functions. A case study. Journal of Approximation Theory, 48(1):3-167, 1986.
55. Paul Nevai and Vilmos Totik. Weighted polynomial inequalities. Constructive Approximation, 2(1):113-127, 1986.
56. N. Nisan and M. Szegedy. On the degree of boolean functions as real polynomials. Computational Complexity, 4:301-313, 1994.
57. Noam Nisan. RL ⊂ SC. In STOC, pages 619-623, 1992.
58. Ryan O'Donnell. Analysis of Boolean Functions. Cambridge University Press, New York, NY, USA, 2014.
59. Ramamohan Paturi. On the degree of polynomials that approximate symmetric boolean functions (preliminary version). In STOC, pages 468-474, 1992.
60. Omer Reingold. Undirected connectivity in log-space. J. ACM, 55(4):17:1-17:24, September 2008.
61. Omer Reingold, Thomas Steinke, and Salil Vadhan. Pseudorandomness for regular branching programs via fourier analysis. In APPROX-RANDOM, pages 655-670, 2013.
62. T. J. Rivlin and E. W. Cheney. A comparison of uniform approximations on an interval and a finite subset thereof. SIAM J. Numer. Anal., 3(2):311-320, 1966.
63. Sushant Sachdeva and Nisheeth K. Vishnoi. Faster algorithms via approximation theory. Foundations and Trends in Theoretical Computer Science, 9(2):125-210, 2014.
64. J. Schmidt, A. Siegel, and A. Srinivasan. Chernoff-Hoeffding bounds for applications with limited independence. SIAM J. Discrete Mathematics, 8(2):223-250, 1995.
65. Alexander A. Sherstov. Communication lower bounds using dual polynomials. Bulletin of the EATCS, 95:59-93, 2008.
66. Alexander A. Sherstov. Separating AC^0 from depth-2 majority circuits. SIAM J. Comput., 38(6):2113-2129, 2009.
67. Leslie G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134-1142, 1984.
68. Karl Wimmer. Agnostically learning under permutation invariant distributions. In Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, FOCS'10, pages 113-122, Washington, DC, USA, 2010. IEEE Computer Society.
X

Feedback for Dagstuhl Publishing