Document Open Access Logo

Weighted Polynomial Approximations: Limits for Learning and Pseudorandomness

Authors Mark Bun, Thomas Steinke

PDF
Thumbnail PDF

File

LIPIcs.APPROX-RANDOM.2015.625.pdf
  • Filesize: 0.52 MB
  • 20 pages

Document Identifiers

Author Details

Mark Bun
Thomas Steinke

Cite AsGet BibTex

Mark Bun and Thomas Steinke. Weighted Polynomial Approximations: Limits for Learning and Pseudorandomness. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2015). Leibniz International Proceedings in Informatics (LIPIcs), Volume 40, pp. 625-644, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2015)
https://doi.org/10.4230/LIPIcs.APPROX-RANDOM.2015.625

Abstract

Low-degree polynomial approximations to the sign function underlie pseudorandom generators for halfspaces, as well as algorithms for agnostically learning halfspaces. We study the limits of these constructions by proving inapproximability results for the sign function. First, we investigate the derandomization of Chernoff-type concentration inequalities. Schmidt et al. (SIAM J. Discrete Math. 1995) showed that a tail bound of delta can be established for sums of Bernoulli random variables with only O(log(1/delta))-wise independence. We show that their results are tight up to constant factors. Secondly, the “polynomial regression” algorithm of Kalai et al. (SIAM J. Comput. 2008) shows that halfspaces can be efficiently learned with respect to log-concave distributions on R^n in the challenging agnostic learning model. The power of this algorithm relies on the fact that under log-concave distributions, halfspaces can be approximated arbitrarily well by low-degree polynomials. In contrast, we exhibit a large class of non-log-concave distributions under which polynomials of any degree cannot approximate the sign function to within arbitrarily low error.
Keywords
  • Polynomial Approximations
  • Pseudorandomness
  • Concentration
  • Learning Theory
  • Halfspaces

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Scott Aaronson and Yaoyun Shi. Quantum lower bounds for the collision and the element distinctness problems. J. ACM, 51(4):595-605, 2004. Google Scholar
  2. Noga Alon, László Babai, and Alon Itai. A fast and simple randomized parallel algorithm for the maximal independent set problem. Journal of Algorithms, 7(4):567-583, 1986. Google Scholar
  3. Noga Alon and Asaf Nussboim. K-wise independent random graphs. In Foundations of Computer Science, 2008. FOCS'08. IEEE 49th Annual IEEE Symposium on, pages 813-822. IEEE, 2008. Google Scholar
  4. Louay M. J. Bazzi. Polylogarithmic independence can fool DNF formulas. SIAM J. Comput., 38(6):2220-2272, March 2009. Google Scholar
  5. Robert Beals, Harry Buhrman, Richard Cleve, Michele Mosca, and Ronald de Wolf. Quantum lower bounds by polynomials. J. ACM, 48(4):778-797, 2001. Google Scholar
  6. Richard Beigel. The polynomial method in circuit complexity. In Structure in Complexity Theory Conference, pages 82-95, 1993. Google Scholar
  7. Richard Beigel. Perceptrons, PP, and the polynomial hierarchy. Computational Complexity, 4:339-349, 1994. Google Scholar
  8. M. Bellare and J. Rompel. Randomness-efficient oblivious sampling. In FOCS, pages 276-287, Nov 1994. Google Scholar
  9. Itai Benjamini, Ori Gurel-Gurevich, and Ron Peled. On k-wise independent distributions and boolean functions. arXiv preprint arXiv:1201.3261, 2012. Google Scholar
  10. S. N. Bernstein. Le problème de l'approximation des fonctions continues sur tout l'axe réel et l'une de ses applications. Bull. Math. Soc. France, 52:399-410, 1924. Google Scholar
  11. Eric Blais, Ryan O'Donnell, and Karl Wimmer. Polynomial regression under arbitrary product distributions. Machine Learning, 80(2-3):273-294, 2010. Google Scholar
  12. Aline Bonami. Étude des coefficients de fourier des fonctions de l^p(g). Annales de l'institut Fourier, 20(2):335-402, 1970. Google Scholar
  13. Mark Braverman. Polylogarithmic independence fools AC0 circuits. J. ACM, 57(5):28:1-28:10, June 2008. Google Scholar
  14. Mark Braverman, Anup Rao, Ran Raz, and Amir Yehudayoff. Pseudorandom generators for regular branching programs. FOCS, pages 40-47, 2010. Google Scholar
  15. Joshua Brody and Elad Verbin. The coin problem and pseudorandomness for branching programs. In FOCS, pages 30-39, 2010. Google Scholar
  16. Lennart Carleson. Bernstein’s approximation problem. Proc. Amer. Math. Soc., 2:953-961, 1951. Google Scholar
  17. E.W. Cheney. Introduction to Approximation Theory. AMS Chelsea Publishing Series. AMS Chelsea Pub., 1982. Google Scholar
  18. Dana Dachman-Soled, Vitaly Feldman, Li-Yang Tan, Andrew Wan, and Karl Wimmer. Approximate resilience, monotonicity, and the complexity of agnostic learning. CoRR, abs/1405.5268, 2014. To appear in SODA 2015. Google Scholar
  19. Amit Daniely, Nati Linial, and Shai Shalev-Shwartz. The complexity of learning halfspaces using generalized linear methods. CoRR, abs/1211.0616, 2014. Google Scholar
  20. Anindya De, Omid Etesami, Luca Trevisan, and Madhur Tulsiani. Improved pseudorandom generators for depth 2 circuits. In Maria Serna, Ronen Shaltiel, Klaus Jansen, and Jose Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, volume 6302 of Lecture Notes in Computer Science, pages 504-517, 2010. Google Scholar
  21. Ilias Diakonikolas, Parikshit Gopalan, Ragesh Jaiswal, Rocco A. Servedio, and Emanuele Viola. Bounded independence fools halfspaces. In In Proc. 50th Annual Symposium on Foundations of Computer Science (FOCS), pages 171-180, 2009. Google Scholar
  22. Ilias Diakonikolas, Rocco A. Servedio, Li-Yang Tan, and Andrew Wan. A regularity lemma, and low-weight approximators, for low-degree polynomial threshold functions. In Proceedings of the 2010 IEEE 25th Annual Conference on Computational Complexity, CCC'10, pages 211-222, Washington, DC, USA, 2010. IEEE Computer Society. Google Scholar
  23. H. Ehlich and K. Zeller. Schwankung von polynomen zwischen gitterpunkten. Mathematische Zeitschrift, 86:41-44, 1964. Google Scholar
  24. Vitaly Feldman, Parikshit Gopalan, Subhash Khot, and Ashok Kumar Ponnuswami. New results for learning noisy parities and halfspaces. In Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science, FOCS'06, pages 563-574, Washington, DC, USA, 2006. IEEE Computer Society. Google Scholar
  25. Vitaly Feldman and Pravesh Kothari. Agnostic learning of disjunctions on symmetric distributions. CoRR, abs/1405.6791, 2014. Google Scholar
  26. Parikshit Gopalan, Adam Tauman Kalai, and Adam R. Klivans. Agnostically learning decision trees. In STOC, pages 527-536, 2008. Google Scholar
  27. Parikshit Gopalan, Daniel Kane, and Raghu Meka. Pseudorandomness for concentration bounds and signed majorities. CoRR, abs/1411.4584, 2014. Google Scholar
  28. Parikshit Gopalan, Ryan O'Donnell, Yi Wu, and David Zuckerman. Fooling functions of halfspaces under product distributions. In Proceedings of the 2010 IEEE 25th Annual Conference on Computational Complexity, CCC'10, pages 223-234, Washington, DC, USA, 2010. IEEE Computer Society. Google Scholar
  29. Parikshit Gopalan and Jaikumar Radhakrishnan. Finding duplicates in a data stream. In Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 402-411. Society for Industrial and Applied Mathematics, 2009. Google Scholar
  30. V. Guruswami and P. Raghavendra. Hardness of learning halfspaces with noise. In Proceedings of FOCS'06, pages 543-552, 2006. Google Scholar
  31. Nicholas J. A. Harvey, Jelani Nelson, and Krzysztof Onak. Sketching and streaming entropy via approximation theory. In FOCS, pages 489-498, 2008. Google Scholar
  32. Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):pp. 13-30, 1963. Google Scholar
  33. Russell Impagliazzo, Noam Nisan, and Avi Wigderson. Pseudorandomness for network algorithms. In STOC, pages 356-364, 1994. Google Scholar
  34. S. Izumi and T. Kawata. Quasi-analytic class and closure of tⁿ in the interval (-∞, ∞). Tohoku Math. J., 43:267-273, 1937. Google Scholar
  35. Adam Tauman Kalai, Adam R. Klivans, Yishay Mansour, and Rocco A. Servedio. Agnostically learning halfspaces. SIAM J. Comput., 37(6):1777-1805, 2008. Google Scholar
  36. Daniel M. Kane, Adam Klivans, and Raghu Meka. Learning halfspaces under log-concave densities: Polynomial approximations and moment matching. In COLT, pages 522-545, 2013. Google Scholar
  37. Daniel M Kane and Jelani Nelson. A derandomized sparse Johnson-Lindenstrauss transform. arXiv preprint arXiv:1006.3585, 2010. Google Scholar
  38. Michael Kearns, Robert E. Schapire, and Linda M. Sellie. Toward efficient agnostic learning. In Machine Learning, pages 341-352. ACM Press, 1994. Google Scholar
  39. Adam R. Klivans, Philip M. Long, and Alex K. Tang. Baum’s algorithm learns intersections of halfspaces with respect to log-concave distributions. In Irit Dinur, Klaus Jansen, Joseph Naor, and José Rolim, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, volume 5687 of Lecture Notes in Computer Science, pages 588-600. Springer Berlin Heidelberg, 2009. Google Scholar
  40. Adam R. Klivans, Ryan O'Donnell, and Rocco A. Servedio. Learning geometric concepts via gaussian surface area. In FOCS, pages 541-550, 2008. Google Scholar
  41. Adam R. Klivans and Rocco A. Servedio. Learning DNF in time 2^õ(n^1/3). J. Comput. Syst. Sci., 68(2):303-318, 2004. Google Scholar
  42. Adam R. Klivans and Alexander A. Sherstov. Lower bounds for agnostic learning via approximate rank. Computational Complexity, 19(4):581-604, 2010. Google Scholar
  43. Michal Koucký, Prajakta Nimbhorkar, and Pavel Pudlák. Pseudorandom generators for group products. In STOC, pages 263-272, 2011. Google Scholar
  44. Wee Sun Lee, Peter L. Bartlett, and Robert C. Williamson. On efficient agnostic learning of linear combinations of basis functions. In Proceedings of the Eighth Annual Conference on Computational Learning Theory, COLT'95, pages 369-376, New York, NY, USA, 1995. ACM. Google Scholar
  45. Doron Lubinsky. A survey of weighted polynomial approximation with exponential weights. Surveys in Approximation Theory, 3:1-105, 2007. Google Scholar
  46. Michael Luby and Avi Wigderson. Pairwise independence and derandomization. Citeseer, 1995. Google Scholar
  47. A. A. Markov. On a question of D. I. Mendeleev. Zapiski Imperatorskoi Akademii Nauk,, 62:1-24, 1890. Google Scholar
  48. Raghu Meka and David Zuckerman. Pseudorandom generators for polynomial threshold functions. In Proceedings of the Forty-second ACM Symposium on Theory of Computing, STOC'10, pages 427-436, New York, NY, USA, 2010. ACM. Google Scholar
  49. Marvin Minsky and Seymour Papert. Perceptrons: An Introduction to Computational Geometry. MIT Press, Cambridge MA, 1972. Google Scholar
  50. Michael Mitzenmacher and Salil Vadhan. Why simple hash functions work: Exploiting the entropy in a data stream. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA'08, pages 746-755, Philadelphia, PA, USA, 2008. Society for Industrial and Applied Mathematics. Google Scholar
  51. Elchanan Mossel and Mesrob I Ohannessian. On the impossibility of learning the missing mass. arXiv preprint arXiv:1503.03613, 2015. Google Scholar
  52. Joseph Naor and Moni Naor. Small-bias probability spaces: Efficient constructions and applications. SIAM J. Computing, 22:838-856, 1993. Google Scholar
  53. P. Nevai and V. Totik. Sharp Nikolskii inequalities with exponential weights. Analysis Mathematica, 13(4):261-267, 1987. Google Scholar
  54. Paul Nevai. Géza Freud, orthogonal polynomials and Christoffel functions. A case study. Journal of Approximation Theory, 48(1):3-167, 1986. Google Scholar
  55. Paul Nevai and Vilmos Totik. Weighted polynomial inequalities. Constructive Approximation, 2(1):113-127, 1986. Google Scholar
  56. N. Nisan and M. Szegedy. On the degree of boolean functions as real polynomials. Computational Complexity, 4:301-313, 1994. Google Scholar
  57. Noam Nisan. RL ⊂ SC. In STOC, pages 619-623, 1992. Google Scholar
  58. Ryan O'Donnell. Analysis of Boolean Functions. Cambridge University Press, New York, NY, USA, 2014. Google Scholar
  59. Ramamohan Paturi. On the degree of polynomials that approximate symmetric boolean functions (preliminary version). In STOC, pages 468-474, 1992. Google Scholar
  60. Omer Reingold. Undirected connectivity in log-space. J. ACM, 55(4):17:1-17:24, September 2008. Google Scholar
  61. Omer Reingold, Thomas Steinke, and Salil Vadhan. Pseudorandomness for regular branching programs via fourier analysis. In APPROX-RANDOM, pages 655-670, 2013. Google Scholar
  62. T. J. Rivlin and E. W. Cheney. A comparison of uniform approximations on an interval and a finite subset thereof. SIAM J. Numer. Anal., 3(2):311-320, 1966. Google Scholar
  63. Sushant Sachdeva and Nisheeth K. Vishnoi. Faster algorithms via approximation theory. Foundations and Trends in Theoretical Computer Science, 9(2):125-210, 2014. Google Scholar
  64. J. Schmidt, A. Siegel, and A. Srinivasan. Chernoff-Hoeffding bounds for applications with limited independence. SIAM J. Discrete Mathematics, 8(2):223-250, 1995. Google Scholar
  65. Alexander A. Sherstov. Communication lower bounds using dual polynomials. Bulletin of the EATCS, 95:59-93, 2008. Google Scholar
  66. Alexander A. Sherstov. Separating AC^0 from depth-2 majority circuits. SIAM J. Comput., 38(6):2113-2129, 2009. Google Scholar
  67. Leslie G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134-1142, 1984. Google Scholar
  68. Karl Wimmer. Agnostically learning under permutation invariant distributions. In Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, FOCS'10, pages 113-122, Washington, DC, USA, 2010. IEEE Computer Society. Google Scholar
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact