Document Open Access Logo

Computational Hardness of Certifying Bounds on Constrained PCA Problems

Authors Afonso S. Bandeira, Dmitriy Kunisky, Alexander S. Wein

PDF

File

Thumbnail PDF
PDF
LIPIcs.ITCS.2020.78.pdf
  • Filesize: 0.62 MB
  • 29 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Computational complexity and cryptography
Keywords
  • Certification
  • Sherrington-Kirkpatrick model
  • spiked Wishart model
  • low-degree likelihood ratio

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Given a random n × n symmetric matrix ? drawn from the Gaussian orthogonal ensemble (GOE), we consider the problem of certifying an upper bound on the maximum value of the quadratic form ?^⊤ ? ? over all vectors ? in a constraint set ? ⊂ ℝⁿ. For a certain class of normalized constraint sets we show that, conditional on a certain complexity-theoretic conjecture, no polynomial-time algorithm can certify a better upper bound than the largest eigenvalue of ?. A notable special case included in our results is the hypercube ? = {±1/√n}ⁿ, which corresponds to the problem of certifying bounds on the Hamiltonian of the Sherrington-Kirkpatrick spin glass model from statistical physics. Our results suggest a striking gap between optimization and certification for this problem.
Our proof proceeds in two steps. First, we give a reduction from the detection problem in the negatively-spiked Wishart model to the above certification problem. We then give evidence that this Wishart detection problem is computationally hard below the classical spectral threshold, by showing that no low-degree polynomial can (in expectation) distinguish the spiked and unspiked models. This method for predicting computational thresholds was proposed in a sequence of recent works on the sum-of-squares hierarchy, and is conjectured to be correct for a large class of problems. Our proof can be seen as constructing a distribution over symmetric matrices that appears computationally indistinguishable from the GOE, yet is supported on matrices whose maximum quadratic form over ? ∈ ? is much larger than that of a GOE matrix.

Cite As Get BibTex

Afonso S. Bandeira, Dmitriy Kunisky, and Alexander S. Wein. Computational Hardness of Certifying Bounds on Constrained PCA Problems. In 11th Innovations in Theoretical Computer Science Conference (ITCS 2020). Leibniz International Proceedings in Informatics (LIPIcs), Volume 151, pp. 78:1-78:29, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2020) https://doi.org/10.4230/LIPIcs.ITCS.2020.78

Author Details

Afonso S. Bandeira
  • Dept. of Mathematics, ETH Zurich, Switzerland
Dmitriy Kunisky
  • Dept. of Mathematics, Courant Institute of Mathematical Sciences, New York University, USA
Alexander S. Wein
  • Dept. of Mathematics, Courant Institute of Mathematical Sciences, New York University, USA

Funding

  • Bandeira, Afonso S.: Part of this work was done while with the Courant Institute of Mathematical Sciences and the Center for Data Science at New York University and supported by NSF grants DMS-1712730, DMS-1719545, and by a grant from the Sloan Foundation.
  • Kunisky, Dmitriy: Partially supported by NSF grants DMS-1712730 and DMS-1719545.
  • Wein, Alexander S.: Partially supported by NSF grant DMS-1712730 and by the Simons Collaboration on Algorithms and Geometry.

Acknowledgements

We thank Andrea Montanari and Samuel B. Hopkins for insightful discussions.

References

  1. Dimitris Achlioptas and Amin Coja-Oghlan. Algorithmic barriers from phase transitions. In Foundations of Computer Science, 2008. FOCS'08. IEEE 49th Annual IEEE Symposium on, pages 793-802. IEEE, 2008. Google Scholar
  2. Louigi Addario-Berry and Pascal Maillard. The algorithmic hardness threshold for continuous random energy models. arXiv preprint, 2018. URL: http://arxiv.org/abs/1810.05129.
  3. Michael Aizenman, Joel L Lebowitz, and David Ruelle. Some rigorous results on the Sherrington-Kirkpatrick spin glass model. Communications in mathematical physics, 112(1):3-20, 1987. Google Scholar
  4. Arash A Amini and Martin J Wainwright. High-dimensional analysis of semidefinite relaxations for sparse principal components. In 2008 IEEE International Symposium on Information Theory, pages 2454-2458. IEEE, 2008. Google Scholar
  5. Greg W Anderson, Alice Guionnet, and Ofer Zeitouni. An introduction to random matrices, 2010. Google Scholar
  6. Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 33(5):1643-1697, 2005. Google Scholar
  7. Jinho Baik and Jack W Silverstein. Eigenvalues of large sample covariance matrices of spiked population models. Journal of multivariate analysis, 97(6):1382-1408, 2006. Google Scholar
  8. Jess Banks, Robert Kleinberg, and Cristopher Moore. The Lovász Theta Function for Random Regular Graphs and Community Detection in the Hard Regime. arXiv preprint, 2017. URL: http://arxiv.org/abs/1705.01194.
  9. Boaz Barak, Samuel B Hopkins, Jonathan Kelner, Pravesh Kothari, Ankur Moitra, and Aaron Potechin. A nearly tight sum-of-squares lower bound for the planted clique problem. In Foundations of Computer Science (FOCS), 2016 IEEE 57th Annual Symposium on, pages 428-437. IEEE, 2016. Google Scholar
  10. Nicolas Boumal, Tamir Bendory, Roy R Lederman, and Amit Singer. Heterogeneous multireference alignment: A single pass approach. In Information Sciences and Systems (CISS), 2018 52nd Annual Conference on, pages 1-6. IEEE, 2018. Google Scholar
  11. Amin Coja-Oghlan, Andreas Goerdt, and André Lanka. Strong refutation heuristics for random k-SAT. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pages 310-321. Springer, 2004. Google Scholar
  12. Nadia Creignou and Hervé Daude. Satisfiability threshold for random XOR-CNF formulas. Discrete Applied Mathematics, 96:41-53, 1999. Google Scholar
  13. A Crisanti, H Horner, and H-J Sommers. The spherical p-spin interaction spin-glass model. Zeitschrift für Physik B Condensed Matter, 92(2):257-271, 1993. Google Scholar
  14. A Crisanti and HJ Sommers. The spherical p-spin interaction spin glass model: the statics. Phys. B, 87:341, 1992. Google Scholar
  15. Andrea Crisanti and Tommaso Rizzo. Analysis of the infinity-replica symmetry breaking solution of the Sherrington-Kirkpatrick model. Physical Review E, 65(4):046137, 2002. Google Scholar
  16. Aurelien Decelle, Florent Krzakala, Cristopher Moore, and Lenka Zdeborová. Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Physical Review E, 84(6):066106, 2011. Google Scholar
  17. Yash Deshpande and Andrea Montanari. Information-theoretically optimal sparse PCA. arXiv preprint, 2014. URL: http://arxiv.org/abs/1402.2238.
  18. Yash Deshpande and Andrea Montanari. Sparse PCA via covariance thresholding. In Advances in Neural Information Processing Systems, pages 334-342, 2014. Google Scholar
  19. Yash Deshpande and Andrea Montanari. Improved sum-of-squares lower bounds for hidden clique and hidden submatrix problems. In Conference on Learning Theory, pages 523-562, 2015. Google Scholar
  20. Yash Deshpande, Andrea Montanari, and Emile Richard. Cone-constrained principal component analysis. In Advances in Neural Information Processing Systems, pages 2717-2725, 2014. Google Scholar
  21. Yunzi Ding, Dmitriy Kunisky, Alexander S Wein, and Afonso S Bandeira. Subexponential-Time Algorithms for Sparse PCA. arXiv preprint, 2019. URL: http://arxiv.org/abs/1907.11635.
  22. Uriel Feige. Relations between average case complexity and approximation complexity. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 534-543. ACM, 2002. Google Scholar
  23. Delphine Féral and Sandrine Péché. The largest eigenvalue of rank one deformation of large Wigner matrices. Communications in mathematical physics, 272(1):185-228, 2007. Google Scholar
  24. Alyson K Fletcher and Sundeep Rangan. Iterative reconstruction of rank-one matrices in noise. Information and Inference: A Journal of the IMA, 7(3):531-562, 2018. Google Scholar
  25. David Gamarnik and Quan Li. Finding a large submatrix of a Gaussian random matrix. The Annals of Statistics, 46(6A):2511-2561, 2018. Google Scholar
  26. Dima Grigoriev. Linear lower bound on degrees of Positivstellensatz calculus proofs for the parity. Theoretical Computer Science, 259(1-2):613-622, 2001. Google Scholar
  27. Dima Grigoriev and Nicolai Vorobjov. Complexity of Null-and Positivstellensatz proofs. Annals of Pure and Applied Logic, 113(1-3):153-160, 2001. Google Scholar
  28. Samuel Hopkins. Statistical Inference and the Sum of Squares Method. PhD thesis, Cornell University, 2018. Google Scholar
  29. Samuel B Hopkins, Pravesh K Kothari, Aaron Potechin, Prasad Raghavendra, Tselil Schramm, and David Steurer. The power of sum-of-squares for detecting hidden structures. In Foundations of Computer Science (FOCS), 2017 IEEE 58th Annual Symposium on, pages 720-731. IEEE, 2017. Google Scholar
  30. Samuel B Hopkins, Jonathan Shi, and David Steurer. Tensor principal component analysis via sum-of-squares proofs. In Conference on Learning Theory, pages 956-1006, 2015. Google Scholar
  31. Samuel B Hopkins and David Steurer. Bayesian estimation from few samples: community detection and related problems. arXiv preprint, 2017. URL: http://arxiv.org/abs/1710.00264.
  32. Vishesh Jain, Frederic Koehler, and Andrej Risteski. Mean-field approximation, convex hierarchies, and the optimality of correlation rounding: a unified perspective. arXiv preprint, 2018. URL: http://arxiv.org/abs/1808.07226.
  33. Mark Jerrum. Large cliques elude the Metropolis process. Random Structures & Algorithms, 3(4):347-359, 1992. Google Scholar
  34. Iain M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics, 29(2):295-327, 2001. Google Scholar
  35. Iain M Johnstone and Arthur Yu Lu. Sparse principal components analysis. Unpublished manuscript, 7, 2004. Google Scholar
  36. Subhash Khot, Guy Kindler, Elchanan Mossel, and Ryan O’Donnell. Optimal inapproximability results for MAX-CUT and other 2-variable CSPs? SIAM Journal on Computing, 37(1):319-357, 2007. Google Scholar
  37. Chiheon Kim, Afonso S Bandeira, and Michel X Goemans. Community detection in hypergraphs, spiked tensor models, and Sum-of-Squares. In Sampling Theory and Applications (SampTA), 2017 International Conference on, pages 124-128. IEEE, 2017. Google Scholar
  38. Pravesh K Kothari, Ryuhei Mori, Ryan O'Donnell, and David Witmer. Sum of squares lower bounds for refuting any CSP. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pages 132-145. ACM, 2017. Google Scholar
  39. Dmitriy Kunisky and Afonso S Bandeira. A Tight Degree 4 Sum-of-Squares Lower Bound for the Sherrington-Kirkpatrick Hamiltonian. arXiv preprint, 2019. URL: http://arxiv.org/abs/1907.11686.
  40. Dmitriy Kunisky, Alexander S Wein, and Afonso S Bandeira. Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratio. arXiv preprint, 2019. URL: http://arxiv.org/abs/1907.11636.
  41. Jean B Lasserre. An explicit exact SDP relaxation for nonlinear 0-1 programs. In International Conference on Integer Programming and Combinatorial Optimization, pages 293-303. Springer, 2001. Google Scholar
  42. Thibault Lesieur, Florent Krzakala, and Lenka Zdeborová. MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel. arXiv preprint, 2015. URL: http://arxiv.org/abs/1507.03857.
  43. Thibault Lesieur, Florent Krzakala, and Lenka Zdeborová. Phase transitions in sparse PCA. In 2015 IEEE International Symposium on Information Theory (ISIT), pages 1635-1639. IEEE, 2015. Google Scholar
  44. Raghu Meka, Aaron Potechin, and Avi Wigderson. Sum-of-squares lower bounds for planted clique. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 87-96. ACM, 2015. Google Scholar
  45. Sidhanth Mohanty, Prasad Raghavendra, and Jeff Xu. Lifting Sum-of-Squares Lower Bounds: Degree-2 to Degree-4, 2019. URL: http://arxiv.org/abs/1911.01411.
  46. Andrea Montanari. Optimization of the Sherrington-Kirkpatrick Hamiltonian. arXiv preprint, 2018. URL: http://arxiv.org/abs/1812.10897.
  47. Andrea Montanari and Emile Richard. Non-negative principal component analysis: Message passing algorithms and sharp asymptotics. IEEE Transactions on Information Theory, 62(3):1458-1484, 2016. Google Scholar
  48. Andrea Montanari and Subhabrata Sen. Semidefinite programs on sparse random graphs and their application to community detection. arXiv preprint, 2015. URL: http://arxiv.org/abs/1504.05910.
  49. Dmitry Panchenko. The Sherrington-Kirkpatrick model. Springer Science & Business Media, 2013. Google Scholar
  50. Giorgio Parisi. Infinite number of order parameters for spin-glasses. Physical Review Letters, 43(23):1754, 1979. Google Scholar
  51. Giorgio Parisi. A sequence of approximated solutions to the SK model for spin glasses. Journal of Physics A: Mathematical and General, 13(4):L115, 1980. Google Scholar
  52. Pablo A Parrilo. Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization. PhD thesis, California Institute of Technology, 2000. Google Scholar
  53. Amelia Perry, Alexander S Wein, Afonso S Bandeira, and Ankur Moitra. Message-Passing Algorithms for Synchronization Problems over Compact Groups. Communications on Pure and Applied Mathematics, 71(11):2275-2322, 2018. Google Scholar
  54. Amelia Perry, Alexander S Wein, Afonso S Bandeira, and Ankur Moitra. Optimality and sub-optimality of PCA I: Spiked random matrix models. The Annals of Statistics, 46(5):2416-2451, 2018. Google Scholar
  55. Prasad Raghavendra, Tselil Schramm, and David Steurer. High-dimensional estimation via sum-of-squares proofs. arXiv preprint, 2018. URL: http://arxiv.org/abs/1807.11419.
  56. Philippe Rigollet and Jan-Christian Hütter. High-dimensional statistics. Lecture notes, 2018. Google Scholar
  57. Steven Roman. The umbral calculus. Springer, 2005. Google Scholar
  58. Grant Schoenebeck. Linear level Lasserre lower bounds for certain k-CSPs. In 2008 49th Annual IEEE Symposium on Foundations of Computer Science, pages 593-602. IEEE, 2008. Google Scholar
  59. David Sherrington and Scott Kirkpatrick. Solvable model of a spin-glass. Physical review letters, 35(26):1792, 1975. Google Scholar
  60. Naum Z Shor. Class of global minimum bounds of polynomial functions. Cybernetics, 23(6):731-734, 1987. Google Scholar
  61. Eliran Subag. Following the ground-states of full-RSB spherical spin glasses. arXiv preprint, 2018. URL: http://arxiv.org/abs/1812.04588.
  62. Michel Talagrand. The Parisi formula. Annals of mathematics, pages 221-263, 2006. Google Scholar
  63. Tengyao Wang, Quentin Berthet, and Yaniv Plan. Average-case hardness of RIP certification. In Advances in Neural Information Processing Systems, pages 3819-3827, 2016. Google Scholar
  64. Eugene P Wigner. Characteristic vectors of bordered matrices with infinite dimensions I. In The Collected Works of Eugene Paul Wigner, pages 524-540. Springer, 1993. Google Scholar
  65. Lenka Zdeborova and Florent Krzakala. Hiding quiet solutions in random constraint satisfaction problems. Physical Review Letters, 102(LA-UR-08-08090; LA-UR-08-8090), 2008. Google Scholar
  66. Lenka Zdeborová and Florent Krzakala. Quiet planting in the locked constraint satisfaction problems. SIAM Journal on Discrete Mathematics, 25(2):750-770, 2011. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact