Document Open Access Logo

Sublinear Time Eigenvalue Approximation via Random Sampling

Authors Rajarshi Bhattacharjee, Gregory Dexter, Petros Drineas, Cameron Musco, Archan Ray

PDF
Thumbnail PDF

File

LIPIcs.ICALP.2023.21.pdf
  • Filesize: 0.79 MB
  • 18 pages

Document Identifiers

Author Details

Rajarshi Bhattacharjee
  • Manning College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA
Gregory Dexter
  • Department of Computer Science, Purdue University, West Lafayette, IN, USA
Petros Drineas
  • Department of Computer Science, Purdue University, West Lafayette, IN, USA
Cameron Musco
  • Manning College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA
Archan Ray
  • Manning College of Information and Computer Sciences, University of Massachusetts, Amherst, MA, USA

Funding

RB, CM and AR were partially supported by an Adobe Research grant, along with NSF Grants 2046235 and 1763618. PD and GD were partially supported by NSF AF 1814041, NSF FRG 1760353, and DOE-SC0022085.

Acknowledgements

We thank Ainesh Bakshi, Rajesh Jayaram, Anil Damle, Nicholas Monath and Christopher Musco for helpful conversations about this work.

Cite AsGet BibTex

Rajarshi Bhattacharjee, Gregory Dexter, Petros Drineas, Cameron Musco, and Archan Ray. Sublinear Time Eigenvalue Approximation via Random Sampling. In 50th International Colloquium on Automata, Languages, and Programming (ICALP 2023). Leibniz International Proceedings in Informatics (LIPIcs), Volume 261, pp. 21:1-21:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023)
https://doi.org/10.4230/LIPIcs.ICALP.2023.21

Abstract

We study the problem of approximating the eigenspectrum of a symmetric matrix A ∈ ℝ^{n×n} with bounded entries (i.e., ‖A‖_∞ ≤ 1). We present a simple sublinear time algorithm that approximates all eigenvalues of A up to additive error ±εn using those of a randomly sampled Õ((log³ n)/ε³)×Õ((log³ n)/ε³) principal submatrix. Our result can be viewed as a concentration bound on the complete eigenspectrum of a random submatrix, significantly extending known bounds on just the singular values (the magnitudes of the eigenvalues). We give improved error bounds of ± ε √{nnz(A)} and ±ε‖A‖_F when the rows of A can be sampled with probabilities proportional to their sparsities or their squared 𝓁₂ norms respectively. Here nnz(A) is the number of non-zero entries in A and ‖A‖_F is its Frobenius norm. Even for the strictly easier problems of approximating the singular values or testing the existence of large negative eigenvalues (Bakshi, Chepurko, and Jayaram, FOCS '20), our results are the first that take advantage of non-uniform sampling to give improved error bounds. From a technical perspective, our results require several new eigenvalue concentration and perturbation bounds for matrices with bounded entries. Our non-uniform sampling bounds require a new algorithmic approach, which judiciously zeroes out entries of a randomly sampled submatrix to reduce variance, before computing the eigenvalues of that submatrix as estimates for those of A. We complement our theoretical results with numerical simulations, which demonstrate the effectiveness of our algorithms in practice.

Subject Classification

ACM Subject Classification
  • Theory of computation → Sketching and sampling
  • Mathematics of computing → Computations on matrices
Keywords
  • sublinear algorithms
  • eigenvalue approximation
  • randomized linear algebra

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

References

  1. Dimitris Achlioptas and Frank McSherry. Fast computation of low-rank matrix approximations. Journal of the ACM (JACM), 54(2):9-es, 2007. Google Scholar
  2. Josh Alman and Virginia Vassilevska Williams. A refined laser method and faster matrix multiplication. In Proceedings of the 32nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2021. Google Scholar
  3. Alexandr Andoni and Huy L Nguyên. Eigenvalues of a matrix in the streaming model. In Proceedings of the 24th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2013. Google Scholar
  4. Arturs Backurs, Piotr Indyk, Cameron Musco, and Tal Wagner. Faster kernel matrix algebra via density estimation. Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. Google Scholar
  5. Ainesh Bakshi, Nadiia Chepurko, and Rajesh Jayaram. Testing positive semi-definiteness via random submatrices. Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2020. Google Scholar
  6. Maria-Florina Balcan, Yi Li, David P Woodruff, and Hongyang Zhang. Testing matrix rank, optimally. In Proceedings of the 30th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2019. Google Scholar
  7. Itai Benjamini, Oded Schramm, and Asaf Shapira. Every minor-closed property of sparse graphs is testable. Advances in Mathematics, 223(6):2200-2218, 2010. Google Scholar
  8. Serge Bernstein. Sur l'extension du théorème limite du calcul des probabilités aux sommes de quantités dépendantes. Mathematische Annalen, 97(1):1-59, 1927. Google Scholar
  9. Rajendra Bhatia. Matrix analysis. Springer Science & Business Media, 2013. Google Scholar
  10. Rajarshi Bhattacharjee, Gregory Dexter, Petros Drineas, Cameron Musco, and Archan Ray. Sublinear time eigenvalue approximation via random sampling. arXiv preprint, 2021. URL: https://arxiv.org/abs/2109.07647.
  11. Vladimir Braverman, Aditya Krishnan, and Christopher Musco. Linear and sublinear time spectral density estimation. Proceedings of the 54th Annual ACM Symposium on Theory of Computing (STOC), 2022. Google Scholar
  12. Nadiia Chepurko, Kenneth L Clarkson, Lior Horesh, Honghao Lin, and David P Woodruff. Quantum-inspired algorithms from randomized numerical linear algebra. arXiv, 2020. URL: https://arxiv.org/abs/2011.04125.
  13. David Cohen-Steiner, Weihao Kong, Christian Sohler, and Gregory Valiant. Approximating the spectrum of a graph. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2018. Google Scholar
  14. James Demmel, Ioana Dumitriu, Olga Holtz, and Robert Kleinberg. Fast matrix multiplication is stable. Numerische Mathematik, 2007. Google Scholar
  15. Kun Dong, Austin R Benson, and David Bindel. Network density of states. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2019. Google Scholar
  16. Petros Drineas and Ravi Kannan. Fast monte-carlo algorithms for approximate matrix multiplication. In Proceedings of the 42nd Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2001. Google Scholar
  17. Talya Eden and Will Rosenbaum. On sampling edges almost uniformly. SIAM Symposium on Simplicty in Algorithms (SOSA), 2018. Google Scholar
  18. Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38(1):1-50, 2010. Google Scholar
  19. Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast Monte-Carlo algorithms for finding low-rank approximations. Journal of the ACM (JACM), 51(6):1025-1041, 2004. Google Scholar
  20. Semyon Aranovich Gershgorin. Uber die abgrenzung der eigenwerte einer matrix. Izvestiya Rossiyskoy akademii nauk. Seriya matematicheskaya, 6:749-754, 1931. Google Scholar
  21. Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. Google Scholar
  22. Alex Gittens and Joel A Tropp. Tail bounds for all eigenvalues of a sum of random matrices. arXiv, 2011. URL: https://arxiv.org/abs/1104.4513.
  23. Oded Goldreich and Dana Ron. Property testing in bounded degree graphs. In Proceedings of the 29th Annual ACM Symposium on Theory of Computing (STOC), 1997. Google Scholar
  24. Oded Goldreich and Dana Ron. Approximating average parameters of graphs. Random Structures & Algorithms, 32(4):473-493, 2008. Google Scholar
  25. Leslie Greengard and John Strain. The fast gauss transform. SIAM Journal on Scientific and Statistical Computing, 1991. Google Scholar
  26. Ming Gu and Stanley C Eisenstat. A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem. SIAM Journal on Matrix Analysis and Applications, 1995. Google Scholar
  27. Moritz Hardt and Eric Price. The noisy power method: A meta algorithm with applications. Advances in Neural Information Processing Systems 27 (NIPS), 2014. Google Scholar
  28. Jonas Helsen, Francesco Battistel, and Barbara M Terhal. Spectral quantum tomography. Quantum Information, 2019. Google Scholar
  29. Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, USA, 2nd edition, 2012. Google Scholar
  30. Robert Krauthgamer and Ori Sasson. Property testing of data dimensionality. In Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2003. Google Scholar
  31. Ruipeng Li, Yuanzhe Xi, Lucas Erlandson, and Yousef Saad. The eigenvalues slicing library (EVSL): Algorithms, implementation, and software. SIAM Journal on Scientific Computing, 2019. Google Scholar
  32. Yi Li, Huy L Nguyên, and David P Woodruff. On sketching matrix norms and the top singular vector. In Proceedings of the 25th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2014. Google Scholar
  33. Yi Li, Zhengyu Wang, and David P Woodruff. Improved testing of low rank matrices. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2014. Google Scholar
  34. Yi Li and David P Woodruff. On approximating functions of the singular values in a stream. In Proceedings of the 48th Annual ACM Symposium on Theory of Computing (STOC), 2016. Google Scholar
  35. Lin Lin, Yousef Saad, and Chao Yang. Approximating spectral densities of large matrices. SIAM Review, 2016. Google Scholar
  36. Milena Mihail and Christos Papadimitriou. On the eigenvalue power law. In International Workshop on Randomization and Approximation Techniques in Computer Science, pages 254-262. Springer, 2002. Google Scholar
  37. Deanna Needell, William Swartworth, and David P Woodruff. Testing positive semidefiniteness using linear measurements. In Proceedings of the 63rd Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2022. Google Scholar
  38. Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. Journal of the ACM (JACM), 2007. Google Scholar
  39. Yousef Saad. Numerical methods for large eigenvalue problems: revised edition. SIAM, 2011. Google Scholar
  40. Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv, 2016. URL: https://arxiv.org/abs/1611.07476.
  41. RN Silver and H Röder. Densities of states of mega-dimensional Hamiltonian matrices. International Journal of Modern Physics C, 1994. Google Scholar
  42. G. W. Stewart and Ji guang Sun. Matrix Perturbation Theory. Academic Press, 1990. Google Scholar
  43. Ewin Tang. Quantum-inspired classical algorithms for principal component analysis and supervised clustering. arXiv, 2018. URL: https://arxiv.org/abs/1811.00414.
  44. Joel A Tropp. Norms of random submatrices and sparse approximation. Comptes Rendus Mathematique, 2008. Google Scholar
  45. Joel A. Tropp. The random paving property for uniformly bounded matrices. Studia Mathematica, 185:67-82, 2008. Google Scholar
  46. Joel A Tropp. An introduction to matrix concentration inequalities. arXiv, 2015. URL: https://arxiv.org/abs/1501.01571.
  47. Madeleine Udell and Alex Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-160, 2019. Google Scholar
  48. Lin-Wang Wang. Calculating the density of states and optical-absorption spectra of large quantum systems by the plane-wave moments method. Physical Review B, 1994. Google Scholar
  49. Alexander Weiße, Gerhard Wellein, Andreas Alvermann, and Holger Fehske. The kernel polynomial method. Reviews of Modern Physics, 2006. Google Scholar
  50. Hermann Weyl. The asymptotic distribution law of the eigenvalues of linear partial differential equations (with an application to the theory of cavity radiation). Mathematical Annals, 1912. Google Scholar
  51. Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. arXiv, 2018. URL: https://arxiv.org/abs/1802.08241.
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact