Document

# Universal Matrix Sparsifiers and Fast Deterministic Algorithms for Linear Algebra

## File

LIPIcs.ITCS.2024.13.pdf
• Filesize: 0.83 MB
• 24 pages

## Cite As

Rajarshi Bhattacharjee, Gregory Dexter, Cameron Musco, Archan Ray, Sushant Sachdeva, and David P. Woodruff. Universal Matrix Sparsifiers and Fast Deterministic Algorithms for Linear Algebra. In 15th Innovations in Theoretical Computer Science Conference (ITCS 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 287, pp. 13:1-13:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/LIPIcs.ITCS.2024.13

## Abstract

Let S ∈ ℝ^{n × n} be any matrix satisfying ‖1-S‖₂ ≤ εn, where 1 is the all ones matrix and ‖⋅‖₂ is the spectral norm. It is well-known that there exists S with just O(n/ε²) non-zero entries achieving this guarantee: we can let 𝐒 be the scaled adjacency matrix of a Ramanujan expander graph. We show that, beyond giving a sparse approximation to the all ones matrix, 𝐒 yields a universal sparsifier for any positive semidefinite (PSD) matrix. In particular, for any PSD A ∈ ℝ^{n×n} which is normalized so that its entries are bounded in magnitude by 1, we show that ‖A-A∘S‖₂ ≤ ε n, where ∘ denotes the entrywise (Hadamard) product. Our techniques also yield universal sparsifiers for non-PSD matrices. In this case, we show that if S satisfies ‖1-S‖₂ ≤ (ε²n)/(c log²(1/ε)) for some sufficiently large constant c, then ‖A-A∘S‖₂ ≤ ε⋅max(n,‖ A‖₁), where ‖A‖₁ is the nuclear norm. Again letting 𝐒 be a scaled Ramanujan graph adjacency matrix, this yields a sparsifier with Õ(n/ε⁴) entries. We prove that the above universal sparsification bounds for both PSD and non-PSD matrices are tight up to logarithmic factors. Since 𝐀∘𝐒 can be constructed deterministically without reading all of A, our result for PSD matrices derandomizes and improves upon established results for randomized matrix sparsification, which require sampling a random subset of O((n log n)/ε²) entries and only give an approximation to any fixed A with high probability. We further show that any randomized algorithm must read at least Ω(n/ε²) entries to spectrally approximate general A to error εn, thus proving that these existing randomized algorithms are optimal up to logarithmic factors. We leverage our deterministic sparsification results to give the first {deterministic algorithms} for several problems, including singular value and singular vector approximation and positive semidefiniteness testing, that run in faster than matrix multiplication time. This partially addresses a significant gap between randomized and deterministic algorithms for fast linear algebraic computation. Finally, if A ∈ {-1,0,1}^{n × n} is PSD, we show that a spectral approximation Ã with ‖A-Ã‖₂ ≤ ε n can be obtained by deterministically reading Õ(n/ε) entries of A. This improves the 1/ε dependence on our result for general PSD matrices by a quadratic factor and is information-theoretically optimal up to a logarithmic factor.

## Subject Classification

##### ACM Subject Classification
• Theory of computation → Sketching and sampling
• Mathematics of computing → Computations on matrices
• Theory of computation → Expander graphs and randomness extractors
##### Keywords
• sublinear algorithms
• randomized linear algebra
• spectral sparsification
• expanders

## Metrics

• Access Statistics
• Total Accesses (updated on a weekly basis)
0

## References

1. Dimitris Achlioptas, Zohar Shay Karnin, and Edo Liberty. Near-optimal entrywise sampling for data matrices. In Advances in Neural Information Processing Systems 26 (NeurIPS), 2013.
2. Dimitris Achlioptas and Frank McSherry. Fast computation of low rank matrix approximations. In Proceedings of the 39th Annual ACM Symposium on Theory of Computing (STOC), 2007.
3. Josh Alman and Virginia Vassilevska Williams. A refined laser method and faster matrix multiplication. In Proceedings of the 32nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2021.
4. Noga Alon. Eigenvalues and expanders. Combinatorica, 1986.
5. Noga Alon. Explicit expanders of every degree and size. Combinatorica, 2021.
6. Alexandr Andoni, Jiecao Chen, Robert Krauthgamer, Bo Qin, David P Woodruff, and Qin Zhang. On sketching quadratic forms. In Proceedings of the 7th Conference on Innovations in Theoretical Computer Science (ITCS), 2016.
7. Sanjeev Arora, Elad Hazan, and Satyen Kale. Fast algorithms for approximate semidefinite programming using the multiplicative weights update method. In Proceedings of the 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2005.
8. Sanjeev Arora, Elad Hazan, and Satyen Kale. A fast random sampling algorithm for sparsifying matrices. In Proceedings of the 10th International Workshop on Randomization and Computation (RANDOM), 2006.
9. Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random Fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
10. Ainesh Bakshi, Nadiia Chepurko, and Rajesh Jayaram. Testing positive semi-definiteness via random submatrices. In Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2020.
11. Ziv Bar-Yossef, T. S. Jayram, Ravi Kumar, and D. Sivakumar. An information statistics approach to data stream and communication complexity. Journal of Computer and System Sciences, 2004.
12. Joshua Batson, Daniel A Spielman, Nikhil Srivastava, and Shang-Hua Teng. Spectral sparsification of graphs: theory and algorithms. Communications of the ACM, 2013.
13. Rajarshi Bhattacharjee, Gregory Dexter, Petros Drineas, Cameron Musco, and Archan Ray. Sublinear time eigenvalue approximation via random sampling. Proceedings of the 50th International Colloquium on Automata, Languages and Programming (ICALP), 2023.
14. Rajarshi Bhattacharjee, Gregory Dexter, Cameron Musco, Archan Ray, Sushant Sachdeva, and David P Woodruff. Universal matrix sparsifiers and fast deterministic algorithms for linear algebra, 2023. URL: https://arxiv.org/abs/2305.05826.
15. Mark Braverman, Ankit Garg, Denis Pankratov, and Omri Weinstein. Information lower bounds via self-reducibility. In Proceedings of the 8th International Computer Science Symposium in Russia (CSR), 2013.
16. Vladimir Braverman, Robert Krauthgamer, Aditya R Krishnan, and Shay Sapir. Near-optimal entrywise sampling of numerically sparse matrices. In Proceedings of the 34th Annual Conference on Computational Learning Theory (COLT), 2021.
17. Vladimir Braverman, Aditya Krishnan, and Christopher Musco. Sublinear time spectral density estimation. In Proceedings of the 54th Annual ACM Symposium on Theory of Computing (STOC), 2022.
18. Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. Communications of the ACM, 2012.
19. Emmanuel J. Candès and Terence Tao. The power of convex relaxation: near-optimal matrix completion. IEEE Transations on Information Theory, 2010.
20. Amit Chakrabarti and Oded Regev. An optimal lower bound on the communication complexity of gap-Hamming-distance. SIAM Journal on Computing, 2012.
21. Françoise Chatelin. Spectral approximation of linear operators. SIAM, 2011.
22. Julia Chuzhoy, Yu Gao, Jason Li, Danupon Nanongkai, Richard Peng, and Thatchaphol Saranurak. A deterministic algorithm for balanced cut with applications to dynamic connectivity, flows, and beyond. In Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2020.
23. Kenneth L Clarkson and David P Woodruff. Low-rank PSD approximation in input-sparsity time. In Proceedings of the 28th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2017.
24. Michael B Cohen. Ramanujan graphs in polynomial time. In Proceedings of the 57th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2016.
25. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
26. Alexandre d'Aspremont. Subsampling algorithms for semidefinite programming. Stochastic Systems, 2011.
27. Petros Drineas and Anastasios Zouzias. A note on element-wise matrix sparsification via a matrix-valued Bernstein inequality. Information Processing Letters, 2011.
28. Jacek Kuczyński and Henryk Woźniakowski. Estimating the largest eigenvalue by the power and Lanczos algorithms with a random start. SIAM Journal on Matrix Analysis and Applications, 1992.
29. Abhisek Kundu. Element-wise matrix sparsification and reconstruction. PhD thesis, Rensselaer Polytechnic Institute, USA, 2015.
30. Lin Lin, Yousef Saad, and Chao Yang. Approximating spectral densities of large matrices. SIAM Review, 2016.
31. Alexander Lubotzky, Ralph Phillips, and Peter Sarnak. Ramanujan graphs. Combinatorica, 1988.
32. Grigorii Aleksandrovich Margulis. Explicit constructions of concentrators. Problemy Peredachi Informatsii, 1973.
33. Raphael A Meyer, Cameron Musco, Christopher Musco, and David P Woodruff. Hutch++: Optimal stochastic trace estimation. In Symposium on Simplicity in Algorithms (SOSA). SIAM, 2021.
34. Moshe Morgenstern. Existence and explicit constructions of q+1 regular Ramanujan graphs for every prime power q. Journal of Combinatorial Theory, Series B, 1994.
35. Cameron Musco and Christopher Musco. Randomized block Krylov methods for stronger and faster approximate singular value decomposition. Advances in Neural Information Processing Systems 28 (NeurIPS), 2015.
36. Cameron Musco and Christopher Musco. Recursive sampling for the Nyström method. Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
37. Cameron Musco and David P. Woodruff. Sublinear time low-rank approximation of positive semidefinite matrices. In Proceedings of the 58th Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2017.
38. Deanna Needell, William Swartworth, and David P. Woodruff. Testing positive semidefiniteness using linear measurements. Proceedings of the 63rd Annual IEEE Symposium on Foundations of Computer Science (FOCS), 2022.
39. Tim Roughgarden. Communication complexity (for algorithm designers). Foundations and Trends in Theoretical Computer Science, 2016.
40. Yousef Saad. Numerical methods for large eigenvalue problems: Revised edition. SIAM, 2011.
41. Florian Schäfer, Matthias Katzfuss, and Houman Owhadi. Sparse Cholesky factorization by Kullback-Leibler minimization. SIAM Journal on Scientific Computing, 2021.
42. Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. In Proceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC), 2008.
43. Daniel A. Spielman and Shang-Hua Teng. Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems. In Proceedings of the 36th Annual ACM Symposium on Theory of Computing (STOC), 2004.
44. Vaidehi Srinivas, David P Woodruff, Ziyu Xu, and Samson Zhou. Memory bounds for the experts problem. Proceedings of the 54th Annual ACM Symposium on Theory of Computing (STOC), 2022.
45. JA Tropp, A Yurtsever, M Udell, and V Cevher. Randomized single-view algorithms for low-rank matrix approximation. acm report 2017-01, caltech, pasadena, jan. 2017, 2017.
46. Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge University Press, 2018.
47. Shusen Wang, Luo Luo, and Zhihua Zhang. SPSD matrix approximation vis column selection: Theories, algorithms, and extensions. The Journal of Machine Learning Research, 2016.
48. Alexander Weisse, Gerhard Wellein, Andreas Alvermann, and Holger Fehske. The kernel polynomial method. Reviews of modern physics, 2006.
49. Hermann Weyl. The asymptotic distribution law of the eigenvalues of linear partial differential equations (with an application to the theory of cavity radiation). Mathematical Annals, 1912.
50. Christopher Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. Advances in Neural Information Processing Systems 13 (NeurIPS), 2000.
51. David Woodruff and William Swartworth. Optimal eigenvalue approximation via sketching. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing (STOC), 2023.
52. Jianlin Xia and Ming Gu. Robust approximate Cholesky factorization of rank-structured symmetric positive definite matrices. SIAM Journal on Matrix Analysis and Applications, 2010.
X

Feedback for Dagstuhl Publishing