Document Open Access Logo

Nearly Instance Optimal Sparse Matrix Approximation from Matrix-Vector Products

Authors Christopher Musco , Indu Ramesh

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESA.2026.85.pdf
  • Filesize: 1 MB
  • 14 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Design and analysis of algorithms
Keywords
  • Matrix learning
  • sparse approximation
  • implicit methods

Metrics

Abstract

A large body of work studies the problem of learning an approximation to an implicit matrix 𝐀 ∈ ℝ^{m×n} that is only accessible implicitly via matrix-vector product queries (matvec queries) of the form 𝐱 → 𝐀𝐱 or 𝐱 → 𝐀^T𝐱. Of particular interest are methods that learn a near-optimal approximation with a fixed sparsity pattern. For example, we might want to learn a near-optimal diagonal, banded, or arrow-head approximation to an implicit matrix 𝐀. 
Naturally, the number of matvec queries required to solve this problem depends on the sparsity pattern, which can be encoded as a binary matrix 𝐒 ∈ {0,1}^{m× n}. The query complexity of previous algorithms scales with quantities like the total number of ones in 𝐒, its maximum column/row sparsity, or the chromatic number of its "conflict graph". These quantities are incomparable: for a given 𝐒, parameterizing by one might yield lower query complexity than another.
In this work, we unify and tighten these prior results by providing a nearly sharp characterization of the matvec query complexity of sparse matrix approximation. Generalizing a definition from graph algorithms, let the degeneracy, degen(𝐒), denote the smallest number k so that, if we iteratively delete all rows and columns of 𝐒 with ≤ k ones, we are left with an empty matrix. We show that a near-optimal approximation to 𝐀 with sparsity pattern 𝐒 can be learned with Õ(degen(𝐒)) matrix-vector product queries, and Ω(degen(𝐒)) queries are necessary, for any sparsity pattern 𝐒. Moreover, unlike prior work based on graph coloring, all of our methods run in polynomial time.

Cite As Get BibTex

Christopher Musco and Indu Ramesh. Nearly Instance Optimal Sparse Matrix Approximation from Matrix-Vector Products. In 34th Annual European Symposium on Algorithms (ESA 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 388, pp. 85:1-85:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESA.2026.85

Author Details

Christopher Musco
  • New York University, NY, USA
Indu Ramesh
  • New York University, NY, USA

Funding

  • Musco, Christopher: partially supported by NSF Award #2427363.

Acknowledgements

We would like to thank Cameron Musco for helpful discussions.

References

  1. Noah Amsel, Pratyush Avi, Tyler Chen, Feyza Duman Keles, Chinmay Hegde, Cameron Musco, Christopher Musco, and David Persson. Query efficient structured matrix learning. In Proceedings of the 39th Annual Conference on Computational Learning Theory (COLT), 2026. Google Scholar
  2. Noah Amsel, Tyler Chen, Feyza Duman Keles, Diana Halikias, Cameron Musco, and Christopher Musco. Fixed-sparsity matrix approximation from matrix-vector products. SIAM Journal on Matrix Analysis and Applications, 2026. Google Scholar
  3. Noah Amsel, Tyler Chen, Feyza Duman Keles, Diana Halikias, Cameron Musco, Christopher Musco, and David Persson. Quasi-optimal hierarchically semi-separable matrix approximation. SIAM Journal on Matrix Analysis and Applications, 2026. Google Scholar
  4. Ainesh Bakshi, Kenneth L. Clarkson, and David P. Woodruff. Low-rank approximation with 1/ε^1/3 matrix-vector products. In Proceedings of the 54th Annual ACM Symposium on Theory of Computing (STOC), 2022. Google Scholar
  5. Robert A. Baston and Yuji Nakatsukasa. Stochastic diagonal estimation: probabilistic bounds and an improved algorithm. http://arxiv.org/abs/2201.10684, 2022. Google Scholar
  6. C. Bekas, E. Kokiopoulou, and Y. Saad. An estimator for the diagonal of a matrix. Applied Numerical Mathematics, 57(11-12):1214-1229, 2007. Google Scholar
  7. Mathieu Blondel and Vincent Roulet. The elements of differentiable programming. http://arxiv.org/abs/2403.14606, 2024. Google Scholar
  8. Nicolas Boullé, Diana Halikias, Samuel E. Otto, and Alex Townsend. Operator learning without the adjoint. Journal of Machine Learning Research, 25(364):1-54, 2024. URL: https://jmlr.org/papers/v25/24-0162.html.
  9. Nicolas Boullé, Diana Halikias, and Alex Townsend. Elliptic PDE learning is provably data-efficient. Proceedings of the National Academy of Sciences, 120(39), 2023. Google Scholar
  10. Nicolas Boullé and Alex Townsend. Learning elliptic partial differential equations with randomized linear algebra. Foundations of Computational Mathematics, 23(2):709-739, 2023. URL: https://doi.org/10.1007/S10208-022-09556-W.
  11. Mark Braverman, Elad Hazan, Max Simchowitz, and Blake Woodworth. The gradient complexity of linear regression. In Proceedings of the 33rd Annual Conference on Computational Learning Theory (COLT), 2020. Google Scholar
  12. Vladimir Braverman, Aditya Krishnan, and Christopher Musco. Sublinear time spectral density estimation. In Proceedings of the 54th Annual ACM Symposium on Theory of Computing (STOC), 2022. Google Scholar
  13. Tyler Chen, Feyza Duman Keles, Diana Halikias, Cameron Musco, Christopher Musco, and David Persson. Near-optimal hierarchical matrix approximation from matrix-vector products. In Proceedings of the 36th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2025. Google Scholar
  14. Tyler Chen, Thomas Trogdon, and Shashanka Ubaru. Analysis of stochastic Lanczos quadrature for spectrum approximation. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. Google Scholar
  15. Sinho Chewi, Jaume de Dios Pont, Jerry Li, Chen Lu, and Shyam Narayanan. Query lower bounds for log-concave sampling. Journal of the ACM, 71(4), 2024. URL: https://doi.org/10.1145/3673651.
  16. Kenneth L. Clarkson and David P. Woodruff. Numerical linear algebra in the streaming model. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC), 2009. Google Scholar
  17. Thomas F. Coleman and Jin-Yi Cai. The cyclic coloring problem and estimation of sparse Hessian matrices. SIAM Journal on Algebraic Discrete Methods, 7(2):221-235, 1986. Google Scholar
  18. Thomas F. Coleman and Jorge J. Moré. Estimation of sparse Jacobian matrices and graph coloring problems. SIAM Journal on Numerical Analysis, 20(1):187-209, 1983. Google Scholar
  19. A. R. Curtis, M. J. D. Powell, and J. K. Reid. On the estimation of sparse Jacobian matrices. IMA Journal of Applied Mathematics, 13(1):117-119, 1974. Google Scholar
  20. Gautam Dasarathy, Parikshit Shah, Badri Narayan Bhaskar, and Robert D. Nowak. Sketching sparse matrices, covariances, and graphs via tensor products. IEEE Transactions on Information Theory, 61(3):1373-1388, 2015. URL: https://doi.org/10.1109/TIT.2015.2391251.
  21. Yann N. Dauphin, Harm de Vries, and Yoshua Bengio. Equilibrated adaptive learning rates for non-convex optimization. In Advances in Neural Information Processing Systems 28 (NeurIPS), 2015. Google Scholar
  22. Maarten V. de Hoop, Nikola B. Kovachki, Nicholas H. Nelsen, and Andrew M. Stuart. Convergence rates for learning linear operators from noisy data. SIAM/ASA Journal on Uncertainty Quantification, 11(2):480-513, 2023. URL: https://doi.org/10.1137/21M1442942.
  23. Michał Dereziński, Ethan N. Epperly, and Raphael A. Meyer. The matrix-vector complexity of Ax = b. http://arxiv.org/abs/2602.04842, 2026. Google Scholar
  24. Prathamesh Dharangutte and Christopher Musco. A tight analysis of hutchinson’s diagonal estimator. In Proceedings of the 6th Symposium on Simplicity in Algorithms (SOSA), 2023. Google Scholar
  25. Alan Edelman. Eigenvalues and condition numbers of random matrices. SIAM Journal on Matrix Analysis and Applications, 9(4):543-560, 1988. Google Scholar
  26. David Eppstein. Coloring, register allocation, Strahler number, and interval graphs. Lecture notes, CS 163/CS 265, University of California, Irvine, 2025. URL: https://ics.uci.edu/~eppstein/163/lecture6c.pdf.
  27. P. Erdős and A. Hajnal. On chromatic number of graphs and set-systems. Acta Mathematica Academiae Scientiarum Hungarica, 17(1):61-99, 1966. Google Scholar
  28. David Eriksson, Kun Dong, Eric Lee, David Bindel, and Andrew G Wilson. Scaling Gaussian process regression with derivatives. In Advances in Neural Information Processing Systems 31 (NeurIPS), volume 31, 2018. Google Scholar
  29. D. Goldfarb and Ph. L. Toint. Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations. Mathematics of Computation, 43(167):69-88, 1984. Google Scholar
  30. Diana Halikias and Alex Townsend. Structured matrix recovery from matrix‐vector products. Numerical Linear Algebra with Applications, 31(1), 2023. URL: https://doi.org/10.1002/NLA.2531.
  31. N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-288, 2011. URL: https://doi.org/10.1137/090771806.
  32. Arun Jambulapati, Jerry Li, Christopher Musco, Aaron Sidford, and Kevin Tian. Structured semidefinite programming for recovering structured preconditioners. In Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. Google Scholar
  33. Lefteris M. Kirousis and Dimitris M. Thilikos. The linkage of a graph. SIAM J. Comput., 25(3):626-647, 1996. URL: https://doi.org/10.1137/S0097539793255709.
  34. James Levitt and Per-Gunnar Martinsson. Linear-complexity black-box randomized compression of rank-structured matrices. SIAM Journal on Scientific Computing, 46(3):A1747-A1763, 2024. Google Scholar
  35. James Levitt and Per-Gunnar Martinsson. Randomized compression of rank-structured matrices accelerated with graph coloring. Journal of Computational and Applied Mathematics, 451, 2024. URL: https://doi.org/10.1016/J.CAM.2024.116044.
  36. D. R. Lick and A. T. White. k-degenerate graphs. Canadian Journal of Mathematics, 22:1082-1096, 1970. Google Scholar
  37. Lin Lin, Jianfeng Lu, and Lexing Ying. Fast construction of hierarchical matrix representation from matrix-vector multiplication. Journal of Computational Physics, 230(10):4071-4087, 2011. URL: https://doi.org/10.1016/J.JCP.2011.02.033.
  38. Lin Lin, Jianfeng Lu, Lexing Ying, Roberto Car, and Weinan E. Fast algorithm for extracting the diagonal of the inverse matrix with application to the electronic structure analysis of metallic systems. Communications in Mathematical Sciences, 7(3):755-777, 2009. Google Scholar
  39. Yang Liu, Xin Xing, Han Guo, Eric Michielssen, Pieter Ghysels, and Xiaoye Sherry Li. Butterfly factorization via randomized matrix-vector multiplications. SIAM Journal on Scientific Computing, 43(2), 2021. URL: https://doi.org/10.1137/20M1315853.
  40. David W. Matula and Leland L. Beck. Smallest-last ordering and clustering and graph coloring algorithms. J. ACM, 30(3):417-427, 1983. URL: https://doi.org/10.1145/2402.322385.
  41. S. Thomas Mccormick. Optimal approximation of sparse Hessians and its equivalence to a graph coloring problem. Math. Program., 26(2):153-171, 1983. URL: https://doi.org/10.1007/BF02592052.
  42. L. Métivier, F. Bretaudeau, R. Brossier, S. Operto, and J. Virieux. Full waveform inversion and the truncated newton method: quantitative imaging of complex subsurface structures. Geophysical Prospecting, 62(6):1353-1375, 2014. Google Scholar
  43. Raphael A. Meyer, Cameron Musco, Christopher Musco, and David Woodruff. Hutch++: optimal stochastic trace estimation. In Proceedings of the 4th Symposium on Simplicity in Algorithms (SOSA), 2021. Google Scholar
  44. Raphael A. Meyer, William Swartworth, and David Woodruff. Understanding the Kronecker matrix-vector complexity of linear algebra. In Proceedings of the 42ndInternational Conference on Machine Learning (ICML), 2025. Google Scholar
  45. Cameron Musco and Christopher Musco. Randomized block Krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems 28 (NeurIPS), 2015. Google Scholar
  46. Cameron Musco, Christopher Musco, Lucas Rosenblatt, and Apoorv Vikram Singh. Sharper bounds for chebyshev moment matching, with applications. In Proceedings of the 38th Annual Conference on Computational Learning Theory (COLT), 2025. Google Scholar
  47. Katherine J. Pearce, Anna Yesypenko, James Levitt, and Per-Gunnar Martinsson. Randomized rank-structured matrix compression by tagging. http://arxiv.org/abs/2501.05528, 2025. Google Scholar
  48. Barak A. Pearlmutter. Fast exact multiplication by the Hessian. Neural computation, 6(1):147-160, 1994. URL: https://doi.org/10.1162/NECO.1994.6.1.147.
  49. M. J. D. Powell and Ph. L. Toint. On the estimation of sparse hessian matrices. SIAM Journal on Numerical Analysis, 16(6):1060-1074, 1979. Google Scholar
  50. Florian Schäfer, Matthias Katzfuss, and Houman Owhadi. Sparse cholesky factorization by kullback-leibler minimization. SIAM Journal on Scientific Computing, 43(3):A2019-A2046, 2021. URL: https://doi.org/10.1137/20M1336254.
  51. Florian Schäfer and Houman Owhadi. Sparse recovery of elliptic solvers from matrix-vector products. SIAM Journal on Scientific Computing, 46(2), 2024. URL: https://doi.org/10.1137/22M154226X.
  52. Max Simchowitz, Ahmed El Alaoui, and Benjamin Recht. Tight query complexity lower bounds for PCA via finite sample deformed Wigner law. In Proceedings of the 50th Annual ACM Symposium on Theory of Computing (STOC), 2018. Google Scholar
  53. Xiaoming Sun, David P. Woodruff, Guang Yang, and Jialin Zhang. Querying a matrix through matrix-vector products. In Proceedings of the 46th International Colloquium on Automata, Languages and Programming (ICALP), 2019. Google Scholar
  54. Jok M. Tang and Yousef Saad. A probing method for computing the diagonal of a matrix inverse. Numerical Linear Algebra with Applications, 19(3):485-501, 2011. URL: https://doi.org/10.1002/NLA.779.
  55. Umberto Villa, Noemi Petra, and Omar Ghattas. hIPPYlib: an extensible software framework for large-scale inverse problems governed by PDEs: Part i: Deterministic inversion and linearized Bayesian inference. ACM Trans. Math. Softw., 47(2), 2021. URL: https://doi.org/10.1145/3428447.
  56. David Woodruff, Fred Zhang, and Richard Zhang. Optimal query complexities for dynamic trace estimation. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. Google Scholar
  57. Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W Mahoney. ADAHESSIAN: An adaptive second order optimizer for machine learning. In Proceedings of the AAAI Conference on Artificial (AAAI)2021, 2021. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact