Document Open Access Logo

Optimal Oblivious Subspace Embeddings with Near-Optimal Sparsity

Authors Shabarish Chenakkod , Michał Dereziński , Xiaoyu Dong

PDF

File

Thumbnail PDF
PDF
LIPIcs.ICALP.2025.55.pdf
  • Filesize: 0.94 MB
  • 20 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Sketching and sampling
  • Theory of computation → Random projections and metric embeddings
Keywords
  • Randomized linear algebra
  • matrix sketching
  • subspace embeddings

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

An oblivious subspace embedding is a random m× n matrix Π such that, for any d-dimensional subspace, with high probability Π preserves the norms of all vectors in that subspace within a 1±ε factor. In this work, we give an oblivious subspace embedding with the optimal dimension m = Θ(d/ε²) that has a near-optimal sparsity of Õ(1/ε) non-zero entries per column of Π. This is the first result to nearly match the conjecture of Nelson and Nguyen [FOCS 2013] in terms of the best sparsity attainable by an optimal oblivious subspace embedding, improving on a prior bound of Õ(1/ε⁶) non-zeros per column [Chenakkod et al., STOC 2024]. We further extend our approach to the non-oblivious setting, proposing a new family of Leverage Score Sparsified embeddings with Independent Columns, which yield faster runtimes for matrix approximation and regression tasks.
In our analysis, we develop a new method which uses a decoupling argument together with the cumulant method for bounding the edge universality error of isotropic random matrices. To achieve near-optimal sparsity, we combine this general-purpose approach with new trace inequalities that leverage the specific structure of our subspace embedding construction.

Cite As Get BibTex

Shabarish Chenakkod, Michał Dereziński, and Xiaoyu Dong. Optimal Oblivious Subspace Embeddings with Near-Optimal Sparsity. In 52nd International Colloquium on Automata, Languages, and Programming (ICALP 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 334, pp. 55:1-55:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ICALP.2025.55

Author Details

Shabarish Chenakkod
  • University of Michigan, Ann Arbor, MI, USA
Michał Dereziński
  • University of Michigan, Ann Arbor, MI, USA
Xiaoyu Dong
  • National University of Singapore, Singapore

Funding

  • Chenakkod, Shabarish: NSF grant DMS 20544.
  • Dereziński, Michał: NSF grant CCF 2338655.

Acknowledgements

The authors are grateful for the generous help and support of Mark Rudelson throughout the duration of this work.

References

  1. Dimitris Achlioptas. Database-friendly random projections: Johnson-lindenstrauss with binary coins. Journal of computer and System Sciences, 66(4):671-687, 2003. URL: https://doi.org/10.1016/S0022-0000(03)00025-4.
  2. Nir Ailon and Bernard Chazelle. The fast johnson-lindenstrauss transform and approximate nearest neighbors. SIAM Journal on computing, 39(1):302-322, 2009. URL: https://doi.org/10.1137/060673096.
  3. Jean Bourgain, Sjoerd Dirksen, and Jelani Nelson. Toward a unified theory of sparse dimensionality reduction in euclidean space. In Proceedings of the forty-seventh annual ACM symposium on Theory of Computing, pages 499-508, 2015. URL: https://doi.org/10.1145/2746539.2746541.
  4. Tatiana Brailovskaya and Ramon van Handel. Universality and sharp matrix concentration inequalities. Geometric and Functional Analysis, pages 1-105, 2024. Google Scholar
  5. Coralia Cartis, Jan Fiala, and Zhen Shao. Hashing embeddings of optimal dimension, with applications to linear least squares. arXiv preprint arXiv:2105.11815, 2021. URL: https://arxiv.org/abs/2105.11815.
  6. Shabarish Chenakkod, Michał Dereziński, Xiaoyu Dong, and Mark Rudelson. Optimal embedding dimension for sparse subspace embeddings. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 1106-1117, 2024. Google Scholar
  7. Shabarish Chenakkod, Michał Dereziński, and Xiaoyu Dong. Optimal oblivious subspace embeddings with near-optimal sparsity, 2024. URL: https://doi.org/10.48550/arXiv.2411.08773.
  8. Nadiia Chepurko, Kenneth L Clarkson, Praneeth Kacham, and David P Woodruff. Near-optimal algorithms for linear algebra in the current matrix multiplication time. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 3043-3068. SIAM, 2022. URL: https://doi.org/10.1137/1.9781611977073.118.
  9. Yeshwanth Cherapanamjeri, Sandeep Silwal, David P Woodruff, and Samson Zhou. Optimal algorithms for linear algebra in the current matrix multiplication time. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 4026-4049. SIAM, 2023. URL: https://doi.org/10.1137/1.9781611977554.CH154.
  10. Kenneth L Clarkson and David P Woodruff. Low rank approximation and regression in input sparsity time. In Proceedings of the forty-fifth annual ACM symposium on Theory of Computing, pages 81-90, 2013. URL: https://doi.org/10.1145/2488608.2488620.
  11. Michael B Cohen. Nearly tight oblivious subspace embeddings by trace inequalities. In Proc. of the 27th annual ACM-SIAM Symposium on Discrete Algorithms, pages 278-287. SIAM, 2016. URL: https://doi.org/10.1137/1.9781611974331.CH21.
  12. Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 163-172, 2015. URL: https://doi.org/10.1145/2746539.2746569.
  13. Michael B Cohen, Jelani Nelson, and David P Woodruff. Optimal approximate matrix product in terms of stable rank. In International Colloquium on Automata, Languages, and Programming. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Dagstuhl Publishing, 2016. Google Scholar
  14. Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. A sparse johnson: Lindenstrauss transform. In Proceedings of the forty-second ACM symposium on Theory of computing, pages 341-350, 2010. URL: https://doi.org/10.1145/1806689.1806737.
  15. Michał Dereziński. Algorithmic gaussianization through sketching: Converting data into sub-gaussian random designs. In The Thirty Sixth Annual Conference on Learning Theory, pages 3137-3172. PMLR, 2023. Google Scholar
  16. Michał Dereziński, Jonathan Lacotte, Mert Pilanci, and Michael W Mahoney. Newton-less: Sparsification without trade-offs for the sketched newton update. Advances in Neural Information Processing Systems, 34:2835-2847, 2021. Google Scholar
  17. Michał Dereziński, Zhenyu Liao, Edgar Dobriban, and Michael Mahoney. Sparse sketches with small inversion bias. In Conference on Learning Theory, pages 1467-1510. PMLR, 2021. Google Scholar
  18. Michał Dereziński and Michael W Mahoney. Recent and upcoming developments in randomized numerical linear algebra for machine learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6470-6479, 2024. Google Scholar
  19. Petros Drineas, Malik Magdon-Ismail, Michael W Mahoney, and David P Woodruff. Fast approximation of matrix coherence and statistical leverage. The Journal of Machine Learning Research, 13(1):3475-3506, 2012. URL: https://doi.org/10.5555/2503308.2503352.
  20. Petros Drineas and Michael W Mahoney. Randnla: randomized numerical linear algebra. Communications of the ACM, 59(6):80-90, 2016. URL: https://doi.org/10.1145/2842602.
  21. Petros Drineas, Michael W Mahoney, and S Muthukrishnan. Sampling algorithms for 𝓁₂ regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm, pages 1127-1136, 2006. Google Scholar
  22. Daniel M Kane and Jelani Nelson. Sparser johnson-lindenstrauss transforms. Journal of the ACM (JACM), 61(1):1-23, 2014. URL: https://doi.org/10.1145/2559902.
  23. Yi Li and Mingmou Liu. Lower bounds for sparse oblivious subspace embeddings. In Proceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pages 251-260, 2022. URL: https://doi.org/10.1145/3517804.3526224.
  24. Per-Gunnar Martinsson and Joel A Tropp. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403-572, 2020. URL: https://doi.org/10.1017/S0962492920000021.
  25. Xiangrui Meng and Michael W Mahoney. Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 91-100, 2013. URL: https://doi.org/10.1145/2488608.2488621.
  26. Jelani Nelson and Huy L Nguyên. Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 2013 ieee 54th annual symposium on foundations of computer science, pages 117-126. IEEE, 2013. Google Scholar
  27. Jelani Nelson and Huy L Nguyên. Lower bounds for oblivious subspace embeddings. In International Colloquium on Automata, Languages, and Programming, pages 883-894. Springer, 2014. URL: https://doi.org/10.1007/978-3-662-43948-7_73.
  28. Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In 2006 47th annual IEEE symposium on foundations of computer science (FOCS'06), pages 143-152. IEEE, 2006. Google Scholar
  29. Joel A Tropp. Improved analysis of the subsampled randomized hadamard transform. Advances in Adaptive Data Analysis, 3(01n02):115-126, 2011. URL: https://doi.org/10.1142/S1793536911000787.
  30. Joel A. Tropp. An introduction to matrix concentration inequalities. Found. Trends Mach. Learn., 8(1–2):1-230, May 2015. URL: https://doi.org/10.1561/2200000048.
  31. Joel A Tropp. Comparison theorems for the minimum eigenvalue of a random positive-semidefinite matrix. arXiv preprint arXiv:2501.16578, 2025. URL: https://doi.org/10.48550/arXiv.2501.16578.
  32. David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, 2014. URL: https://doi.org/10.1561/0400000060.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact