Document Open Access Logo

Guessing Efficiently for Constrained Subspace Approximation

Authors Aditya Bhaskara , Sepideh Mahabadi , Madhusudhan Reddy Pittu , Ali Vakilian , David P. Woodruff

PDF

File

Thumbnail PDF
PDF
LIPIcs.ICALP.2025.29.pdf
  • Filesize: 0.85 MB
  • 20 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Continuous optimization
Keywords
  • parameterized complexity
  • low rank approximation
  • fairness
  • non-negative matrix factorization
  • clustering

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

In this paper we study constrained subspace approximation problem. Given a set of n points {a₁,…,a_n} in ℝ^d, the goal of the subspace approximation problem is to find a k dimensional subspace that best approximates the input points. More precisely, for a given p ≥ 1, we aim to minimize the pth power of the 𝓁_p norm of the error vector (‖a₁-Pa₁‖,…,‖a_n-Pa_n‖), where P denotes the projection matrix onto the subspace and the norms are Euclidean. In constrained subspace approximation (CSA), we additionally have constraints on the projection matrix P. In its most general form, we require P to belong to a given subset 𝒮 that is described explicitly or implicitly.
We introduce a general framework for constrained subspace approximation. Our approach, that we term coreset-guess-solve, yields either (1+ε)-multiplicative or ε-additive approximations for a variety of constraints. We show that it provides new algorithms for partition-constrained subspace approximation with applications to fair subspace approximation, k-means clustering, and projected non-negative matrix factorization, among others. Specifically, while we reconstruct the best known bounds for k-means clustering in Euclidean spaces, we improve the known results for the remainder of the problems.

Cite As Get BibTex

Aditya Bhaskara, Sepideh Mahabadi, Madhusudhan Reddy Pittu, Ali Vakilian, and David P. Woodruff. Guessing Efficiently for Constrained Subspace Approximation. In 52nd International Colloquium on Automata, Languages, and Programming (ICALP 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 334, pp. 29:1-29:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ICALP.2025.29

Author Details

Aditya Bhaskara
  • University of Utah, Salt Lake City, UT, USA
Sepideh Mahabadi
  • Microsoft Research, Redmond, WA, US
Madhusudhan Reddy Pittu
  • Carnegie Mellon University, Pittsburgh, PA, USA
Ali Vakilian
  • Toyota Technological Institute at Chicago, IL, USA
David P. Woodruff
  • Carnegie Mellon University, Pittsburgh, PA, USA

Funding

  • Bhaskara, Aditya: Supported by NSF CCF-2047288.
  • Woodruff, David P.: Supported by a Simons Investigator Award and Office of Naval Research award number N000142112647.

References

  1. Jason Altschuler, Aditya Bhaskara, Gang Fu, Vahab Mirrokni, Afshin Rostamizadeh, and Morteza Zadimoghaddam. Greedy column subset selection: New bounds and distributed algorithms. In International Conference on Machine Learning, pages 2539-2548, 2016. URL: http://proceedings.mlr.press/v48/altschuler16.html.
  2. Megasthenis Asteris, Dimitris Papailiopoulos, and Alexandros Dimakis. Nonnegative sparse pca with provable guarantees. In International Conference on Machine Learning, pages 1728-1736. PMLR, 2014. URL: http://proceedings.mlr.press/v32/asteris14.html.
  3. Frank Ban, Vijay Bhattiprolu, Karl Bringmann, Pavel Kolev, Euiwoong Lee, and David P. Woodruff. A PTAS for 𝓁_p-low rank approximation. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 747-766, 2019. Google Scholar
  4. Frank Ban, David P. Woodruff, and Qiuyi (Richard) Zhang. Regularized weighted low rank approximation. CoRR, abs/1911.06958, 2019. URL: https://arxiv.org/abs/1911.06958.
  5. Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Sparse features for pca-like linear regression. Advances in Neural Information Processing Systems, 24, 2011. Google Scholar
  6. Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near-optimal column-based matrix reconstruction. SIAM Journal on Computing, 43(2):687-717, 2014. URL: https://doi.org/10.1137/12086755X.
  7. Christos Boutsidis, Michael W Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms, pages 968-977, 2009. URL: https://doi.org/10.1137/1.9781611973068.105.
  8. Christos Boutsidis, Anastasios Zouzias, Michael W Mahoney, and Petros Drineas. Randomized dimensionality reduction for k-means clustering. IEEE Transactions on Information Theory, 61(2):1045-1062, 2014. URL: https://doi.org/10.1109/TIT.2014.2375327.
  9. Jorge Cadima and Ian T Jolliffe. Loading and correlations in the interpretation of principle compenents. Journal of applied Statistics, 22(2):203-214, 1995. Google Scholar
  10. Ashish Chiplunkar, Sagar Kale, and Sivaramakrishnan Natarajan Ramamoorthy. How to solve fair k-center in massive data models. In International Conference on Machine Learning, pages 1877-1886, 2020. Google Scholar
  11. Ali Civril and Malik Magdon-Ismail. Column subset selection via sparse approximation of SVD. Theoretical Computer Science, 421:1-14, 2012. URL: https://doi.org/10.1016/J.TCS.2011.11.019.
  12. Kenneth L Clarkson and David P Woodruff. Numerical linear algebra in the streaming model. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pages 205-214, 2009. URL: https://doi.org/10.1145/1536414.1536445.
  13. Kenneth L. Clarkson and David P. Woodruff. Input sparsity and hardness for robust subspace approximation. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 310-329, 2015. URL: https://doi.org/10.1109/FOCS.2015.27.
  14. Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 163-172, 2015. URL: https://doi.org/10.1145/2746539.2746569.
  15. Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, Chris Schwiegelshohn, and Omar Ali Sheikh-Omar. Improved coresets for euclidean k-means. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2024. Google Scholar
  16. Vincent Cohen-Addad, David Saulpic, and Chris Schwiegelshohn. A new coreset framework for clustering. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 169-182, 2021. URL: https://doi.org/10.1145/3406325.3451022.
  17. Alberto Del Pia. Sparse pca on fixed-rank matrices. Mathematical Programming, 198(1):139-157, 2023. URL: https://doi.org/10.1007/S10107-022-01769-9.
  18. Amit Deshpande and Luis Rademacher. Efficient volume sampling for row/column subset selection. In 2010 ieee 51st annual symposium on foundations of computer science, pages 329-338, 2010. URL: https://doi.org/10.1109/FOCS.2010.38.
  19. Amit Deshpande, Madhur Tulsiani, and Nisheeth K. Vishnoi. Algorithms and hardness for subspace approximation. In Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, pages 482-496, USA, 2011. URL: https://doi.org/10.1137/1.9781611973082.39.
  20. Petros Drineas, Alan Frieze, Ravi Kannan, Santosh Vempala, and Vishwanathan Vinay. Clustering large graphs via the singular value decomposition. Machine learning, 56:9-33, 2004. URL: https://doi.org/10.1023/B:MACH.0000033113.59016.96.
  21. Dan Feldman, Morteza Monemizadeh, and Christian Sohler. A ptas for k-means clustering based on weak coresets. In Proceedings of the twenty-third annual symposium on Computational geometry, pages 11-18, 2007. URL: https://doi.org/10.1145/1247069.1247072.
  22. Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, 4th edition, 2013. Google Scholar
  23. Venkatesan Guruswami, Prasad Raghavendra, Rishi Saket, and Yi Wu. Bypassing ugc from some optimal geometric inapproximability results. ACM Trans. Algorithms, 12(1), February 2016. URL: https://doi.org/10.1145/2737729.
  24. Venkatesan Guruswami and Ali Kemal Sinop. Optimal column-based low-rank matrix reconstruction. In Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms, pages 1207-1214, 2012. URL: https://doi.org/10.1137/1.9781611973099.95.
  25. Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity. Monographs on statistics and applied probability, 143(143):8, 2015. Google Scholar
  26. Sedjro Salomon Hotegni, Sepideh Mahabadi, and Ali Vakilian. Approximation algorithms for fair range clustering. In International Conference on Machine Learning, pages 13270-13284. PMLR, 2023. Google Scholar
  27. Lingxiao Huang, Jian Li, and Xuan Wu. On optimal coreset construction for euclidean (k,z)-clustering. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 1594-1604, 2024. URL: https://doi.org/10.1145/3618260.3649707.
  28. Matthew Jones, Huy Nguyen, and Thy Nguyen. Fair k-centers via maximum matching. In International Conference on Machine Learning, pages 4940-4949, 2020. Google Scholar
  29. Matthäus Kleindessner, Pranjal Awasthi, and Jamie Morgenstern. Fair k-center clustering for data summarization. In International Conference on Machine Learning, pages 3448-3457, 2019. URL: http://proceedings.mlr.press/v97/kleindessner19a.html.
  30. Arvind V. Mahankali and David P. Woodruff. Optimal 𝓁₁ column subset selection and a fast PTAS for low rank approximation. CoRR, abs/2007.10307, 2020. Google Scholar
  31. Antonis Matakos, Bruno Ordozgoiti, and Suhas Thejaswi. Fair column subset selection. arXiv preprint arXiv:2306.04489, 2023. URL: https://doi.org/10.48550/arXiv.2306.04489.
  32. Ankur Moitra. An almost optimal algorithm for computing nonnegative rank. SIAM J. Comput., 45(1):156-173, 2016. URL: https://doi.org/10.1137/140990139.
  33. Dimitris Papailiopoulos, Alexandros Dimakis, and Stavros Korokythakis. Sparse pca through low-rank approximations. In International Conference on Machine Learning, pages 747-755. PMLR, 2013. URL: http://proceedings.mlr.press/v28/papailiopoulos13.html.
  34. Ilya Razenshteyn, Zhao Song, and David P. Woodruff. Weighted low rank approximations with provable guarantees. In Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing, pages 250-263, 2016. URL: https://doi.org/10.1145/2897518.2897639.
  35. Samira Samadi, Uthaipon Tantipongpipat, Jamie H Morgenstern, Mohit Singh, and Santosh Vempala. The price of fair PCA: One extra dimension. In Advances in neural information processing systems, pages 10976-10987, 2018. Google Scholar
  36. Ignacio Santamaria, Javier Vía, Michael Kirby, Tim Marrinan, Chris Peterson, and Louis Scharf. Constrained subspace estimation via convex optimization. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 1200-1204. IEEE, 2017. URL: https://doi.org/10.23919/EUSIPCO.2017.8081398.
  37. Zhao Song, Ali Vakilian, David Woodruff, and Samson Zhou. On socially fair regression and low-rank approximation. In Advances in Neural Information Processing Systems, 2024. Google Scholar
  38. Uthaipon Tantipongpipat, Samira Samadi, Mohit Singh, Jamie H Morgenstern, and Santosh Vempala. Multi-criteria dimensionality reduction with applications to fairness. In Advances in Neural Information Processing Systems, pages 15135-15145, 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/2201611d7a08ffda97e3e8c6b667a1bc-Abstract.html.
  39. Joel A Tropp. Column subset selection, matrix factorization, and eigenvalue optimization. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms, pages 978-986. SIAM, 2009. URL: https://doi.org/10.1137/1.9781611973068.106.
  40. Ameya Velingker, Maximilian Vötsch, David P. Woodruff, and Samson Zhou. Fast (1+ε)-approximation algorithms for binary matrix factorization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023. Google Scholar
  41. David P. Woodruff and Taisuke Yasuda. Nearly linear sparsification of 𝓁_p subspace approximation, 2024. URL: https://doi.org/10.48550/arXiv.2407.03262.
  42. David P. Woodruff and Taisuke Yasuda. Ridge leverage score sampling for 𝓁_p subspace approximation, 2025. URL: https://arxiv.org/abs/2407.03262.
  43. Zhirong Yang and Erkki Oja. Linear and nonlinear projective nonnegative matrix factorization. IEEE Transactions on Neural Networks, 21:734-749, 2010. URL: https://doi.org/10.1109/TNN.2010.2041361.
  44. Xiao-Tong Yuan and Tong Zhang. Truncated power method for sparse eigenvalue problems. Journal of Machine Learning Research, 14(4), 2013. URL: https://doi.org/10.5555/2567709.2502610.
  45. Zhijian Yuan and Erkki Oja. Projective nonnegative matrix factorization for image compression and feature extraction. In Image Analysis: 14th Scandinavian Conference, SCIA 2005, Joensuu, Finland, June 19-22, 2005. Proceedings 14, pages 333-342. Springer, 2005. URL: https://doi.org/10.1007/11499145_35.
  46. Zhijian Yuan, Zhirong Yang, and Erkki Oja. Projective nonnegative matrix factorization: Sparseness, orthogonality, and clustering, 2009. Google Scholar
  47. Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. Journal of computational and graphical statistics, 15(2):265-286, 2006. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact