Document Open Access Logo

Acceleration Meets Inverse Maintenance: Faster ๐“_โˆž-Regression

Authors Deeksha Adil , Shunhua Jiang , Rasmus Kyng

PDF

File

Thumbnail PDF
PDF
LIPIcs.ICALP.2025.5.pdf
  • Filesize: 1.44 MB
  • 16 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation โ†’ Continuous optimization
  • Theory of computation โ†’ Data structures design and analysis
  • Theory of computation โ†’ Sketching and sampling
Keywords
  • Regression
  • Inverse Maintenance
  • Multiplicative Weights Update

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

We propose a randomized multiplicative weight update (MWU) algorithm for ๐“_{โˆž} regression that runs in Oฬƒ(n^{2+1/22.5} poly(1/ฮต)) time when ฯ‰ = 2+o(1), improving upon the previous best Oฬƒ(n^{2+1/18} polylog(1/ฮต)) runtime in the low-accuracy regime. Our algorithm combines state-of-the-art inverse maintenance data structures with acceleration. In order to do so, we propose a novel acceleration scheme for MWU that exhibits stability and robustness, which are required for the efficient implementations of the inverse maintenance data structures.
We also design a faster deterministic MWU algorithm that runs in Oฬƒ(n^{2+1/12}poly(1/ฮต)) time when ฯ‰ = 2+o(1), improving upon the previous best Oฬƒ(n^{2+1/6} poly log(1/ฮต)) runtime in the low-accuracy regime. We achieve this by showing a novel stability result that goes beyond previously known works based on interior point methods (IPMs).
Our work is the first to use acceleration and inverse maintenance together efficiently, finally making the two most important building blocks of modern structured convex optimization compatible.

Cite As Get BibTex

Deeksha Adil, Shunhua Jiang, and Rasmus Kyng. Acceleration Meets Inverse Maintenance: Faster ๐“_โˆž-Regression. In 52nd International Colloquium on Automata, Languages, and Programming (ICALP 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 334, pp. 5:1-5:16, Schloss Dagstuhl โ€“ Leibniz-Zentrum fรผr Informatik (2025) https://doi.org/10.4230/LIPIcs.ICALP.2025.5

Author Details

Deeksha Adil
  • ETH Zรผrich, Switzerland
Shunhua Jiang
  • Columbia University, New York, NY, USA
Rasmus Kyng
  • ETH Zรผrich, Switzerland

Funding

  • Adil, Deeksha: Supported by Dr. Max Rรถssler, the Walter Haefner Foundation and the ETH Zรผrich Foundation.
  • Jiang, Shunhua: Supported by NSF grants (CCF-1617955, CCF-1740833, CCF-2008733), NSF CAREER award CCF-1844887, and by Google PhD fellowship.
  • Kyng, Rasmus: The research leading to these results has received funding from grant no. 200021 204787 and from the starting grant "A New Paradigm for Flow and Cut Algorithms" (no. TMSGI2_218022) of the Swiss National Science Foundation.

References

  1. Deeksha Adil, Brian Bullins, and Sushant Sachdeva. Unifying width-reduced methods for quasi-self-concordant optimization. Advances in Neural Information Processing Systems, 34:19122-19133, 2021. URL: https://proceedings.neurips.cc/paper/2021/hash/9f9e8cba3700df6a947a8cf91035ab84-Abstract.html.
  2. Deeksha Adil, Rasmus Kyng, Richard Peng, and Sushant Sachdeva. Iterative refinement for ๐“_p-norm regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1405-1424. SIAM, 2019. Google Scholar
  3. Deeksha Adil, Rasmus Kyng, Richard Peng, and Sushant Sachdeva. Fast algorithms for ๐“_p-regression. Journal of the ACM, 71(5):1-45, 2024. Google Scholar
  4. Sanjeev Arora, Elad Hazan, and Satyen Kale. The Multiplicative Weights Update Method: A Meta-Algorithm and Applications. Theory of Computing, 8(6):121-164, May 2012. URL: https://doi.org/10.4086/toc.2012.v008a006.
  5. Jan Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 259-278. SIAM, 2020. URL: https://doi.org/10.1137/1.9781611975994.16.
  6. Jan van den Brand, Danupon Nanongkai, and Thatchaphol Saranurak. Dynamic matrix inverse: Improved algorithms and matching conditional lower bounds. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS), pages 456-480. IEEE, 2019. URL: https://doi.org/10.1109/FOCS.2019.00036.
  7. G. W. Brown and J. Von Neumann. 6. SOLUTIONS OF GAMES BY DIFFERENTIAL EQUATIONS. In Harold William Kuhn and Albert William Tucker, editors, Contributions to the Theory of Games (AM-24), Volume I, pages 73-80. Princeton University Press, December 1951. URL: https://doi.org/10.1515/9781400881727-007.
  8. Brian Bullins. Fast minimization of structured convex quartics. arXiv preprint arXiv:1812.10349, 2018. URL: https://arxiv.org/abs/1812.10349.
  9. Yair Carmon, Arun Jambulapati, Qijia Jiang, Yujia Jin, Yin Tat Lee, Aaron Sidford, and Kevin Tian. Acceleration with a ball optimization oracle. Advances in Neural Information Processing Systems, 33:19052-19063, 2020. Google Scholar
  10. Hui Han Chin, Aleksander Madry, Gary L Miller, and Richard Peng. Runtime guarantees for regression problems. In Proceedings of the 4th conference on Innovations in Theoretical Computer Science, pages 269-282, 2013. URL: https://doi.org/10.1145/2422436.2422469.
  11. Paul Christiano, Jonathan A Kelner, Aleksander Madry, Daniel A Spielman, and Shang-Hua Teng. Electrical flows, laplacian systems, and faster approximation of maximum flow in undirected graphs. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 273-282, 2011. URL: https://doi.org/10.1145/1993636.1993674.
  12. Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM), 68(1):1-39, 2021. URL: https://doi.org/10.1145/3424305.
  13. Alina Ene and Adrian Vladu. Improved convergence for ๐“_infty and ๐“โ‚ regression via iteratively reweighted least squares. Proceedings of Machine Learning Research, 97, 2019. Google Scholar
  14. Mehrdad Ghadiri, Richard Peng, and Santosh S Vempala. The bit complexity of efficient continuous optimization. In 2023 IEEE 64th Annual Symposium on Foundations of Computer Science (FOCS), pages 2059-2070. IEEE, 2023. URL: https://doi.org/10.1109/FOCS57990.2023.00125.
  15. M. R. Hestenes and E. Stiefel. On the convergence of the conjugate gradient method for singular liner operator equations. J. Research Nat. Bur. Standards, 49:409-436, 1952. Google Scholar
  16. Shunhua Jiang, Zhao Song, Omri Weinstein, and Hengjie Zhang. A faster algorithm for solving general lps. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 823-832, 2021. URL: https://doi.org/10.1145/3406325.3451058.
  17. Narendra Karmarkar. A new polynomial-time algorithm for linear programming. In Proceedings of the Sixteenth Annual ACM Symposium on Theory of Computing, pages 302-311, 1984. URL: https://doi.org/10.1145/800057.808695.
  18. Jonathan A Kelner, Yin Tat Lee, Lorenzo Orecchia, and Aaron Sidford. An almost-linear-time algorithm for approximate max flow in undirected graphs, and its multicommodity generalizations. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms, pages 217-226. SIAM, 2014. URL: https://doi.org/10.1137/1.9781611973402.16.
  19. Cornelius Lanczos. Solution of systems of linear equations by minimized iterations. J. Res. Nat. Bur. Standards, 49(1):33-53, 1952. Google Scholar
  20. Yin Tat Lee and Aaron Sidford. Efficient inverse maintenance and faster algorithms for linear programming. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 230-249. IEEE, 2015. URL: https://doi.org/10.1109/FOCS.2015.23.
  21. Yin Tat Lee, Zhao Song, and Qiuyi Zhang. Solving empirical risk minimization in the current matrix multiplication time. In Conference on Learning Theory, pages 2140-2157. PMLR, 2019. URL: http://proceedings.mlr.press/v99/lee19a.html.
  22. Yin Tat Lee and Santosh S Vempala. Tutorial on the robust interior point method. arXiv preprint arXiv:2108.04734, 2021. URL: https://arxiv.org/abs/2108.04734.
  23. Yang P Liu and Aaron Sidford. Faster energy maximization for faster maximum flow. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 803-814, 2020. URL: https://doi.org/10.1145/3357713.3384247.
  24. Aleksander Madry. Navigating central path with electrical flows: From flows to matchings, and back. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 253-262. IEEE, 2013. URL: https://doi.org/10.1109/FOCS.2013.35.
  25. Aleksander Madry. Computing maximum flow with augmenting electrical flows. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pages 593-602. IEEE, 2016. URL: https://doi.org/10.1109/FOCS.2016.70.
  26. Renato D. C. Monteiro and B. F. Svaiter. An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods. SIAM Journal on Optimization, 23(2):1092-1125, January 2013. URL: https://doi.org/10.1137/110833786.
  27. Y Nesterov. A method for solving the convex programming problem with convergence rate o(1/k2). Dokl Akad Nauk SSSR, 269:543, 1983. Google Scholar
  28. Yu E. Nesterov and Arkadii Nemirovskii. Self-concordant functions and polynomial-time methods in convex programming. Report, Central Economic and Mathematic Institute, USSR Acad. Sci, 1989. Google Scholar
  29. Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM, 1994. URL: https://doi.org/10.1137/1.9781611970791.
  30. James Renegar. A polynomial-time algorithm, based on Newtonโ€™s method, for linear programming. Mathematical programming, 40(1):59-93, 1988. URL: https://doi.org/10.1007/BF01580724.
  31. Jonah Sherman. Nearly maximum flows in nearly linear time. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 263-269. IEEE, 2013. URL: https://doi.org/10.1109/FOCS.2013.36.
  32. Aaron Sidford and Kevin Tian. Coordinate methods for accelerating ๐“_โˆž regression and faster approximate maximum flow. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 922-933. IEEE, 2018. URL: https://doi.org/10.1109/FOCS.2018.00091.
  33. Pravin M. Vaidya. Speeding-up linear programming using fast matrix multiplication. In 30th Annual Symposium on Foundations of Computer Science, pages 332-337. IEEE Computer Society, 1989. Google Scholar
  34. David Woodruff and Qin Zhang. Subspace embeddings and$$1ell_p-regression using exponential random variables. In Conference on Learning Theory, pages 546-567. PMLR, 2013. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl โ€“ LZI GmbH About DROPS Imprint Privacy Contact