Document Open Access Logo

Extracting Dual Solutions via Primal Optimizers

Authors Yair Carmon , Arun Jambulapati, Liam O'Carroll, Aaron Sidford

PDF
Thumbnail PDF

File

LIPIcs.ITCS.2025.29.pdf
  • Filesize: 1.6 MB
  • 24 pages

Document Identifiers

Author Details

Yair Carmon
  • Tel Aviv University, Israel
Arun Jambulapati
  • University of Michigan, Ann Arbor, MI, USA
Liam O'Carroll
  • Stanford University, CA, USA
Aaron Sidford
  • Stanford University, CA, USA

Funding

  • Carmon, Yair: YC acknowledges support from the Israeli Science Foundation (ISF) grant no. 2486/21, and the Alon Fellowship.
  • O'Carroll, Liam: LO acknowledges support from NSF Grant CCF-1955039.
  • Sidford, Aaron: AS acknowledges support from a Microsoft Research Faculty Fellowship, NSF CAREER Grant CCF-1844855, NSF Grant CCF-1955039, and a PayPal research award.

Cite As Get BibTex

Yair Carmon, Arun Jambulapati, Liam O'Carroll, and Aaron Sidford. Extracting Dual Solutions via Primal Optimizers. In 16th Innovations in Theoretical Computer Science Conference (ITCS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 325, pp. 29:1-29:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ITCS.2025.29

Abstract

We provide a general method to convert a "primal" black-box algorithm for solving regularized convex-concave minimax optimization problems into an algorithm for solving the associated dual maximin optimization problem. Our method adds recursive regularization over a logarithmic number of rounds where each round consists of an approximate regularized primal optimization followed by the computation of a dual best response. We apply this result to obtain new state-of-the-art runtimes for solving matrix games in specific parameter regimes, obtain improved query complexity for solving the dual of the CVaR distributionally robust optimization (DRO) problem, and recover the optimal query complexity for finding a stationary point of a convex function.

Subject Classification

ACM Subject Classification
  • Theory of computation → Mathematical optimization
Keywords
  • Minimax optimization
  • black-box optimization
  • matrix games
  • distributionally robust optimization

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

References

  1. Jacob D Abernethy and Jun-Kun Wang. On frank-wolfe and equilibrium computation. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. Google Scholar
  2. Ilan Adler. The equivalence of linear programs and zero-sum games. International Journal of Games Theory, Volume 42:165-177, February 2013. Google Scholar
  3. Zeyuan Allen-Zhu and Elad Hazan. Optimal black-box reductions between optimization objectives. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pages 1614-1622, Red Hook, NY, USA, 2016. Curran Associates Inc. Google Scholar
  4. Hilal Asi, Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Stochastic bias-reduced gradient methods. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024. Curran Associates Inc. Google Scholar
  5. Yair Carmon and Danielle Hausler. Distributionally robust optimization via ball oracle acceleration. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. Google Scholar
  6. Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Thinking inside the ball: Near-optimal minimization of the maximal loss. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 866-882. PMLR, 15-19 August 2021. URL: http://proceedings.mlr.press/v134/carmon21a.html.
  7. Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Recapp: Crafting a more efficient catalyst for convex optimization. In International Conference on Machine Learning, 2022. Google Scholar
  8. Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. A Whole New Ball Game: A Primal Accelerated Method for Matrix Games and Minimizing the Maximum of Smooth Functions, pages 3685-3723. Society for Industrial and Applied Mathematics, 2024. URL: https://doi.org/10.1137/1.9781611977912.130.
  9. Yair Carmon, Yujia Jin, Aaron Sidford, and Kevin Tian. Variance reduction for matrix games. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dquotesingle Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. Google Scholar
  10. Kenneth L. Clarkson, Elad Hazan, and David P. Woodruff. Sublinear optimization for machine learning. J. ACM, 59(5), November 2012. URL: https://doi.org/10.1145/2371656.2371658.
  11. Michael B. Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, pages 938-942, New York, NY, USA, 2019. Association for Computing Machinery. URL: https://doi.org/10.1145/3313276.3316303.
  12. Michael B. Cohen, Aaron Sidford, and Kevin Tian. Relative lipschitzness in extragradient methods and a direct recipe for acceleration. In James R. Lee, editor, 12th Innovations in Theoretical Computer Science Conference, ITCS 2021, January 6-8, 2021, Virtual Conference, volume 185 of LIPIcs, pages 62:1-62:18. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021. URL: https://doi.org/10.4230/LIPICS.ITCS.2021.62.
  13. Sebastian Curi, Kfir Y. Levy, Stefanie Jegelka, and Andreas Krause. Adaptive sampling for stochastic risk-averse learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. Google Scholar
  14. G. B. Dantzig. Linear programming and extensions, 1953. Google Scholar
  15. Jelena Diakonikolas and Puqian Wang. Potential function-based framework for minimizing gradients in convex and min-max optimization. SIAM Journal on Optimization, 32(3):1668-1697, 2022. URL: https://doi.org/10.1137/21M1395302.
  16. John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49, June 2021. Google Scholar
  17. I. Ekeland, Roger Temam, Society for Industrial, and Applied Mathematics. Convex analysis and variational problems. Classics in applied mathematics. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, Pa., 1999. Google Scholar
  18. Roy Frostig, Rong Ge, Sham M. Kakade, and Aaron Sidford. Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pages 2540-2548. JMLR.org, 2015. URL: http://proceedings.mlr.press/v37/frostig15.html.
  19. Michael B. Giles. Multilevel monte carlo path simulation. Operations Research, 56(3):607-617, June 2008. URL: https://doi.org/10.1287/OPRE.1070.0496.
  20. Michael B. Giles. Multilevel monte carlo methods. Acta Numerica, 24:259-328, May 2015. URL: https://doi.org/10.1017/S096249291500001X.
  21. G. N. Grapiglia and Yurii Nesterov. Tensor methods for finding approximate stationary points of convex functions. Optimization Methods and Software, 37(2):605-638, 2022. URL: https://doi.org/10.1080/10556788.2020.1818082.
  22. Michael D. Grigoriadis and Leonid G. Khachiyan. A sublinear-time randomized approximation algorithm for matrix games. Operations Research Letters, 18(2):53-58, 1995. URL: https://doi.org/10.1016/0167-6377(95)00032-0.
  23. Yujia Jin, Aaron Sidford, and Kevin Tian. Sharper rates for separable minimax and finite sum optimization via primal-dual extragradient methods. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 4362-4415. PMLR, 02-05 July 2022. URL: https://proceedings.mlr.press/v178/jin22b.html.
  24. Donghwan Kim and Jeffrey A. Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of Optimization Theory and Applications, 188(1):192-219, January 2021. URL: https://doi.org/10.1007/S10957-020-01770-2.
  25. Jaeyeon Kim, Asuman Ozdaglar, Chanwoo Park, and Ernest K. Ryu. Time-reversed dissipation induces duality between minimizing gradient norm and function value, 2023. URL: https://arxiv.org/abs/2305.06628.
  26. Jaeyeon Kim, Chanwoo Park, Asuman Ozdaglar, Jelena Diakonikolas, and Ernest K. Ryu. Mirror duality in convex optimization, 2024. URL: https://arxiv.org/abs/2311.17296.
  27. Guanghui Lan, Yuyuan Ouyang, and Zhe Zhang. Optimal and parameter-free gradient minimization methods for convex and nonconvex optimization, 2023. URL: https://arxiv.org/abs/2310.12139.
  28. Jongmin Lee, Chanwoo Park, and Ernest K. Ryu. A geometric structure of acceleration and its role in making gradients small fast. In Neural Information Processing Systems, 2021. Google Scholar
  29. Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8847-8860. Curran Associates, Inc., 2020. Google Scholar
  30. Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 3384-3392, Cambridge, MA, USA, 2015. MIT Press. Google Scholar
  31. M. Minsky and S. Papert. Perceptrons: An introduction to computational geometry, 1988. Google Scholar
  32. Hongseok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pages 2216-2224, Red Hook, NY, USA, 2016. Curran Associates Inc. Google Scholar
  33. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009. URL: https://doi.org/10.1137/070704277.
  34. A.S Nemirovsky. On optimality of krylov’s information when solving linear operator equations. Journal of Complexity, 7(2):121-130, 1991. URL: https://doi.org/10.1016/0885-064X(91)90001-E.
  35. A.S Nemirovsky. Information-based complexity of linear operator equations. Journal of Complexity, 8(2):153-175, 1992. URL: https://doi.org/10.1016/0885-064X(92)90013-2.
  36. Yurii Nesterov. Lectures on Convex Optimization. Springer Publishing Company, Incorporated, 2nd edition, 2018. Google Scholar
  37. Yurii Nesterov, Alexander Gasnikov, Sergey Guminov, and Pavel Dvurechensky. Primal-dual accelerated gradient methods with small-dimensional relaxation oracle. Optimization Methods and Software, 36(4):773-810, July 2021. URL: https://doi.org/10.1080/10556788.2020.1731747.
  38. Francesco Orabona. A modern introduction to online learning, 2023. arXiv 1912.13213. URL: https://arxiv.org/abs/1912.13213.
  39. Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8:171-176, 1958. Google Scholar
  40. Paul Tseng. On accelerated proximal gradient methods for convex-concave optimization, 2008. Google Scholar
  41. J. v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295-320, December 1928. Google Scholar
  42. Jan van den Brand, Yin Tat Lee, Yang P. Liu, Thatchaphol Saranurak, Aaron Sidford, Zhao Song, and Di Wang. Minimum cost flows, mdps, and 𝓁₁-regression in nearly linear time for dense instances. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021, pages 859-869, New York, NY, USA, 2021. Association for Computing Machinery. Google Scholar
  43. Jun-Kun Wang, Jacob Abernethy, and Kfir Y. Levy. No-regret dynamics in the fenchel game: a unified framework for algorithmic convex optimization. Mathematical Programming, 205(1-2):203-268, May 2024. URL: https://doi.org/10.1007/S10107-023-01976-Y.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact