Document Open Access Logo

Optimal Rates for Robust Stochastic Convex Optimization

Authors Changyu Gao, Andrew Lowy, Xingyu Zhou, Stephen J. Wright

PDF

File

Thumbnail PDF
PDF
LIPIcs.FORC.2025.9.pdf
  • Filesize: 0.84 MB
  • 21 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Mathematical optimization
Keywords
  • Adversarial Robustness
  • Machine Learning
  • Optimization Algorithms
  • Robust Optimization
  • Stochastic Convex Optimization

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Machine learning algorithms in high-dimensional settings are highly susceptible to the influence of even a small fraction of structured outliers, making robust optimization techniques essential. In particular, within the ε-contamination model, where an adversary can inspect and replace up to an ε-fraction of the samples, a fundamental open problem is determining the optimal rates for robust stochastic convex optimization (SCO) under such contamination. We develop novel algorithms that achieve minimax-optimal excess risk (up to logarithmic factors) under the ε-contamination model. Our approach improves over existing algorithms, which are not only suboptimal but also require stringent assumptions, including Lipschitz continuity and smoothness of individual sample functions. By contrast, our optimal algorithms do not require these stringent assumptions, assuming only population-level smoothness of the loss. Moreover, our algorithms can be adapted to handle the case in which the covariance parameter is unknown, and can be extended to nonsmooth population risks via convolutional smoothing. We complement our algorithmic developments with a tight information-theoretic lower bound for robust SCO.

Cite As Get BibTex

Changyu Gao, Andrew Lowy, Xingyu Zhou, and Stephen J. Wright. Optimal Rates for Robust Stochastic Convex Optimization. In 6th Symposium on Foundations of Responsible Computing (FORC 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 329, pp. 9:1-9:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.FORC.2025.9

Author Details

Changyu Gao
  • University of Wisconsin-Madison, Madison, WI, USA
Andrew Lowy
  • University of Wisconsin-Madison, Madison, WI, USA
Xingyu Zhou
  • Wayne State University, Detroit, MI, USA
Stephen J. Wright
  • University of Wisconsin-Madison, Madison, WI, USA

Funding

Research of CG, AL, and SW was supported by NSF Awards 2023239 and 2224213 and AFOSR Award FA9550-21-1-0084. XZ is supported in part by NSF CNS-2153220 and CNS-2312835.

Acknowledgements

Changyu would like to thank Shuyao Li for helpful discussions.

References

  1. Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012. Google Scholar
  2. Yeshwanth Cherapanamjeri, Efe Aras, Nilesh Tripuraneni, Michael I Jordan, Nicolas Flammarion, and Peter L Bartlett. Optimal robust linear regression in nearly linear time. arXiv preprint arXiv:2007.08137, 2020. URL: https://arxiv.org/abs/2007.08137.
  3. Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 48(2):742-864, 2019. URL: https://doi.org/10.1137/17M1126680.
  4. Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. SEVER: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pages 1596-1606. PMLR, 2019. URL: http://proceedings.mlr.press/v97/diakonikolas19a.html.
  5. Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning, pages 999-1008. PMLR, 2017. URL: http://proceedings.mlr.press/v70/diakonikolas17a.html.
  6. Ilias Diakonikolas and Daniel M Kane. Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint arXiv:1911.05911, 2019. URL: https://arxiv.org/abs/1911.05911.
  7. Ilias Diakonikolas, Daniel M Kane, and Ankit Pensia. Outlier robust mean estimation with subgaussian rates via stability. Advances in Neural Information Processing Systems, 33:1830-1840, 2020. Google Scholar
  8. Ilias Diakonikolas, Weihao Kong, and Alistair Stewart. Efficient algorithms and lower bounds for robust linear regression. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2745-2754. SIAM, 2019. URL: https://doi.org/10.1137/1.9781611975482.170.
  9. Vitaly Feldman. Generalization of erm in stochastic convex optimization: The dimension strikes back. Advances in Neural Information Processing Systems, 29, 2016. Google Scholar
  10. Samuel B Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 497-506, 2023. URL: https://doi.org/10.1145/3564246.3585115.
  11. Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution, pages 492-518. Springer, 1992. Google Scholar
  12. Arun Jambulapati, Jerry Li, Tselil Schramm, and Kevin Tian. Robust regression revisited: Acceleration and improved estimation rates. Advances in Neural Information Processing Systems, 34:4475-4488, 2021. URL: https://proceedings.neurips.cc/paper/2021/hash/23b023b22d0bf47626029d5961328028-Abstract.html.
  13. Adam Klivans, Pravesh K Kothari, and Raghu Meka. Efficient algorithms for outlier-robust regression. In Conference On Learning Theory, pages 1420-1430. PMLR, 2018. URL: http://proceedings.mlr.press/v75/klivans18a.html.
  14. Shuyao Li, Yu Cheng, Ilias Diakonikolas, Jelena Diakonikolas, Rong Ge, and Stephen Wright. Robust second-order nonconvex optimization and its application to low rank matrix sensing. Advances in Neural Information Processing Systems, 36, 2024. Google Scholar
  15. Andrew Lowy and Meisam Razaviyayn. Private stochastic optimization with large worst-case lipschitz parameter: Optimal rates for (non-smooth) convex losses and extension to non-convex losses. In International Conference on Algorithmic Learning Theory, pages 986-1054. PMLR, 2023. URL: https://proceedings.mlr.press/v201/lowy23a.html.
  16. Adarsh Prasad, Arun Sai Suggala, Sivaraman Balakrishnan, and Pradeep Ravikumar. Robust estimation via robust gradient estimation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(3):601-627, 2020. Google Scholar
  17. John Wilder Tukey. A survey of sampling from contaminated distributions. Contributions to probability and statistics, pages 448-485, 1960. Google Scholar
  18. Farzad Yousefian, Angelia Nedić, and Uday V Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica, 48(1):56-67, 2012. URL: https://doi.org/10.1016/J.AUTOMATICA.2011.09.043.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact