Document Open Access Logo

Stability Yields Sublinear Time Algorithms for Geometric Optimization in Machine Learning

Author Hu Ding

PDF
Thumbnail PDF

File

LIPIcs.ESA.2021.38.pdf
  • Filesize: 0.92 MB
  • 19 pages

Document Identifiers

Author Details

Hu Ding
  • School of Computer Science and Technology, University of Science and Technology of China, Anhui, China

Acknowledgements

The author wants to thank Jinhui Xu and the anonymous reviewers for their helpful comments and suggestions for improving the paper.

Cite AsGet BibTex

Hu Ding. Stability Yields Sublinear Time Algorithms for Geometric Optimization in Machine Learning. In 29th Annual European Symposium on Algorithms (ESA 2021). Leibniz International Proceedings in Informatics (LIPIcs), Volume 204, pp. 38:1-38:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021)
https://doi.org/10.4230/LIPIcs.ESA.2021.38

Abstract

In this paper, we study several important geometric optimization problems arising in machine learning. First, we revisit the Minimum Enclosing Ball (MEB) problem in Euclidean space ℝ^d. The problem has been extensively studied before, but real-world machine learning tasks often need to handle large-scale datasets so that we cannot even afford linear time algorithms. Motivated by the recent developments on beyond worst-case analysis, we introduce the notion of stability for MEB, which is natural and easy to understand. Roughly speaking, an instance of MEB is stable, if the radius of the resulting ball cannot be significantly reduced by removing a small fraction of the input points. Under the stability assumption, we present two sampling algorithms for computing radius-approximate MEB with sample complexities independent of the number of input points n. In particular, the second algorithm has the sample complexity even independent of the dimensionality d. We also consider the general case without the stability assumption. We present a hybrid algorithm that can output either a radius-approximate MEB or a covering-approximate MEB, which improves the running time and the number of passes for the previous sublinear MEB algorithms. Further, we extend our proposed notion of stability and design sublinear time algorithms for other geometric optimization problems including MEB with outliers, polytope distance, one-class and two-class linear SVMs (without or with outliers). Our proposed algorithms also work fine for kernels.

Subject Classification

ACM Subject Classification
  • Theory of computation → Approximation algorithms analysis
Keywords
  • stability
  • sublinear time
  • geometric optimization
  • machine learning

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Pankaj K. Agarwal, Sariel Har-Peled, and Kasturi R. Varadarajan. Geometric approximation via coresets. Combinatorial and Computational Geometry, 52:1-30, 2005. Google Scholar
  2. Pankaj K. Agarwal, Sariel Har-Peled, and Hai Yu. Embeddings of surfaces, curves, and moving points in euclidean space. In Proceedings of the 23rd ACM Symposium on Computational Geometry, Gyeongju, South Korea, June 6-8, 2007, pages 381-389, 2007. Google Scholar
  3. Pankaj K. Agarwal, Sariel Har-Peled, and Hai Yu. Robust shape fitting via peeling and grating coresets. Discrete & Computational Geometry, 39(1-3):38-58, 2008. Google Scholar
  4. Pankaj K. Agarwal and R. Sharathkumar. Streaming algorithms for extent problems in high dimensions. Algorithmica, 72(1):83-98, 2015. Google Scholar
  5. Alok Aggarwal, Hiroshi Imai, Naoki Katoh, and Subhash Suri. Finding k points with minimum diameter and related problems. Journal of algorithms, 12(1):38-56, 1991. Google Scholar
  6. Zeyuan Allen Zhu, Zhenyu Liao, and Yang Yuan. Optimization algorithms for faster computational geometry. In 43rd International Colloquium on Automata, Languages, and Programming, ICALP 2016, July 11-15, 2016, Rome, Italy, pages 53:1-53:6, 2016. Google Scholar
  7. Noga Alon, Seannie Dar, Michal Parnas, and Dana Ron. Testing of clustering. SIAM Journal on Discrete Mathematics, 16(3):393-417, 2003. Google Scholar
  8. Pranjal Awasthi, Avrim Blum, and Or Sheffet. Stability yields a PTAS for k-median and k-means clustering. In 51th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2010, October 23-26, 2010, Las Vegas, Nevada, USA, pages 309-318, 2010. Google Scholar
  9. Pranjal Awasthi, Avrim Blum, and Or Sheffet. Center-based clustering under perturbation stability. Inf. Process. Lett., 112(1-2):49-54, 2012. Google Scholar
  10. Maria-Florina Balcan, Avrim Blum, and Anupam Gupta. Clustering under approximation stability. Journal of the ACM (JACM), 60(2):8, 2013. Google Scholar
  11. Maria-Florina Balcan and Mark Braverman. Finding low error clusterings. In COLT 2009 - The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21, 2009, 2009. Google Scholar
  12. Maria-Florina Balcan, Nika Haghtalab, and Colin White. k-center clustering under perturbation resilience. In 43rd International Colloquium on Automata, Languages, and Programming, ICALP 2016, July 11-15, 2016, Rome, Italy, pages 68:1-68:14, 2016. Google Scholar
  13. Maria-Florina Balcan and Yingyu Liang. Clustering under perturbation resilience. SIAM J. Comput., 45(1):102-155, 2016. Google Scholar
  14. Dimitris Bertsimas and Melvyn Sim. The price of robustness. Oper. Res., 52(1):35-53, 2004. Google Scholar
  15. Yonatan Bilu and Nathan Linial. Are stable instances easy? Combinatorics, Probability & Computing, 21(5):643-660, 2012. Google Scholar
  16. Mihai Bădoiu and Kenneth L. Clarkson. Smaller core-sets for balls. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 801-802, 2003. Google Scholar
  17. Mihai Bădoiu and Kenneth L. Clarkson. Optimal core-sets for balls. Computational Geometry, 40(1):14-22, 2008. Google Scholar
  18. Mihai Bădoiu, Sariel Har-Peled, and Piotr Indyk. Approximate clustering via core-sets. In Proceedings of the ACM Symposium on Theory of Computing (STOC), pages 250-257, 2002. Google Scholar
  19. Giuseppe Carlo Calafiore and Marco C. Campi. Uncertain convex programs: randomized solutions and confidence levels. Math. Program., 102(1):25-46, 2005. Google Scholar
  20. Matteo Ceccarello, Andrea Pietracaprina, and Geppino Pucci. Solving k-center clustering (with outliers) in mapreduce and streaming, almost as accurately as sequentially. PVLDB, 12(7):766-778, 2019. Google Scholar
  21. Timothy M. Chan and Vinayak Pathak. Streaming and dynamic algorithms for minimum enclosing balls in high dimensions. Comput. Geom., 47(2):240-247, 2014. Google Scholar
  22. Moses Charikar, Samir Khuller, David M Mount, and Giri Narasimhan. Algorithms for facility location problems with outliers. In Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms, pages 642-651. Society for Industrial and Applied Mathematics, 2001. Google Scholar
  23. Moses Charikar, Liadan O'Callaghan, and Rina Panigrahy. Better streaming algorithms for clustering problems. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing, pages 30-39. ACM, 2003. Google Scholar
  24. Kenneth L. Clarkson. Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Transactions on Algorithms, 6(4):63, 2010. Google Scholar
  25. Kenneth L. Clarkson, Elad Hazan, and David P. Woodruff. Sublinear optimization for machine learning. J. ACM, 59(5):23:1-23:49, 2012. Google Scholar
  26. Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20:273, 1995. Google Scholar
  27. David J. Crisp and Christopher J. C. Burges. A geometric interpretation of v-SVM classifiers. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller, editors, NIPS, pages 244-250. The MIT Press, 1999. Google Scholar
  28. Artur Czumaj and Christian Sohler. Sublinear-time algorithms. Survey paper, 2004. Google Scholar
  29. Artur Czumaj and Christian Sohler. Sublinear-time approximation for clustering via random sampling. In International Colloquium on Automata, Languages, and Programming, pages 396-407. Springer, 2004. Google Scholar
  30. Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Structures & Algorithms, 22(1):60-65, 2003. Google Scholar
  31. Hu Ding. A sub-linear time framework for geometric optimization with outliers in high dimensions. In Fabrizio Grandoni, Grzegorz Herman, and Peter Sanders, editors, 28th Annual European Symposium on Algorithms, ESA 2020, September 7-9, 2020, Pisa, Italy (Virtual Conference), volume 173 of LIPIcs, pages 38:1-38:21, 2020. Google Scholar
  32. Hu Ding and Jinhui Xu. Sub-linear time hybrid approximations for least trimmed squares estimator and related problems. In Proceedings of the International Symposium on Computational geometry (SoCG), page 110, 2014. Google Scholar
  33. Hu Ding, Haikuo Yu, and Zixiu Wang. Greedy strategy works for k-center clustering with outliers and coreset construction. In 27th Annual European Symposium on Algorithms, ESA 2019, September 9-11, 2019, Munich/Garching, Germany., pages 40:1-40:16, 2019. Google Scholar
  34. Alon Efrat, Micha Sharir, and Alon Ziv. Computing the smallest k-enclosing circle and related problems. Computational Geometry, 4(3):119-136, 1994. Google Scholar
  35. Dan Feldman. Core-sets: An updated survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 10(1), 2020. Google Scholar
  36. Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Lance Fortnow and Salil P. Vadhan, editors, Proceedings of the 43rd ACM Symposium on Theory of Computing, STOC 2011, San Jose, CA, USA, 6-8 June 2011, pages 569-578. ACM, 2011. Google Scholar
  37. Dan Feldman, Chongyuan Xiang, Ruihao Zhu, and Daniela Rus. Coresets for differentially private k-means clustering and applications to privacy in mobile sensor networks. In Proceedings of the 16th ACM/IEEE International Conference on Information Processing in Sensor Networks, IPSN 2017, Pittsburgh, PA, USA, April 18-21, 2017, pages 3-15, 2017. Google Scholar
  38. Kaspar Fischer, Bernd Gärtner, and Martin Kutz. Fast smallest-enclosing-ball computation in high dimensions. In Algorithms - ESA 2003, 11th Annual European Symposium, Budapest, Hungary, September 16-19, 2003, Proceedings, pages 630-641, 2003. Google Scholar
  39. Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(1-2):95-110, 1956. Google Scholar
  40. Dan Garber and Elad Hazan. Approximating semidefinite programs in sublinear time. In John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 1080-1088, 2011. Google Scholar
  41. Bernd Gärtner and Martin Jaggi. Coresets for polytope distance. In Proceedings of the International Symposium on Computational geometry (SoCG), pages 33-42, 2009. Google Scholar
  42. Elmer G. Gilbert. An iterative procedure for computing the minimum of a quadratic form on a convex set. SIAM Journal on Control, 4(1):61-80, 1966. Google Scholar
  43. Ashish Goel, Piotr Indyk, and Kasturi R Varadarajan. Reductions among high dimensional proximity problems. In SODA, volume 1, pages 769-778. Citeseer, 2001. Google Scholar
  44. Oded Goldreich, Shafi Goldwasser, and Dana Ron. Property testing and its connection to learning and approximation. J. ACM, 45(4):653-750, 1998. Google Scholar
  45. Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293-306, 1985. Google Scholar
  46. Laszlo Gyongyosi and Sandor Imre. Geometrical analysis of physically allowed quantum cloning transformations for quantum cryptography. Information Sciences, 285:1-23, 2014. Google Scholar
  47. Sariel Har-Peled and Soham Mazumdar. Fast algorithms for computing the smallest k-enclosing circle. Algorithmica, 41(3):147-157, 2005. Google Scholar
  48. Sariel Har-Peled and Yusu Wang. Shape fitting with outliers. SIAM Journal on Computing, 33(2):269-285, 2004. Google Scholar
  49. David Haussler and Emo Welzl. eps-nets and simplex range queries. Discrete & Computational Geometry, 2(2):127-151, 1987. Google Scholar
  50. Kohei Hayashi and Yuichi Yoshida. Minimizing quadratic functions in constant time. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2217-2225, 2016. Google Scholar
  51. Elad Hazan, Tomer Koren, and Nati Srebro. Beating SGD: learning svms in sublinear time. In John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 1233-1241, 2011. Google Scholar
  52. Dorit S Hochbaum and David B Shmoys. A best possible heuristic for the k-center problem. Mathematics of operations research, 10(2):180-184, 1985. Google Scholar
  53. Lingxiao Huang, Shaofeng Jiang, Jian Li, and Xuan Wu. Epsilon-coresets for clustering (with outliers) in doubling metrics. In 59th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2018, Paris, France, October 7-9, 2018, pages 814-825, 2018. Google Scholar
  54. Piotr Indyk. Sublinear time algorithms for metric space problems. In Proceedings of the Thirty-First Annual ACM Symposium on Theory of Computing, May 1-4, 1999, Atlanta, Georgia, USA, pages 428-434, 1999. Google Scholar
  55. Piotr Indyk. A sublinear time approximation scheme for clustering in metric spaces. In 40th Annual Symposium on Foundations of Computer Science, FOCS '99, 17-18 October, 1999, New York, NY, USA, pages 154-159, 1999. Google Scholar
  56. Michael Kerber and Sharath Raghvendra. Approximation and streaming algorithms for projective clustering via random projections. In Proceedings of the 27th Canadian Conference on Computational Geometry, CCCG 2015, Kingston, Ontario, Canada, August 10-12, 2015, 2015. Google Scholar
  57. Michael Kerber and R. Sharathkumar. Approximate čech complex in low and high dimensions. In Algorithms and Computation - 24th International Symposium, ISAAC 2013, Hong Kong, China, December 16-18, 2013, Proceedings, pages 666-676, 2013. Google Scholar
  58. Amer Krivosija and Alexander Munteanu. Probabilistic smallest enclosing ball in high dimensions via subgradient sampling. In Gill Barequet and Yusu Wang, editors, 35th International Symposium on Computational Geometry, SoCG 2019, June 18-21, 2019, Portland, Oregon, USA, volume 129 of LIPIcs, pages 47:1-47:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019. Google Scholar
  59. Amit Kumar and Ravindran Kannan. Clustering with spectral norm and the k-means algorithm. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 299-308. IEEE, 2010. Google Scholar
  60. Piyush Kumar, Joseph S. B. Mitchell, and E. Alper Yildirim. Approximate minimum enclosing balls in high dimensions using core-sets. ACM Journal of Experimental Algorithmics, 8, 2003. Google Scholar
  61. Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129-137, 1982. Google Scholar
  62. Jiří Matoušek. On enclosing k points by a circle. Information Processing Letters, 53(4):217-221, 1995. Google Scholar
  63. Richard Matthew McCutchen and Samir Khuller. Streaming algorithms for k-center clustering with outliers and with anonymity. In Approximation, Randomization and Combinatorial Optimization. Algorithms and Techniques, pages 165-178. Springer, 2008. Google Scholar
  64. Adam Meyerson, Liadan O'callaghan, and Serge Plotkin. A k-median algorithm with running time independent of data size. Machine Learning, 56(1-3):61-87, 2004. Google Scholar
  65. Nina Mishra, Dan Oblinger, and Leonard Pitt. Sublinear time approximate clustering. In Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms, pages 439-447. Society for Industrial and Applied Mathematics, 2001. Google Scholar
  66. Rajeev Motwani and Prabhakar Raghavan. Randomized Algorithms. Cambridge University Press, USA, 1995. Google Scholar
  67. David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. On the least trimmed squares estimator. Algorithmica, 69(1):148-183, 2014. Google Scholar
  68. Frank Nielsen and Richard Nock. Approximating smallest enclosing balls with applications to machine learning. Int. J. Comput. Geom. Appl., 19(5):389-414, 2009. Google Scholar
  69. Kobbi Nissim, Uri Stemmer, and Salil P. Vadhan. Locating a small cluster privately. In Proceedings of the 35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, PODS 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 413-427, 2016. Google Scholar
  70. Rafail Ostrovsky, Yuval Rabani, Leonard J Schulman, and Chaitanya Swamy. The effectiveness of lloyd-type methods for the k-means problem. Journal of the ACM (JACM), 59(6):28, 2012. Google Scholar
  71. Rina Panigrahy. Minimum enclosing polytope in high dimensions. arXiv preprint cs/0407020, 2004. Google Scholar
  72. Jeff M. Phillips. Coresets and sketches. Computing Research Repository, 2016. Google Scholar
  73. J. Platt. Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors, Advances in Kernel Methods - Support Vector Learning, pages 185-208, Cambridge, MA, 1999. MIT Press. Google Scholar
  74. Tim Roughgarden. Beyond worst-case analysis. Commun. ACM, 62(3):88-96, 2019. Google Scholar
  75. Ronitt Rubinfeld. Sublinear time algorithms. Citeseer, 2006. Google Scholar
  76. Ankan Saha, S. V. N. Vishwanathan, and Xinhua Zhang. New approximation algorithms for minimum enclosing convex shapes. In Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2011, San Francisco, California, USA, January 23-25, 2011, pages 1146-1160, 2011. Google Scholar
  77. B. Scholkopf, A. J. Smola, K. R. Muller, and P. L. Bartlett. New support vector algorithms. Neural Computation, 12:1207-1245, 2000. Google Scholar
  78. Bernhard Schölkopf and Alexander Johannes Smola. Learning with Kernels: support vector machines, regularization, optimization, and beyond. Adaptive computation and machine learning series. MIT Press, 2002. Google Scholar
  79. Donald R. Sheehy. The persistent homology of distance functions under random projection. In 30th Annual Symposium on Computational Geometry, SOCG'14, Kyoto, Japan, June 08 - 11, 2014, page 328, 2014. Google Scholar
  80. Ivor W. Tsang, James T. Kwok, and Pak-Ming Cheung. Core vector machines: Fast SVM training on very large data sets. Journal of Machine Learning Research, 6:363-392, 2005. Google Scholar
  81. Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In Measures of complexity, pages 11-30. Springer, 2015. Google Scholar
  82. Hamid Zarrabi-Zadeh and Asish Mukhopadhyay. Streaming 1-center with outliers in high dimensions. In Proceedings of the Canadian Conference on Computational Geometry (CCCG), pages 83-86, 2009. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact