Document Open Access Logo

Matrix Completion and Related Problems via Strong Duality

Authors Maria-Florina Balcan, Yingyu Liang, David P. Woodruff, Hongyang Zhang

PDF
Thumbnail PDF

File

LIPIcs.ITCS.2018.5.pdf
  • Filesize: 1.57 MB
  • 22 pages

Document Identifiers

Author Details

Maria-Florina Balcan
Yingyu Liang
David P. Woodruff
Hongyang Zhang

Cite AsGet BibTex

Maria-Florina Balcan, Yingyu Liang, David P. Woodruff, and Hongyang Zhang. Matrix Completion and Related Problems via Strong Duality. In 9th Innovations in Theoretical Computer Science Conference (ITCS 2018). Leibniz International Proceedings in Informatics (LIPIcs), Volume 94, pp. 5:1-5:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2018)
https://doi.org/10.4230/LIPIcs.ITCS.2018.5

Abstract

This work studies the strong duality of non-convex matrix factorization problems: we show that under certain dual conditions, these problems and its dual have the same optimum. This has been well understood for convex optimization, but little was known for non-convex problems. We propose a novel analytical framework and show that under certain dual conditions, the optimal solution of the matrix factorization program is the same as its bi-dual and thus the global optimality of the non-convex program can be achieved by solving its bi-dual which is convex. These dual conditions are satisfied by a wide class of matrix factorization problems, although matrix factorization problems are hard to solve in full generality. This analytical framework may be of independent interest to non-convex optimization more broadly. We apply our framework to two prototypical matrix factorization problems: matrix completion and robust Principal Component Analysis (PCA). These are examples of efficiently recovering a hidden matrix given limited reliable observations of it. Our framework shows that exact recoverability and strong duality hold with nearly-optimal sample complexity guarantees for matrix completion and robust PCA.
Keywords
  • Non-Convex Optimization
  • Strong Duality
  • Matrix Completion
  • Robust PCA
  • Sample Complexity

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Alekh Agarwal, Sahand Negahban, and Martin J Wainwright. Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions. The Annals of Statistics, pages 1171-1197, 2012. Google Scholar
  2. Naman Agarwal, Zeyuan Allen-Zhu, Brian Bullins, Elad Hazan, and Tengyu Ma. Finding approximate local minima for nonconvex optimization in linear time. In ACM Symposium on Theory of Computing, pages 1195-1199, 2017. Google Scholar
  3. Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. In ACM Symposium on Theory of Computing, 2017. Google Scholar
  4. Christoph Ambühl, Monaldo Mastrolilli, and Ola Svensson. Inapproximability results for maximum edge biclique, minimum linear arrangement, and sparsest cut. SIAM Journal on Computing, 40(2):567-596, 2011. Google Scholar
  5. Anima Anandkumar and Rong Ge. Efficient approaches for escaping higher order saddle points in non-convex optimization. Annual Conference on Learning Theory, pages 81-102, 2016. Google Scholar
  6. Pranjal Awasthi, Maria-Florina Balcan, Nika Haghtalab, and Hongyang Zhang. Learning and 1-bit compressed sensing under asymmetric noise. In Annual Conference on Learning Theory, pages 152-192, 2016. Google Scholar
  7. Maria-Florina Balcan and Hongyang Zhang. Noise-tolerant life-long matrix completion via adaptive sampling. In Advances in Neural Information Processing Systems, pages 2955-2963, 2016. Google Scholar
  8. Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53-58, 1989. Google Scholar
  9. Saugata Basu, Richard Pollack, and Marie Francoise Roy. On the combinatorial and algebraic complexity of quantifier elimination. Journal of the ACM, 43(6):1002-1045, 1996. Google Scholar
  10. Amir Beck and Yonina C Eldar. Strong duality in nonconvex quadratic optimization with two quadratic constraints. SIAM Journal on Optimization, 17(3):844-860, 2006. Google Scholar
  11. Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Global optimality of local search for low rank matrix recovery. In Advances in Neural Information Processing Systems, pages 3873-3881, 2016. Google Scholar
  12. Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. Google Scholar
  13. Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM, 58(3):11, 2011. Google Scholar
  14. Emmanuel J. Candès and Ben Recht. Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9(6):717-772, 2009. Google Scholar
  15. Emmanuel J. Candès and Terence Tao. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory, 56(5):2053-2080, 2010. Google Scholar
  16. Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, and Alan S Willsky. The convex geometry of linear inverse problems. Foundations of Computational Mathematics, 12(6):805-849, 2012. Google Scholar
  17. Ji Chen and Xiaodong Li. Memory-efficient kernel PCA via partial matrix sampling and nonconvex optimization: a model-free analysis of local minima. arXiv preprint arXiv:1711.01742, 2017. Google Scholar
  18. Yudong Chen. Incoherence-optimal matrix completion. IEEE Transactions on Information Theory, 61(5):2909-2923, 2015. Google Scholar
  19. Uriel Feige. Relations between average case complexity and approximation complexity. In Annual IEEE Conference on Computational Complexity, page 5, 2002. Google Scholar
  20. David Gamarnik, Quan Li, and Hongyi Zhang. Matrix completion from O(n) samples in linear time. In Annual Conference on Learning Theory, 2017. Google Scholar
  21. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points - online stochastic gradient for tensor decomposition. In Annual Conference on Learning Theory, pages 797-842, 2015. Google Scholar
  22. Rong Ge, Chi Jin, and Zheng Yi. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. International Conference on Machine Learning, 2017. Google Scholar
  23. Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. In Advances in Neural Information Processing Systems, pages 2973-2981, 2016. Google Scholar
  24. Andreas Goerdt and André Lanka. An approximation hardness result for bipartite clique. In Electronic Colloquium on Computational Complexity, Report, volume 48, 2004. Google Scholar
  25. D. Gross. Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory, 57(3):1548-1566, 2011. Google Scholar
  26. Christian Grussler, Anders Rantzer, and Pontus Giselsson. Low-rank optimization with convex constraints. arXiv preprint arXiv:1606.01793, 2016. Google Scholar
  27. Quanquan Gu, Zhaoran Wang, and Han Liu. Low-rank and sparse structure pursuit via alternating minimization. In International Conference on Artificial Intelligence and Statistics, pages 600-609, 2016. Google Scholar
  28. Benjamin Haeffele, Eric Young, and Rene Vidal. Structured low-rank matrix factorization: Optimality, algorithm, and applications to image processing. In International Conference on Machine Learning, pages 2007-2015, 2014. Google Scholar
  29. Benjamin D Haeffele and René Vidal. Global optimality in neural network training. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7331-7339, 2017. Google Scholar
  30. Moritz Hardt. Understanding alternating minimization for matrix completion. In IEEE Symposium on Foundations of Computer Science, pages 651-660, 2014. Google Scholar
  31. Moritz Hardt, Raghu Meka, Prasad Raghavendra, and Benjamin Weitz. Computational limits for matrix completion. In Annual Conference on Learning Theory, pages 703-725, 2014. Google Scholar
  32. Moritz Hardt and Ankur Moitra. Algorithms and hardness for robust subspace recovery. Annual Conference on Learning Theory, 2013. Google Scholar
  33. Bingsheng He and Xiaoming Yuan. On the O(1/n) convergence rate of the douglas-rachford alternating direction method. SIAM Journal on Numerical Analysis, 50(2):700-709, 2012. Google Scholar
  34. Russell Impagliazzo and Avi Wigderson. P = BPP if E requires exponential circuits: Derandomizing the XOR lemma. In ACM Symposium on the Theory of Computing, pages 220-229, 1997. Google Scholar
  35. Johannes Jahn. Introduction to the theory of nonlinear optimization. Springer Berlin Heidelberg, 2007. Google Scholar
  36. Prateek Jain, Raghu Meka, and Inderjit S Dhillon. Guaranteed rank minimization via singular value projection. In Advances in Neural Information Processing Systems, pages 937-945, 2010. Google Scholar
  37. Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In ACM Symposium on Theory of Computing, pages 665-674, 2013. Google Scholar
  38. Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. International Conference on Machine Learning, 2017. Google Scholar
  39. Kenji Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems, pages 586-594, 2016. Google Scholar
  40. Raghunandan H Keshavan, Andrea Montanari, and Sewoong Oh. Matrix completion from a few entries. IEEE Transactions on Information Theory, 56(6):2980-2998, 2010. Google Scholar
  41. Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. IEEE Computer, 42(8):30-37, 2009. Google Scholar
  42. Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank approximation via alternating minimization. In International Conference on Machine Learning, pages 2358-2367, 2016. Google Scholar
  43. Praneeth Netrapalli, UN Niranjan, Sujay Sanghavi, Animashree Anandkumar, and Prateek Jain. Non-convex robust PCA. In Advances in Neural Information Processing Systems, pages 1107-1115, 2014. Google Scholar
  44. Michael L Overton and Robert S Womersley. On the sum of the largest eigenvalues of a symmetric matrix. SIAM Journal on Matrix Analysis and Applications, 13(1):41-45, 1992. Google Scholar
  45. Ilya Razenshteyn, Zhao Song, and David P. Woodruff. Weighted low rank approximations with provable guarantees. In ACM Symposium on Theory of Computing, pages 250-263, 2016. Google Scholar
  46. Benjamin Recht. A simpler approach to matrix completion. Journal of Machine Learning Research, 12:3413-3430, 2011. Google Scholar
  47. James Renegar. On the computational complexity and geometry of the first-order theory of the reals, part I: introduction. preliminaries. the geometry of semi-algebraic sets. the decision problem for the existential theory of the reals. Journal of symbolic computation, 13(3):255-300, 1992. Google Scholar
  48. James Renegar. On the computational complexity and geometry of the first-order theory of the reals, part II: the general decision problem. preliminaries for quantifier elimination. Journal of Symbolic Computation, 13(3):301-328, 1992. Google Scholar
  49. Reinhold Schneider and André Uschmajew. Convergence results for projected line-search methods on varieties of low-rank matrices via Lojasiewicz inequality. SIAM Journal on Optimization, 25(1):622-646, 2015. Google Scholar
  50. Yuan Shen, Zaiwen Wen, and Yin Zhang. Augmented lagrangian alternating direction method for matrix separation based on low-rank factorization. Optimization Methods and Software, 29(2):239-263, 2014. Google Scholar
  51. Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. In IEEE International Symposium on Information Theory, pages 2379-2383, 2016. Google Scholar
  52. Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere I: Overview and the geometric picture. IEEE Transactions on Information Theory, 63(2):853-884, 2017. Google Scholar
  53. Ruoyu Sun and Zhi-Quan Luo. Guaranteed matrix completion via nonconvex factorization. In IEEE Symposium on Foundations of Computer Science, pages 270-289, 2015. Google Scholar
  54. Stephen Tu, Ross Boczar, Mahdi Soltanolkotabi, and Benjamin Recht. Low-rank solutions of linear matrix equations via procrustes flow. International Conference on Machine Learning, 2016. Google Scholar
  55. Roman Vershynin. Lectures in geometric functional analysis, 2009. URL: https://www.math.uci.edu/~rvershyn/papers/GFA-book.pdf.
  56. Roman Vershynin. Estimation in high dimensions: A geometric perspective. In Sampling theory, a renaissance, pages 3-66. Springer, 2015. Google Scholar
  57. Yu-Xiang Wang and Huan Xu. Stability of matrix factorization for collaborative filtering. In International Conference on Machine Learning, pages 417-424, 2012. Google Scholar
  58. Zaiwen Wen, Wotao Yin, and Yin Zhang. Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm. Mathematical Programming Computation, 4(4):333-361, 2012. Google Scholar
  59. Xinyang Yi, Dohyung Park, Yudong Chen, and Constantine Caramanis. Fast algorithms for robust PCA via gradient descent. In Advances in neural information processing systems, pages 4152-4160, 2016. Google Scholar
  60. Hongyang Zhang, Zhouchen Lin, and Chao Zhang. A counterexample for the validity of using nuclear norm as a convex surrogate of rank. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, volume 8189, pages 226-241, 2013. Google Scholar
  61. Hongyang Zhang, Zhouchen Lin, and Chao Zhang. Completing low-rank matrices with corrupted samples from few coefficients in general basis. IEEE Transactions on Information Theory, 62(8):4748-4768, 2016. Google Scholar
  62. Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Edward Chang. Exact recoverability of robust PCA via outlier pursuit with tight recovery bounds. In AAAI Conference on Artificial Intelligence, pages 3143-3149, 2015. Google Scholar
  63. Xiao Zhang, Lingxiao Wang, and Quanquan Gu. A nonconvex free lunch for low-rank plus sparse matrix recovery. arXiv preprint arXiv:1702.06525, 2017. Google Scholar
  64. Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. In Advances in Neural Information Processing Systems, pages 559-567, 2015. Google Scholar
  65. Qinqing Zheng and John Lafferty. Convergence analysis for rectangular matrix completion using Burer-Monteiro factorization and gradient descent. arXiv preprint arXiv:1605.07051, 2016. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact