Document Open Access Logo

Near-Linear Time and Fixed-Parameter Tractable Algorithms for Tensor Decompositions

Authors Arvind V. Mahankali, David P. Woodruff, Ziyu Zhang

PDF
Thumbnail PDF

File

LIPIcs.ITCS.2024.79.pdf
  • Filesize: 1.24 MB
  • 23 pages

Document Identifiers

Author Details

Arvind V. Mahankali
  • Stanford University, CA, USA
David P. Woodruff
  • Carnegie Mellon University, Pittsburgh, PA, USA
Ziyu Zhang
  • MIT CSAIL, Cambridge, MA, USA

Funding

  • Woodruff, David P.: Simons Investigator Award, NIH Grant 5R01 HG 10798-2, NSF Grant CCF-1815840, and ONR Grant N00014-18-1-2562.

Acknowledgements

D. Woodruff would like to thank partial support from a Simons Investigator Award.

Cite AsGet BibTex

Arvind V. Mahankali, David P. Woodruff, and Ziyu Zhang. Near-Linear Time and Fixed-Parameter Tractable Algorithms for Tensor Decompositions. In 15th Innovations in Theoretical Computer Science Conference (ITCS 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 287, pp. 79:1-79:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/LIPIcs.ITCS.2024.79

Abstract

We study low rank approximation of tensors, focusing on the Tensor Train and Tucker decompositions, as well as approximations with tree tensor networks and general tensor networks. As suggested by hardness results also shown in this work, obtaining (1+ε)-approximation algorithms for rank k tensor train and Tucker decompositions efficiently may be computationally hard for these problems. Therefore, we propose different algorithms that respectively satisfy some of the objectives above while violating some others within a bound, known as bicriteria algorithms. On the one hand, for rank-k tensor train decomposition for tensors with q modes, we give a (1 + ε)-approximation algorithm with a small bicriteria rank (O(qk/ε) up to logarithmic factors) and O(q ⋅ nnz(A)) running time, up to lower order terms. Here nnz(A) denotes the number of non-zero entries in the input tensor A. We also show how to convert the algorithm of [Huber et al., 2017] into a relative error approximation algorithm, but their algorithm necessarily has a running time of O(qr² ⋅ nnz(A)) + n ⋅ poly(qk/ε) when converted to a (1 + ε)-approximation algorithm with bicriteria rank r. Thus, the running time of our algorithm is better by at least a k² factor. To the best of our knowledge, our work is the first to achieve a near-input-sparsity time relative error approximation algorithm for tensor train decomposition. Our key technique is a method for efficiently obtaining subspace embeddings for a matrix which is the flattening of a Tensor Train of q tensors - the number of rows in the subspace embeddings is polynomial in q, thus avoiding the curse of dimensionality. We extend our algorithm to tree tensor networks and tensor networks on arbitrary graphs. Another way of coping with intractability is by looking at fixed-parameter tractable (FPT) algorithms. We give FPT algorithms for the tensor train, Tucker, and Canonical Polyadic (CP) decompositions, which are simpler than the FPT algorithms of [Song et al., 2019], since our algorithms do not make use of polynomial system solvers. Our technique of using an exponential number of Gaussian subspace embeddings with exactly k rows (and thus exponentially small success probability) may be of independent interest.

Subject Classification

ACM Subject Classification
  • Theory of computation
Keywords
  • Low rank approximation
  • Sketching algorithms
  • Tensor decomposition

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

References

  1. Thomas D. Ahle, Michael Kapralov, Jakob Bæk Tejs Knudsen, Rasmus Pagh, Ameya Velingker, David P. Woodruff, and Amir Zandieh. Oblivious sketching of high-degree polynomial kernels. In Shuchi Chawla, editor, Proceedings of the 2020 ACM-SIAM Symposium on Discrete Algorithms, SODA 2020, Salt Lake City, UT, USA, January 5-8, 2020, pages 141-160. SIAM, 2020. URL: https://doi.org/10.1137/1.9781611975994.9.
  2. Sanjeev Arora, Rong Ge, Ravi Kannan, and Ankur Moitra. Computing a nonnegative matrix factorization - provably. SIAM J. Comput., 45(4):1582-1611, 2016. URL: https://doi.org/10.1137/130913869.
  3. Haim Avron, Vikas Sindhwani, and David P. Woodruff. Sketching structured matrices for faster nonlinear regression. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2994-3002, 2013. Google Scholar
  4. Frank Ban, David P. Woodruff, and Qiuyi (Richard) Zhang. Regularized weighted low rank approximation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4061-4071, 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/90aef91f0d9e7c3be322bd7bae41617d-Abstract.html.
  5. Boaz Barak, Fernando GSL Brandao, Aram W Harrow, Jonathan Kelner, David Steurer, and Yuan Zhou. Hypercontractivity, sum-of-squares proofs, and their applications. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing, pages 307-326, 2012. Google Scholar
  6. Saugata Basu, Richard Pollack, and Marie-Françoise Roy. On the combinatorial and algebraic complexity of quantifier elimination. Journal of the ACM, 43(6):1002-1045, 1996. Google Scholar
  7. Karl Bringmann. Fine-grained complexity theory (tutorial). In Rolf Niedermeier and Christophe Paul, editors, 36th International Symposium on Theoretical Aspects of Computer Science, STACS 2019, March 13-16, 2019, Berlin, Germany, volume 126 of LIPIcs, pages 4:1-4:7. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2019. URL: https://doi.org/10.4230/LIPIcs.STACS.2019.4.
  8. Dimitris G. Chachlakis, Ashley Prater-Bennette, and Panos P. Markopoulos. L1-norm tucker tensor decomposition. CoRR, abs/1904.06455, 2019. URL: https://arxiv.org/abs/1904.06455.
  9. Sitan Chen, Jerry Li, Yuanzhi Li, and Anru R. Zhang. Learning polynomial transformations, 2022. URL: https://doi.org/10.48550/arXiv.2204.04209.
  10. Andrzej Cichocki, Namgil Lee, Ivan V. Oseledets, Anh Huy Phan, Qibin Zhao, and Danilo P. Mandic. Low-rank tensor networks for dimensionality reduction and large-scale optimization problems: Perspectives and challenges PART 1. CoRR, abs/1609.00893, 2016. URL: https://arxiv.org/abs/1609.00893.
  11. Kenneth L. Clarkson and David P. Woodruff. Numerical linear algebra in the streaming model. In Michael Mitzenmacher, editor, Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009, Bethesda, MD, USA, May 31 - June 2, 2009, pages 205-214. ACM, 2009. URL: https://doi.org/10.1145/1536414.1536445.
  12. Michael B. Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing, STOC '15, pages 163-172, New York, NY, USA, 2015. Association for Computing Machinery. URL: https://doi.org/10.1145/2746539.2746569.
  13. Hussam Al Daas, Grey Ballard, Paul Cazeaux, Eric Hallman, Agnieszka Miedlar, Mirjeta Pasha, Tim W. Reid, and Arvind K. Saibaba. Randomized algorithms for rounding in the tensor-train format, 2021. URL: https://doi.org/10.48550/arXiv.2110.04393.
  14. Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4):1253-1278, 2000. URL: https://doi.org/10.1137/S0895479896305696.
  15. Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 and rank-(r1 ,r2 ,. . .,rn) approximation of higher-order tensors. SIAM Journal on Matrix Analysis and Applications, 21(4):1324-1342, 2000. URL: https://doi.org/10.1137/S0895479898346995.
  16. A Dektor, A Rodgers, and D Venturi. Rank-adaptive tensor methods for high-dimensional nonlinear pdes. J. Sci. Comput., 88:36, 2021. Google Scholar
  17. Huaian Diao, Rajesh Jayaram, Zhao Song, Wen Sun, and David P. Woodruff. Optimal sketching for kronecker product regression and low rank approximation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4739-4750, 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/0c215f194276000be6a6df6528067151-Abstract.html.
  18. S. V. Dolgov, B. N. Khoromskij, and I. V. Oseledets. Fast solution of parabolic problems in the tensor train/quantized tensor train format with initial application to the fokker-planck equation. SIAM Journal on Scientific Computing, 34(6):A3016-A3038, 2012. Google Scholar
  19. Sergey Dolgov, Boris N. Khoromskij, Alexander Litvinenko, and Hermann G. Matthies. Polynomial chaos expansion of random coefficients and the solution of stochastic partial differential equations in the tensor train format. SIAM/ASA Journal on Uncertainty Quantification, 3(1):1109-1135, 2015. Google Scholar
  20. Matthew Fahrbach, Mehrdad Ghadiri, and Thomas Fu. Fast low-rank tensor decomposition by ridge leverage score sampling, 2021. URL: https://arxiv.org/abs/2107.10654.
  21. Alex Gorodetsky, Sertac Karaman, and Youssef Marzouk. High-dimensional stochastic optimal control using continuous tensor decompositions. The International journal of robotics research, 37(2-3), 2018. Google Scholar
  22. Alex Gorodetsky, Sertac Karaman, and Youssef Marzouk. High-dimensional stochastic optimal control using continuous tensor decompositions. The International Journal of Robotics Research, 37(2-3):340-377, 2018. Google Scholar
  23. Lars Grasedyck. Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl., 31(4):2029-2054, 2010. URL: https://doi.org/10.1137/090764189.
  24. Jonas Haferkamp, Dominik Hangleiter, Jens Eisert, and Marek Gluza. Contracting projected entangled pair states is average-case hard. Physical Review Research, 2(1), January 2020. URL: https://doi.org/10.1103/physrevresearch.2.013010.
  25. C. J. Hillar and L.-H. Lim. Most tensor problems are np-hard. ACM 60, 6(45), 2013. URL: https://doi.org/10.1145/2512329.
  26. M K Horowitz, A Damle, and J W Burdick. Linear Hamilton Jacobi Bellman equations in high dimensions. In 53rd IEEE Conference on Decision and Control, pages 5880-5887, 2014. Google Scholar
  27. Johan Håstad. Tensor rank is np-complete. Journal of Algorithms, 11(4):644-654, 1990. URL: https://doi.org/10.1016/0196-6774(90)90014-6.
  28. Benjamin Huber, Reinhold Schneider, and Sebastian Wolf. A Randomized Tensor Train Singular Value Decomposition, pages 261-290. Springer International Publishing, Cham, 2017. URL: https://doi.org/10.1007/978-3-319-69802-1_9.
  29. Rajesh Jayaram, Alireza Samadian, David P. Woodruff, and Peng Ye. In-database regression in input sparsity time. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4797-4806. PMLR, 2021. Google Scholar
  30. Vladimir A. Kazeev and Boris N. Khoromskij. Low-rank explicit qtt representation of the laplace operator and its inverse. SIAM Journal on Matrix Analysis and Applications, 33(3):742-758, 2012. Google Scholar
  31. Boris N. Khoromskij. Tensors-structured numerical methods in scientific computing: Survey on recent advances. Chemometrics and Intelligent Laboratory Systems, 110(1):1-19, 2012. URL: https://doi.org/10.1016/j.chemolab.2011.09.001.
  32. Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3):455-500, 2009. Google Scholar
  33. Katharina Kormann. A semi-lagrangian vlasov solver in tensor train format. SIAM Journal on Scientific Computing, 37(4):B613-B632, 2015. Google Scholar
  34. Lingjie Li, Wenjian Yu, and Kim Batselier. Faster tensor train decomposition for sparse data. CoRR, abs/1908.02721, 2019. URL: https://arxiv.org/abs/1908.02721.
  35. Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2022, pages 1248-1261, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3519935.3520012.
  36. Michael Lubasch, Pierre Moinier, and Dieter Jaksch. Multigrid renormalization. Journal of Computational Physics, 372:587-602, 2018. URL: https://doi.org/10.1016/j.jcp.2018.06.065.
  37. Ningyi Lyu, Micheline B. Soley, and Victor S. Batista. Tensor-train split-operator ksl (tt-soksl) method for quantum dynamics simulations. Journal of Chemical Theory and Computation, 18(6):3327-3346, 2022. PMID: 35649210. URL: https://doi.org/10.1021/acs.jctc.2c00209.
  38. Linjian Ma and Edgar Solomonik. Cost-efficient gaussian tensor network embeddings for tensor-structured inputs, 2022. URL: https://doi.org/10.48550/arXiv.2205.13163.
  39. Osman Asif Malik and Stephen Becker. Low-rank tucker decomposition of large tensors using tensorsketch. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL: https://proceedings.neurips.cc/paper/2018/file/45a766fa266ea2ebeb6680fa139d2a3d-Paper.pdf.
  40. Osman Asif Malik and Stephen Becker. A sampling-based method for tensor ring decomposition. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7400-7411. PMLR, 18-24 July 2021. URL: https://proceedings.mlr.press/v139/malik21b.html.
  41. Igor L. Markov and Yaoyun Shi. Simulating quantum computation by contracting tensor networks. SIAM J. Comput., 38(3):963-981, 2008. URL: https://doi.org/10.1137/050644756.
  42. Raphael A. Meyer, Cameron Musco, Christopher Musco, David P. Woodruff, and Samson Zhou. Fast regression for structured inputs. CoRR, abs/2203.07557, 2022. Google Scholar
  43. Rachel Minster, Arvind K Saibaba, and Misha E Kilmer. Randomized algorithms for low-rank tensor decompositions in the tucker format. SIAM journal on mathematics of data science, 2(1):189-215, 2020. Google Scholar
  44. Ankur Moitra. An almost optimal algorithm for computing nonnegative rank. SIAM J. Comput., 45(1):156-173, 2016. URL: https://doi.org/10.1137/140990139.
  45. Bryan O'Gorman. Parameterization of tensor network contraction. arXiv preprint arXiv:1906.00013, 2019. Google Scholar
  46. Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295-2317, 2011. Google Scholar
  47. Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 239-247, 2013. Google Scholar
  48. Beheshteh Rakhshan and Guillaume Rabusseau. Tensorized random projections. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3306-3316. PMLR, 26-28 August 2020. URL: https://proceedings.mlr.press/v108/rakhshan20a.html.
  49. Marcus Schaefer and Daniel Stefankovic. The complexity of tensor rank. CoRR, abs/1612.04338, 2016. URL: https://arxiv.org/abs/1612.04338.
  50. Xiaofei Shi and David P. Woodruff. Sublinear time numerical linear algebra for structured matrices. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 4918-4925. AAAI Press, 2019. Google Scholar
  51. Nicholas D. Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E. Papalexakis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing, 65(13):3551-3582, July 2017. URL: https://doi.org/10.1109/tsp.2017.2690524.
  52. Zhao Song, David P. Woodruff, and Peilin Zhong. Relative error tensor low rank approximation. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '19, pages 2772-2789, USA, 2019. Society for Industrial and Applied Mathematics. Google Scholar
  53. Yiming Sun, Yang Guo, Charlene Luo, Joel Tropp, and Madeleine Udell. Low-rank tucker approximation of a tensor from streaming data. SIAM Journal on Mathematics of Data Science, 2(4):1123-1150, 2020. URL: https://doi.org/10.1137/19M1257718.
  54. Jeheon Woo, Woo Youn Kim, and Sunghwan Choi. System-specific separable basis based on tucker decomposition: Application to density functional calculations. Journal of Chemical Theory and Computation, 18(5):2875-2884, 2022. PMID: 35437014. URL: https://doi.org/10.1021/acs.jctc.1c01263.
  55. David P. Woodruff. Sketching as a tool for numerical linear algebra. Found. Trends Theor. Comput. Sci., 10(1-2):1-157, 2014. URL: https://doi.org/10.1561/0400000060.
  56. Wenbin Yang, Zijia Wang, Jiacheng Ni, Qiang Chen, and Zhen Jia. A low-rank tensor bayesian filter framework for multi-modal analysis. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3738-3742, 2022. URL: https://doi.org/10.1109/ICIP46576.2022.9897228.
  57. Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. L2,1-norm regularized discriminative feature selection for unsupervised learning. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Two, IJCAI'11, pages 1589-1594. AAAI Press, 2011. Google Scholar
  58. Ke Ye and Lek-Heng Lim. Tensor network ranks, 2018. URL: https://doi.org/10.48550/arXiv.1801.02662.
  59. Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decomposition-based dnn model compression with optimization framework. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10669-10678, 2021. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact