Tensor Network Complexity of Multilinear Maps

Authors Per Austrin, Petteri Kaski, Kaie Kubjas

Thumbnail PDF


  • Filesize: 0.65 MB
  • 21 pages

Document Identifiers

Author Details

Per Austrin
  • School of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden
Petteri Kaski
  • Department of Computer Science, Aalto University, Helsinki, Finland
Kaie Kubjas
  • Department of Mathematics and Systems Analysis, Aalto University, Helsinki, Finland, and, Laboratoire d'Informatique de Paris 6, Sorbonne Université, Paris, France

Cite AsGet BibTex

Per Austrin, Petteri Kaski, and Kaie Kubjas. Tensor Network Complexity of Multilinear Maps. In 10th Innovations in Theoretical Computer Science Conference (ITCS 2019). Leibniz International Proceedings in Informatics (LIPIcs), Volume 124, pp. 7:1-7:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2019)


We study tensor networks as a model of arithmetic computation for evaluating multilinear maps. These capture any algorithm based on low border rank tensor decompositions, such as O(n^{omega+epsilon}) time matrix multiplication, and in addition many other algorithms such as O(n log n) time discrete Fourier transform and O^*(2^n) time for computing the permanent of a matrix. However tensor networks sometimes yield faster algorithms than those that follow from low-rank decompositions. For instance the fastest known O(n^{(omega +epsilon)t}) time algorithms for counting 3t-cliques can be implemented with tensor networks, even though the underlying tensor has border rank n^{3t} for all t >= 2. For counting homomorphisms of a general pattern graph P into a host graph on n vertices we obtain an upper bound of O(n^{(omega+epsilon)bw(P)/2}) where bw(P) is the branchwidth of P. This essentially matches the bound for counting cliques, and yields small improvements over previous algorithms for many choices of P. While powerful, the model still has limitations, and we are able to show a number of unconditional lower bounds for various multilinear maps, including: b) an Omega(n^{bw(P)}) time lower bound for counting homomorphisms from P to an n-vertex graph, matching the upper bound if omega = 2. In particular for P a v-clique this yields an Omega(n^{ceil[2v/3]}) time lower bound for counting v-cliques, and for P a k-uniform v-hyperclique we obtain an Omega(n^v) time lower bound for k >= 3, ruling out tensor networks as an approach to obtaining non-trivial algorithms for hyperclique counting and the Max-3-CSP problem. c) an Omega(2^{0.918n}) time lower bound for the permanent of an n x n matrix.

Subject Classification

ACM Subject Classification
  • Theory of computation → Models of computation
  • Theory of computation → Computational complexity and cryptography
  • Theory of computation → Design and analysis of algorithms
  • arithmetic complexity
  • lower bound
  • multilinear map
  • tensor network


  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    PDF Downloads


  1. Amir Abboud, Arturs Backurs, Karl Bringmann, and Marvin Künnemann. Fine-Grained Complexity of Analyzing Compressed Data: Quantifying Improvements over Decompress-and-Solve. In Proc. of Foundations of Computer Science (FOCS), pages 192-203, 2017. URL: http://dx.doi.org/10.1109/FOCS.2017.26.
  2. Amir Abboud, Arturs Backurs, and Virginia Vassilevska Williams. If the Current Clique Algorithms are Optimal, So is Valiant’s Parser. In Proc. of Foundations of Computer Science (FOCS), pages 98-117, 2015. URL: http://dx.doi.org/10.1109/FOCS.2015.16.
  3. Michael Alekhnovich, Allan Borodin, Joshua Buresh-Oppenheim, Russell Impagliazzo, Avner Magen, and Toniann Pitassi. Toward a Model for Backtracking and Dynamic Programming. Comput. Complex., 20(4):679-740, 2011. URL: http://dx.doi.org/10.1007/s00037-011-0028-y.
  4. Noga Alon and Shai Gutner. Balanced families of perfect hash functions and their applications. ACM Trans. Algorithms, 6(3):54:1-54:12, 2010. URL: http://dx.doi.org/10.1145/1798596.1798607.
  5. Noga Alon, Raphael Yuster, and Uri Zwick. Finding and Counting Given Length Cycles. Algorithmica, 17(3):209-223, 1997. URL: http://dx.doi.org/10.1007/BF02523189.
  6. Itai Arad and Zeph Landau. Quantum computation and the evaluation of tensor networks. SIAM J. Comput., 39(7):3089-3121, 2010. URL: http://dx.doi.org/10.1137/080739379.
  7. Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach. Cambridge University Press, New York, NY, USA, 1st edition, 2009. URL: http://dx.doi.org/10.1017/CBO9780511804090.
  8. Martin Beaudry and Markus Holzer. The Complexity of Tensor Circuit Evaluation. Comput. Complex., 16(1):60-111, 2007. URL: http://dx.doi.org/10.1007/s00037-007-0222-0.
  9. Stuart J. Berkowitz. On Computing the Determinant in Small Parallel Time Using a Small Number of Processors. Inf. Process. Lett., 18(3):147-150, 1984. URL: http://dx.doi.org/10.1016/0020-0190(84)90018-8.
  10. Andreas Björklund, Thore Husfeldt, Petteri Kaski, and Mikko Koivisto. Fourier meets Möbius: fast subset convolution. In Proc. of Symposium on Theory of Computing (STOC), pages 67-74, 2007. URL: http://dx.doi.org/10.1145/1250790.1250801.
  11. Andreas Björklund, Thore Husfeldt, Petteri Kaski, and Mikko Koivisto. Computing the Tutte Polynomial in Vertex-Exponential Time. In Proc. of Foundations of Computer Science (FOCS), pages 677-686, 2008. URL: http://dx.doi.org/10.1109/FOCS.2008.40.
  12. Andreas Björklund, Thore Husfeldt, Petteri Kaski, and Mikko Koivisto. Counting Paths and Packings in Halves. In Proc. of European Symposium on Algorithms (ESA), pages 578-586, 2009. URL: http://dx.doi.org/10.1007/978-3-642-04128-0_52.
  13. Andreas Björklund, Thore Husfeldt, and Mikko Koivisto. Set Partitioning via Inclusion-Exclusion. SIAM J. Comput., 39(2):546-563, 2009. URL: http://dx.doi.org/10.1137/070683933.
  14. Andreas Björklund, Petteri Kaski, and Łukasz Kowalik. Counting Thin Subgraphs via Packings Faster Than Meet-in-the-Middle Time. In Proc. of Symposium on Discrete Algorithms (SODA), pages 594-603, 2014. URL: http://dx.doi.org/10.1137/1.9781611973402.45.
  15. Hans L. Bodlaender, Erik Jan van Leeuwen, Johan M. M. van Rooij, and Martin Vatshelle. Faster Algorithms on Branch and Clique Decompositions. In Proc. of Mathematical Foundations of Computer Science (MFCS), pages 174-185, 2010. URL: http://dx.doi.org/10.1007/978-3-642-15155-2_17.
  16. James R. Bunch and John E. Hopcroft. Triangular factorization and inversion by fast matrix multiplication. Math. Comp., 28:231-236, 1974. URL: http://dx.doi.org/10.2307/2005828.
  17. Jin-Yi Cai, Heng Guo, and Tyson Williams. A Complete Dichotomy Rises from the Capture of Vanishing Signatures. SIAM J. Comput., 45(5):1671-1728, 2016. URL: http://dx.doi.org/10.1137/15M1049798.
  18. Jin-yi Cai, Pinyan Lu, and Mingji Xia. Computational Complexity of Holant Problems. SIAM J. Comput., 40(4):1101-1132, 2011. URL: http://dx.doi.org/10.1137/100814585.
  19. Florent Capelli, Arnaud Durand, and Stefan Mengel. The Arithmetic Complexity of Tensor Contraction. Theory Comput. Syst., 58(4):506-527, 2016. URL: http://dx.doi.org/10.1007/s00224-015-9630-8.
  20. A. Cayley. On the theory of the analytical forms called trees. Philos. Mag., 13:172-176, 1857. URL: http://dx.doi.org/10.1080/14786445708642275.
  21. A. Cayley. On the analytical forms called trees, part II. Philos. Mag., 18:374-378, 1859. URL: http://dx.doi.org/10.1080/14786445908642782.
  22. A. Clebsch. Ueber symbolische Darstellung algebraischer Formen. J. Reine Angew. Math., 59:1-62, 1861. URL: http://dx.doi.org/10.1515/crll.1861.59.1.
  23. W. Clifford. Extract of a Letter to Mr. Sylvester from Prof. Clifford of University College, London. Amer. J. Math., 1(2):126-128, 1878. URL: http://dx.doi.org/10.2307/2369303.
  24. James W. Cooley and John W. Tukey. An algorithm for the machine calculation of complex Fourier series. Math. Comp., 19:297-301, 1965. URL: http://dx.doi.org/10.2307/2003354.
  25. Radu Curticapean, Holger Dell, and Dániel Marx. Homomorphisms are a good basis for counting small subgraphs. In Proc. of Symposium on Theory of Computing (STOC), pages 210-223, 2017. URL: http://dx.doi.org/10.1145/3055399.3055502.
  26. Carsten Damm, Markus Holzer, and Pierre McKenzie. The complexity of tensor calculus. Comput. Complex., 11(1-2):54-89, 2002. URL: http://dx.doi.org/10.1007/s00037-000-0170-4.
  27. Sashka Davis and Russell Impagliazzo. Models of Greedy Algorithms for Graph Problems. Algorithmica, 54(3):269-317, 2009. URL: http://dx.doi.org/10.1007/s00453-007-9124-4.
  28. Josep Díaz, Maria J. Serna, and Dimitrios M. Thilikos. Counting H-colorings of partial k-trees. Theor. Comput. Sci., 281(1-2):291-309, 2002. URL: http://dx.doi.org/10.1016/S0304-3975(02)00017-8.
  29. Frederic Dorn. Dynamic Programming and Fast Matrix Multiplication. In Proc. of European Symposium on Algorithms (ESA), pages 280-291, 2006. URL: http://dx.doi.org/10.1007/11841036_27.
  30. Friedrich Eisenbrand and Fabrizio Grandoni. On the complexity of fixed parameter clique and dominating set. Theoret. Comput. Sci., 326(1-3):57-67, 2004. URL: http://dx.doi.org/10.1016/j.tcs.2004.05.009.
  31. Norman P. Jouppi et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proc. of International Symposium on Computer Architecture (ISCA), pages 1-12, 2017. URL: http://dx.doi.org/10.1145/3079856.3080246.
  32. Peter Floderus, Miroslaw Kowaluk, Andrzej Lingas, and Eva-Marta Lundell. Detecting and Counting Small Pattern Graphs. SIAM J. Discrete Math., 29(3):1322-1339, 2015. URL: http://dx.doi.org/10.1137/140978211.
  33. Fedor V. Fomin, Daniel Lokshtanov, Venkatesh Raman, Saket Saurabh, and B. V. Raghavendra Rao. Faster algorithms for finding and counting subgraphs. J. Comput. Syst. Sci., 78(3):698-706, 2012. URL: http://dx.doi.org/10.1016/j.jcss.2011.10.001.
  34. William I. Gasarch. The Second P =? NP Poll. SIGACT News Complexity Theory Column, 74, 2012. URL: http://dx.doi.org/10.1145/2261417.2261434.
  35. Russell Impagliazzo and Ramamohan Paturi. On the Complexity of k-SAT. J. Comput. Syst. Sci., 62(2):367-375, 2001. URL: http://dx.doi.org/10.1006/jcss.2000.1727.
  36. Russell Impagliazzo, Ramamohan Paturi, and Francis Zane. Which Problems Have Strongly Exponential Complexity? J. Comput. Syst. Sci., 63(4):512-530, 2001. URL: http://dx.doi.org/10.1006/jcss.2001.1774.
  37. Alon Itai and Michael Rodeh. Finding a Minimum Circuit in a Graph. SIAM J. Comput., 7(4):413-423, 1978. URL: http://dx.doi.org/10.1137/0207033.
  38. Matti Karppa, Petteri Kaski, and Jukka Kohonen. A Faster Subquadratic Algorithm for Finding Outlier Correlations. ACM Trans. Algorithms, 14(3):31:1-31:26, 2018. URL: http://dx.doi.org/10.1145/3174804.
  39. L. H. Kauffman. Knots, abstract tensors and the Yang-Baxter equation. In Knots, topology and quantum field theories, pages 179-334. World Scientific, 1989. Google Scholar
  40. A. B. Kempe. On the Application of Clifford’s Graphs to Ordinary Binary Quantics. P. Lond. Math. Soc., 17:107-121, 1885/86. URL: http://dx.doi.org/10.1112/plms/s1-17.1.107.
  41. A. B. Kempe. On the application of the Sylvester-Clifford Graphs to Ordinary Binary Quantics. (Second Part.). P. Lond. Math. Soc., 24:97-118, 1892/93. URL: http://dx.doi.org/10.1112/plms/s1-24.1.97.
  42. Ton Kloks, Dieter Kratsch, and Haiko Müller. Finding and counting small induced subgraphs efficiently. Inf. Process. Lett., 74(3-4):115-121, 2000. URL: http://dx.doi.org/10.1016/S0020-0190(00)00047-8.
  43. Tamara G. Kolda. Multilinear operators for higher-order decompositions. Technical Report SAND2006-2081, Sandia National Laboratories, 2006. URL: http://dx.doi.org/10.2172/923081.
  44. Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev., 51(3):455-500, 2009. URL: http://dx.doi.org/10.1137/07070111X.
  45. Daphne Koller and Nir Friedman. Probabilistic Graphical Models. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2009. Google Scholar
  46. Joseph B. Kruskal. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl., 18(2):95-138, 1977. URL: http://dx.doi.org/10.1016/0024-3795(77)90069-6.
  47. Greg Kuperberg. Involutory Hopf algebras and 3-manifold invariants. Internat. J. Math., 2(1):41-66, 1991. URL: http://dx.doi.org/10.1142/S0129167X91000053.
  48. J. M. Landsberg. Tensors: Geometry and Applications, volume 128 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, 2012. URL: http://dx.doi.org/10.1090/gsm/128.
  49. Joseph M. Landsberg, Yang Qi, and Ke Ye. On the geometry of tensor network states. Quantum Inf. Comput., 12(3-4):346-354, 2012. Google Scholar
  50. François Le Gall. Powers of tensors and fast matrix multiplication. In Proc. of International Symposium on Symbolic and Algebraic Computation (ISSAC), pages 296-303, 2014. URL: http://dx.doi.org/10.1145/2608628.2608664.
  51. James R. Lee, Prasad Raghavendra, and David Steurer. Lower Bounds on the Size of Semidefinite Programming Relaxations. In Proc. of Symposium on Theory of Computing (STOC), pages 567-576, 2015. URL: http://dx.doi.org/10.1145/2746539.2746599.
  52. Andrea Lincoln, Virginia Vassilevska Williams, and R. Ryan Williams. Tight Hardness for Shortest Cycles and Paths in Sparse Graphs. In Proc. of Symposium on Discrete Algorithms (SODA), pages 1236-1252, 2018. URL: http://dx.doi.org/10.1137/1.9781611975031.80.
  53. Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S. Vetter. NVIDIA tensor core programmability, performance & precision. In Proc. of International Parallel and Distributed Processing Symposium Workshops (IPDPS), pages 522-531, 2018. URL: http://dx.doi.org/10.1109/IPDPSW.2018.00091.
  54. J. Nešetřil and S. Poljak. On the complexity of the subgraph problem. Comment. Math. Univ. Carolin., 26(2):415-419, 1985. Google Scholar
  55. Stephan Olariu. Paw-Free Graphs. Inf. Process. Lett., 28(1):53-54, 1988. URL: http://dx.doi.org/10.1016/0020-0190(88)90143-3.
  56. R. Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Ann. Phys., 349:117-158, 2014. URL: http://dx.doi.org/10.1016/j.aop.2014.06.013.
  57. Roger Penrose. Tensor Methods in Algebraic Geometry. PhD thesis, St. John’s College, 1956. Google Scholar
  58. Roger Penrose. Applications of negative dimensional tensors. In Combinatorial Mathematics and its Applications, pages 221-244. Academic Press, 1971. Google Scholar
  59. Robert N. C. Pfeifer, Jutho Haegeman, and Frank Verstraete. Faster identification of optimal contraction sequences for tensor networks. Phys. Rev. E, 90:033315, 2014. URL: http://dx.doi.org/10.1103/PhysRevE.90.033315.
  60. Ran Raz. Tensor-Rank and Lower Bounds for Arithmetic Formulas. J. ACM, 60(6):40:1-40:15, 2013. URL: http://dx.doi.org/10.1145/2535928.
  61. Jürgen Richter-Gebert and Peter Lebmeir. Diagrams, tensors and geometric reasoning. Discrete Comput. Geom., 42(2):305-334, 2009. URL: http://dx.doi.org/10.1007/s00454-009-9188-9.
  62. Elina Robeva and Anna Seigal. Duality of graphical models and tensor networks. Inf. Inference, page iay009, 2018. URL: http://dx.doi.org/10.1093/imaiai/iay009.
  63. Günter Rote. Division-Free Algorithms for the Determinant and the Pfaffian: Algebraic and Combinatorial Approaches. In Computational Discrete Mathematics, Advanced Lectures, pages 119-135, 2001. URL: http://dx.doi.org/10.1007/3-540-45506-X_9.
  64. Herbert John Ryser. Combinatorial Mathematics. The Carus Mathematical Monographs, No. 14. Mathematical Association of America, 1963. Google Scholar
  65. Alexander Schrijver. On traces of tensor representations of diagrams. Linear Algebra Appl., 476:28-41, 2015. URL: http://dx.doi.org/10.1016/j.laa.2015.02.037.
  66. Edgar Solomonik and Torsten Hoefler. Sparse Tensor Algebra as a Parallel Programming Model. arXiv:1512.00066, 2015. Google Scholar
  67. Edgar Solomonik, Devin Matthews, Jeff R. Hammond, John F. Stanton, and James Demmel. A massively parallel tensor contraction framework for coupled-cluster computations. J. Parallel Distrib. Comput., 74(12):3176-3190, 2014. URL: http://dx.doi.org/10.1016/j.jpdc.2014.06.002.
  68. Volker Strassen. Gaussian elimination is not optimal. Numer. Math., 13:354-356, 1969. URL: http://dx.doi.org/10.1007/BF02165411.
  69. J. J. Sylvester. On an Application of the New Atomic Theory to the Graphical Representation of the Invariants and Covariants of Binary Quantics, with Three Appendices. Amer. J. Math., 1(1):64-125, 1878. URL: http://dx.doi.org/10.2307/2369436.
  70. Leslie G. Valiant. The complexity of computing the permanent. Theor. Comput. Sci., 8:189-201, 1979. URL: http://dx.doi.org/10.1016/0304-3975(79)90044-6.
  71. Leslie G. Valiant. Holographic Algorithms. SIAM J. Comput., 37(5):1565-1594, 2008. URL: http://dx.doi.org/10.1137/070682575.
  72. Charles Van Loan. Computational Frameworks for the Fast Fourier Transform. SIAM, 1992. URL: http://dx.doi.org/10.1137/1.9781611970999.
  73. Virginia Vassilevska Williams. Multiplying matrices faster than Coppersmith-Winograd. In Proc. of Symposium on Theory of Computing (STOC), pages 887-898, 2012. URL: http://dx.doi.org/10.1145/2213977.2214056.
  74. Virginia Vassilevska Williams and Ryan Williams. Finding, Minimizing, and Counting Weighted Subgraphs. SIAM J. Comput., 42(3):831-854, 2013. URL: http://dx.doi.org/10.1137/09076619X.
  75. Ryan Williams. A new algorithm for optimal 2-constraint satisfaction and its implications. Theor. Comput. Sci., 348(2-3):357-365, 2005. URL: http://dx.doi.org/10.1016/j.tcs.2005.09.023.
  76. Ryan Williams. Faster all-pairs shortest paths via circuit complexity. In Proc. of Symposium on Theory of Computing (STOC), pages 664-673, 2014. URL: http://dx.doi.org/10.1145/2591796.2591811.
  77. Virginia Vassilevska Williams, Joshua R. Wang, Richard Ryan Williams, and Huacheng Yu. Finding Four-Node Subgraphs in Triangle Time. In Proc. of Symposium on Discrete Algorithms (SODA), pages 1671-1680, 2015. URL: http://dx.doi.org/10.1137/1.9781611973730.111.