Document Open Access Logo

The Hierarchy of Manifolds in a Stratification of the Set of Equivalent Linear Neural Networks

Authors Jonathan Richard Shewchuk , Sagnik Bhattacharya

PDF

File

Thumbnail PDF
PDF
LIPIcs.SoCG.2026.91.pdf
  • Filesize: 1.1 MB
  • 19 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Computing methodologies → Neural networks
  • Mathematics of computing → Nonlinear equations
  • Theory of computation → Computational geometry
Keywords
  • Linear neural network
  • real algebraic variety
  • stratification
  • multilinear algebra
  • product of matrices
  • persistence barcode
  • real algebraic geometry
  • discrete geometry

Metrics

Abstract

A linear neural network computes a linear transformation of its input vector. Given a fully-connected linear network, the set of all weight vectors for which the network computes the same linear transformation is an algebraic variety in weight space, called a fiber under the matrix multiplication map. Sometimes this variety is a manifold, but usually not. The rank stratification of a fiber is a natural partition of the fiber into manifolds of various dimensions called strata. We characterize how these strata are connected to each other. They satisfy the frontier condition: if a stratum intersects the closure of another stratum, then the former stratum is a subset of the closure of the latter stratum. This subset relationship can be expressed as a partial order with a single minimal element. Our main result describes the relationship between this partial order and the ranks of certain matrices in the network. Each stratum represents a different pattern of information flow through the network, expressed as a barcode. Connections among the strata are best understood through simple transformations of the barcodes called barcode moves.

Cite As Get BibTex

Jonathan Richard Shewchuk and Sagnik Bhattacharya. The Hierarchy of Manifolds in a Stratification of the Set of Equivalent Linear Neural Networks. In 42nd International Symposium on Computational Geometry (SoCG 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 367, pp. 91:1-91:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.SoCG.2026.91

Author Details

Jonathan Richard Shewchuk
  • University of California, Berkeley, CA, USA
Sagnik Bhattacharya
  • University of California, Berkeley, CA, USA

Funding

Supported in part by the National Science Foundation under Award CCF-1909204.

Acknowledgements

We thank Marc Khoury for initiating this research collaboration, and we thank Herbert Edelsbrunner, Jeff Erickson, Marc Glisse, and Ling Zhou for conversations helping us to understand the relationship between this work and persistence modules.

References

  1. Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 244-253, Stockholm, Sweden, July 2018. URL: https://proceedings.mlr.press/v80/arora18a.html.
  2. Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit Regularization in Deep Matrix Factorization. In Advances in Neural Information Processing Systems, volume 32, pages 7411-7422, 2019. URL: https://proceedings.neurips.cc/paper/2019/hash/c0c783b5fc0d7d808f1d14a6e9c8280d-Abstract.html.
  3. Frédéric Chazal, Vin de Silva, Marc Glisse, and Steve Oudot. The Structure and Stability of Persistence Modules. Springer, 2016. URL: https://doi.org/10.1007/978-3-319-42545-0.
  4. Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun. The Loss Surfaces of Multilayer Networks. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 192-204, San Diego, California, USA, May 2015. URL: https://proceedings.mlr.press/v38/choromanska15.html.
  5. Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, and Holger Rauhut. Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank. Applied and Computational Harmonic Analysis, 68:101595, 2024. URL: https://doi.org/10.1016/j.acha.2023.101595.
  6. Franklin C. Crow. Summed-Area Tables for Texture Mapping. In Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '84), pages 207-212, Minneapolis, Minnesota, July 1984. URL: https://doi.org/10.1145/800031.808600.
  7. Herbert Edelsbrunner and John Harer. Computational Topology: An Introduction. American Mathematical Society, 2010. URL: http://www.ams.org/bookstore-getitem/item=MBK-69.
  8. Herbert Edelsbrunner, David Letscher, and Afra Zomorodian. Topological Persistence and Simplification. Discrete & Computational Geometry, 28(4):511-533, November 2002. URL: https://doi.org/10.1007/s00454-002-2885-2.
  9. Robert Ghrist. Barcodes: The Persistent Topology of Data. Bulletin of the American Mathematical Society, 45(1):61-75, 2008. URL: https://doi.org/10.1090/S0273-0979-07-01191-3.
  10. Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit Regularization in Matrix Factorization. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6152-6160, 2017. URL: https://doi.org/10.1109/ITA.2018.8503198.
  11. Moritz Hardt and Tengyu Ma. Identity Matters in Deep Learning, July 2018. URL: https://arxiv.org/abs/1611.04231.
  12. Ziwei Ji and Matus Telgarsky. Gradient Descent Aligns the Layers of Deep Linear Networks. In 7th International Conference on Learning Representations, New Orleans, Louisiana, May 2019. URL: https://openreview.net/forum?id=HJflg30qKX.
  13. Kenji Kawaguchi. Deep Learning without Poor Local Minima. In Proceedings of the 30th International Conference on Neural Information Processing Systems, volume 29, pages 586-594, 2016. URL: https://proceedings.neurips.cc/paper/2016/hash/f2fc990265c712c49d51a18a32b39f0c-Abstract.html.
  14. Qianyi Li and Haim Sompolinsky. Statistical Mechanics of Deep Linear Neural Networks: The Backpropagating Kernel Renormalization. Physical Review X, 11:031059, September 2021. URL: https://doi.org/10.1103/PhysRevX.11.031059.
  15. Clara Löh. A Comment on the Structure of Graded Modules over Graded Principal Ideal Domains in the Context of Persistent Homology, February 2023. URL: https://arxiv.org/abs/2301.11756.
  16. Stanisław Łojasiewicz. Ensembles Semi-Analytiques. Unpublished manuscript. A version typeset by Michel Coste is available at https://perso.univ-rennes1.fr/michel.coste/Lojasiewicz.pdf, July 1970.
  17. Haihao Lu and Kenji Kawaguchi. Depth Creates No Bad Local Minima, May 2017. URL: https://arxiv.org/abs/1702.08580.
  18. John Mather. Notes on Topological Stability. Lecture Notes. Reprinted in Bulletin of the American Mathematical Society 49(4):475-506, October 2012, 1970. URL: https://doi.org/10.1090/S0273-0979-2012-01383-6.
  19. Adityanarayanan Radhakrishnan, Eshaan Nichani, Daniel Irving Bernstein, and Caroline Uhler. On Alignment in Deep Linear Neural Networks, June 2020. URL: https://arxiv.org/abs/2003.06340v2.
  20. Jonathan Richard Shewchuk and Sagnik Bhattacharya. The Geometry of the Set of Equivalent Linear Neural Networks, April 2024. URL: https://doi.org/10.48550/arXiv.2404.14855.
  21. Gilbert Strang. The Fundamental Theorem of Linear Algebra. The American Mathematical Monthly, 100(9):848-855, November 1993. URL: https://doi.org/10.1080/00029890.1993.11990500.
  22. René Thom. La Stabilité Topologique des Applications Polynomiales. L'Enseignement Mathématique, 8(1-2):24-33, 1962. Google Scholar
  23. Matthew Trager, Kathlén Kohn, and Joan Bruna. Pure and Spurious Critical Points: A Geometric Study of Linear Networks. In Eighth International Conference on Learning Representations, Addis Ababa, Ethiopia, April 2020. URL: https://openreview.net/forum?id=rkgOlCVYvB.
  24. Hassler Whitney. Elementary Structure of Real Analytic Varieties. Annals of Mathematics, 66(2):545-556, 1957. URL: https://doi.org/10.2307/1969908.
  25. Hassler Whitney. Local Properties of Analytic Varieties. In Stewart Scott Cairns, editor, Differential and Combinatorial Topology, pages 205-244. Princeton University Press, 1965. URL: https://doi.org/10.1515/9781400874842-014.
  26. Hassler Whitney. Tangents to an Analytic Variety. Annals of Mathematics, 81(2):496-549, 1965. URL: https://doi.org/10.2307/1970400.
  27. Li Zhang. Depth Creates No More Spurious Local Minima, January 2020. URL: https://arxiv.org/abs/1901.09827.
  28. Afra Zomorodian and Gunnar E. Carlsson. Computing Persistent Homology. In Proceedings of the Twentieth Annual Symposium on Computational Geometry, pages 347-356, Brooklyn, New York, June 2004. URL: https://doi.org/10.1145/997817.997870.
  29. Afra Zomorodian and Gunnar E. Carlsson. Computing Persistent Homology. Discrete & Computational Geometry, 33(2):249-274, February 2005. URL: https://doi.org/10.1007/s00454-004-1146-y.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact