Document Open Access Logo

A Universal Uniform Approximation Theorem for Neural Networks

Authors Olivier Bournez , Johanne Cohen , Adrian Wurm

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.MFCS.2025.29.pdf
  • Filesize: 0.86 MB
  • 20 pages
HTML5 Logo HTML (experimental)
LIPIcs.MFCS.2025.29.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Models of computation
  • Theory of computation → Computability
  • Theory of computation → Complexity classes
  • Computing methodologies → Neural networks
Keywords
  • Models of computation
  • Complexity theory
  • Formal neural networks

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

We show the existence of a fixed recurrent network capable of approximating any computable function with arbitrary precision, provided that an encoding of the function is given in the initial input. While uniform approximation over a compact domain is a well-known property of neural networks, we go further by proving that our network ensures effective uniform approximation - simultaneously ensuring:  
- Uniform approximation in the sup-norm sense, guaranteeing precision across the compact domain {[0,1]^d}; 
- Uniformity in the sense of computability theory (also referred to as effectivity or universality), meaning the same network works for all computable functions.  Our result is obtained constructively, using original arguments. Moreover, our construction bridges computation theory with neural network approximation, providing new insights into the fundamental connections between circuit complexity and function representation. 
Furthermore, this connection extends beyond computability to complexity theory. The obtained network is efficient: if a function is computable or approximable in polynomial time in the Turing machine model, then the network requires only a polynomial number of recurrences or iterations to achieve the same level of approximation, and conversely. Moreover, the recurrent network can be assumed to be very narrow, strengthening the link our results and existing models of very deep learning, where uniform approximation properties have already been established.

Cite As Get BibTex

Olivier Bournez, Johanne Cohen, and Adrian Wurm. A Universal Uniform Approximation Theorem for Neural Networks. In 50th International Symposium on Mathematical Foundations of Computer Science (MFCS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 345, pp. 29:1-29:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.MFCS.2025.29

Author Details

Olivier Bournez
  • Institut Polytechnique de Paris, Ecole Polytechnique, LIX, 91128 Palaiseau Cedex, France
Johanne Cohen
  • Équipe GALaC, LISN, CNRS, Université Paris-Saclay, 91190 Gif-sur-Yvette, France
Adrian Wurm
  • Computer Science Institute, BTU Cottbus-Senftenberg, Cottbus, Germany

Funding

O. Bournez and J. Cohen received support from the Programme Gaspard Monge (PGMO). O. Bournez received support from ANR Project OTC ANR-24-CE48-0335. A. Wurm received partial support from the BTU Graduate Research School (Conference Travel Grant).

References

  1. Eric Allender. Circuit complexity, Kolmogorov complexity, and prospects for lower bounds. In DCFS, pages 7-13. Citeseer, 2008. Google Scholar
  2. Eric Allender, Harry Buhrman, Michal Kouckỳ, Dieter Van Melkebeek, and Detlef Ronneburger. Power from random strings. SIAM Journal on Computing, 35(6):1467-1493, 2006. URL: https://doi.org/10.1137/050628994.
  3. Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491, 2016. URL: https://arxiv.org/abs/1611.01491.
  4. Midhun T Augustine. A survey on universal approximation theorems. arXiv preprint arXiv:2407.12895, 2024. URL: https://doi.org/10.48550/arXiv.2407.12895.
  5. Daniel Bertschinger, Christoph Hertrich, Paul Jungeblut, Tillmann Miltzow, and Simon Weber. Training Fully Connected Neural Networks is ∃ℝ-Complete. Advances in Neural Information Processing Systems (NeurIPS 2023), 36:36222-36237, 2023. Google Scholar
  6. Manon Blanc and Olivier Bournez. A characterisation of functions computable in polynomial time and space over the reals with discrete ordinary differential equations: Simulation of Turing machines with analytic discrete ODEs. In Jérôme Leroux, Sylvain Lombardy, and David Peleg, editors, 48th International Symposium on Mathematical Foundations of Computer Science, MFCS 2023, August 28 to September 1, 2023, Bordeaux, France, volume 272 of LIPIcs, pages 21:1-21:15. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. URL: https://doi.org/10.4230/LIPIcs.MFCS.2023.21.
  7. Vincent D. Blondel, Olivier Bournez, Pascal Koiran, and John Tsitsiklis. The stability of saturated linear dynamical systems is undecidable. Journal of Computer and System Science, 62(3):442-462, May 2001. URL: https://doi.org/10.1006/jcss.2000.1737.
  8. Digvijay Boob, Santanu S Dey, and Guanghui Lan. Complexity of training ReLU neural network. Discrete Optimization, 44:100620, 2022. URL: https://doi.org/10.1016/J.DISOPT.2020.100620.
  9. Olivier Bournez and Michel Cosnard. On the computational power of dynamical systems and hybrid systems. Theoretical Computer Science, 168(2):417-459, November 1996. URL: https://doi.org/10.1016/S0304-3975(96)00086-2.
  10. Cornelius Brand, Robert Ganian, and Mathis Rocton. New complexity-theoretic frontiers of tractability for neural network training. Advances in Neural Information Processing Systems, 36:56456-56468, 2023. Google Scholar
  11. Vasco Brattka. Computability & complexity in analysis. Tutorial donné à Computability in Europe (CIE'2005), 2005. URL: https://doi.org/10.1016/j.jco.2006.10.001.
  12. Vasco Brattka, Peter Hertling, and Klaus Weihrauch. A tutorial on computable analysis. In New computational paradigms, pages 425-491. Springer, 2008. URL: https://doi.org/10.1007/978-0-387-68546-5_18.
  13. Yongqiang Cai. Achieve the minimum width of neural networks for universal approximation. In The Eleventh International Conference on Learning Representations, 2023. URL: https://openreview.net/forum?id=hfUJ4ShyDEU.
  14. Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems, pages 6571-6583, 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/69386f6bb1dfed68692a24c8686939b9-Abstract.html.
  15. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989. URL: https://doi.org/10.1007/BF02551274.
  16. Ingrid Daubechies, Ronald DeVore, Simon Foucart, Boris Hanin, and Guergana Petrova. Nonlinear approximation and (deep) ReLU networks. Constructive Approximation, 55(1):127-172, 2022. Google Scholar
  17. Vincent Froese and Christoph Hertrich. Training neural networks is np-hard in fixed dimension. Advances in Neural Information Processing Systems, 36:44039-44049, 2023. Google Scholar
  18. Vincent Froese, Christoph Hertrich, and Rolf Niedermeier. The computational complexity of ReLU network training parameterized by data dimensionality. Journal of Artificial Intelligence Research, 74:1775-1790, 2022. URL: https://doi.org/10.1613/JAIR.1.13547.
  19. Vincent Froese, Christoph Hertrich, and Rolf Niedermeier. The computational complexity of ReLU network training parameterized by data dimensionality. Journal of Artificial Intelligence Research, 74:1775-1790, 2022. URL: https://doi.org/10.1613/JAIR.1.13547.
  20. Robert Ganian, Mathis Rocton, and Simon Wietheger. Training one-dimensional graph neural networks is NP-hard. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. Google Scholar
  21. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010. URL: http://proceedings.mlr.press/v9/glorot10a.html.
  22. Surbhi Goel, Adam Klivans, Pasin Manurangsi, and Daniel Reichman. Tight hardness results for training depth-2 ReLU networks. In 12th Innovations in Theoretical Computer Science Conference (ITCS 2021), volume 185 of Leibniz International Proceedings in Informatics, pages 22:1-22:14, 2021. URL: https://doi.org/10.4230/LIPICS.ITCS.2021.22.
  23. Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual network: Backpropagation without storing activations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, volume 30, pages 2214-2224, 2017. URL: https://proceedings.neurips.cc/paper/2017/hash/f9be311e65d81a9ad8150a60844bb94c-Abstract.html.
  24. Riccardo Gozzi. Analog Characterization of Complexity Classes. PhD thesis, Instituto Superior Técnico, Lisbon, Portugal and University of Algarve, Faro, Portugal, 2022. Google Scholar
  25. Daniel S. Graça, Manuel L. Campagnolo, and J. Buescu. Robust simulations of Turing machines with analytic maps and flows. In S. B. Cooper, B. Löwe, and L. Torenvliet, editors, CiE 2005: New Computational Paradigms, LNCS 3526, pages 169-179. Springer, 2005. URL: https://doi.org/10.1007/11494645_21.
  26. Boris Hanin and Mark Sellke. Approximating continuous functions by ReLU nets of minimal width. arXiv preprint arXiv:1710.11278, 2017. URL: https://arxiv.org/abs/1710.11278.
  27. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. URL: https://doi.org/10.1109/cvpr.2016.90.
  28. Ruiyang Hong and Anastasis Kratsios. Bridging the gap between approximation and learning via optimal approximation by ReLU MLPS of maximal regularity. arXiv preprint arXiv:2409.12335, 2024. URL: https://doi.org/10.48550/arXiv.2409.12335.
  29. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251-257, 1991. URL: https://doi.org/10.1016/0893-6080(91)90009-T.
  30. Patrick Kidger. On Neural Differential Equations. PhD thesis, Mathematical Institute, University of Oxford, 2022. URL: https://doi.org/10.48550/arXiv.2202.02435.
  31. Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 2306-2327. PMLR, 2020. URL: http://proceedings.mlr.press/v125/kidger20a.html.
  32. Namjun Kim, Chanho Min, and Sejun Park. Minimum width for universal approximation using ReLU networks on compact domain. In The Twelfth International Conference on Learning Representations, 2024. URL: https://openreview.net/forum?id=dpDw5U04SU.
  33. Ker-I Ko. Complexity Theory of Real Functions. Progress in Theoretical Computer Science. Birkhaüser, Boston, 1991. URL: https://doi.org/10.1007/978-1-4684-6802-1.
  34. Pascal Koiran, Michel Cosnard, and Max Garzon. Computability with low-dimensional dynamical systems. Theoretical Computer Science, 132(1-2):113-128, September 1994. URL: https://doi.org/10.1016/0304-3975(94)90229-1.
  35. Anastasis Kratsios and Léonie Papon. Universal approximation theorems for differentiable geometric deep learning. Journal of Machine Learning Research, 23(196):1-73, 2022. URL: https://jmlr.org/papers/v23/21-0716.html.
  36. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. ICLR, 2016. Google Scholar
  37. Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is a universal approximator. Advances in neural information processing systems, 31, 2018. Google Scholar
  38. Chenghao Liu, Enming Liang, and Minghua Chen. Characterizing resnet’s universal approximation capability. In Forty-first International Conference on Machine Learning, 2024. Google Scholar
  39. Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings, pages 3276-3285. PMLR, OpenReview.net, 2018. URL: https://openreview.net/forum?id=B1LatDVUM.
  40. Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017. Google Scholar
  41. Cristopher Moore. Generalized shifts: unpredictability and undecidability in dynamical systems. Nonlinearity, 4(3):199-230, 1991. URL: https://doi.org/10.1088/0951-7715/4/2/002.
  42. Anirbit Mukherjee, Amitabh Basu, Raman Arora, and Poorya Mianjy. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations (ICLR 2018), 2018. Google Scholar
  43. Ian Parberry. Circuit complexity and neural networks. MIT press, 1994. URL: https://doi.org/10.7551/mitpress/1836.001.0001.
  44. Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143-195, 1999. Google Scholar
  45. Amaury Pouly. Continuous models of computation: from computability to complexity. PhD thesis, Ecole Polytechnique and Unidersidade Do Algarve, defended on july 6, 2015 2015. https://pastel.archives-ouvertes.fr/tel-01223284, Prix de Thèse de l'Ecole Polytechnique 2016, Ackermann Award 2017. Google Scholar
  46. Hava T. Siegelmann. Neural networks and analog computation - beyond the Turing limit. Progress in theoretical computer science. Birkhäuser, 1999. Google Scholar
  47. Hava T. Siegelmann and Eduardo D. Sontag. Turing computability with neural nets. Applied Mathematics Letters, 4(6):77-80, 1991. URL: https://doi.org/10.1016/0893-9659(91)90080-f.
  48. Hava T. Siegelmann and Eduardo D. Sontag. Analog computation via neural networks. Theoretical Computer Science, 131(2):331-360, September 1994. URL: https://doi.org/10.1016/0304-3975(94)90178-3.
  49. Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1):132-150, February 1995. URL: https://doi.org/10.1006/jcss.1995.1013.
  50. Michael Sipser. Introduction to the Theory of Computation. PWS Publishing Company, 1997. Google Scholar
  51. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015. URL: https://arxiv.org/abs/1505.00387.
  52. Heribert Vollmer. Introduction to circuit complexity: a uniform approach. Springer Science & Business Media, 1999. Google Scholar
  53. Klaus Weihrauch. Computable Analysis - An Introduction. Texts in Theoretical Computer Science. An EATCS Series. Springer, 2000. URL: https://doi.org/10.1007/978-3-642-56999-9.
  54. Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural networks, 94:103-114, 2017. URL: https://doi.org/10.1016/J.NEUNET.2017.07.002.
  55. Dmitry Yarotsky. Elementary superexpressive activations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11932-11940. PMLR, 18-24 July 2021. URL: https://proceedings.mlr.press/v139/yarotsky21a.html.
  56. Dmitry Yarotsky. Structure of universal formulas. Advances in Neural Information Processing Systems, 36:54876-54902, 2023. Google Scholar
  57. Shijun Zhang, Jianfeng Lu, and Hongkai Zhao. Deep network approximation: Beyond ReLU to diverse activation functions. Journal of Machine Learning Research, 25(35):1-39, 2024. URL: https://jmlr.org/papers/v25/23-0912.html.
  58. Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. Polynet: A pursuit of structural diversity in very deep networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3900-3908. IEEE Computer Society, 2017. URL: https://doi.org/10.1109/CVPR.2017.415.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact