A Universal Uniform Approximation Theorem for Neural Networks

As mentioned, we do not attempt a full survey and we briefly highlight why deep narrow networks have only recently gained attention. Increasing depth to boost performance is intuitive, but only recently became practical after [27]. The main obstacle was the use of gradient-based training: as depth grows – especially beyond hundreds of layers – gradients vanish [21], leaving only the final layers effectively updated during backpropagation.

A key breakthrough was the recent introduction of skip connections, which allow layers to connect beyond their immediate predecessor. Skip connections were popularized about nine years ago by Residual Networks (ResNets) introduced in [27], and by Highway Nets introduced in [51]. It has been shown that this structure allows ResNets, or some considered architectures, to be interpreted as discrete approximations of continuous-time dynamical systems, particularly through the forward Euler method. As a result, such networks inherit desirable properties from numerical methods, including stability, convergence, and robustness. It was later mathematically proven that many efficient models are, in fact, reformulations of discretisation schemes for ODEs. For example, following [39], PolyNet [58] can be viewed as an approximation to the backward Euler scheme for solving $\mathrm{ODE}{u}_t=f(u)$. Fractalnet [36] follows a well-known Runge-Kutta scheme. ResNet [23] corresponds to a forward Euler discretization of a simple dynamical system.

This perspective has also inspired models such as Neural Ordinary Differential Equations [14], in which the dynamics evolve continuously and are governed by the network parameters: ${𝐱}_t^{\prime }=𝐟(\theta ,{𝐱}_t)$. The parameters $\theta$ can be trained using methods such as the adjoint state method, which serves as a continuous-time analogue of backpropagation [14]: see e.g. [30] for a survey. All these models fall within the deep narrow framework considered here. Such approaches have contributed significantly to the widespread use of very deep networks today.

It is known that a deep $\mathrm{ReLU}$ neural network [26] requires a minimum width of at least the input dimension plus one to ensure uniform approximation. The uniform norm requirement can also be relaxed: it is known that width of $d+4$ is sufficient for approximation using the $L^1$-norm [40]. Deep ReLU networks approximate smooth functions more efficiently than shallow networks, as demonstrated by adaptive depth-6 architectures that outperform standard shallow designs [54]. Furthermore, recent work establishes connections between network depth and width [28], while also revealing that neural networks possess even greater approximation power than traditional nonlinear approximation methods [16]. ResNet, even with just one neuron per layer, has been proved to approximate any Lebesgue-integrable function arbitrarily well as the number of tunable weights goes to infinity [37]. Furthermore, [38] establishes that ResNets can be reformulated as feedforward neural networks, enabling the derivation of lower bounds on the number of tunable parameters required for ResNets to approximate different function classes. Universal approximation with general activation functions has been studied in [13, 16, 31, 32, 35, 55, 56, 57], focusing on which types of functions allow for approximation and, in some cases, achieving optimal bounds.

In the mid-1990s, recurrent neural networks (RNNs) were widely used. It is known that if an ideal sigmoid function – often modeled as the saturated linear function, defined by $\sigma (x)=0$ for $x\le 0$, $\sigma (x)=x$ for $0\le x\le 1$, and $\sigma (x)=1$ for $x\ge 1$ is allowed, then such networks can simulate Turing machines using rational weights [47]. The computational power of these networks has also been extensively investigated in the case where real or even non-computable weights are allowed, leading to connections with non-uniform complexity classes such as $P/poly$ [48]. Refer to [46] for an overview. Since the 1990s, research has shifted toward alternative network architectures. Today, architectures using $\mathrm{ReLU}$ activation, defined as $\mathrm{ReLU}(x)=\max (x,0)$ are widely adopted. It is easy to see that deep $\mathrm{ReLU}$ neural networks can simulate ideal sigmoid networks and conversely, as illustrated by the identity $\sigma (x)=\mathrm{ReLU}(x)-\mathrm{ReLU}(x-1)$. Moreover, any continuous piecewise linear function can be represented by a neural network with $\mathrm{ReLU}$ activation [3]. Hence, clearly the previous statements about the possibility of simulating Turing machines hold for such “ideal” activation functions.

While it is known that neural networks can simulate Turing machines – implying universality in the computational sense (at least for ideal sigmoids) – this does not provide statements about their capabilities of universal approximations. The point is that uniform approximation is about “every inputs”, while these statements about their computational capabilities are about “some inputs”. Existing results only show that certain inputs can encode arbitrary Turing machine computations.

In this paper, we investigate neural networks $\mathrm{𝐍𝐍}:{ℝ}^d\to {ℝ}^d$ with matching input and output dimensions, focusing on their behaviour when repeatedly applied to the same input $x\in {ℝ}^d$, as seen in all these contexts of the deep narrow scenario. We prove that uniform approximation is also achievable in the sense of computation theory. Note that we are playing with two distinct notions of uniformity: one refers to the uniform norm, as in the classical uniform approximation theorem for neural networks; the other stems from computability theory, where uniformity means that the approximation is effective – that is, computable. We establish that both hold: we have universality, in the sense that a single, fixed neural network can serve as a universal approximator.

Neural networks are intrinsically computing over real numbers. This leads to various possibilities of considering computability, when stating results: either by restricting to networks with rational coefficients, and seeing them as functions over the rational numbers, or even more generally as functions from the reals to the reals, with possibly real coefficients. In the first case, we are in the framework of classical computability over finite words. In the second case, we are in the framework of computable analysis [11, 12, 53, 33]. Our results work for both, including aspects of time complexity.

First, we recall more formally the uniform approximation theorem in the context of neural networks:

In this statement, the width of the hidden layer can be arbitrary large and depends on both the function being approximated and the precision parameter $ϵ>0$. This result holds in particular for $\mathrm{ReLU}$-networks: a $\mathrm{ReLU}$-Neural-Net or $\mathrm{ReLU}$-Net is a neural network whose activation function $\rho$ is defined as $\mathrm{ReLU}(x)=\max (x,0)$. In this article, we mainly focus on such networks, and for conciseness, we write then $\mathrm{𝐍𝐍}(x)$ for $\mathrm{𝐍𝐍}[\mathrm{ReLU}](x)$. Note that Theorem 1 does not actually require the output and input dimensions to be equal. However, we have chosen this version to be more comparable to our setting (and both problems are mutually reducible). A dyadic is a number $d\in ℝ$ that has a finite binary expansion, meaning it can be expressed as $d=m/2^p$ for some integers $m$ and $p$. In this case, we denote the precision of $d$ by $p=prec(d)$. The set of all dyadic rational numbers (respecitvely of precision $p$) is denoted by $𝔻$ (resp. ${𝔻}_p$). The binary length of a dyadic rational number is defined as $\mathrm{size}(m/2^p)=\mathrm{size}(m)+p$, in consistency with the standard convention for rational numbers.

Let ${𝒞}_{PAF}^{0,\mathbb{Q}}$ (respectively: ${𝒞}_{PAF}^{0,𝔻}$) denote the set of continuous piecewise affine functions with rational coefficients (resp. dyadic coefficients, preserving dyadics). We fix a representation $⟨𝐟⟩$ for functions in ${𝒞}_{PAF}^{0,\mathbb{Q}}$ (resp. ${𝒞}_{PAF}^{0,𝔻}$): this is a surjective function from words over a finite alphabet to ${𝒞}_{PAF}^{0,\mathbb{Q}}$ (resp. ${𝒞}_{PAF}^{0,𝔻}$), so that, as expected, given a rational $𝐱$ and $⟨𝐟⟩$, we can compute $𝐟(𝐱)$ easily (say in polynomial time). We need only this: so, for the coming discussions when talking about circuit complexity, as an alternative to see $⟨𝐟⟩$ as only a description of $𝐟$, this possibly helps to consider that $⟨𝐟⟩$ can also be seen as a description of a circuit to compute $𝐟$.

We emphasize here that while it is known that any function in ${𝒞}_{PAF}^{0,\mathbb{Q}}$ can be approximated by a neural network, we state here that a stronger statement holds: a single neural network architecture is sufficient to approximate all such functions. We have universal uniform approximation theorem.

Notice, that there is “no encoding involved in variable $𝐱$”: we state that there is uniform approximation of $𝐟(𝐱)$ by the function ${\pi }_{[1,d]}({\mathrm{𝐍𝐍}}^{[T]}(𝐱,⟨𝐟⟩,ϵ)$: Assuming only this (with no a priori encoding involved) is precisely one of the main difficulties we solve.

Moreover, this result can be further strengthened within the framework of computable analysis: the class ${𝒞}_{PAF}^{0,𝔻}$ is dense in the space of continuous functions ${𝒞}^0({[0,1]}^d)$ with respect to the uniform norm. That is, any function $𝐟\in {𝒞}^0({[0,1]}^d)$ admits a sequence ${\{{𝐟}_i\}}_{i\ge 0}$ in ${𝒞}_{PAF}^{0,𝔻}$ such that $\Vert 𝐟-{𝐟}_i\Vert \le 2^{-i}$ for all $i\ge 0$. As a result, any function $𝐟$ in ${𝒞}^0({[0,1]}^d)$ can be encoded as the infinite word $⟨𝐟⟩=⟨{𝐟}_0⟩\mathrm{\#}⟨{𝐟}_1⟩\mathrm{\#}⟨{𝐟}_2⟩\mathrm{\dots }$.This infinite word, when interpreted as the binary expansion of a real number in $[0,1]$, is computable as an infinite word if and only if the function $𝐟$ is computable in the sense of computable analysis [11, 12, 53, 33]. Hence, $⟨𝐟⟩$ can be viewed as a (possibly irrational) real number in $[0,1]$ whose digits encode $𝐟$, and this number is computable if and only if $𝐟$ is computable.

In other words, this result establishes uniform approximation with uniformity in the sense of computability for computable analysis: given any description of a function – even if the function itself is not computable – the neural network $\mathrm{𝐍𝐍}$ will approximate it to any prescribed precision, as determined by an input parameter. This goes beyond classical approximation: it guarantees that a single, fixed network architecture can approximate any described function with explicit, parameter-driven control over the precision parameter.

From a complexity-theoretic perspective, beyond mere computability, we prove that the parameter $T$ admits a natural interpretation: for functions in ${𝒞}_{PAF}^{0,\mathbb{Q}}$, $T$ corresponds – up to a polynomial factor – to the size of the family of Boolean circuits that compute $𝐟$ over the dyadic rationals. In this sense, $T$ captures the inherent computational complexity of the function. In the next statement, we say that $𝐟$ is $t$-computable, if we can produce in time $t(p)$ some $2^{-p}$ approximation of $𝐟$ by some function in ${𝒞}_{PAF}^{0,\mathbb{Q}}$:

Unlike previous deep learning results, this is established by computability-theoretic arguments, specifically by simulating Circuit Machines. Furthermore, for more general functions, we provably relate $T$ to the complexity of the function $f$ to be approximated. Using recent results, this $T$ can be connected to the Kolmogorov complexity of $𝐟$ [1, 2].

Our results are obtained by programming with $\mathrm{ReLU}$-nets. While the final conclusions may seem intuitive – essentially that $\mathrm{ReLU}$-nets can be viewed as specific circuit models, and that we have universality – one might wonder why such statements were not well established. We now outline the exact issues we addressed and why they were not immediately apparent from existing work.

In short, we provide a method to map from reals to the world of Turing machines, and conversely, using only $\mathrm{ReLU}$ functions. The problem lies in the uniform approximation, which requires approximation over all inputs (i.e. real, rational and irrational). Restricting to a suitable subset would have been much easier, as existing results from the 1990s already address this case. However, achieving true uniform approximation demands mapping all inputs into a domain where Turing machines can operate effectively.

More precisely, one has to realise first that functions constructed by a $\mathrm{ReLU}$-net must be specific ${𝒞}_{PAF}^{0,\mathbb{Q}}$ functions. In particular, even when iterated a bounded number of times, they always remain Lipschitz continuous. This forbids to obtain some simple functions. For example, the Heaviside function, being discontinuous, is impossible to compute with a $\mathrm{ReLU}$-net in a finite number of iterations. This limitation extends beyond Heaviside, excluding many other functions as well:

That stated, we indeed rely on the well-established result from the 1990s that Turing machines can be simulated by $\mathrm{ReLU}$-nets. However, this leads us to discuss some coding issues, and the need to encode/decode inputs, if we want to come to a settings where we can use these results for uniform approximation.

This issues arises from the nature of the input data type: the input $x$ of a network is a real number, while the Turing machine expects a discrete input (e.g. a word encoding $x$). A central first idea to bridge this gap is to use the binary expansion of $x=0.x_1{x}_2{x}_3\mathrm{\dots }{x}_n$. The Turing machine would then proceed the input bit by bit. However, extracting the first bit of a real number is generally impossible due to discontinuity. For example, consider $x=\frac{1}{2}+\epsilon$, which has first bit $x_1=1$, while $x^{\prime }=\frac{1}{2}-\epsilon$ yields $x_1^{\prime }=0$. For $\epsilon \to 0$, both values converge to $1/2$, but their first bits remain distinct. Thus, a network $\mathrm{𝐍𝐍}$ satisfying $\mathrm{𝐍𝐍}(x)=x_1$ cannot exist as it would necessarily compute a discontinuous function – an operation fundamentally incompatible with the continuity and piecewise affine structure enforced by $\mathrm{ReLU}$-nets. A trick, adapted from Turing machine simulations of the 1990s [47, 48], uses $1,3-quadradics$. The discontinuity issue disappears if we encode $x_1,x_2,x_3\mathrm{\dots },x_n$ as a $1,3-$quadric $\overline{x}=\sum _{i=1}^n(2x_i+1)4^{-i}$. In this encoding, the first bit can be extracted by a function $extract(\overline{x})=x_1$ that equals 0 on $[1/4,1/2]$ and 1 on $[3/4,1]$. Since $extract$ is not constrained in $(1/2,3/4)$, it can be extended arbitrarily on this domain as a continuous, semilinear function – thereby making it compatible with computation by a $\mathrm{ReLU}$ neural network.

Such encoding using $1,3$-quadradics is employed throughout the Turing machine simulations presented in [47, 48, 46], where all problems mentioned above were already present.

In other words, while it is well understood that, within the Turing machine framework (computability theory and computable analysis), we can achieve uniform approximation by mapping the encoding of $x$ to an arbitrarily precise encoding of $f(x)$, this does not, however, immediately imply that we can compute $f(x)$ from $x$ itself for all $x$ from a neural network. The key difficulty lies in converting a real or dyadic number into a suitable $1,3$-quadric encoding that a Turing machine can process: this is exactly what we solve.

This transformation use only $\mathrm{ReLU}$-nets. Our main technical result is stated below:

The proof of the following proposition is similar and simpler.

These constructions yield both decoders and encoders between dyadics and $1,3$-quadradics, with a major advantage: the latter encoding can be tailored to be robust against local errors. One important remaining issue is that they will work for most of the dyadic $d$, but not all.

We overcome this limitation by introducing an error indicator $E$, allowing two decoders to run in parallel and always ensure a correct result. Formally:

Notably, as a side effect, we also prove that universality can be achieved with width $3$ (Theorem 23). To the best of our knowledge, this result is original. Furthermore, the ability to map neural computations onto the Turing machine framework gives rise to numerous variants of the previous statements. In essence, many of these variations become valid as soon as the corresponding operations are feasible on a Turing machine.

Coming back to classical complexity, our results give a way to represent continuous functions using circuits, where the time $T$ is polynomially related to the circuit size $\mathrm{SIZE}$. In this sense, $T$ measures the circuit complexity of the uniform approximation of a function; see [52] for a comprehensive reference on circuit complexity. More generally, several natural measures can be used to quantify the complexity of a function – among them, Kolmogorov complexity. It is commonly defined via the function $\mathrm{C}(x)$, which denotes the length of the shortest binary description $d\in {\{0,1\}}^{\ast }$ such that a fixed universal Turing machine $U$ satisfies $U(d)=x$. If we adopt the slightly different definition of $\mathrm{C}(x)$ and $\mathrm{KT}(x)$ proposed in [2], which departs from the classical formulation, then, Kolmogorov complexity can be directly related to circuit complexity. In this sense, we establish that the time parameter $T$ is polynomially related to both the size of the circuits computing the function (possibly relativized) and to its Kolmogorov complexity: It is known that $\mathrm{KT}(x)\approx \mathrm{SIZE}(x)$ and $C(x)\approx {\mathrm{SIZE}}^H(x)$, where $H$ is the halting problem, and $\approx$ means “polynomially related” [2, 1].

Let all basic commands $C_1,\mathrm{\dots },C_{\mathrm{\ell }}:{[0,1]}^k\times {[0,1]}^k\to {[0,1]}^k$ be labeled with indices $1,\mathrm{\dots },\mathrm{\ell }$ according to the order in which they appear in the algorithm. We first define a command-finding function $c:\{1,\mathrm{\dots },\mathrm{\ell }\}\times {[0,1]}^k\to \{0,\mathrm{\dots },\mathrm{\ell }\}$ which, given the index of the current command and the current state, returns the index of the next command to execute.

Case 1:

The return-command is followed by terminating the network, meaning $c(\mathrm{\ell },𝐱)=0$ for all $𝐱$.

Case 2:

Let $i$ be the index of the last command before entering a while-loop with condition $x_i\ge \beta$, so $i+1$ is the index of the first command in that loop. Let $j$ be the index of the first command after leaving the loop, making $j-1$ the last command in the loop. The command-finding function in this case is defined as:

Case 3:

If-conditions follow the same structure as in Case 2.

Case 4:

All other commands are mapped to their immediate successor, that is, $c(i,𝐱)=i+1$.

The function $c$ defined above can be extended to a continuous piecewise affine map (CPAM) and is therefore computable by a $\mathrm{ReLU}$-net by Theorem 13.

Second, we construct the network ${\mathrm{𝐍𝐍}}_g$. Given input $(i,x_1,\mathrm{\dots },x_k)$, ${\mathrm{𝐍𝐍}}_g$ first computes $c(i,𝐱)$ as well as the outputs of all commands $C_1,\mathrm{\dots },C_{\mathrm{\ell }}$. It then subtracts $1-{\delta }_{i,j}$ from the result of each command $C_j$ where ${\delta }_{i,j}$ denotes the Kronecker delta. In the first layer, this yields the vector $(c(i,𝐱),\mathrm{ReLU}(C_1(𝐱)-(1-{\delta }_{i,1})),\mathrm{ReLU}(C_2(𝐱)-(1-{\delta }_{i,2})),\mathrm{\dots },\mathrm{ReLU}(C_{\mathrm{\ell }}(𝐱)-(1-{\delta }_{i,\mathrm{\ell }})))$. Since $\mathrm{ReLU}(C_j(𝐱)-(1-{\delta }_{i,j}))=0$ for $i\ne j$ and $\mathrm{ReLU}(C_j(𝐱)-(1-{\delta }_{i,j}))=C_j(𝐱)$ for $i=j$, the first layer simplifies to $(c(i,𝐱),0,0,\mathrm{\dots },0,C_i(𝐱),0,\mathrm{\dots },0)$. In the output layer, a component-wise sum over all $C_j(𝐱)-(1-{\delta }_{i,j})$ ensures that the final output is $(c(i,𝐱),C_i(𝐱))$. $◀$ The above statement and its proof may at first appear to be a trivial observation. However, the restriction to the domain ${[0,1]}^k$ instead of ${ℝ}^k$ and the requirement for gaps in conditional queries, are essential for the construction to be valid. In this sense, the difficulty lies in ensuring that all computations are carried out using functions that can be explicitly implemented by $\mathrm{ReLU}$-nets.

The above proof does not aim to optimize the width of the constructed $\mathrm{ReLU}$-nets. We present it in this form as it facilitates the understanding of subsequent arguments. Later, we will apply several techniques to reduce the network width (e.g., in the proof of Theorem 23). Essentially, if we had applied the same optimizations in the above proof, we would obtain feasibility within a width that is at most the maximum width of the involved commands, as established by Lemma 21, by executing the commands sequentially.

To achieve width reduction, we will actually also leverage techniques similar to those used in the proof of the following statement, but sometimes even more optimized.

Observe that the function $f:ℝ\to ℝ$, defined by $f(x)=2\mathrm{ReLU}\left(x-\frac{1}{4}\right)-2\mathrm{ReLU}\left(x-\frac{3}{4}\right)$, can be computed by a $\mathrm{ReLU}$-network. This function satisfies $f^{2p}(x)=0$ for all $x\le \frac{1}{2}-\frac{1}{2^{2p+1}}$, and $f^{2p}(x)=1$ for all $x\ge \frac{1}{2}+\frac{1}{2^{2p+1}}$. Define the function $h(x)=\frac{1}{2}-\mathrm{ReLU}\left(\sigma (x)-\frac{1}{2}\right)-\mathrm{ReLU}\left(\frac{1}{2}-\sigma (x)\right)$. The function $h(x)$ satisfies the property that $h(x)=0$ if and only if $x\notin (0,1)$. Both $\sigma (x)$ and $h(x)$ are computable by neural networks of width 2.

We describe, at a precise level, how to transform a dyadic number in binary form into a $1,3$-quadradic of the same precision.

The step of $d\leftarrow d+a/2$ ensures that the new value of dyadic $d$ is in the middle of the interval of all real numbers, whose binary expansion starts with $0.d_1{d}_2\mathrm{\dots }{d}_n$.

The above algorithm returns $\overline{d}$, but with the digits in reversed order. This issue is easily corrected, as the POP-operation can be performed in base $4$ in constant time. By Lemma 16, the code suffices to show that a recurrent network can implement this transformation.

Note that the above algorithm computes the first $2p$ digits, so it would have returned the same result as long as digits $p$ to $2p$ do not all have the same value. This holds for any input in the interval $[d,d+2^{-(p+1)}-2^{-(2p+1)}]$. Therefore, if the algorithm fails for a non-dyadic $d$, it will still correctly a nearby value $d^{\prime }=d+a/4$, because .

This program has $7$ variables (including the index of the current command). It can hence be implemented by a width 8 $\mathrm{ReLU}$-net, by the techniques used in the proof of Theorem 23 (actually Lemma 21), and above remarks about the fact that $\sigma (x)$ and $h(x)$ can be computed with only one more variable. $◀$

By lack of space, we assume that $x\in ℝ$ has dimension $1$. Furthermore, we assume that $f$ is Lipschitz-continuous with constant $L=1$. If $L\ne 1$, we replace $\epsilon$ by a largest power of 2 smaller than $\epsilon /L$. We also assume that $⟨f⟩$ ends with a unique symbol # whose binary encoding is metrically isolated in the sense that $|⟨\mathrm{\#}⟩-⟨w⟩|\ge 2^{-length(⟨\mathrm{\#}⟩)}$ for every other string $w\in {\mathrm{\Sigma }}^{\ast },w\ne \mathrm{\#}$ that does not contain #. We now argue that the following pseudocode can be implemented within the constraints of Lemma 16 (we assume the decoding back into the dyadic numbers to be done implicitly in the return command; the same holds for the next pseudocode).

Note that at least one of $x$ and $x^{\prime }$ is correctly encoded; without loss of generality, assume it is for $x$. This ensures that $E_x=0$ at all times. Since the computation of $f(x)$ eventually halts, $x-E_x$ becomes static and is $\epsilon$-close to $f(x)$. For $x^{\prime }$, two cases arise:

Case 1:

$x^{\prime }$ is also correctly encoded in base $4$. In this case, $E_x^{\prime }=0$ and by continuity of $f$, $x^{\prime }-E_x^{\prime }$ is also $\epsilon$-close to $f(x)$. Consequently $max\{x-E_x,x^{\prime }-E_x^{\prime }\}$ is $\epsilon$-close to $f(x)$ as well.

Case 2:

$x^{\prime }$ is incorrectly transformed into base 4, which means that $E_x^{\prime }>0$. After $n$ iterations of the last loop, $2^n{E}_x^{\prime }>1$, leading to . This ensures that $max\{x-E_x,x^{\prime }-E_x^{\prime }\}=x$, which is again $\epsilon$-close to $f(x)$.

However, if an error remains nonzero, there is no bound on the runtime: for example, if $E_x$ is very small, it may require an arbitrarily long time before successive multiplications by $2$ make it exceed $1$.

To establish a polynomial bound on the number of network iterations when $x$ is a dyadic number, consider the following simplified version of the above algorithm:

To transform $x$ into base 4, the outer loop runs $p$ times, so the inner loop executes a total of $p^2$ iterations. The transformation of $y$ into base $4$ requires polynomially many steps in the precision of $y$ that we need. The final loop depends polynomially on the complexity $t$ of computing $f(x)$. Thus, Theorem 3 follows directly: instead of extracting the function immediately, we delete the first bits of $⟨f⟩=⟨f_1⟩\mathrm{\#}⟨f_1⟩\mathrm{\#}\mathrm{\dots }\mathrm{\#}⟨f_{p-1}⟩\mathrm{\#}⟨f_p⟩\mathrm{\#}⟨f_{p+1}0⟩\mathrm{\#}\mathrm{\dots }$ until reaching the $n$-th $\mathrm{\#}$, and then extract $f_p$, which, due to of fast convergence is $2^{-p}$-close to the desired function. We then proceed as in Theorem 2. This program as written above has $11$ variables (including the index of the current command). It can hence be implemented by a width $12$ $\mathrm{ReLU}$-net by techniques similar to the proof of Theorem 23. $◀$