Function approximation with neural networks
The central claim is about approximation capacity: a sufficiently large single hidden layer with sigmoid units can approximate any continuous function on a compact input region to any desired accuracy.
From simple mappings to approximation
A neural network can be viewed as a system that maps inputs to outputs. The important question is not merely whether a network produces an output, but what kinds of input-output functions it can represent or approximate. A network with no hidden layers can represent only a very small fraction of possible input-output functions. The universal approximation property is significant because it describes how a sufficiently wide hidden layer with suitable nonlinear units greatly expands that capability.
The central claim is about approximation capacity: a sufficiently large single hidden layer with sigmoid units can approximate any continuous function on a compact input region to any desired accuracy.
Tracing a target function
Suppose a continuous target function is defined over a bounded input region. The universal approximation property says that a feedforward network with one hidden layer can approximate that target as closely as desired when the hidden layer is sufficiently large and its units use sigmoid activation functions. The result does not say that one small network automatically matches the target. It says that a network with enough capacity exists within this architectural family.
Reading the approximation claim
Consider a continuous target function defined on a bounded input region. What does the universal approximation property allow us to claim about a feedforward network with one hidden layer?
- 1Identify the target
The target is a continuous function whose inputs are restricted to a compact, or bounded, region.
- 2Choose the stated architecture
Use a feedforward network with one hidden layer containing sigmoid units.
- 3Apply the capacity condition
The hidden layer must be sufficiently large. The property is not presented as a guarantee for an arbitrarily small network.
- 4State the result
With sufficient size, the network can approximate the target function to any desired accuracy.
The property establishes the existence of a sufficiently large single-hidden-layer network that can approximate the continuous target on the compact input region to any desired accuracy.
Why activation nonlinearity matters
An activation function determines how a unit responds to the activation pattern arriving from the network's input units. The universal approximation result depends on using nonlinear activation functions. Sigmoid units are one stated example, and other nonlinear activation functions can work when they satisfy mild conditions. Nonlinearity is therefore the ingredient that allows hidden units to contribute more than a purely linear input-output relationship.
Adding a hidden layer by itself is not enough. A hidden layer helps expand the class of functions only when its units introduce nonlinearity. This is why the phrase single hidden layer with sigmoid units is important: both the hidden layer and the nonlinear activation behavior are part of the approximation claim.
Tracing a linear stack
Now remove the nonlinear behavior and let every activation function be linear. The network may still contain several layers, but each layer performs a linear transformation. When linear functions are composed with other linear functions, the result remains linear. Consequently, the entire multi-layer network is equivalent to a network with no hidden layers for this purpose.
Several layers, unchanged function class
A feedforward network has several layers, but every unit uses a linear activation function. Does the number of layers by itself make the network capable of representing a richer class of functions?
- 1Inspect each layer
Every layer applies a linear transformation because every activation function is linear.
- 2Compose the layers
The output of one linear transformation becomes the input to the next, and linear functions composed with linear functions remain linear.
- 3Compare with the baseline
The complete stack is equivalent to a network with no hidden layers for the resulting function class.
Several linear layers do not overcome the limitation of a network with no hidden layers. Nonlinearity, rather than layer count alone, is the crucial ingredient.
Capacity is not a guarantee for every network
The wording sufficiently large is essential. The universal approximation property is a capacity claim: it says that a suitable network of sufficient size can approximate any continuous function on the stated compact input region to any desired accuracy. It does not say that every single-hidden-layer network can do so, that every width is sufficient, or that an arbitrary network will automatically approximate a chosen target.
Common mistake Interpreting universal approximation to mean that every network can approximate every continuous function.
What it looks likeA small network is treated as automatically capable of approximating any chosen target.
Why it failsThe result applies to a sufficiently large network, not to every individual network.
Do this insteadState the claim as the existence of a sufficiently large suitable network with one hidden layer.
Common mistake Assuming that adding hidden layers is enough to create complex function representations.
What it looks likeSeveral layers are added while every activation function remains linear.
Why it failsComposing linear functions with linear functions still produces a linear function.
Do this insteadCheck whether the units introduce nonlinearity; layer count alone is not the crucial ingredient.
Common mistake Treating the hidden layer as useful even when its units use only linear activation functions.
What it looks likeA hidden layer is present, so the network is assumed to have the approximation ability in the theorem.
Why it failsThe stated approximation result depends on nonlinear activation functions, with sigmoid units as one example.
Do this insteadInclude suitable nonlinear units when reasoning about the universal approximation property.
Common mistake Forgetting the domain conditions in the approximation statement.
What it looks likeThe property is stated for an arbitrary target without mentioning continuity or the input region.
Why it failsThe source states the result for continuous functions on a compact input region.
Do this insteadInclude both conditions when stating the property.
Practice the distinction
A learner says: A network with five linear layers must be more expressive than a network with no hidden layers, and the universal approximation property proves that every such network can approximate any continuous target. Identify both errors and replace the statement with a source-grounded version.
Hints
- Consider what happens when linear functions are composed.
- Look closely at the words sufficiently large, single hidden layer, sigmoid units, continuous function, and compact input region.
- Separate a claim that a suitable network exists from a claim about every individual network.
What do you think happens?
A network has several layers, but all activation functions are linear. After the layers are composed, what kind of function does the whole network represent?
Reveal answer
A linear function
Linear functions composed with linear functions remain linear, so the stack is equivalent to a network with no hidden layers for this purpose.
What to remember
- 1A sufficiently large single hidden layer with sigmoid units can approximate any continuous function on a compact input region to any desired accuracy.
- 2Nonlinear activation functions are the crucial ingredient behind this approximation ability.
- 3A network with no hidden layers represents only a very small fraction of possible input-output functions.
- 4Several layers with only linear activation functions still compose to a linear function and do not create a richer function class.
- 5The universal approximation property is a capacity claim about the existence of a sufficiently large suitable network, not a guarantee about every individual network.
At a glance
- The universal approximation property concerns sufficiently large single-hidden-layer feedforward networks with sigmoid units.
- The target must be continuous and defined on a compact input region in the stated result.
- Nonlinearity allows hidden layers to expand the class of functions that can be approximated.
- Stacking linear layers does not help because their composition remains linear.
- The property describes what some suitable networks can achieve, not what every individual network automatically achieves.