Expressive Power of Neural Networks
Neural networks with sign activation can be understood through geometric illustrations.
From Computation to Geometry
A neural network is often described as a sequence of numerical transformations. With sign activation, there is another useful viewpoint: the computation can be understood as dividing the input space into regions and assigning outputs from the set {−1, 1}. For two-dimensional real inputs, these regions can be illustrated geometrically. The key is to follow two stages: hidden neurons first perform geometric tests, and the output neuron then combines their binary results.
Hidden Neurons as Halfspaces
Each neuron in the hidden layer of a depth 2 network implements a halfspace predictor. Geometrically, its computation uses a hyperplane as a boundary. The input space is divided into two sides of that boundary, and the sign activation reports a binary result for the side containing the input. Thus, a hidden neuron is not merely producing an intermediate number: it is testing whether an input lies in one of two geometric regions.
The word halfspace is important because the hidden neuron’s decision is geometric rather than arbitrary. One hyperplane supplies the boundary, and the two sides of that boundary supply the possible regions recognized by the sign-activated neuron. With several hidden neurons, the network has several such tests available for the same input.
Combining Hidden Decisions
The output layer provides the second stage of the geometric interpretation. The single output neuron applies a halfspace to the binary outputs produced by the hidden neurons. In this representation, the output neuron does not inspect the original input space directly. It receives the hidden neurons’ binary decisions and applies another halfspace test to that binary-output space.
A halfspace can implement conjunction, so the output stage can require multiple hidden-layer tests to agree. In the input space, this combination is interpreted as an intersection of halfspaces. The resulting accepted region can therefore be a convex polytope rather than just the region from one individual hidden neuron.
Depth 2 as a Polytope Builder
Counting Faces from Hidden Neurons
A depth 2 network has k hidden neurons and represents a convex polytope under the stated geometric interpretation. How many faces does the polytope have?
- 1Identify the hidden tests
Each hidden neuron implements a halfspace predictor, so the hidden layer supplies k geometric constraints.
- 2Interpret the output stage
The output neuron applies a halfspace to the hidden neurons' binary outputs. This can combine the hidden tests as an intersection of halfspaces.
- 3Use the network-to-polytope relationship
For k hidden neurons, the resulting network can express a convex polytope with k − 1 faces.
The convex polytope represented by the depth 2 network has k − 1 faces.
What do you think happens?
A depth 2 network has 6 hidden neurons and follows the stated relationship between hidden neurons and polytope faces. How many faces does the represented convex polytope have?
Reveal answer
5
The relationship is k − 1 faces for k hidden neurons. Substituting k = 6 gives 6 − 1 = 5 faces.
Changing Hidden-Layer Capacity
| Network description | Geometric interpretation |
|---|---|
| One hidden neuron | One halfspace predictor |
| Several hidden neurons | Several halfspace tests whose results can be combined |
| Depth 2 network with k hidden neurons | An intersection-of-halfspaces representation that can express a convex polytope with k − 1 faces |
The geometric interpretation connects hidden-neuron tests to the shape represented by the network.
Increasing the number of hidden neurons increases the number of halfspace predictors available to the network. The hidden layer can therefore supply more geometric tests for the output stage to combine. In the specific depth 2 relationship described here, k hidden neurons correspond to a convex polytope with k − 1 faces. This gives a direct way to connect a network-size question with a geometric face-count question.
When analyzing a sign-activated network geometrically, separate the two stages. First identify the halfspace test performed by each hidden neuron. Then examine how the output neuron combines the resulting binary values. This prevents the common mistake of treating the entire network as one unexplained boundary.
Common Interpretation Errors
Common mistake Treating a hidden neuron as the complete polytope.
What it looks likeOne hidden neuron is interpreted as producing the entire final shape.
Why it failsA hidden neuron implements one halfspace predictor. The final geometric region comes from combining hidden-layer results through the output neuron.
Do this insteadInterpret each hidden neuron as one geometric test, then analyze their intersection as produced by the output stage.
Common mistake Ignoring the binary nature of sign-activated hidden outputs.
What it looks likeThe hidden layer is described only as producing arbitrary numerical features.
Why it failsThe geometric interpretation in the source uses binary outputs in {−1, 1}.
Do this insteadTrack each hidden neuron as returning one of the two sign outputs associated with the two sides of its boundary.
Common mistake Counting k faces for k hidden neurons.
What it looks likeA network with 6 hidden neurons is said to represent a polytope with 6 faces.
Why it failsThe stated relationship for the described depth 2 network is k − 1 faces.
Do this insteadSubstitute the number of hidden neurons into k − 1. For 6 hidden neurons, the count is 5 faces.
Common mistake Describing the output neuron as operating directly on the original input coordinates.
What it looks likeThe output stage is explained as if it independently redraws every hidden boundary in the input space.
Why it failsThe output neuron applies a halfspace to the binary outputs produced by the hidden neurons.
Do this insteadExplain the output neuron as the second-stage combiner of hidden-layer decisions.
Check Your Understanding
A depth 2 network uses sign activation and has 9 hidden neurons. Explain the geometric role of one hidden neuron, describe what the output neuron does with the hidden neurons’ binary results, and determine the number of faces in the convex polytope represented by the network under the stated relationship.
Hints
- Begin by identifying the hyperplane and the two halfspaces associated with one hidden neuron.
- The output neuron applies a halfspace to the hidden neurons’ binary outputs and can implement their conjunction.
- Use k − 1 with k = 9.
- 1A sign-activated neural network can be read as a geometric construction. Each hidden-layer neuron is a halfspace predictor: a hyperplane divides the input space, and the sign output identifies one of the two sides. The output neuron applies a halfspace to the hidden neurons’ binary outputs, allowing their tests to be combined as an intersection of halfspaces. For the described depth 2 network, k hidden neurons can represent a convex polytope with k − 1 faces.
Key Takeaways
- Sign activation allows a neural network to be interpreted through regions and shapes in the input space.
- Each hidden-layer neuron in the depth 2 network acts as a halfspace predictor.
- The output neuron applies a halfspace to the hidden neurons’ binary outputs and can combine their tests as an intersection.
- For k hidden neurons, the described network can express a convex polytope with k − 1 faces.