Introduction to Feedforward Neural Networks
A neural network architecture is specified by (V, E, σ), consisting of the graph and the activation function.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
From Structure to Prediction
A neural network can be viewed as a function that maps inputs to outputs. To describe a collection of possible neural-network predictors, separate two decisions: first choose the network structure, and then choose the numerical weights assigned to its edges. The structure determines the architecture; the weights select one particular function within that architecture.
The central distinction is simple: the architecture is the fixed structure, while the weights are the adjustable parameters that select an individual hypothesis.
Architecture as a Three-Part Description
A neural network architecture is specified by the tuple (V, E, σ). The graph contributes its vertices V and directed edges E. The activation function contributes σ. Together, these describe the structure and activation behavior that are held fixed when defining the architecture.
Reading an Architecture Description
Suppose an architecture is described as (V, E, σ). Identify which part represents the graph and which part represents the activation function.
- 1Identify the graph
The graph is represented by V and E: V denotes the vertices and E denotes the directed edges.
- 2Identify the activation function
The symbol σ denotes the activation function.
- 3Keep the categories separate
V, E, and σ together specify the architecture. Numerical edge weights are not part of this three-part architecture description.
The architecture consists of the graph described by V and E together with the activation function σ.
Tracing a Network Function
To understand the network as a function, follow the directed structure from an input toward an output. The graph specifies which vertices are connected by directed edges, and the activation function is part of the computation associated with the architecture. The result is a mapping from inputs to outputs.
Weights Select One Hypothesis
Weights are assigned to the network edges. They act as the parameters of an individual hypothesis. Once the architecture (V, E, σ) and a particular assignment of weights w are fixed, the tuple (V, E, σ, w) defines one function, written h_V,E,σ,w.
Comparing Two Weight Assignments
Consider one fixed architecture (V, E, σ). Compare two possible weight assignments, w₁ and w₂.
- 1Hold the architecture fixed
The vertices V, directed edges E, and activation function σ remain the same in both cases.
- 2Change the weights
The first network uses w₁, while the second uses w₂.
- 3Name the resulting functions
The first choice defines h_V,E,σ,w₁, and the second defines h_V,E,σ,w₂.
The two weight assignments can specify two different hypotheses even though the architecture is unchanged.
Building the Hypothesis Class
A hypothesis class is formed by fixing the architecture and allowing the edge weights to vary. Every possible choice of weights gives one predictor, and the collection of all such predictors is the hypothesis class H_V,E,σ. The subscript records the architecture components that stay fixed; the varying weights are represented by the individual functions contained in the class.
| Item | What happens to it? | Role |
|---|---|---|
| V | Held fixed | Vertices in the graph |
| E | Held fixed | Directed edges in the graph |
| σ | Held fixed | Activation function |
| w | Allowed to vary | Weights assigned to network edges |
| h_V,E,σ,w | Changes when w changes | One individual hypothesis |
| H_V,E,σ | Collects the resulting functions | Hypothesis class |
The fixed-versus-varying distinction in a neural-network hypothesis class
Reading the Class Notation
Read H_V,E,σ as “the hypothesis class for the architecture specified by V, E, and σ.” Read h_V,E,σ,w as “the individual function obtained from that architecture with the particular weight assignment w.” The notation makes the modeling decision visible: V, E, and σ identify the fixed architecture, while w identifies one choice within the class.
The absence of w from the subscript of H_V,E,σ is meaningful: the class keeps the architecture fixed while collecting the functions produced by varying w.
Mistakes Beginners Make
Common mistake Treating the weights as part of the fixed architecture
What it looks likeDescribing (V, E, σ, w) as the architecture rather than as one fully specified network function
Why it failsThe architecture is specified by (V, E, σ). Adding w identifies one individual hypothesis based on that architecture.
Do this insteadKeep (V, E, σ) fixed when defining the architecture, then use w to select h_V,E,σ,w.
Common mistake Confusing one hypothesis with the hypothesis class
What it looks likeCalling h_V,E,σ,w the entire hypothesis class
Why it failsh_V,E,σ,w denotes one function for one particular weight assignment. H_V,E,σ denotes the collection obtained by varying the weights.
Do this insteadUse h for an individual predictor and H for the collection of predictors.
Common mistake Assuming a changed weight creates a changed graph
What it looks likeConcluding that changing w necessarily changes V or E
Why it failsThe important modeling distinction is that weights can select a different function while the graph and activation function remain fixed.
Do this insteadAsk whether the vertices, edges, or activation function changed. If only w changed, the architecture is still the same.
Common mistake Ignoring the activation function in the architecture description
What it looks likeRepresenting the architecture using only V and E
Why it failsThe architecture is specified by the graph together with the activation function σ.
Do this insteadInclude all three components: V, E, and σ.
When reading a neural-network description, classify every symbol as either structural or parameter-related. V, E, and σ describe the fixed architecture; w describes the edge-weight assignment used to obtain one hypothesis.
Check Your Understanding
A neural-network model uses a fixed architecture (V, E, σ). You are given two weight assignments, w₁ and w₂. State what remains fixed, name the two individual hypotheses, and identify the hypothesis class that contains them.
Hints
- The architecture is the part described by V, E, and σ.
- Attach each weight assignment to the architecture to name one hypothesis.
- The hypothesis class keeps the architecture in its subscript and collects the resulting functions.
What do you think happens?
If the graph and activation function stay fixed but the edge weights change, do you still have the same individual hypothesis?
Reveal answer
No, because the weight assignment selects the individual hypothesis.
The fixed architecture can support multiple hypotheses. Each possible weight assignment w defines an individual function h_V,E,σ,w, and the collection of those functions forms H_V,E,σ.
Summary
- 1A neural-network architecture is specified by (V, E, σ): vertices, directed edges, and an activation function.
- 2Weights are assigned to network edges and act as parameters of an individual hypothesis.
- 3The tuple (V, E, σ, w) defines one function h_V,E,σ,w.
- 4Fixing V, E, and σ while varying w produces the hypothesis class H_V,E,σ.
- 5The architecture determines the fixed structure; the weight assignment selects one function within that structure.
At a glance
- The architecture of a feedforward neural network is represented by (V, E, σ).
- The graph and activation function describe the fixed structure, while edge weights are parameters.
- One weight assignment produces one hypothesis h_V,E,σ,w.
- Varying the weights while keeping the architecture fixed produces the hypothesis class H_V,E,σ.
- The notation distinguishes structural choices from the parameters that select an individual predictor.