Concepts / Training Feedforward Neural Networks

Training Feedforward Neural Networks

A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.

  • Programming
Interactive lab

Try it: Neural Network Forward Pass

How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.

How it works

  1. Each hidden neuron computes z = w1·x1 + w2·x2 + b.
  2. It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
  3. The output neuron does the same with the hidden activations as its inputs.
  4. Changing any input, weight or bias changes every value downstream of it.

Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.

Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).

Educational simulation

Loading the simulation…

The One-Way Route

A feedforward neural network provides a one-way route for information. Values enter through the input layer, pass through neurons connected by weighted links, and eventually reach the output layer. To understand how such a network is trained, first understand the structure that information travels through: neurons arranged in layers and connections that point forward without forming cycles.

NeuronnodeWeighted connectiondirected edgeNeuronnodesends outputcarries value
What makes a feedforward neural network a directed acyclic graph?

Layer Order

The full set of neurons is divided into layer sets named V0 through VT. The input layer is V0. Hidden layers lie between the input and output layers, and the output layer is VT. The layer index records processing order: information starts at V0, moves through any hidden layers, and finishes at VT. These names describe each layer's role in the flow rather than merely giving it a position.

V0input layerV1hidden layerVT-1hidden layerVToutput layerforwardforwardforward
How are the layers arranged, and what role does each layer play?
LayerNotationRole in the flow
InputV0Exposes the starting feature values
HiddenV1 through VT-1Performs intermediate transformations
OutputVTContains the network's final output

The layer names describe where information is in the processing route.

Input Representation

For an n-dimensional input space, the described input layer contains n feature neurons and one additional constant neuron. Each feature neuron represents one input feature. The constant neuron always outputs 1. Thus, the input layer does more than hold feature values: it also provides this constant value as part of the information available to the next layer.

Feature 1x1Hidden neuronreceives weighted inputsFeature 2x2Constant1weighted connectionweighted connectionweighted connection
How are input features and the constant neuron represented and connected to the next layer?

Neuron Processing

A neuron receives outputs from earlier neurons through weighted connections. It combines those weighted incoming outputs and then processes the result with an activation function. The activation function therefore sits between a neuron's combined incoming information and the output that the neuron passes onward. The exact activation function is not needed to identify its role: it transforms the combined incoming result into the neuron's output.

Weighted input sumcombined incoming outputsNeuron outputactivation function appliedactivation function
What changes when a neuron's weighted input sum passes through its activation function?

A Small Network Trace

Worked example

Following Two Features to One Output

Trace an illustrative network with two feature neurons, one constant neuron, two hidden neurons, and one output neuron.

  1. 1
    Start at V0

    The input layer contains Feature 1, Feature 2, and the constant neuron. The first two nodes expose the example's feature values, while the constant neuron outputs 1.

  2. 2
    Move to the hidden layer

    Each hidden neuron receives information through weighted connections from the input layer. The hidden neurons combine their incoming weighted outputs and process the results with activation functions.

  3. 3
    Continue forward

    The outputs of the hidden neurons travel through weighted connections to the output neuron. The output neuron again combines its incoming weighted outputs and applies its activation function.

  4. 4
    Arrive at VT

    The output neuron's result is the network's final output at the output layer. No information needs to travel backward or return to an earlier layer in this feedforward route.

Result

The trace is V0 to hidden neurons to VT: feature values and the constant value enter first, hidden neurons perform intermediate transformations, and the output neuron produces the final output.

Feature 1input valueHidden 1intermediate outputOutputfinal outputFeature 2input valueHidden 2intermediate outputConstant1weightedweightedweightedweightedweightedweightedweightedweighted
How does each input value move through hidden neurons and reach the output?
Key point

When tracing a feedforward network, follow the direction of the connections. First identify the values exposed by V0, then track the weighted incoming information received by each hidden neuron, then follow the hidden outputs to VT. The network's graph structure determines this order.

Common Misunderstandings

  • Common mistake

    Treating the network as a loose collection of independent neurons

    What it looks like

    Looking at each neuron separately without tracing the connections between layers.

    Why it fails

    The neurons and their communication links form a directed graph. The links determine how one stage supplies information to the next.

    Do this instead

    Trace both the neurons and the directed weighted connections that carry information forward.

  • Common mistake

    Calling every layer an input or output layer

    What it looks like

    Describing a middle processing layer as an output layer because it produces a value.

    Why it fails

    The input, hidden, and output names describe positions and roles in the complete flow. Hidden layers perform intermediate transformations, while the output layer contains the network's final output.

    Do this instead

    Use V0 for the input layer, V1 through VT-1 for hidden layers, and VT for the output layer.

  • Common mistake

    Counting only the feature neurons in the input layer

    What it looks like

    Forgetting the constant neuron when describing an n-dimensional input.

    Why it fails

    The described input layer contains n feature neurons plus one constant neuron that always outputs 1.

    Do this instead

    List the feature neurons and the additional constant neuron separately.

  • Common mistake

    Treating a neuron's output as just its incoming weighted combination

    What it looks like

    Stopping after combining the weighted incoming outputs.

    Why it fails

    A neuron processes the combined result with an activation function before producing its output.

    Do this instead

    Include the activation-function step when tracing what a neuron sends onward.

  • Common mistake

    Assuming information can move in cycles

    What it looks like

    Following a connection from a later layer back to an earlier layer in a feedforward network.

    Why it fails

    A feedforward neural network is a directed acyclic graph, and its information route moves forward through the layers.

    Do this instead

    Follow the directed route from V0 through hidden layers to VT without returning to an earlier stage.

Practice Trace

MEDIUM

Describe the information route in a feedforward network whose input layer contains three feature neurons and one constant neuron, followed by one hidden layer and an output layer. Name what happens at the input layer, at each hidden neuron, and at the output neuron.

Hints
  • Start with V0 and identify the three feature neurons and the constant neuron.
  • Remember that each neuron combines weighted incoming outputs and processes the result with an activation function.
  • End at VT, where the network's final output is found.

Key Takeaways

  1. 1
    A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.
  2. 2
    The input layer is V0, hidden layers lie between the input and output layers, and the output layer is VT.
  3. 3
    For an n-dimensional input, the described input layer contains n feature neurons plus a constant neuron that always outputs 1.
  4. 4
    A neuron combines weighted incoming outputs and processes the result with an activation function.
  5. 5
    Information moves from the input layer through hidden transformations to the final output layer.

At a glance

  • Feedforward neural networks organize neurons and weighted connections as a directed acyclic graph.
  • The layer sequence is V0 for input, V1 through VT-1 for hidden layers, and VT for output.
  • The input layer represents features and includes a constant neuron that always outputs 1.
  • Each neuron combines weighted incoming outputs and applies an activation function.
  • Tracing the directed connections reveals how information moves from inputs to the final output.

More concepts starting with T