Weights and Biases in Neural Networks

A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.

Programming

Interactive lab

Try it: Neural Network Forward Pass

How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.

How it works

Each hidden neuron computes z = w1·x1 + w2·x2 + b.
It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
The output neuron does the same with the hidden activations as its inputs.
Changing any input, weight or bias changes every value downstream of it.

Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.

Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).

Educational simulation

Loading the simulation…

A One-Way Computational Route

A feedforward neural network is best understood as a one-way route for information. Values enter through the input layer, travel along directed weighted connections, pass through one or more hidden layers, and eventually reach the output layer. The network is not a loose collection of independent neurons: its neurons are nodes in a directed acyclic graph, and its connections determine how one stage supplies information to the next.

What contains what in a feedforward network, and how do directed weighted connections link neurons without forming cycles?

Layer Order and Information Flow

The full neuron set is divided into layer sets V0 through VT. V0 is the input layer. The hidden layers are the layers between the input and output layers, represented as V1 through VT-1. VT is the output layer. The index records processing order: information starts in V0, moves through the hidden layers in order, and finishes in VT.

How are the input, hidden, and output layers positioned and distinguished in the network?

Layer	Role in the flow
Input layer V0	Exposes the starting feature values and the constant value
Hidden layers V1 through VT-1	Perform intermediate transformations
Output layer VT	Contains the network's final output

The layer names describe positions and roles in the computation.

Features and the Constant Neuron

For an n-dimensional input space, the described input layer contains n feature neurons plus one additional constant neuron. Each feature neuron represents one input feature. The constant neuron always outputs 1. Because it supplies a fixed value through a weighted connection, it gives the network a constant contribution to a neuron's combined input; this is the role associated with the bias contribution in this representation.

How are individual input features and the constant value represented in the input layer?

Worked example

Counting the Input Neurons

Consider an input space with n features. What belongs in the input layer?

1
Represent the features
Place one feature neuron in the input layer for each of the n input features.
2
Add the constant neuron
Add one more neuron whose output is always 1.
3
Interpret the extra connection
The constant neuron's weighted connection supplies a fixed contribution to downstream neurons.

Result

The described input layer contains n feature neurons plus one constant neuron.

From Weighted Inputs to Neuron Output

A neuron receives outputs from earlier neurons through weighted connections. It combines those weighted incoming outputs and then processes the combined result with an activation function. The activation function therefore sits between the neuron's incoming information and the output that it passes to the next layer. The constant neuron's weighted contribution is included in the same combination as the other incoming contributions.

How do weighted inputs and the constant contribution combine before a neuron produces its output?

What changes when a neuron applies its activation function to the combined weighted input?

Tracing a Layered Network

Use a small network with an input layer, one hidden layer, and an output layer. The input layer exposes its feature values and the constant value. Directed weighted connections carry those outputs to hidden neurons. Each hidden neuron combines its incoming weighted outputs, applies its activation function, and sends its result forward. The output neuron then receives the hidden outputs, combines them through its own weighted connections, applies its activation function, and produces the final network output.

How does an input value move from the input layer through hidden neurons to the output neuron?

Worked example

Following One Value Forward

Trace the route of an input feature through a network with one hidden layer.

1
Start at V0
The feature value is represented by a neuron in the input layer, alongside the constant neuron that outputs 1.
2
Reach the hidden layer
Directed weighted connections carry the feature output and the constant contribution to hidden neurons.
3
Process at a hidden neuron
Each hidden neuron combines its weighted incoming outputs and processes the combination with an activation function.
4
Continue to VT
The hidden outputs travel through directed weighted connections to the output neuron.
5
Produce the final output
The output neuron performs the same general pattern: combine weighted incoming outputs, apply an activation function, and expose the result in the output layer.

Result

The value moves forward layer by layer; it does not travel backward or enter a cycle.

What do you think happens?

If a value begins in the input layer, can it skip the ordered feedforward route and return to an earlier layer before reaching the output layer?

Yes, because neurons operate independently
Yes, because directed connections can form cycles
No, the network is organized as a directed acyclic graph with forward layer order

Reveal answer

Answer:

No, the network is organized as a directed acyclic graph with forward layer order.

The input layer starts the route, hidden layers lie between input and output, and the output layer finishes it. The directed acyclic structure prevents the information route from forming a cycle.

Mistakes in Reading the Graph

Common mistake
Treating the network as a collection of independent neurons
What it looks like
Describing a hidden neuron without considering the directed connections that supply its inputs.
Why it fails
The network's computation depends on the graph formed by neurons and weighted communication links.
Do this instead
Trace both the neurons and the directed connections between layers.
Common mistake
Confusing the input layer with the first hidden layer
What it looks like
Calling every neuron before the output neuron a hidden neuron.
Why it fails
The input layer V0 has the special role of representing starting features and the constant value.
Do this instead
Identify V0 first, then identify the layers between V0 and VT as hidden layers.
Common mistake
Forgetting the constant neuron
What it looks like
Counting only the n feature neurons in an n-dimensional input space.
Why it fails
The described input layer contains n feature neurons plus one constant neuron that always outputs 1.
Do this instead
Include the constant neuron when describing the input layer.
Common mistake
Stopping before the activation function
What it looks like
Treating the combined weighted incoming value as the neuron's final output.
Why it fails
A neuron processes the combined incoming result with an activation function before sending its output onward.
Do this instead
Separate the combination step from the activation step.
Common mistake
Assuming layer names are merely labels
What it looks like
Ignoring the order represented by V0 through VT.
Why it fails
The layer index records the processing order of the feedforward route.
Do this instead
Read the layer index as a guide to where information is in the computation.

Practice the Forward Route

MEDIUM

Describe the route through a network with input layer V0, hidden layers V1 and V2, and output layer V3. Include the feature neurons, the constant neuron, the weighted connections, and the activation step at a neuron in V1 and at the output layer.

Hints

Begin with the role of V0 and state what the constant neuron outputs.
Name the order in which V1, V2, and V3 receive information.
At each receiving neuron, mention weighted incoming outputs followed by the activation function.

Your attempt (not saved)

Key Takeaways

1
A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.
2
The input layer is V0, hidden layers lie between the input and output layers, and the output layer is VT.
3
For an n-dimensional input space, the described input layer contains n feature neurons and one constant neuron that always outputs 1.
4
A neuron combines weighted incoming outputs and processes the combination with an activation function.
5
Information moves forward from input to hidden layers to output; the layer structure records the order of that computation.

At a glance

A feedforward neural network is a directed acyclic graph of neurons connected by directed weighted edges.
Input, hidden, and output layers describe the positions and roles of neurons in the forward flow.
The input layer represents n features with n feature neurons and includes a constant neuron that outputs 1.
Each neuron combines weighted incoming outputs and then applies an activation function.
Tracing the layer order makes it possible to follow information from the input layer to the final output.

Related concepts

Activation Functions

More concepts starting with W