Artificial Neural Networks
The forward pass tells the network what activations result from the current input and the current network state.
From Input to Training
An artificial neural network is not one isolated computational rule. It is a network of interconnected units whose computations work together to represent a relationship between inputs and outputs. During training, the network repeatedly computes activations in one direction and then computes information about its weights in the opposite direction. This alternating process is called backpropagation.
Interconnected Units
An artificial neural network is a network of interconnected units. Its units have some properties similar to neurons in nervous systems, but the key idea for this topic is their organization as a connected system rather than as a single isolated unit.
Each unit receives activations from units before it and contributes an activation that can be used by later units. Because the units are connected, the network represents a relationship through the combined activity of many units. This connected structure is important when the relationship being represented is not a simple, straight, easily described rule.
The Forward Pass
A forward pass begins with the current activations of the network's input units. Computation then proceeds toward later units. Each unit's activation is determined from the activations available before it. In this direction, information moves from the input side through the network.
The forward pass tells the network what activations result from the current input and the current network state.
Tracing a Forward Pass
Consider a generated ANN with input units, hidden units, and a later unit. What information is computed when one input is sent through the network?
- 1Start at the input units
The current input provides the current activations of the input units.
- 2Compute hidden-unit activations
Each hidden unit determines its activation from the activations available before it.
- 3Continue toward later units
The computation proceeds through the network, using earlier activations to determine later activations.
- 4Record the resulting activations
The forward pass produces the activation state that results from this input and the current network state.
A forward pass computes unit activations from the input side toward later units.
The Backward Pass
After the forward pass, backpropagation performs a backward pass. Its task is not to recompute the forward activations. Instead, it efficiently computes a partial derivative for each weight in the network. The collected partial derivatives form an estimate of the true gradient.
This information is useful because a network with hidden layers contains weights whose effects are separated from the later parts of the computation. The backward pass organizes the calculation of weight-specific derivatives across the network, giving the training process a gradient estimate rather than only the activations produced by the forward pass.
Why the Directions Alternate
Training an ANN with hidden layers requires more than sending an input through the network once. A forward pass computes the activations produced by the current input and current network state. A backward pass then computes a partial derivative for every weight. Alternating these passes connects what the network currently does with information that the training process can use to adjust its parameters.
- Forward: compute the activation of each unit from the activations available before it.
- Backward: compute a partial derivative for each weight.
- Collect: treat the derivatives together as an estimate of the true gradient.
- Repeat during training: alternate the two directions as the network is trained.
Nonlinear Function Approximation
Many machine learning tasks require a system to represent a relationship that is not simply a straight, easily described rule. Artificial neural networks are widely used for nonlinear function approximation: the network of connected units serves as the tool for representing that relationship.
The important connection is between structure and purpose. A single isolated unit offers only one local computation. An interconnected network combines the activations of many units across the network, allowing the system to represent a more complex relationship. Hidden layers are part of this connected organization, and backpropagation provides a way to train the weights in such a network.
Deep Layered Networks
Artificial neural networks have a long history. More recent progress in training deeply layered ANNs has contributed to some of the most impressive abilities of machine learning systems. The basic idea remains a network of interconnected units; advances in training networks with many layers have helped those systems develop stronger capabilities.
Common Mistakes
Common mistake Treating an ANN as a single isolated unit.
What it looks likeDescribing the network as though one unit alone represents the complete relationship.
Why it failsThe definition emphasizes a network of interconnected units.
Do this insteadThink about how activations move among connected units and combine across the network.
Common mistake Calling the forward pass the entire training process.
What it looks likeStopping after computing the activations produced by the current input.
Why it failsA network with hidden layers requires a backward pass that computes a partial derivative for each weight.
Do this insteadDescribe training as alternating forward and backward passes.
Common mistake Saying that the backward pass recomputes the forward activations.
What it looks likeDefining the backward pass as another calculation of the unit activations.
Why it failsIts task is instead to compute partial derivatives for the weights.
Do this insteadSeparate the forward-pass result, unit activations, from the backward-pass result, weight derivatives.
Common mistake Confusing individual derivatives with the gradient estimate.
What it looks likeCalling one partial derivative the complete gradient.
Why it failsThe collected derivatives form the gradient estimate.
Do this insteadDistinguish each weight's partial derivative from the collection of derivatives.
Check Your Understanding
Explain the training cycle for an ANN with hidden layers. Your answer should name what is computed during the forward pass, what is computed during the backward pass, and how the collected backward-pass results support training.
Hints
- Begin with the current activations of the input units.
- Explain how computation proceeds toward later units.
- State what is computed for each weight during the backward pass.
- Finish by explaining what the collected derivatives represent.
A learner says, "An ANN is just one rule applied to an input, and backpropagation means running that rule backward." Rewrite the statement so that it correctly describes interconnected units and the separate roles of the forward and backward passes.
Hints
- Use the phrase network of interconnected units.
- Mention activations for the forward pass.
- Mention a partial derivative for each weight in the backward pass.
Key Takeaways
- 1An artificial neural network is a network of interconnected units, not a single isolated unit.
- 2A forward pass computes unit activations from the current input-unit activations toward later units.
- 3A backward pass computes a partial derivative for each weight rather than recomputing the forward activations.
- 4The collected derivatives form an estimate of the true gradient used by the training process.
- 5ANNs are widely used for nonlinear function approximation, and advances in training deeply layered networks have expanded their abilities in machine learning systems.
At a glance
- Artificial neural networks represent relationships through interconnected units.
- The forward pass computes the activations produced by the current input and network state.
- The backward pass computes one partial derivative for each weight.
- Together, the collected derivatives form a gradient estimate, so alternating forward and backward passes supports training.
- Deep-training advances have contributed to stronger abilities in machine learning systems.