Deep Neural Networks
The rise of deep learning depended on two improvements arriving together: much more available data and progress in the algorithms used to train neural networks.
Learning Needs Material
A deep-learning system begins with data rather than with intelligence already built in. Data provides the material from which the system learns. The system can improve only when training methods can make useful progress from that material.
The rise of deep learning depended on two improvements arriving together: much more available data and progress in the algorithms used to train neural networks.
The Internet’s Dataset Effect
Before internet-scale collection and distribution, the supply of material available for machine learning was more limited. The rise of the internet changed the scale of that material. Very large datasets became feasible to collect and distribute for machine learning.
Following the Data Story
Explain why internet-scale material mattered to the rise of deep learning.
- 1Start with material
A learning system needs data as the material from which it can learn.
- 2Increase the supply
The internet made very large collections of images, videos, and natural-language material feasible to collect and distribute.
- 3Connect material to training
Those large datasets supplied more material for machine-learning systems, provided that training algorithms could use it effectively.
Internet-scale data addressed the supply side of deep learning, while training-algorithm progress addressed whether neural networks could improve from that supply.
Why Depth Became Difficult
A neural network with several layers must use feedback to improve. That feedback passes through the stack of layers. Adding layers made the path through which the feedback had to travel longer, creating a training difficulty: the feedback signal could fade before it had a useful effect throughout the network.
The important contrast is not simply that a deeper network contains more parts. Its feedback must cross more layers before it can affect the whole network. As the stack became deeper, that feedback could fade away before reaching layers where it needed to have a useful effect.
The Fading Feedback Signal
Think of training feedback as a message sent backward through the network. In a network with several layers, the message must pass through the stack so that the network can improve. If the message becomes weaker as it travels, it may no longer have a useful effect by the time it reaches parts of the network farther along the backward path.
Tracing One Training Attempt
Trace what happens when a deep network tries to use feedback to improve.
- 1Feedback enters the stack
The network uses feedback to improve after processing data.
- 2Feedback crosses layers
The feedback must pass through the network's layers rather than affecting every layer directly.
- 3The path becomes longer
With more layers, the feedback has more of the stack to cross.
- 4The signal fades
The source identifies fading feedback as the reason training very deep networks was difficult.
- 5Improvement is limited
If the feedback fades before having a useful effect throughout the network, the network cannot reliably make useful progress across all those layers.
Increasing depth created a longer feedback path, and a fading signal limited the training effect across the network.
The central obstacle was a training problem, not merely a shortage of possible network designs: feedback could fade across many layers before it had a useful effect throughout the network.
Before Reliable Deep Training
Before a reliable way to train very deep neural networks was available, neural networks generally remained shallow, using only one or two layers of representations. This limitation mattered in comparison with more-refined shallow methods such as support vector machines and random forests.
| Situation | Network depth | Training consequence |
|---|---|---|
| Earlier neural-network practice | Generally one or two layers of representations | Very deep networks were difficult to train reliably |
| Increasing the layer count | A deeper stack of layers | Feedback had farther to travel and could fade |
| More-refined shallow methods | Shallow methods such as support vector machines and random forests | Neural networks did not consistently outperform them before reliable deep training |
Depth increased the potential training path, but fading feedback limited reliable improvement.
Check Your Understanding
A learner says, “Deep learning became possible simply because computers could use more layers.” Correct this explanation using the roles of data and training algorithms.
Hints
- Identify what supplies the material for learning.
- Identify what determines whether a network can make useful progress from that material.
- Explain what happens to feedback when the layer stack becomes deeper.
Common mistake Treating data as optional background information.
What it looks likeDescribing the training algorithm while leaving out the dataset.
Why it failsData provides the material from which a deep-learning system learns.
Do this insteadExplain both sides: data supplies the material, and training methods determine whether the network can improve from it.
Common mistake Assuming that more layers automatically make training better.
What it looks likeSaying that adding layers guarantees a more capable network.
Why it failsAs the stack became deeper, feedback could fade before it had a useful effect throughout the network.
Do this insteadDescribe greater depth together with the longer feedback path and its training difficulty.
Common mistake Explaining the rise of deep learning through internet data alone.
What it looks likeSaying that large datasets were sufficient by themselves.
Why it failsThe source identifies advances in training algorithms as the second condition that improved alongside data availability.
Do this insteadConnect internet-scale datasets with progress in neural-network training methods.
Key Takeaways
- 1Data is the material from which deep-learning systems learn.
- 2The internet made very large datasets feasible to collect and distribute for machine learning.
- 3The rise of deep learning required both more available data and progress in training algorithms.
- 4Adding layers made the feedback path longer, and the feedback signal could fade before affecting the whole network usefully.
- 5Before reliable training for very deep networks was available, neural networks generally remained shallow and did not consistently outperform refined shallow methods.
At a glance
- Data supplies the material from which deep-learning systems learn.
- Internet-scale collection and distribution expanded the supply of images, videos, and natural-language material available in large datasets.
- Deep learning advanced when data availability and training algorithms improved together.
- Feedback had to travel through every layer, and increasing depth could cause that signal to fade before it had a useful effect.