Machine Learning Datasets
Data provides the material from which deep-learning systems learn.
From Examples to Rules
A deep-learning system begins with data rather than with intelligence already built in. Data provides the material from which the system learns. In machine learning, people provide data together with expected answers, and the computer uses that material to produce rules. Those learned rules can then be applied to new data to generate answers.
Why Data Became Abundant
The rise of deep learning depended on improvements in two areas: the amount of available data and the algorithms used to train neural networks. The internet changed the scale of the material available for machine learning because internet-scale collection and distribution made very large datasets feasible. These datasets can contain images, videos, or natural-language material.
| Condition | Role in learning |
|---|---|
| More available data | Supplies more material from which a learning system can improve |
| Better training algorithms | Determine whether a network can make useful progress from that material |
Deep learning became more promising when data availability and training methods improved together.
When Depth Weakens Feedback
Training a neural network requires feedback so the network can improve. That feedback must pass through the network's layers. As the stack becomes deeper, the feedback signal can fade before it has a useful effect throughout the network. This made training deep networks difficult.
Shallow network One or two layers of representations
The feedback path crosses fewer layers.
Deeper network Many layers of representations
The feedback path crosses more layers, so the signal can fade before affecting the whole network.
Before a reliable way to train very deep neural networks was available, neural networks generally remained shallow, using only one or two layers of representations. Because of this limitation, they did not consistently outperform more-refined shallow methods such as support vector machines and random forests.
Two Ways to Program
| Classical programming | Machine learning |
|---|---|
| Humans provide a program containing rules and data. | Humans provide data together with expected answers. |
| The computer follows the supplied rules. | The computer uses the paired data and answers to produce rules. |
| The computer produces answers from the human-designed path. | The learned rules are applied to new data to produce answers. |
Machine learning changes where the rules come from.
In classical programming, the path from input to output is designed by the programmer before the computer runs the program. In machine learning, humans do not provide the complete set of rules in advance. They provide examples and expected answers, and the computer produces rules from that material. The rules can then be used with information that was not part of the original material.
A Message-Classification Trace
Learning from labeled messages
A human wants a computer to assign each message to one of two categories. How can machine learning approach this task?
- 11. Supply data
The human gives the computer many messages as data.
- 22. Supply expected answers
The human supplies the expected category for each message, creating paired examples.
- 33. Produce rules
The computer uses the paired messages and answers to learn rules.
- 44. Apply the rules
When a new message arrives, the learned rules are applied to that new case.
- 55. Generate an answer
The rules produce an answer for the new message.
The computer has moved from paired examples to learned rules and then used those rules to answer a case that was not part of the original material.
The important distinction is the direction of construction. In classical programming, a human writes the rules first. In this machine-learning scenario, the human supplies examples and expected answers, while the computer produces rules that can later be applied to new messages.
What do you think happens?
In the message scenario, what does the computer receive before it can produce learned rules?
Reveal answer
Messages together with expected answers
Machine learning begins with data and expected answers. The computer uses that paired material to produce rules, which can then be applied to new data.
Lovelace and Computer Learning
Machine learning connects to a historical question: can a computer learn rather than merely follow explicitly ordered instructions? In 1843, Ada Lovelace argued that Charles Babbage's Analytical Engine could perform whatever people knew how to order it to perform, but could not originate anything. Alan Turing later discussed this idea as Lady Lovelace's objection while considering whether general-purpose computers could learn and show originality.
Machine learning addresses the learning part of that question by changing where the rules come from. Instead of having a human explicitly order every rule, the computer examines data and expected answers, produces useful rules, and uses those rules on information it has not seen before.
Common Reasoning Mistakes
Common mistake Treating machine learning as classical programming with a different name.
What it looks likeSaying that humans provide a complete rule set and the computer merely follows it in both approaches.
Why it failsClassical programming starts with human-written rules, while machine learning uses data and expected answers to produce rules.
Do this insteadAsk who supplies the rules. In classical programming, humans supply them; in machine learning, the computer produces them from paired examples.
Common mistake Describing data as optional training material.
What it looks likeAssuming a deep-learning system already contains the needed intelligence before seeing data.
Why it failsData provides the material from which deep-learning systems learn.
Do this insteadTreat data as the material a learning system uses to improve.
Common mistake Explaining difficult deep-network training only by counting layers.
What it looks likeSaying that extra layers are difficult simply because there are more of them.
Why it failsThe key issue described here is that feedback must pass through the stack, and the signal can fade before it has a useful effect throughout the network.
Do this insteadTrace the feedback path through the layers and ask whether the signal remains useful across the stack.
Common mistake Assuming that more data alone caused the rise of deep learning.
What it looks likeLeaving training algorithms out of the explanation.
Why it failsThe rise of deep learning depended on both access to much more data and progress in training algorithms.
Do this insteadExplain the two conditions together: data supplies material, while training methods determine whether the network can make useful progress.
Check Your Understanding
Describe the inputs and outputs in each approach. For classical programming, state what humans provide and what the computer produces. For machine learning, state what humans provide, what the computer produces during learning, and what happens when the learned rules meet new data.
Hints
- Use the words rules, data, expected answers, and new data.
- Remember that machine learning changes where the rules come from.
Explain why adding many layers made neural-network training difficult. Your answer should mention the feedback signal and what can happen as it passes through the stack.
Hints
- Feedback is needed for improvement.
- The source describes the signal as capable of fading before it has a useful effect throughout the network.
Key Takeaways
- 1Data provides the material from which deep-learning systems learn.
- 2The internet made very large datasets feasible to collect and distribute, including images, videos, and natural-language material.
- 3Training deep networks was difficult because feedback could fade as it passed through many layers.
- 4Classical programming gives the computer human-written rules and data; machine learning gives it data and expected answers so it can produce rules.
- 5Machine learning connects to the historical question of whether computers can learn rather than merely follow explicitly ordered instructions.
At a glance
- Deep learning needs data as the material from which its systems learn.
- Internet-scale collection and distribution expanded the supply and variety of machine-learning datasets.
- Feedback can fade across many neural-network layers, limiting the training of deep networks.
- Machine learning uses data and expected answers to produce rules that can be applied to new data.
- This shift in the source of rules connects machine learning to Lady Lovelace's historical question about whether computers can learn or originate something new.