Concepts / Representation Learning

Representation Learning

The word representation refers to how data is expressed at a particular learned layer. Deep learning emphasizes a sequence of these learned expressions rather than a single learned representation.

  • Programming

From Data to Representations

Deep learning extends machine learning by emphasizing successive learned representations. Instead of asking one learned representation to carry an entire task, a deep-learning model learns a succession of representations from data. The central idea is therefore not simply that a model is large or complicated. The important change is how learning is organized across layers.

Input datadataRepresentation 1learned layerRepresentation 2learned layerRepresentation nlearned layerlearnspasses topasses to
How does data move through successive learned representations in a deep-learning model?
Key point

The word representation refers to how data is expressed at a particular learned layer. Deep learning emphasizes a sequence of these learned expressions rather than a single learned representation.

Tracing Layered Learning

Worked example

An Abstract Layer Sequence

Trace what happens when abstract input data is processed by a model that learns several successive representations.

  1. 1
    Start with input data

    The model receives data that must be represented for a task.

  2. 2
    Learn the first representation

    A neural-network layer learns a representation of the data.

  3. 3
    Pass the representation onward

    The representation produced by one layer becomes the input to the next layer.

  4. 4
    Continue through later layers

    Additional layers learn further representations, creating a succession rather than a single representation.

Result

The model contains an organized sequence of learned representations. The number of contributing layers gives the model its depth.

This example is intentionally abstract. The essential pattern does not depend on naming a particular application. What matters is that data is represented through successive learned layers, and that the number of those layers determines the model's depth.

Neural layer 1representationNeural layer 2representationNeural layer 3representationpasses representationpasses representation
How do neural-network layers transform and pass representations from one layer to the next?

What Deep Measures

Definition

Model depth is the number of contributing layers. In deep learning, deep refers to the presence of many layers in the succession of learned representations. It does not mean deeper understanding.

Input dataInput dataRepresentationone or two layersRepresentation 1learned layerRepresentation 2learned layerRepresentation nmany layerslearnslearnspasses topasses to
What is the difference between an approach with one or two representation layers and a deep model with many stacked layers?

An approach that learns only one or two representation layers has a shorter representation sequence. Deep learning emphasizes many stacked layers. Modern deep-learning models often contain tens or even hundreds of such layers, and those layers are learned automatically from training data.

Neural Networks as Layered Structures

Neural networks provide the stacked-layer structure used to learn successive representations. A layer contributes to the sequence, and its learned representation can be passed to another layer. This arrangement lets the model organize learning across multiple representations instead of assigning the whole task to one learned representation.

Input dataInput dataRepresentationone or two layersRepresentation 1learned layerRepresentation 2learned layerRepresentation nmany learned layerslearnslearnspasses topasses to
How does the number and arrangement of learned representation layers differ between shallow and deep approaches?

Neural Does Not Mean Biological

A common first impression is that deep learning is a computer version of the brain. The name neural network can encourage that interpretation, but it is misleading. In this context, deep learning is a mathematical framework for learning representations from data.

  • Common mistake

    Treating a neural network as a model of the brain.

    What it looks like

    Assuming that a neural-network layer works in the same way as a biological brain structure.

    Why it fails

    Deep-learning models are not models of the brain, and their learning mechanisms should not be treated as the same as biological learning mechanisms.

    Do this instead

    Understand a neural network here as the stacked-layer structure used to learn representations from data.

  • Common mistake

    Interpreting deep as deeper understanding.

    What it looks like

    Assuming that a deeper model necessarily understands a subject more deeply because it has the word deep in its name.

    Why it fails

    In this context, depth refers to the number of contributing layers.

    Do this instead

    Count the contributing layers when interpreting model depth.

  • Common mistake

    Calling a model deep merely because it uses a neural network.

    What it looks like

    Treating any neural network as deep learning without considering how many representation layers contribute.

    Why it fails

    Deep learning emphasizes a succession of representations and the number of contributing layers.

    Do this instead

    Focus on the layered organization and the model's depth, not only on the term neural network.

Check the Representation Sequence

EASY

A model learns an input representation, passes it to a second learned representation, and then stops. Another model learns a succession of many stacked representations. Which model better matches the source definition of deep learning, and what does its depth refer to?

Hints
  • Look for the number of contributing layers.
  • Deep learning emphasizes successive learned representations.
  • Do not use the word deep to mean deeper understanding.
Worked example

Practice Answer

Compare the two abstract models.

  1. 1
    Identify the representation sequence

    The first model learns only a short sequence of representations. The second learns many stacked representations.

  2. 2
    Apply the definition

    The second model better matches deep learning because deep learning emphasizes successive learned representations.

  3. 3
    Interpret depth

    Its depth refers to the number of contributing layers.

Result

The model with many stacked learned representation layers better matches the described deep-learning approach.

Key Takeaways

  1. 1
    Deep learning extends machine learning by emphasizing successive learned representations.
  2. 2
    Deep refers to the number of contributing layers, not to deeper understanding.
  3. 3
    Neural networks provide the stacked-layer structure used to learn these representations.
  4. 4
    Approaches with one or two representation layers differ from deep approaches that organize many layers in succession.
  5. 5
    Deep-learning models are mathematical frameworks for learning from data, not models of the biological brain.

At a glance

  • Deep learning is a subfield of machine learning focused on successive learned representations.
  • A model's depth is the number of contributing layers.
  • Neural networks arrange layers so representations can be learned successively.
  • Deep learning should not be interpreted as a computer version of the brain.
  • The difference between shallow and deep approaches is the organization and number of learned representation layers.

Related Topics

Related concepts

More concepts starting with R