Concepts / Interpreting Deep Learning Models

Interpreting Deep Learning Models

Visualization makes convnet learning and classification decisions easier to inspect.

  • Programming

From Label to Evidence

A convolutional network can produce a class decision, but the class label alone does not show what the network learned or what visual evidence influenced that decision. Visualization makes the network's learning and classification process easier to inspect. The goal is not merely to display a result. It is to examine the route from the input image, through the network's convolutional layers, to the final class decision.

VisualizationWhat the networklearnsFeature mapsHow the networkdecidesInput evidenceClass comparisoninspectsinspectsthroughthroughthrough
How can visual evidence connect image patterns to what a convnet learns and to the decision it produces?
Key point

A useful interpretation asks two related questions: what visual patterns does the convnet learn, and how do those patterns relate to a particular classification decision?

The Route Through the Network

Begin with the complete route rather than with an isolated visualization. An image enters the convnet. Visual information is processed through convolutional layers. The network then produces a class decision. At every stage, ask not only what output the layer produces, but also what visual information that output represents for the next stage. This creates a state-by-state view of the path toward the decision.

Input imagevisual informationConvolutional layerfirst representationConvolutional layerlater representationClass decisionselected classprocessedtransformedsupports
How does image data move through successive convolutional layers and eventually become a class prediction?
Worked example

Tracing One Classification

An illustrative convnet receives an image and produces a class decision. How should the interpretation be organized?

  1. 1
    Inspect the input

    Start with the image that entered the convnet. The image is the source of the visual information being processed.

  2. 2
    Inspect layer outputs

    Follow the visual information through convolutional layers. For each layer, ask what its output represents for the next stage.

  3. 3
    Inspect the decision

    Record the class selected by the network, then relate that outcome to the visual evidence observed along the route.

Result

The interpretation is a trace from input image to convolutional representations to class decision, not a list of visualization names detached from the route.

Reading Feature Maps

Feature-map visualizations show which patterns activate across network layers. Early layers can be examined for simpler visual patterns such as edges. Deeper layers can be examined for more developed patterns, including object parts. Comparing layers helps you observe how the representation changes as the image moves through the convnet.

Early layeredgesMiddle layerdeveloped patternsDeeper layerobject partsChangedrepresentationevidence for the next stagecombinesdevelopscontributes
What patterns are activated at early, middle, and late layers, and how do those patterns change as depth increases?
In the real world

For an illustrative feature-map reading, an early-layer visualization might highlight edge-like patterns, while a deeper-layer visualization might highlight more developed patterns, including parts of an object. The important observation is the change in representation across layers, not the claim that one isolated map completely explains the final decision.

Choosing the Right View

Different visualizations answer different questions about a convnet decision. Saliency and attribution maps focus on which pixels or regions of the input contributed most to the decision. Activation maximization reverses the usual inspection direction: instead of starting with an image and asking what activates, it asks what input pattern would produce the strongest activation for a particular neuron or class. Comparative views examine why the convnet favors one class over another and whether the visual evidence differs between those predictions.

Saliency viewimportant pixels or regionsAttribution viewinput evidenceActivationmaximizationstrongest patternComparative viewdifference between classesConvnet decisionselected classinspectsinspectsrelates tocontrasts
Which parts of the input, internal features, or learned activations does each visualization identify as relevant to the predicted class?
ViewQuestion it asksWhat to inspect
Feature mapWhat patterns activate at this layer?The changing representation across layers
Saliency or attributionWhich parts of the input contributed to the decision?Highlighted pixels or regions
Activation maximizationWhat input pattern would produce the strongest response?A pattern associated with a neuron or class
ComparisonWhy is one class favored over another?Differences in visual evidence between predictions

The useful visualization depends on the question being asked about the convnet.

Best practice

Name the question before choosing the visualization. If the question concerns changing internal representations, compare feature maps across layers. If it concerns input evidence, inspect saliency or attribution. If it concerns a preferred response pattern, use activation maximization. If it concerns competing classes, compare the explanations.

Decision and Explanation

Definition

The classification decision is the class selected by the convnet. An explanation is a visual inspection of information related to that decision, such as activated patterns, highlighted input regions, or differences between class predictions. The explanation helps examine the decision, but it is not the decision itself.

Selected classclassification decisionActivated patternsfeature-map evidenceHighlighted regionssaliency or attributionevidenceClass contrastcomparative evidencehelps inspecthelps inspecthelps inspect
What is the difference between the class selected by the model and a visual explanation of evidence related to that selection?
  • Common mistake

    Treating the class label as a complete explanation

    What it looks like

    The network selects a class, so the label is treated as proof of what visual evidence caused the result.

    Why it fails

    The class label does not show what the network learned or what evidence influenced the decision.

    Do this instead

    Trace the route through convolutional layers and use an appropriate visualization to inspect related evidence.

  • Common mistake

    Treating every highlighted region as a complete explanation

    What it looks like

    An attribution map highlights a region, and the entire decision is assumed to be explained by that region alone.

    Why it fails

    The source describes the map as one way to inspect the decision alongside the layer-by-layer view.

    Do this instead

    Ask whether the highlighted evidence helps explain the selected class, then consider it together with other views.

  • Common mistake

    Listing technique names without tracing the decision

    What it looks like

    An interpretation lists feature maps, saliency, attribution, and comparison views without saying what each one investigates.

    Why it fails

    A strong answer describes what is inspected first, what changes through the network, and how visual evidence relates to the decision.

    Do this instead

    State the question, follow the image from input to decision, and connect each visualization to the part of the route it helps inspect.

Practice the Trace

MEDIUM

A convnet has selected a class for an image. You want to investigate both how its internal representation changes and which parts of the input are associated with the decision. Describe the order of your investigation and choose the visualization for each question.

Hints
  • Begin with the complete route from input image to class decision.
  • Use feature maps to compare activated patterns across layers.
  • Use saliency or attribution when the question concerns input pixels or regions.
  • Treat the selected class as the outcome and the visualizations as evidence used to inspect it.
Worked example

A Strong Interpretation Plan

How can the investigation answer both what the convnet learns and how it reaches a decision?

  1. 1
    Establish the outcome

    Record the class selected by the convnet. This is the decision being examined.

  2. 2
    Trace internal change

    Compare feature-map visualizations across layers, looking for simpler patterns in early layers and more developed patterns, including object parts, in deeper layers.

  3. 3
    Inspect input evidence

    Use saliency or attribution to examine which pixels or regions are associated with the selected class.

  4. 4
    Add contrast when needed

    If the question is why one class is favored over another, use a comparative view to examine differences between the predictions.

Result

The explanation connects the class decision with both the changing internal representation and the visual evidence associated with the input.

Summary

  1. 1
    Visualization helps inspect both what a convnet learns and how it makes classification decisions.
  2. 2
    A complete interpretation traces visual information from the input image through convolutional layers to the final class decision.
  3. 3
    Feature maps reveal how activated patterns change across network depth, from simpler patterns such as edges to more developed patterns including object parts.
  4. 4
    Saliency and attribution focus on input pixels or regions, activation maximization asks what pattern produces a strong response, and comparison views examine competing class explanations.
  5. 5
    The classification decision is the outcome; visual explanations are evidence used to inspect and interpret that outcome.

At a glance

  • Visualization turns a convnet classification from an isolated label into a traceable path from image input to class decision.
  • Feature-map comparisons show how the patterns represented by the network change across layers.
  • Saliency, attribution, activation maximization, and comparison views answer different questions and should be selected accordingly.
  • An explanation provides visual evidence about a decision; it is not the decision itself.

Related Topics

Related concepts

More concepts starting with I