Concepts / Introduction to Artificial Neural Networks

Introduction to Artificial Neural Networks

If at least one loop exists, the network is recurrent instead.

  • Programming

The Decisive Structural Question

When people first see a neural network, they may focus on the fact that information enters through one side and eventually produces an output. That description is not enough to classify the network as feedforward. The decisive question is whether the network's connections contain a loop. A feedforward artificial neural network has no loop in its connections. If at least one loop exists, the network is recurrent instead.

InputHiddenOutputInputHiddenOutputconnectionconnectionconnectionconnectionloop
How can you tell from a network diagram whether connections only move forward or whether information can loop back?
Key point

Feedforward describes the connection pattern, not merely the existence of an input and an output. The absence of loops is the defining feature.

Tracing a Feedforward Network

To trace a feedforward network, begin at the input layer and follow its connections toward the output layer. The network can contain one or more hidden layers between them. The important restriction is that the connections do not create a path through which a unit's output can influence its own input. Information therefore moves through the network without returning to an earlier point in a loop.

Input unitsinput layerHidden unitshidden layerOutput unitsoutput layerforward connectionforward connection
How does data move from the input units through hidden units to the output units without returning to an earlier layer?
Worked example

Classifying a Three-Layer Network

A network has an input layer, one hidden layer, and an output layer. Its connections lead from the input layer to the hidden layer and from the hidden layer to the output layer. No connection creates a loop. How should the network be classified?

  1. 1
    Locate the entrance

    The input layer is the network's entrance because it receives the input.

  2. 2
    Locate the internal layer

    The layer between the input and output layers is a hidden layer. It is internal to the network, not disconnected or inactive.

  3. 3
    Locate the exit

    The output layer is the network's exit because it produces the output.

  4. 4
    Check for a loop

    The connections do not form a loop, so the network satisfies the defining structural condition for a feedforward ANN.

Result

The network is a feedforward artificial neural network.

Layer Positions and Roles

A feedforward ANN is organized into input, hidden, and output layers. The input layer receives the input. The output layer produces the output. Any layer that is neither the input layer nor the output layer is a hidden layer. A feedforward ANN may have one or more hidden layers.

Input layerreceives inputHidden layerinternal layerOutput layerproduces outputpositioned beforepositioned before
What contains the input, hidden, and output units, and where is each layer positioned in the network?

Weights on the Connections

The units in an ANN are connected by links. Every link has an associated real-valued weight. In terminology used for biological neural networks, a weight roughly corresponds to the efficacy of a synaptic connection. These weights are part of the network's specification, while the presence or absence of loops determines whether the network is feedforward or recurrent.

Unit AUnit BWeightreal-valuedUnit CWeightreal-valuedweight_aweight_b
Where is each connection's weight located, and how can different real-valued weights be represented on links between units?
Key point

For classification, inspect the connection pattern first: a loop makes the network recurrent. The real-valued weights describe the links but do not remove or create the structural loop criterion by themselves.

When One Loop Changes the Classification

A network can still have recognizable input, hidden, and output layers while being recurrent. The deciding issue is not whether most connections move toward the output. The presence of even one loop changes the classification to recurrent.

Worked example

A Mostly Forward Network with a Loop

A network has an input layer, a hidden layer, and an output layer. Most connections move from input to hidden to output, but one connection leads from the output back to a hidden unit. Is the network feedforward?

  1. 1
    Identify the layers

    The input, hidden, and output layers can still be identified from their positions and roles.

  2. 2
    Inspect the extra connection

    The connection from the output back to a hidden unit returns to an earlier part of the network.

  3. 3
    Apply the loop rule

    That return connection creates a loop. The rule depends on whether any loop exists, not on whether most connections point forward.

Result

The correct classification is recurrent, not feedforward.

  • Common mistake

    Calling every network with an input and an output feedforward.

    What it looks like

    A diagram has an input side and an output side, but one connection returns from a later unit to an earlier unit.

    Why it fails

    Having an input and an output does not establish that the connections are loop-free.

    Do this instead

    Trace the connections and check whether any path forms a loop.

  • Common mistake

    Treating hidden layers as disconnected.

    What it looks like

    A learner sees the word hidden and assumes the units do not participate in the network.

    Why it fails

    Hidden means that the layer is internal rather than the entrance or exit.

    Do this instead

    Identify hidden layers as layers that are neither the input layer nor the output layer.

  • Common mistake

    Using weights alone to classify the network.

    What it looks like

    A learner focuses on the real-valued weights attached to links and ignores the direction and structure of the links.

    Why it fails

    Weights are part of the network specification, but the feedforward or recurrent classification is determined by whether the connections contain a loop.

    Do this instead

    Check the connection pattern for loops, then note that each link also has an associated real-valued weight.

Check Your Classification

EASY

A diagram contains an input layer, two hidden layers, and an output layer. Every connection moves toward a later layer, and no connection returns to an earlier layer. Classify the network and explain which observation supports your answer.

Hints
  • Start by checking whether any connection creates a loop.
  • The number of hidden layers does not decide the classification.
  • Use the absence or presence of a loop as your main evidence.

What do you think happens?

A network has input, hidden, and output layers, but one link creates a loop. What is its classification?

  • Feedforward

  • Recurrent

  • Neither

Reveal answer
Answer:

Recurrent

The presence of even one loop changes the classification to recurrent. The network may still have identifiable input, hidden, and output layers.

Essential Takeaways

  1. 1
    A feedforward ANN has no loops in its connections.
  2. 2
    The input layer receives input, the output layer produces output, and internal layers are hidden layers.
  3. 3
    A feedforward ANN may contain one or more hidden layers.
  4. 4
    Every connection between units has an associated real-valued weight.
  5. 5
    The presence of even one loop makes the network recurrent rather than feedforward.

At a glance

  • Feedforward refers to the absence of loops in an ANN's connections.
  • A feedforward ANN is organized into input, hidden, and output layers.
  • Each connection between units has an associated real-valued weight.
  • A single loop is enough to classify the network as recurrent.

Related Topics

Related concepts

More concepts starting with I