Understanding Module Structure
Figure 3.24 illustrates the structure of a multi-head attention module, which consists of multiple single-head attention modules, as previously depicted in figure 3.18, stacked on top of each other.
From Layers to Modules
A useful way to understand a neural-network architecture is to look at its modules as organized layers. A layer is a data-processing module that receives one or more tensors as input and produces one or more tensors as output. Some layers are stateless, while others have state in the form of weights learned with stochastic gradient descent. These weights contain the network's learned knowledge.
The multi-head attention module is one part of the larger GPT model structure. In the surrounding architecture, input tokenization and embedding and masked multi-head attention have already been introduced. The next structural step is to understand what the multi-head attention module itself contains before considering the transformer blocks that form part of the GPT model.
Tracing the Module Contents
What do you think happens?
What would you expect to find inside a multi-head attention module?
Reveal answer
Multiple single-head attention modules
The structure shown for the multi-head attention module consists of multiple single-head attention modules arranged on top of one another.
The key structural relationship is containment. A multi-head attention module is made from multiple single-head attention modules. The single-head modules are represented as being stacked on top of one another. Therefore, “multi-head” describes a module that includes several single-head modules rather than a single indivisible attention operation.
A Layer Has Structure and State
Reading a Module as a Layer
Suppose you are examining the multi-head attention module as a layer in a neural network. What structural questions should you ask?
- 1Identify the module
Start with the multi-head attention module as the layer or data-processing module being examined.
- 2Look inside
The module contains multiple single-head attention modules stacked on top of one another.
- 3Place it in the layer model
As a layer, the module belongs to the general pattern of receiving one or more tensors and producing one or more tensors.
- 4Check for state
Layers may have state in the form of weights. These weights are learned with stochastic gradient descent and together contain the network's knowledge.
The module can be understood at two levels: internally, it is composed of multiple single-head attention modules; generally, it fits the neural-network layer pattern of data processing with possible learned weight state.
Checking the Structure
Imagine inspecting a diagram from the outside inward. The outer object is the multi-head attention module. Moving inward reveals several copies of the single-head attention module arranged as a stack. The diagram therefore communicates a hierarchy: one larger module is composed of multiple smaller modules.
Multi-head attention
?The internal structure is unknown.
Multi-head attention
Single-head attention
Single-head attention
Additional single-head attention modulesThe module is understood as multiple single-head attention modules stacked on top of one another.
The important change is not a change to the name of the outer module. The change is in your structural understanding: the multi-head module is no longer treated as a single unexplained box. Its internal organization is now visible as a collection of single-head attention modules.
Mistakes in Structural Reading
Common mistake Treating multi-head attention as one single-head attention module.
What it looks likeReading the name “multi-head attention” as if the module contained only one attention head.
Why it failsThe structure consists of multiple single-head attention modules.
Do this insteadRepresent the multi-head module as the larger structure and place multiple single-head modules inside it.
Common mistake Ignoring the stacked arrangement.
What it looks likeListing single-head modules without showing that they are arranged on top of one another.
Why it failsThe module structure is described as single-head attention modules stacked on top of each other.
Do this insteadShow the single-head modules as a stack within the multi-head attention module.
Common mistake Confusing a module's contents with its learned state.
What it looks likeSaying that the single-head modules are the weights of the multi-head module.
Why it failsThe source distinguishes the modules that make up a layer from the weights that can provide a layer's learned state.
Do this insteadDescribe composition and weights separately: the module contains single-head modules, while weights are learned state.
Practice and Recap
Draw a two-level structure for a multi-head attention module. Put the multi-head attention module at the top level, then place several single-head attention modules beneath it in a vertical stack. Beside your drawing, write one sentence explaining how this structure differs from treating multi-head attention as one single module with no internal organization.
Hints
- Use containment: the larger module should contain the smaller modules.
- Show more than one single-head attention module.
- Use the word stacked when describing their arrangement.
- 1A neural-network layer is a data-processing module that takes one or more tensors as input and produces one or more tensors as output. Layers may also have learned state in the form of weights. The multi-head attention module has a specific internal structure: it consists of multiple single-head attention modules stacked on top of one another. Reading this hierarchy makes the module easier to place within the broader GPT architecture.
Key Takeaways
- A layer is a neural-network data-processing module with inputs and outputs.
- A layer may have learned state in the form of weights.
- A multi-head attention module consists of multiple single-head attention modules.
- The single-head modules are arranged as a stack within the larger multi-head module.
- Module composition and learned weight state are related but distinct ideas.