Adding Tensors
Before addition can be performed on tensors with different shapes, broadcasting may be needed to make their shapes compatible.
Try it: Tensor Shapes and Broadcasting
How NumPy decides result shapes for element-wise addition with broadcasting, np.dot, reshape and transpose on small tensors of rank 0 to 3, including the exact errors it raises for incompatible shapes.
How it works
- Broadcasting: prepend size-1 broadcast axes to the lower-rank shape so the trailing dimensions line up.
- Compare dimensions right to left: equal ones stay, a 1 is stretched, anything else raises ValueError.
- np.dot sums over A's last axis and B's only (1-D) or second-to-last axis; those sizes must match.
- reshape regroups the same values in row-major order (one -1 may be inferred); transpose reverses the axes.
Default run (5 steps): A has shape (2, 3) (rank 2), B has shape (3,) (rank 1). Operation: A + B. … Step 2: the stretched tensors are added element by element, giving shape (2, 3). For example result[1, 2] = 36.
Simplified: Tensors are filled with 1, 2, 3, ... (A) and 10, 20, 30, ... (B); ranks 0 to 3 with dimensions 1 to 4. Only these four operations are modelled.
Loading the simulation…
Why Shape Matters
Adding tensors is straightforward when both tensors have the same shape. The difficulty appears when their shapes differ. An addition operation that handles only two-dimensional tensors with identical shapes cannot directly handle every addition used in a neural-network layer. For example, a Dense layer can add a two-dimensional tensor to a vector. Broadcasting describes how tensors with different shapes can be made compatible for operations such as addition when that compatibility is possible and unambiguous.
Before addition can be performed on tensors with different shapes, broadcasting may be needed to make their shapes compatible.
What do you think happens?
A smaller tensor must be prepared to work with a larger tensor. What happens first in the broadcasting process?
Reveal answer
Broadcast axes are added to the smaller tensor.
Adding broadcast axes is the first of the two broadcasting steps described in the source. The smaller tensor begins with fewer axes, and this step gives it the same number of dimensions as the larger tensor.
Aligning the Tensor Shapes
Broadcasting changes the dimensional organization of the smaller tensor so that it can be considered toward the shape of the larger tensor. The larger tensor provides the reference shape. The smaller tensor is the one adjusted during broadcasting; the larger tensor is not described as being reduced to fit the smaller one.
The diagram shows the central direction of the adjustment: the smaller tensor is broadcast toward the larger tensor's shape. Broadcasting does not begin by shrinking the larger tensor. It begins by organizing the smaller tensor so that its dimensional structure can match the larger tensor's number of dimensions.
Adding Broadcast Axes
A tensor's ndim is its number of dimensions. When the smaller tensor has fewer dimensions than the larger tensor, broadcast axes are added to the smaller tensor to match the larger tensor's ndim. This is the first of the two broadcasting steps. The important state change is dimensional: the smaller tensor starts with fewer axes and then has additional axes in its organization.
Finding the First Broadcasting Change
A larger tensor has more dimensions than a smaller tensor. Identify the first change made to the smaller tensor before addition.
- 1Compare dimensional organization
The smaller tensor begins with fewer axes than the larger tensor.
- 2Add broadcast axes
Broadcast axes are added to the smaller tensor so that its number of dimensions matches the larger tensor's ndim.
- 3Continue toward compatibility
The smaller tensor is now organized with the same number of dimensions as the larger tensor. This is the first broadcasting step; the source describes broadcasting as a two-step process.
The first step adjusts the smaller tensor by adding broadcast axes. The larger tensor remains the reference shape.
From Preparation to Addition
The sequence begins by comparing the tensors' dimensional organization. If one tensor has fewer axes, that smaller tensor is the candidate for broadcasting. Broadcast axes are added first, giving the smaller tensor the same number of dimensions as the larger tensor. Broadcasting can then make the tensors compatible for an operation such as addition when the result is possible and unambiguous.
A Dense layer can add a two-dimensional tensor to a vector. This is an example of why tensor addition cannot always be treated as the addition of two tensors that already have identical two-dimensional shapes. Broadcasting provides the description of how tensors with different shapes can be made compatible for this kind of operation.
Smaller tensor: fewer axes Larger tensor: more axes
The tensors have different dimensional organizations.
Smaller tensor: broadcast axes added Larger tensor: reference shape retained
The smaller tensor has the same number of dimensions as the larger tensor.
Mistakes About Broadcasting
Common mistake Assuming tensors must already have identical shapes before they can be added.
What it looks likeTreating every addition as valid only when both tensors begin with the same shape.
Why it failsBroadcasting exists to make tensors with different shapes compatible for operations such as addition when possible and unambiguous.
Do this insteadFirst consider whether broadcasting can make the different shapes compatible.
Common mistake Adjusting the larger tensor instead of the smaller tensor.
What it looks likeTrying to reduce the larger tensor toward the smaller tensor's dimensional organization.
Why it failsThe smaller tensor is broadcast toward the shape of the larger tensor.
Do this insteadUse the larger tensor as the reference shape and identify the smaller tensor as the one being adjusted.
Common mistake Forgetting that broadcast axes are added before later broadcasting work.
What it looks likeJumping directly from different ndim values to addition without identifying the first state change.
Why it failsAdding broadcast axes is the first of the two broadcasting steps.
Do this insteadTrace the first step explicitly: add axes to the smaller tensor until its number of dimensions matches the larger tensor's ndim.
Common mistake Treating a broadcast axis as a value operation rather than a dimensional change.
What it looks likeFocusing on tensor values while ignoring that the smaller tensor began with fewer axes.
Why it failsThe important state change described here concerns the tensor's dimensional organization.
Do this insteadTrack the axes and ndim first; discuss addition only after the shapes are compatible.
When tracing tensor addition, write down three facts in order: which tensor has the larger shape, which tensor has fewer axes, and what broadcast axes must be added to the smaller tensor. This keeps the reference shape and the first broadcasting step distinct.
Check the First Step
A smaller tensor and a larger tensor have different numbers of dimensions. Explain which tensor is adjusted during broadcasting and describe the first change made to it. Then state what the larger tensor contributes to the process.
Hints
- The smaller tensor is the one broadcast toward the larger tensor.
- The first broadcasting step concerns axes and ndim.
- The larger tensor supplies the reference shape.
Practice Answer
Explain the first step when a smaller tensor must be made compatible with a larger tensor for addition.
- 1Identify the adjusted tensor
The smaller tensor is adjusted; the larger tensor remains the reference shape.
- 2Identify the dimensional change
Broadcast axes are added to the smaller tensor.
- 3Identify the target
The added axes bring the smaller tensor to the same number of dimensions as the larger tensor.
The first step is adding broadcast axes to the smaller tensor so that its ndim matches the larger tensor's ndim.
Key Takeaways
- 1Tensors with identical shapes can be added straightforwardly, while tensors with different shapes may need broadcasting.
- 2Broadcasting makes different tensor shapes compatible for operations such as addition when compatibility is possible and unambiguous.
- 3The smaller tensor is broadcast toward the larger tensor's shape.
- 4Broadcast axes are added to the smaller tensor to match the larger tensor's ndim.
- 5Adding broadcast axes is the first of the two broadcasting steps.
At a glance
- Broadcasting is needed when tensors with different shapes must be made compatible for addition.
- The smaller tensor is adjusted toward the larger tensor's reference shape.
- Broadcast axes are added to the smaller tensor to match the larger tensor's number of dimensions.
- Adding broadcast axes is the first broadcasting step.
- A Dense layer may add a two-dimensional tensor to a vector, illustrating why identical starting shapes are not always required.