Tensor Shapes

Broadcasting makes tensors with different shapes compatible for operations such as addition when possible and unambiguous.

Programming

Interactive lab

Try it: Tensor Shapes and Broadcasting

How NumPy decides result shapes for element-wise addition with broadcasting, np.dot, reshape and transpose on small tensors of rank 0 to 3, including the exact errors it raises for incompatible shapes.

How it works

Broadcasting: prepend size-1 broadcast axes to the lower-rank shape so the trailing dimensions line up.
Compare dimensions right to left: equal ones stay, a 1 is stretched, anything else raises ValueError.
np.dot sums over A's last axis and B's only (1-D) or second-to-last axis; those sizes must match.
reshape regroups the same values in row-major order (one -1 may be inferred); transpose reverses the axes.

Default run (5 steps): A has shape (2, 3) (rank 2), B has shape (3,) (rank 1). Operation: A + B. … Step 2: the stretched tensors are added element by element, giving shape (2, 3). For example result[1, 2] = 36.

Simplified: Tensors are filled with 1, 2, 3, ... (A) and 10, 20, 30, ... (B); ranks 0 to 3 with dimensions 1 to 4. Only these four operations are modelled.

Educational simulation

Loading the simulation…

Why Shape Matters

Adding tensors is straightforward when both tensors have the same shape. The situation becomes more involved when their shapes differ. A neural-network layer may need to add a two-dimensional tensor to a vector, so the operation needs a method for making tensors with different shapes compatible when that compatibility is possible and unambiguous.

Definition

Broadcasting is the process of making tensors with different shapes compatible for operations such as addition when the operation is possible and unambiguous.

The Alignment First Step

Broadcasting begins with the tensor that has fewer dimensions. Before compatibility can be considered, the dimensions of the smaller tensor are aligned with the larger tensor's dimensional structure. The first broadcasting step is to add broadcast axes to the smaller tensor until it has the same number of dimensions as the larger tensor.

How are dimensions handled before the tensors are checked for compatibility?

Broadcast Axes

A broadcast axis is an axis added to the smaller tensor so that its number of dimensions can match the larger tensor's number of dimensions. This is a change to dimensional organization: the smaller tensor begins with fewer axes, and the broadcasting process adds axes before the operation is treated as compatible.

Which axes are added so the smaller tensor can reach the larger tensor's number of dimensions?

A Dense Layer Example

Worked example

Making a Vector Compatible

A Dense layer needs to add a two-dimensional tensor to a vector. Which tensor is adjusted in the first broadcasting step?

1
Identify the dimensional difference
The two-dimensional tensor has more axes than the vector, so the vector is the smaller tensor for this operation.
2
Choose the tensor to adjust
Broadcasting adjusts the smaller tensor toward the shape of the larger tensor. The two-dimensional tensor remains the reference for the target dimensional structure.
3
Add broadcast axes
Broadcast axes are added to the vector until it has the same number of dimensions as the two-dimensional tensor. This is the first of the two broadcasting steps.
4
Continue to compatibility
Once the dimensional counts match, the tensors can be considered for compatibility for addition. Broadcasting makes the operation possible only when that compatibility is unambiguous.

Result

The vector is adjusted. The larger two-dimensional tensor keeps its original dimensional structure, while broadcast axes are added to the vector during the first broadcasting step.

Which tensor keeps its original shape, and which tensor is virtually expanded toward it?

What Broadcasting Does Not Guarantee

Different shapes do not automatically make an addition valid. Broadcasting describes how tensors can be made compatible when possible and unambiguous. Adding broadcast axes is the first step, not a guarantee that every pair of different shapes can be added.

How does broadcasting distinguish a pair that can be made compatible from a pair whose addition is not established as valid?

Common Shape Mistakes

Common mistake
Assuming that any two tensors with different shapes can be added.
What it looks like
Treating broadcasting as an automatic solution for every shape mismatch.
Why it fails
Broadcasting makes tensors compatible only when the operation is possible and unambiguous.
Do this instead
After identifying the smaller tensor and adding broadcast axes, still consider whether the resulting operation is unambiguous.
Common mistake
Adjusting the larger tensor instead of the smaller tensor.
What it looks like
Trying to remove or reorganize dimensions from the larger tensor to match the smaller one.
Why it fails
The source describes the smaller tensor as the tensor broadcasted toward the shape of the larger tensor.
Do this instead
Use the larger tensor as the target dimensional structure and adjust the smaller tensor.
Common mistake
Treating broadcast axes as ordinary data values.
What it looks like
Thinking that adding an axis means adding a new independent value to the tensor.
Why it fails
The first broadcasting step concerns dimensional organization: axes are added so the smaller tensor has the same number of dimensions as the larger tensor.
Do this instead
Describe the first step in terms of axes and dimensions rather than new tensor values.
Common mistake
Calling the first step the complete broadcasting process.
What it looks like
Stopping after the smaller tensor has the same number of dimensions as the larger tensor.
Why it fails
Adding broadcast axes is identified as the first of two broadcasting steps.
Do this instead
Recognize the added axes as the first step, then remember that compatibility for addition must still be possible and unambiguous.

Check Your Understanding

EASY

A larger tensor and a smaller tensor have different numbers of axes. Explain the first broadcasting step in three parts: identify which tensor is adjusted, state what is added to it, and describe the dimensional result.

Hints

The tensor with fewer axes is the smaller tensor for this step.
Broadcast axes are added to the smaller tensor.
The target is the same number of dimensions as the larger tensor.

Your attempt (not saved)

MEDIUM

A Dense layer adds a two-dimensional tensor to a vector. Describe why broadcasting may be needed and explain why the vector, rather than the two-dimensional tensor, is adjusted first.

Hints

The tensors have different shapes.
The smaller tensor is broadcasted toward the larger tensor's shape.
The first step adds axes to match the larger tensor's number of dimensions.

Your attempt (not saved)

The Shape-Checking Habit

1
Broadcasting helps tensors with different shapes become compatible for operations such as addition when compatibility is possible and unambiguous.
2
The smaller tensor is broadcasted toward the shape of the larger tensor.
3
Broadcast axes are added to the smaller tensor so it can match the larger tensor's number of dimensions.
4
Adding broadcast axes is the first of the two broadcasting steps.
5
Matching dimensional counts does not by itself guarantee that an addition is valid; the operation must still be possible and unambiguous.

Key Takeaways

Broadcasting addresses shape differences that arise when tensors are used together in operations such as addition.
The smaller tensor is the one adjusted during broadcasting.
The first broadcasting step adds broadcast axes to the smaller tensor.
Those axes give the smaller tensor the same number of dimensions as the larger tensor.
The operation is valid only when the resulting compatibility is possible and unambiguous.

More concepts starting with T