Mathematical Foundations of Neural Networks
Neural networks work with numerical data, so they need a consistent way to hold that data.
From Numbers to Tensors
Neural networks work with numerical data, so they need a consistent way to hold that data. A tensor provides that container. The central idea is not that a tensor is an entirely mysterious new object. It is a broader category that includes matrices and extends the same organizing idea to an arbitrary number of dimensions.
A Tensor as a Numerical Container
A tensor is a container for data, almost always numerical data.
Think of the tensor as the container and the numerical values as the contents. The container may organize a small collection of values or a much more elaborate arrangement. What makes the idea useful is that the same general notion can cover data with different numbers of dimensions.
Reading a Tensor as Organized Data
Imagine a numerical container that begins with one arrangement of values and then gains additional dimensions.
- 1Start with one value
A single numerical item is the smallest illustration of data being held in a container.
- 2Arrange values along one dimension
The values now have one organizing direction. This illustrates a one-dimensional arrangement.
- 3Arrange values along two dimensions
The values now have two organizing directions. A matrix is a two-dimensional tensor.
- 4Continue adding dimensions
The same organizing idea can extend beyond two dimensions because tensors generalize matrices to an arbitrary number of dimensions.
A tensor is best understood as a general numerical container whose data can be organized across different numbers of dimensions.
Dimensions and Axes
In tensor terminology, a dimension is often called an axis. This gives two ways to describe the same structural feature: you can talk about how many dimensions a tensor has, or how many axes organize its numerical data.
The axis language is useful because it focuses attention on organization rather than on a special name for each possible number of dimensions. A matrix has two dimensions, so it has two axes in this terminology. A tensor with more dimensions has more axes while remaining within the same general category of numerical container.
Common mistake Treating tensor and matrix as unrelated kinds of objects
What it looks likeA learner hears tensor and assumes that a matrix cannot be a tensor.
Why it failsThe source defines a matrix as a two-dimensional tensor.
Do this insteadUse tensor as the broader category and matrix as the specific two-dimensional case.
Common mistake Using dimension and axis as completely unrelated terms
What it looks likeA learner counts dimensions but does not recognize the same count when axes are mentioned.
Why it failsIn tensor terminology, a dimension is often called an axis.
Do this insteadWhen discussing tensor structure, treat dimension and axis as two terms for the organizing directions.
Common mistake Thinking tensors are important only because they have many dimensions
What it looks likeA learner focuses on high-dimensional arrangements but overlooks the numerical data they contain.
Why it failsThe defining idea is that a tensor is a container for data, almost always numerical data.
Do this insteadKeep both parts in view: the tensor contains numerical data and organizes it across dimensions.
Why Neural Networks Use Tensors
Machine-learning systems need a way to hold the numerical data they work with. Tensors provide that kind of container, which is why they are the basic data structure of current machine-learning systems. Their importance is practical: machine learning relies on tensors to represent the numerical data used by its systems.
| Term | Meaning in this topic | Relationship |
|---|---|---|
| Tensor | A container for data, almost always numerical data | Broad category |
| Matrix | A two-dimensional tensor | Specific tensor case |
| Dimension | An organizing direction of tensor data | Often called an axis |
| Axis | Another term for a tensor dimension | Describes tensor organization |
The terms describe the container, one specific case, and the ways its dimensions are discussed.
Check Your Understanding
Explain in your own words why a matrix can be described as a tensor. Then explain why the word axis may be used when discussing a tensor's dimensions.
Hints
- Start with the definition of a tensor as a container for data.
- Recall how many dimensions a matrix has.
- Use the terminology that connects a dimension with an axis.
A machine-learning system must work with numerical data. Give a short explanation of why tensors are central to that system rather than merely an optional way to arrange numbers.
Hints
- Focus on what machine-learning systems need to hold.
- Connect that need to the role of tensors as containers for numerical data.
Key Takeaways
- A tensor is a container for data, almost always numerical data.
- Tensors generalize matrices to an arbitrary number of dimensions.
- A matrix is a two-dimensional tensor.
- In tensor terminology, a dimension is often called an axis.
- Tensors are central to current machine-learning systems because those systems need containers for the numerical data they use.