Fundamentals of machine learning

Deep learning became more popular through the combined influence of hardware, data, algorithms, investment, and democratization.

Programming

Why Deep Learning Became Prominent

Deep learning became especially prominent through the combined influence of several factors rather than through one isolated breakthrough. Advances in hardware, greater data availability, improvements in algorithms, a new wave of investment, and the democratization of deep learning reinforced one another. Together, these factors helped make deep learning more popular in recent years.

How did several developments reinforce one another and contribute to the recent growth of deep learning?

Key point

The important idea is combination. The source does not describe popularity as the result of hardware, data, algorithms, investment, or democratization acting alone. It identifies their combined influence.

From Scalars to Tensors

A tensor is a way to organize the data used by a neural network. The organization is described by dimensions and indices. A scalar is a zero-dimensional tensor containing one value. A vector is a one-dimensional tensor, understood as an ordered collection of values. A matrix is a two-dimensional tensor that organizes values across two dimensions. Higher-dimensional tensors extend this arrangement to additional dimensions. Data batches are also part of neural-network data representation.

How are values arranged and indexed differently in a scalar, vector, matrix, and higher-dimensional tensor?

Representation	Dimensions	Organization
Scalar	0D	One value
Vector	1D	An ordered collection of values
Matrix	2D	Values arranged across two dimensions
Higher-dimensional tensor	More than 2D	Values arranged across additional dimensions

A tensor's dimensions describe how its values are organized.

Operations on Tensor Representations

Tensor operations are the computational steps that manipulate both data representations and values inside a neural network. Four important categories are element-wise operations, broadcasting, tensor dot, and reshaping. They do not all change a tensor in the same way: reshaping changes how values are organized, while element-wise operations and tensor dot perform calculations on values. Broadcasting makes some operations possible when tensors have different shapes and the operation's rules allow them to work together.

What changes in tensor shape and values when tensors are combined element by element, broadcast across dimensions, multiplied with a tensor dot, or reshaped?

Worked example

Following One Tensor Transformation

A neural network receives organized data and must transform it before producing a result. Trace what each operation contributes without assuming that every operation has the same effect.

1
Represent
The data enters as a tensor whose dimensions and indices describe how its values are arranged.
2
Calculate
An element-wise operation can calculate on corresponding values. A tensor dot can calculate across tensor inputs. These are value-transforming operations.
3
Align
Broadcasting addresses an operation involving tensors with different shapes when the operation's rules allow them to work together.
4
Reorganize
Reshaping changes the tensor's organization so that the same data representation is arranged differently for a later computation.

Result

Tensor operations are best understood by asking two questions: did the operation calculate new values, or did it change how existing values are organized?

Common mistake
Assuming reshaping is the same as calculating new values.
What it looks like
Describing a reshaped tensor as if its values had necessarily been changed.
Why it fails
The source distinguishes reshaping from operations that calculate on values. Reshaping changes the organization of the tensor.
Do this instead
Describe reshaping as a change in arrangement, and describe element-wise operations or tensor dot as value calculations.
Common mistake
Assuming broadcasting automatically makes any different-shaped tensors compatible.
What it looks like
Treating broadcasting as permission to combine every pair of different-shaped tensors.
Why it fails
Broadcasting applies when the operation's rules allow tensors with different shapes to work together.
Do this instead
Check whether the operation's rules allow the shapes to work together before describing the operation as valid.
Common mistake
Treating all tensor operations as interchangeable.
What it looks like
Using the label tensor operation without distinguishing element-wise operation, broadcasting, tensor dot, or reshaping.
Why it fails
These categories manipulate representations and values in different ways.
Do this instead
Name the operation category and state whether it calculates on values, aligns shapes, or reorganizes the representation.

The Forward Learning State

A neural network does not learn through a single operation. At one point in the training cycle, the network uses its current parameters to perform tensor computations. Organized input data enters as tensors, tensor operations transform that data, and the resulting computation contributes to a prediction. A loss function then measures the result.

How does input data move through tensor operations to produce a prediction, and how do gradients then flow backward to update the model's parameters?

Worked example

One Training Cycle

Trace the state of a neural network during one conceptual training cycle.

1
Current state
The network begins with its current parameters. These parameters are used by the tensor computations.
2
Forward computation
Organized input data enters as tensors. Tensor operations transform the data and produce a prediction.
3
Evaluation
A loss function measures the result of the prediction.
4
Backward information
Backpropagation computes gradients by chaining derivatives through the operations.
5
Update
A gradient-based optimization method, such as stochastic gradient descent, uses the gradients to update the model parameters.
6
Next state
The updated parameters affect later tensor computations, so the next cycle begins with a changed model state.

Result

Training is a repeated state change: compute, evaluate, calculate gradients, update parameters, and use the updated parameters in later computations.

Backpropagation and Parameter Updates

After the loss has been evaluated, the network needs information for changing its parameters. Backpropagation is the algorithm used to compute gradients by chaining derivatives through the operations. The gradients are then used by gradient-based optimization to update the parameters while minimizing the loss. Stochastic gradient descent is identified as one gradient-based optimization method.

What happens next after a neural network makes an incorrect prediction, and how are the error, gradients, and parameter updates connected?

How does the gradient provide information for changing parameters so that the loss can be minimized?

Common mistake
Saying that backpropagation directly changes the parameters.
What it looks like
Describing backpropagation as the step that performs the parameter update.
Why it fails
The source identifies backpropagation as the algorithm that computes gradients. An optimization method uses those gradients to update parameters.
Do this instead
Say that backpropagation computes gradient information and stochastic gradient descent or another optimization method uses that information for the update.
Common mistake
Treating gradients as the loss itself.
What it looks like
Using loss and gradient as interchangeable names for the same quantity.
Why it fails
The loss measures the result, while gradients are computed from the training process to provide information for changing parameters.
Do this instead
Trace the order as prediction, loss evaluation, gradient computation, and parameter update.
Common mistake
Describing training as a one-time computation.
What it looks like
Stopping the explanation after the first prediction and loss.
Why it fails
Training is described as a repeated state change in which updated parameters affect later tensor computations.
Do this instead
Include the repeated cycle of computation, evaluation, gradient calculation, parameter update, and later computation.

Practice the Complete Trace

MEDIUM

Explain the following training trace in your own words: organized input data enters as tensors; tensor operations transform it; the network produces a prediction; a loss function evaluates the result; backpropagation computes gradients; stochastic gradient descent uses the gradients to update parameters; and the updated parameters affect later tensor computations.

Hints

First separate data representation from parameter optimization.
Identify which step produces the gradients.
Identify which step changes the parameters.
End by explaining why the next computation is different from the previous one.

Your attempt (not saved)

What do you think happens?

A learner says, "The network learned as soon as tensor operations produced a prediction." Is that complete?

Yes, because the prediction is the final step
No, because the loss, gradients, and parameter update are also part of training
Yes, because tensors replace optimization
No, because neural networks do not use tensor operations

Reveal answer

Answer:

No, because the loss, gradients, and parameter update are also part of training

Tensor operations transform the data and contribute to a prediction, but training also requires evaluating the result, computing gradients through backpropagation, and updating parameters with a gradient-based optimization method.

The Complete Mental Model

1
Deep learning's recent popularity reflects the combined influence of hardware, data, algorithms, investment, and democratization.
2
Scalars, vectors, matrices, and higher-dimensional structures are tensor representations with different dimensions and indexing arrangements.
3
Element-wise operations and tensor dot calculate on values, broadcasting supports some operations involving different shapes, and reshaping changes organization.
4
Backpropagation computes gradients by chaining derivatives through operations, while stochastic gradient descent is a gradient-based method for updating parameters.
5
The complete training trace is organized input tensors, tensor operations, prediction, loss evaluation, gradient computation, parameter update, and later computation with updated parameters.

Key Takeaways

Deep learning became more popular through the combined influence of hardware, data, algorithms, investment, and democratization.
Tensors organize neural-network data from zero-dimensional scalars through vectors, matrices, and higher-dimensional structures.
Tensor operations transform representations and values through element-wise operations, broadcasting, tensor dot, and reshaping.
Training repeatedly connects tensor computation with loss evaluation, backpropagation, gradients, and parameter updates.
Tensor operations describe how data is transformed, while gradient-based optimization describes how model parameters change.

Related concepts

More concepts starting with F