Gradient of a Function

The gradient identifies the direction in which the function increases fastest around the current vector.

Programming

From a Current Vector to a Decision

Gradient descent does not jump directly to its final answer. It begins with an initial vector, examines the function at that current vector, and repeatedly decides which direction to move next. The gradient provides the direction of greatest local increase, so the algorithm reverses that direction to descend.

One Update in Numbers

What do you think happens?

Suppose the current vector is [10], the gradient at that vector is [2], and the positive learning-rate parameter is 0.5. The scaled movement is therefore [1]. What next vector results when the scaled movement is subtracted?

[11]
[10]
[9]

Reveal answer

Answer:

[9]

The scaled movement is 0.5 × [2] = [1]. Reversing the gradient direction means subtracting that movement from the current vector: [10] − [1] = [9].

Worked example

Applying one descent update

Current vector: [10]. Gradient at the current vector: [2]. Positive parameter η: 0.5.

1
Identify the direction
The gradient [2] indicates the direction in which the function increases fastest around the current vector.
2
Scale the direction
Multiply the gradient by η: 0.5 × [2] = [1]. The parameter controls the size of the movement.
3
Reverse the direction
Gradient descent subtracts the scaled gradient from the current vector: [10] − [1].

Result

The next vector is [9].

How does the current vector change into the next vector when the gradient and learning rate are applied?

What the Gradient Tells You

Definition

For a differentiable function, the gradient identifies the direction in which the function increases fastest around the current vector.

The gradient is a direction signal. At the current vector, it points toward the greatest local increase in the function. Gradient descent needs the opposite direction because its purpose is descent rather than increase. The update therefore reverses the gradient direction and scales the movement using the positive parameter η.

How does the gradient point toward increasing function values, and why does moving in the opposite direction provide the direction used for descent?

Reading the Update Rule

Formula, step by step

1
The current vector is the point where the function and its gradient are inspected.
2
The gradient supplies the direction in which the function increases fastest around that point.
3
η is a positive parameter that scales the movement.
4
Multiplying η by the gradient produces the scaled movement.
5
Subtraction reverses the gradient direction and produces the next vector.

How do the current vector, gradient, learning rate, and subtraction operation connect to produce the next vector?

Repeating the Update

After one update, the newly produced vector becomes the current vector for the next iteration. The algorithm evaluates the gradient again at this new point and applies the same rule again. Starting from an initial value, often written as w(1) = 0 in the description of the algorithm, the process continues through the chosen number of iterations.

How does one update become the starting point for the next update?

The repeated updates create a sequence of vectors. The gradient is not evaluated only once at the starting point; it is evaluated again at each newly produced vector. This is why the algorithm is an iterative process.

Choosing the Final Output

After T iterations, gradient descent needs a rule for selecting its output from the sequence of vectors. The algorithm can return the average vector, the last vector, or the best-performing vector.

Output choice	What it returns	Source note
Average vector	The average of the vectors produced during the iterations	Especially useful when gradient descent is extended to nondifferentiable functions and to the stochastic case
Last vector	The vector produced by the final update	Uses the end of the update sequence
Best-performing vector	The vector selected as the best performer during the process	Selects from the vectors produced during the iterations

The final output choice is part of how the algorithm is defined, not merely a formatting step after the updates.

Mistakes in Direction and Output

Common mistake
Moving in the same direction as the gradient
What it looks like
Adding the scaled gradient to the current vector
Why it fails
The gradient points toward the greatest local increase, while gradient descent uses the opposite direction.
Do this instead
Subtract the scaled gradient from the current vector.
Common mistake
Treating the gradient as the step size
What it looks like
Ignoring η and using the gradient without scaling
Why it fails
The gradient supplies the direction, while the positive parameter η scales the movement.
Do this instead
First scale the gradient with η, then subtract the resulting movement.
Common mistake
Evaluating the gradient only at the initial vector
What it looks like
Using the first gradient for every later update
Why it fails
Each newly produced vector becomes the current point for the next gradient evaluation.
Do this instead
Re-evaluate the gradient at the current vector on every iteration.
Common mistake
Assuming the final output must be the last vector
What it looks like
Discarding the average and best-performing output choices
Why it fails
The algorithm can return an average vector, the last vector, or the best-performing vector.
Do this instead
Choose the output rule that is part of the algorithm's definition.

Check Your Understanding

EASY

A current vector is [12]. The gradient at that vector is [4], and η is 0.25. Determine the scaled movement and the next vector. Then state which vector becomes the current vector for the next iteration.

Hints

Multiply η by the gradient to find the scaled movement.
Subtract the scaled movement from the current vector.
The newly produced vector becomes the current vector for the next iteration.

Your attempt (not saved)

1
The gradient identifies the direction in which a differentiable function increases fastest around the current vector.
2
Gradient descent reverses the gradient direction because it is used to reduce the function value.
3
The positive parameter η scales the movement before subtraction produces the next vector.
4
Each next vector becomes the current vector for the following iteration, creating a sequence of vectors.
5
After the iterations, the algorithm can return an average vector, the last vector, or the best-performing vector.

Key Takeaways

The gradient points toward the fastest local increase of a differentiable function.
Gradient descent moves opposite to that direction by subtracting a scaled gradient.
The current vector, gradient, positive parameter η, and subtraction operation each have a distinct role in the update.
The updated vector becomes the current point for the next iteration.
The final output may be the average vector, the last vector, or the best-performing vector.

More concepts starting with G