Concepts / Gradient Descent and Stochastic Gradient Descent

Gradient Descent and Stochastic Gradient Descent

A gradient at w produces the first-order Taylor approximation of f around w.

  • Programming
Interactive lab

Try it: Gradient Descent: Full-Batch, Mini-Batch and Stochastic

How gradient descent, mini-batch gradient descent and stochastic gradient descent differ only in how many examples estimate the gradient — and how that trades noisy steps for many cheap updates.

How it works

  1. The loss is the mean squared error of the line ŷ = w·x + b over 12 points.
  2. At the start of every epoch, shuffle the examples (seeded) and cut them into batches.
  3. For each batch, average the gradient 2(ŵx + b − y)·(x, 1) over its examples.
  4. Update (w, b) ← (w, b) − learning rate × gradient; the path is drawn over the loss contours.
  5. Batch size 12 is full-batch gradient descent, 1 is SGD; too large a learning rate overshoots and diverges.

Default run (32 steps): mini-batch gradient descent (batch size 4): start at w = -1, b = 3 (MSE 9.823), learning rate 0.1, 3 updates per epoch. The least-squares optimum is w* = 1.595, b* = 0.404. … After 10 epochs (30 updates): w = 1.608, b = 0.468, MSE 0.5665 vs the optimum 0.5613 (distance 0.066). Averaged iterate w̄ = 1.106, b̄ = 0.996.

Simplified: A two-parameter linear-regression loss on 12 fixed points with a constant learning rate; real SGD runs on large datasets and often decays the learning rate.

Educational simulation

Loading the simulation…

Choosing the Next Parameter

Suppose the goal is to minimize a function f by changing its parameter. Looking at the entire function may be difficult when deciding what to do next. Gradient descent begins with a narrower question: what does f look like near the parameter value we currently have?

Key point

The central idea is local decision-making. Use information at the current parameter to estimate which nearby direction could produce a lower function value.

What do you think happens?

If an approximation is created at the current parameter, should it be trusted equally far away from that parameter?

  • Yes, because the gradient describes the whole function

  • No, because the approximation is local and can become loose farther away

Reveal answer
Answer:

No, because the approximation is local and can become loose farther away.

The approximation is formed around the current point. Its usefulness is local, so gradient descent also needs a reason to limit how far the candidate parameter moves.

The Local Taylor View

Let w be the parameter around which the approximation is built, and let u be a nearby candidate parameter. The gradient at w produces a first-order Taylor approximation of f around w. The approximation starts with the current function value f(w), then adds the gradient-based term 〈u − w, ∇f(w)〉. In symbols, the estimate is f(u) ≈ f(w) + 〈u − w, ∇f(w)〉.

Formula, step by step
  1. 1
    w is the point where the approximation is formed.
  2. 2
    u is a nearby candidate parameter whose function value is being estimated.
  3. 3
    f(w) is the function value at the current point.
  4. 4
    ∇f(w) is the gradient at the current point.
  5. 5
    〈u − w, ∇f(w)〉 is the gradient-based change term.
ffunction∇f(w)gradient at wwcurrent parameterf(u) ≈ f(w) + 〈u − w,∇f(w)〉local estimate near wevaluate at wcenter approximationcontributes slope
How does the gradient at w define a local approximation of f near w?

Reading the Approximation

The approximation gives a local way to compare candidate parameters. Starting from f(w), it estimates how the function changes when moving from w toward u. A candidate that receives a lower estimate according to this local expression is a candidate for reducing the function value.

Worked example

Interpreting a Candidate Move

Use the local approximation to describe what is being estimated when the current parameter is w and the candidate parameter is u.

  1. 1
    Start at the current value

    The estimate begins with f(w), the function value at the parameter where the gradient was measured.

  2. 2
    Measure the parameter change

    The difference u − w describes how the candidate differs from the current parameter.

  3. 3
    Use the local gradient

    The term 〈u − w, ∇f(w)〉 estimates the change associated with that candidate movement using the gradient at w.

  4. 4
    Form the local estimate

    Add the gradient-based term to f(w) to obtain the first-order estimate of f(u).

Result

The expression estimates f(u) using only the current value f(w), the candidate displacement u − w, and the gradient ∇f(w) measured at w.

Limiting the Step

A tempting strategy would be to minimize the local approximation directly. The difficulty is that the approximation can become loose when the candidate w is far from the current parameter w(t). A low value predicted by the approximation is therefore not enough by itself to justify an arbitrarily distant move.

Formula, step by step
  1. 1
    w(t) is the current parameter value.
  2. 2
    w is a candidate parameter.
  3. 3
    The gradient ∇f(w(t)) is measured at the current parameter.
  4. 4
    The approximation is intended for candidates close to w(t).
  5. 5
    Moving farther from w(t) increases the concern that the local estimate may be loose.
w(t)current parameterw far from w(t)distant candidatew near w(t)local candidatelocal approximationmay become loosesmall displacementlarge displacementless reliable locally
What changes in the approximation as a candidate parameter moves farther from the point where the gradient was measured?

Balancing Distance and Descent

Gradient descent addresses the problem by balancing two goals. It seeks a lower value according to the local approximation, while also keeping the new parameter close to the current parameter. The update is motivated by jointly minimizing the distance from the current parameter and the approximation of f around that parameter.

The parameter η controls the tradeoff between these two terms. In the usual gradient-descent form, the resulting move is represented as w(t + 1) = w(t) − η∇f(w(t)). The negative gradient direction is the locally downhill direction, while η controls how strongly the update responds to that local gradient. The important motivation is not to trust the approximation arbitrarily far away; it is to use the approximation while limiting the movement.

w(t)current parameterηtradeoff parameterw(t + 1)updated parameter∇f(w(t))local gradientstay closemove downhillcontrols balance
How does balancing closeness to the current parameter with lowering the local approximation produce the gradient descent update?
Key point

The update rule is best understood as a compromise: improve the local estimate, but do not leave the neighborhood in which that estimate was formed without a distance-based constraint.

Convexity and Reliability

Convex functions have a special role in this motivation. For a convex function, the first-order Taylor approximation lower bounds the function. This connects minimizing the local approximation with searching for a lower value of the original function.

convex ffunctionTaylor approximationlower boundgeneral ffunctionTaylor approximationlocal estimatelower boundsapproximates locally
Why does convexity make the relationship between the local approximation and a lower function value more reliable?

Gradient Descent Variants

ApproachGradient information described in the source packWhat the comparison highlights
Gradient descentThe gradient at the current parameter is used to form the local approximation.The update is motivated by balancing the local approximation with distance from the current parameter.
Stochastic gradient descentThe source pack names this approach but does not specify its sampling or update procedure.Do not infer operational details from the name alone; those details require additional material.

This article establishes the local-approximation motivation for gradient descent. It does not provide enough source material to explain the mechanics of stochastic sampling.

Common Reasoning Mistakes

  • Common mistake

    Treating the first-order approximation as an exact description of the whole function

    What it looks like

    Using f(u) ≈ f(w) + 〈u − w, ∇f(w)〉 as though it were equally reliable for every possible u.

    Why it fails

    The approximation is useful locally but can become loose when the candidate is far from the point where the gradient was formed.

    Do this instead

    Interpret the expression as a local estimate and keep the candidate update near the current parameter.

  • Common mistake

    Minimizing the local approximation without considering distance

    What it looks like

    Choosing a distant candidate only because the approximation predicts a lower value there.

    Why it fails

    The prediction may no longer be reliable far from the current parameter.

    Do this instead

    Balance lowering the local approximation with staying close to the current parameter.

  • Common mistake

    Ignoring the role of the gradient's evaluation point

    What it looks like

    Reading ∇f(w(t)) as though it were the gradient at the candidate w.

    Why it fails

    The gradient in the local approximation is measured at the current parameter w(t).

    Do this instead

    Track the evaluation point: the approximation around w(t) uses ∇f(w(t)).

  • Common mistake

    Assuming convexity is irrelevant to the motivation

    What it looks like

    Describing the local approximation as a lower bound for every function.

    Why it fails

    The source specifically gives the lower-bound property for convex functions.

    Do this instead

    Associate the lower-bound connection with convex functions while still remembering that locality matters.

Practice the Motivation

MEDIUM

Explain, in your own words, why gradient descent does not simply minimize the first-order Taylor approximation without a distance constraint.

Hints
  • Identify where the gradient was measured.
  • State what can happen when the candidate parameter moves far from that point.
  • Explain what η controls in the update motivation.
MEDIUM

Complete the reasoning chain: current parameter → local gradient → first-order approximation → distance-aware update. For each arrow, state what information is passed to the next step.

Hints
  • The current parameter identifies where the approximation is formed.
  • The gradient supplies the gradient-based term.
  • The approximation estimates candidate function values locally.
  • The distance-aware update limits how far the next parameter moves.

Key Takeaways

  1. 1
    The gradient at w produces the first-order Taylor approximation of f around w.
  2. 2
    The approximation estimates a candidate value by combining f(w) with a gradient-based change term.
  3. 3
    The estimate is local and can become loose when the candidate moves far from the point where the gradient was measured.
  4. 4
    Gradient descent is motivated by jointly lowering the local approximation and keeping the new parameter close to the current parameter.
  5. 5
    For a convex function, the first-order approximation lower bounds the function, which connects local minimization with the search for a lower original function value.

At a glance

  • A gradient provides a first-order local approximation around the current parameter.
  • The approximation helps identify a locally downhill direction, but it should not be trusted arbitrarily far away.
  • Gradient descent balances improvement according to the local approximation with distance from the current parameter.
  • The parameter η controls the tradeoff in the update.
  • Convexity is important because the first-order approximation lower bounds a convex function.

Related Topics

Related concepts

More concepts starting with G