Concepts / Stochastic Gradient Descent

Stochastic Gradient Descent

The gradient identifies the direction in which the function increases fastest around the current vector.

  • Programming
Interactive lab

Try it: Gradient Descent: Full-Batch, Mini-Batch and Stochastic

How gradient descent, mini-batch gradient descent and stochastic gradient descent differ only in how many examples estimate the gradient — and how that trades noisy steps for many cheap updates.

How it works

  1. The loss is the mean squared error of the line ŷ = w·x + b over 12 points.
  2. At the start of every epoch, shuffle the examples (seeded) and cut them into batches.
  3. For each batch, average the gradient 2(ŵx + b − y)·(x, 1) over its examples.
  4. Update (w, b) ← (w, b) − learning rate × gradient; the path is drawn over the loss contours.
  5. Batch size 12 is full-batch gradient descent, 1 is SGD; too large a learning rate overshoots and diverges.

Default run (32 steps): mini-batch gradient descent (batch size 4): start at w = -1, b = 3 (MSE 9.823), learning rate 0.1, 3 updates per epoch. The least-squares optimum is w* = 1.595, b* = 0.404. … After 10 epochs (30 updates): w = 1.608, b = 0.468, MSE 0.5665 vs the optimum 0.5613 (distance 0.066). Averaged iterate w̄ = 1.106, b̄ = 0.996.

Simplified: A two-parameter linear-regression loss on 12 fixed points with a constant learning rate; real SGD runs on large datasets and often decays the learning rate.

Educational simulation

Loading the simulation…

The Direction of Descent

Optimization begins with a current vector and asks a local question: which direction makes the function increase fastest here? The gradient answers that question. Because the goal is minimization, the algorithm reverses the gradient direction and takes a scaled step away from increasing values. It then treats the new vector as the current vector and repeats the process.

Current vectorw(t)Gradientfastest increaseOpposite directiondescent directionNext vectorw(t+1)inspect locallyreversetake scaled step
How does reversing the gradient direction support minimization?

One Gradient Descent Update

Definition

For a differentiable function, the gradient at the current vector identifies the direction in which the function increases fastest around that vector. Gradient descent uses the opposite direction because it is seeking lower function values.

Formula

The current vector w(t) is the point where the gradient is evaluated. The gradient ∇f(w(t)) supplies the direction of fastest local increase. The positive parameter η, commonly called the learning rate, scales how far the algorithm moves. Subtraction reverses the gradient direction. The resulting vector w(t+1) becomes the current vector for the next iteration.

w(t)current vectorη∇f(w(t))scaled gradientw(t+1)next vectorsubtractupdate
How does the current vector change when the learning rate is multiplied by the gradient and subtracted?

Tracing the Arithmetic

Worked example

A Single Scalar Update

Suppose the current value is w(t) = 10, the gradient is 2, and the positive learning rate is η = 0.5. Find the next value.

  1. 1
    Start with the rule

    Use w(t+1) = w(t) − η∇f(w(t)).

  2. 2
    Scale the gradient

    The scaled movement is 0.5 × 2 = 1.

  3. 3
    Subtract the movement

    The next value is 10 − 1 = 9.

Result

The next value is w(t+1) = 9. The update moves opposite to the positive gradient direction.

The important pattern is not the particular numbers. The current value supplies the location, the gradient supplies the direction, the learning rate controls the scale of the movement, and subtraction produces the next value. In a vector update, the same operations apply component by component through vector arithmetic.

What Makes the Method Stochastic

Definition

Stochastic gradient descent is an optimization algorithm for minimizing a function f(w) using randomly selected update vectors. At iteration t, it uses the current parameters w(t), selects a random vector v(t) from an appropriate distribution, and updates the parameters by subtracting the learning-rate-scaled random vector.

Formula

The random vector v(t) does not need to equal the exact gradient at one iteration. Its conditional expected value must be a subgradient at the current vector. This requirement lets an individual update be random while preserving the required gradient-like direction in expectation. Repeated iterations use these random directions to form a sequence of parameter vectors.

Exact gradientone directionv(t)random directionRepeated v(t)expected directionreplace at one iterationrepeat
How does a randomly selected update direction approximate the full gradient over repeated iterations?

Reading the SGD Rule

Formula
w(t+1)next parameters=becomesw(t)current parameters−reverse directionηpositive learning ratev(t)random update vectorupdatescalesubtract
What roles do the learning rate, random vector, current parameters, and iteration count play in each update?
SymbolRole
w(t)The current parameter vector at iteration t.
ηA positive parameter that scales the movement.
v(t)The randomly selected update vector for iteration t.
w(t+1)The newly calculated parameter vector.
t through TThe iteration range: repeat the update from t = 1 through the chosen final iteration T.

Reading each part of the stochastic gradient descent update

The SGD Procedure

The stated SGD procedure follows a fixed cycle. It begins with an initial parameter vector. For each iteration from t = 1 through T, it uses the current vector to determine the condition the random choice must satisfy, selects v(t) from an appropriate distribution, and calculates the next vector with the update rule. After the final iteration, the procedure outputs the average of the iteration vectors w(t).

Initial vectorw(1)Inspect w(t)current parametersSelect v(t)appropriate distributionUpdate w(t+1)w(t) − ηv(t)Next iterationthrough TAverage vectorfinal outputstartcondition random choicechoose directionadvance tif iterations remainafter T
What happens first, how are the random update and parameters used, and when is the final result returned?

A Symbolic SGD Update

Worked example

Updating a Parameter Vector

Let the current parameter vector be w(t) = (a, b), let the randomly selected update vector be v(t) = (p, q), and let the positive learning rate be η. Find w(t+1).

  1. 1
    Write the SGD rule

    Use w(t+1) = w(t) − ηv(t).

  2. 2
    Substitute the vectors

    Replace w(t) with (a, b) and v(t) with (p, q), giving w(t+1) = (a, b) − η(p, q).

  3. 3
    Distribute the scalar

    The scaled random vector is (ηp, ηq).

  4. 4
    Subtract component by component

    The next vector is (a − ηp, b − ηq).

Result

w(t+1) = (a − ηp, b − ηq).

This symbolic update separates the four roles in the rule. The current parameters provide the starting point, the random vector provides the direction used for this iteration, the learning rate controls the movement's scale, and t identifies which update in the sequence is being performed.

What do you think happens?

If the current vector is (a, b) and the random vector is (p, q), what happens to the first coordinate after applying w(t+1) = w(t) − ηv(t)?

  • It becomes a + ηp

  • It becomes a − ηp

  • It becomes ηa − p

  • It remains a

Reveal answer
Answer:

It becomes a − ηp.

The random vector is first scaled by the learning rate and then subtracted from the current parameter vector.

Implementation Checks

  • Common mistake

    Adding the update vector instead of subtracting it.

    What it looks like
    Using w(t+1) = w(t) + ηv(t).
    Why it fails

    The stated update rule subtracts the learning-rate-scaled random vector, reversing the direction used for the update.

    Do this instead
    Use w(t+1) = w(t) − ηv(t).
  • Common mistake

    Reusing the initial vector at every iteration.

    What it looks like

    Computing every update from w(1) rather than from the newest vector.

    Why it fails

    Each newly produced vector becomes the current vector for the next iteration.

    Do this instead

    After calculating w(t+1), use it as the current vector in the next cycle.

  • Common mistake

    Assuming the random vector must equal the exact gradient.

    What it looks like

    Rejecting an update because v(t) is not identical to the gradient.

    Why it fails

    SGD allows an individual random vector to differ from the exact gradient; its conditional expected value must be a subgradient at the current vector.

    Do this instead

    Check the required expected-value condition rather than demanding equality at every iteration.

  • Common mistake

    Returning the last vector when the specified procedure requires an average.

    What it looks like

    Outputting w(T) without averaging the iteration vectors.

    Why it fails

    The stated SGD procedure outputs the average of the iteration vectors, while last-vector output is only one possible choice in the broader discussion of gradient descent outputs.

    Do this instead

    Preserve the required final averaging step.

  • Common mistake

    Omitting initialization or changing the iteration range.

    What it looks like

    Starting without the initial parameter vector or stopping before the chosen T iterations.

    Why it fails

    A correct implementation must preserve initialization, the iteration process, the update rule, and the final averaging step.

    Do this instead

    Check all four parts against the required procedure.

Best practice

When checking an SGD procedure, audit it in order: confirm the initial vector, verify the iteration range, verify that v(t) is selected under the required condition, check the subtraction update, confirm that each new vector becomes current, and inspect the final output rule.

Practice the Update

MEDIUM

Let w(t) = (6, -2), η = 0.25, and v(t) = (4, -8). Apply the SGD update rule and write w(t+1). Then state which vector must be used as the current vector in the next iteration.

Hints
  • First calculate ηv(t).
  • Subtract the scaled vector from w(t).
  • The newly calculated vector is the next current vector.
Worked example

Practice Check

Using w(t) = (6, -2), η = 0.25, and v(t) = (4, -8), find w(t+1).

  1. 1
    Scale the random vector

    ηv(t) = 0.25(4, -8) = (1, -2).

  2. 2
    Subtract from the current vector

    w(t+1) = (6, -2) − (1, -2).

Result

w(t+1) = (5, 0). The vector (5, 0) becomes the current vector for the next iteration.

Key Takeaways

  1. 1
    The gradient points in the direction of fastest local increase, so gradient descent moves in the opposite direction.
  2. 2
    The gradient descent update subtracts the learning-rate-scaled gradient from the current vector.
  3. 3
    SGD replaces the exact gradient at an individual iteration with a random update vector whose conditional expected value must be a subgradient at the current vector.
  4. 4
    Each new parameter vector becomes the current vector for the next iteration.
  5. 5
    The stated SGD procedure initializes the parameters, repeats the random selection and update through T iterations, and returns the average of the iteration vectors.

At a glance

  • The gradient identifies the fastest local increase, and reversing it gives the direction used for descent.
  • Gradient descent updates the current vector by subtracting a learning-rate-scaled gradient.
  • SGD uses a random update vector whose conditional expected value has the required subgradient direction.
  • The current vector, learning rate, random vector, and iteration count each have a distinct role in the update.
  • A correct SGD procedure preserves initialization, iteration, update, state replacement, and final averaging.

Related Topics

Related concepts

More concepts starting with S