Concepts / Loss Functions

Loss Functions

A derivative explains how a small input change affects the output of a continuous function.

  • Programming

From Input to Outcome

A loss function helps describe how far a candidate result is from a target. To understand how such a result can change, begin with a simpler situation: a function receives one real number and produces another. If the input is adjusted slightly and the function is continuous and smooth, its output changes slightly as well. A derivative gives us a way to describe that local input-output change.

x₀inputy₀outputx₁nearby inputy₁nearby outputfunctionsame function
What changes in a continuous function's output when the input moves slightly from one nearby value to another?
Key point

The important idea is local change: the derivative is not merely the output itself. It describes how the output responds when the input changes slightly.

Reading a Derivative

The basic derivative example uses a real-number input x and an output y. We can describe the relationship as a function that receives x and produces y. The derivative relates a small change in x to the resulting change in y. It therefore answers a local question: if the input moves slightly, what happens to the output?

Change in xsmall adjustmentChange in yresulting adjustmentDerivativelocal relationshipdescribesrelates to
How does the derivative relate the direction and amount of a small input change to the resulting output change?
Worked example

Following a Single-Input Change

A continuous, smooth function receives one real-number input and produces one output. Explain what the derivative contributes when the input is adjusted slightly.

  1. 1
    Identify the input

    The function receives one real number, represented by x.

  2. 2
    Observe the output

    The function produces an output, represented by y.

  3. 3
    Make a small adjustment

    Consider what happens when x is moved slightly rather than replaced by an unrelated value.

  4. 4
    Use the derivative

    The derivative describes the local relationship between the small change in x and the resulting change in y.

Result

The derivative supplies a description of how the output responds locally to a small input change.

Why Tensors Need Gradients

The single-input picture changes when an operation receives multidimensional data. Instead of one real number, a tensor operation may involve several values arranged as tensors. The question remains the same: how does a small change in the input affect the result? A gradient extends the derivative idea to this multidimensional setting.

x₁tensor elementGradient component 1direction and amountx₂tensor elementGradient component 2direction and amountx₃tensor elementGradient component 3direction and amountlocal changelocal changelocal change
How does each element or dimension of a tensor input contribute its own direction and amount of change to the loss?
Definition

A gradient is the derivative of a tensor operation. It extends the single-input derivative idea so that the operation can be considered in a multidimensional input setting.

Tracing a Loss Calculation

A common tensor-based sequence begins with an input vector x and a matrix W. The matrix is used with x to compute a target candidate called y_pred. That candidate is then compared with the target y through a loss function. The loss represents the mismatch between the candidate and the target.

Input vector xmultidimensional inputTarget candidatey_predcomputed resultLossmismatchMatrix Woperation dataTarget ycomparison targetused withused withcomparedcompared
How do the input vector and matrix produce a target candidate, and how are the candidate and target combined to produce a loss?
Worked example

Following the Candidate and Target

Trace the roles of x, W, y_pred, y, and the loss in a tensor-based loss calculation.

  1. 1
    Start with x

    x is the input vector supplied to the operation.

  2. 2
    Use W with x

    The matrix W is used with the input vector to compute a result.

  3. 3
    Name the result

    That computed result is the target candidate y_pred.

  4. 4
    Bring in y

    y is the target against which the candidate is compared.

  5. 5
    Calculate the loss

    The loss represents the mismatch between y_pred and y.

Result

The sequence is x and W produce y_pred, then y_pred and y participate in the loss calculation.

ItemRole in the sequence
xInput vector supplied to the operation
WMatrix used with x
y_predTarget candidate computed from the operation
yTarget used for comparison
LossRepresentation of the mismatch between y_pred and y

Roles of the named quantities in the tensor-based loss sequence

Mistakes in the Change Story

  • Common mistake

    Treating the derivative as the function's output

    What it looks like

    Describing y as though it were the derivative merely because y is produced from x.

    Why it fails

    The output and the derivative answer different questions. The output is what the function produces; the derivative describes how that output changes when x is adjusted slightly.

    Do this instead

    Keep the two roles separate: the function maps x to y, while the derivative describes the local input-output change.

  • Common mistake

    Using the single-number picture for every input

    What it looks like

    Explaining a tensor operation as though it received only one real-number input.

    Why it fails

    A tensor operation may involve several values arranged as tensors, so the input is multidimensional.

    Do this instead

    Use the term gradient for the derivative idea in this multidimensional setting.

  • Common mistake

    Confusing the target candidate with the target

    What it looks like

    Calling y_pred the target y before the comparison occurs.

    Why it fails

    The sequence distinguishes the computed candidate y_pred from the target y.

    Do this instead

    Track the names carefully: x and W produce y_pred, and y_pred is compared with y through a loss.

  • Common mistake

    Treating the loss as the candidate itself

    What it looks like

    Saying that the loss is simply y_pred.

    Why it fails

    The loss represents the mismatch between the candidate and the target.

    Do this instead

    Describe the loss as the result of comparing y_pred with y.

Practice the Trace

MEDIUM

Explain the following sequence in your own words: an input vector x and a matrix W are used to produce y_pred, and y_pred is compared with y through a loss. Then state why a gradient is relevant when the operation accepts tensor inputs.

Hints
  • First identify which items are inputs and which item is the computed candidate.
  • Separate the candidate y_pred from the target y.
  • Connect the gradient to the multidimensional nature of tensor inputs.

What do you think happens?

Before reading the explanation, predict which quantity represents the mismatch: x, W, y_pred, y, or the loss.

  • x

  • W

  • y_pred

  • y

  • The loss

Reveal answer
Answer:

The loss

The input vector and matrix participate in producing the target candidate. The candidate is compared with the target, and the loss represents the resulting mismatch.

Key Takeaways

  1. 1
    For a continuous, smooth function, a small input adjustment produces a small output adjustment.
  2. 2
    A derivative describes the local relationship between a change in one real-number input and the resulting change in its output.
  3. 3
    A gradient extends the derivative idea to operations with multidimensional tensor inputs.
  4. 4
    A matrix W and input vector x can produce a target candidate y_pred.
  5. 5
    The candidate y_pred is compared with target y, and the loss represents their mismatch.

At a glance

  • A derivative explains how a small change in a single input affects a continuous function's output.
  • A gradient applies the same change-focused idea when an operation receives multidimensional tensor inputs.
  • In a common loss sequence, x and W produce y_pred.
  • The candidate y_pred is compared with target y, and the loss represents their mismatch.

Related Topics

Related concepts

More concepts starting with L