Concepts / Classification Loss Functions

Classification Loss Functions

Logistic loss evaluates one prediction on one labeled example. Empirical Risk Minimization then combines those individual evaluations across the training set.

  • Programming
Interactive lab

Try it: Loss Functions of the Margin

How the 0-1, hinge, logistic and squared losses each score a classifier's margin z = y(⟨w,x⟩ + b), and why hinge (and base-2 logistic) are convex surrogates that upper-bound the 0-1 loss.

How it works

  1. For each point compute ⟨w,x⟩ + b and the margin z = y(⟨w,x⟩ + b): positive means correctly classified.
  2. 0-1 loss: 1 if z ≤ 0, else 0.
  3. Hinge loss: ; logistic loss: ; squared loss: , which equals because .
  4. Average each loss over the points to get the empirical risks.
  5. Hinge, squared and base-2 logistic are ≥ the 0-1 loss everywhere; natural-log logistic dips below it near z = 0.

Default run (9 steps): Classifier w = (0.5, 0.866), b = 0. For each point the margin z = y(⟨w,x⟩ + b) is positive when it is on its correct side; every loss is a function of z. … Averages (empirical risks): 0-1 0.286, hinge 0.338, logistic 0.333, squared 1.887. Hinge and squared loss are ≥ the 0-1 loss at every point; natural-log logistic is below it at 2 points (ln 2 ≈ 0.69 < 1 at z = 0) — use base 2 for a true upper bound.

Simplified: Linear classifier in 2-D with at most 10 points; the boundary case z = 0 counts as a mistake.

Educational simulation

Loading the simulation…

Interactive lab

Try it: Logistic Regression

How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.

How it works

  1. The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
  2. Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
  3. ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
  4. Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
  5. Finally the threshold t: predict +1 when , i.e. when ; count the training errors.

Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇LS(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).

Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.

Educational simulation

Loading the simulation…

Why a Prediction Needs a Loss

A logistic regression model produces h_w(x), a value in the interval [0, 1]. The model uses this value in relation to the two possible labels, y ∈ {−1, 1}. A prediction is not equally useful in every situation: it may support the actual label or work against it. Logistic loss turns that relationship into a numerical measure of how bad the prediction is for its true label.

Key point

Logistic loss evaluates one prediction on one labeled example. Empirical Risk Minimization then combines those individual evaluations across the training set.

Reading the Signed Score

Definition

The logistic loss is defined by log(1 + exp(−y〈w, x〉)). The important quantity inside this expression is the signed product y〈w, x〉. It evaluates the model's score relative to the actual label, rather than inspecting 〈w, x〉 by itself.

The sign of y〈w, x〉 tells us whether the model's score agrees with the label. A positive value indicates that the score supports the true label. A negative value indicates disagreement with the label and leads to higher loss. As the signed quantity becomes less favorable and eventually negative, the logistic loss increases.

Negative y〈w, x〉disagreementHigher logistic losspenalized predictionLess favorable y〈w,x〉transition towarddisagreementIncreasing logisticlossas favorability decreasesPositive y〈w, x〉supportLower logistic losssupported predictionleads tomoves towardsupports
How do the sign and favorability of y〈w, x〉 determine the direction of logistic loss?

The diagram shows direction rather than particular numerical loss values. Its central lesson is that the loss evaluates the score relative to the label. The same raw score 〈w, x〉 can therefore be interpreted differently when the actual label changes, because the product y〈w, x〉 changes with y.

From Support to Contradiction

Worked example

Comparing two labeled examples

Consider two examples evaluated by the same logistic-regression model. In the first, the signed quantity y〈w, x〉 is strongly positive. In the second, it is strongly negative. Which example receives the larger logistic loss?

  1. 1
    First example

    A strongly positive y〈w, x〉 means that the model's score supports the true label. The corresponding logistic loss is lower than it would be for a less favorable signed quantity.

  2. 2
    Second example

    A strongly negative y〈w, x〉 means that the score disagrees with the true label. Negative values lead to higher logistic loss.

  3. 3
    Comparison

    The second example receives the larger loss because its signed quantity is less favorable and negative.

Result

Logistic loss is lower when the signed score supports the true label and higher when the signed score contradicts it.

Positive y〈w, x〉strong supportLower logistic losssupported labelNegative y〈w, x〉strong contradictionHigher logistic losscontradicted labelproducesproduces
What changes in logistic loss when a model moves from strongly supporting the true label to strongly contradicting it?

Averaging Loss with ERM

The loss for one example gives a local view of model quality. Empirical Risk Minimization, or ERM, provides the global view: it minimizes the average logistic loss over all training examples. The training process therefore evaluates each labeled example, combines those loss values through their average, and seeks model parameters that minimize that average.

Training example 1x, yLogistic loss 1one predictionAverage logistic losstraining lossERMminimize averageTraining example 2x, yLogistic loss 2one predictionTraining example nx, yLogistic loss none predictionevaluateevaluateevaluateaggregateaggregateaggregateminimize
How do individual examples' logistic losses flow into the average training loss used by Empirical Risk Minimization?

This pipeline connects two levels of reasoning. First, logistic loss asks how bad one prediction is for its true label. Then ERM aggregates those individual judgments so that the model is evaluated over the entire training set rather than on only one example.

Common Interpretation Mistakes

  • Common mistake

    Treating a positive raw score 〈w, x〉 as automatically favorable.

    What it looks like

    Looking only at 〈w, x〉 and ignoring y.

    Why it fails

    Logistic loss evaluates the score relative to the actual label through y〈w, x〉.

    Do this instead

    Determine the sign of y〈w, x〉 before interpreting the prediction.

  • Common mistake

    Assuming that disagreement with the label should produce a smaller loss.

    What it looks like

    Interpreting a negative y〈w, x〉 as a successful prediction.

    Why it fails

    Negative values indicate disagreement with the label and lead to higher loss.

    Do this instead

    Associate negative signed quantities with stronger penalty.

  • Common mistake

    Confusing one example's loss with the ERM objective.

    What it looks like

    Using the logistic loss from a single training example as the complete training evaluation.

    Why it fails

    ERM minimizes the average logistic loss over all training examples.

    Do this instead

    Separate the local loss for one example from the average loss across the training set.

Check Your Reasoning

EASY

Two predictions have signed quantities y〈w, x〉 with opposite signs. Prediction A has a positive signed quantity, while Prediction B has a negative signed quantity. Without calculating numerical values, identify which prediction receives the higher logistic loss and explain why.

Hints
  • Start by interpreting the sign of each signed quantity.
  • Recall how logistic loss responds when the score disagrees with the true label.
MEDIUM

Explain the difference between these two statements: logistic loss evaluates one labeled example, and ERM minimizes average logistic loss.

Hints
  • Identify the local object being evaluated in the first statement.
  • Identify what is aggregated in the second statement.

Key Takeaways

  1. 1
    Logistic loss measures how bad a logistic-regression prediction is for its true label.
  2. 2
    Its defining expression is log(1 + exp(−y〈w, x〉)).
  3. 3
    The sign of y〈w, x〉 matters: negative values indicate disagreement with the label and lead to higher loss.
  4. 4
    Predictions that support the true label receive lower loss than predictions that contradict it, in the directional sense described by the source.
  5. 5
    Empirical Risk Minimization minimizes the average logistic loss over the training set.

At a glance

  • Logistic loss evaluates the quality of a prediction relative to its true label.
  • The signed product y〈w, x〉 is the key quantity because it incorporates both the model score and the actual label.
  • A less favorable or negative signed quantity produces higher logistic loss.
  • ERM uses the average of the individual logistic losses to evaluate and optimize performance across the training set.

Related Topics

Related concepts

More concepts starting with C