Logistic Loss Function
The reasoning chain is: logistic loss measures prediction quality, ERM averages that loss over the training set, and convexity with respect to w makes the resulting minimization efficiently solvable using standard methods.
Try it: Loss Functions of the Margin
How the 0-1, hinge, logistic and squared losses each score a classifier's margin z = y(⟨w,x⟩ + b), and why hinge (and base-2 logistic) are convex surrogates that upper-bound the 0-1 loss.
How it works
- For each point compute ⟨w,x⟩ + b and the margin z = y(⟨w,x⟩ + b): positive means correctly classified.
- 0-1 loss: 1 if z ≤ 0, else 0.
- Hinge loss: ; logistic loss: ; squared loss: , which equals because .
- Average each loss over the points to get the empirical risks.
- Hinge, squared and base-2 logistic are ≥ the 0-1 loss everywhere; natural-log logistic dips below it near z = 0.
Default run (9 steps): Classifier w = (0.5, 0.866), b = 0. For each point the margin z = y(⟨w,x⟩ + b) is positive when it is on its correct side; every loss is a function of z. … Averages (empirical risks): 0-1 0.286, hinge 0.338, logistic 0.333, squared 1.887. Hinge and squared loss are ≥ the 0-1 loss at every point; natural-log logistic is below it at 2 points (ln 2 ≈ 0.69 < 1 at z = 0) — use base 2 for a true upper bound.
Simplified: Linear classifier in 2-D with at most 10 points; the boundary case z = 0 counts as a mistake.
Loading the simulation…
Try it: Logistic Regression
How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.
How it works
- The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
- Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
- ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
- Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
- Finally the threshold t: predict +1 when , i.e. when ; count the training errors.
Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇LS(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).
Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.
Loading the simulation…
Why the Loss Shape Matters
Logistic regression needs a way to measure how bad each prediction is for its true label. Logistic loss provides that measure. The important property in this topic is not only that logistic loss is used for logistic regression, but that it is convex with respect to w. That convexity connects the loss to Empirical Risk Minimization, or ERM: the training problem can be solved efficiently using standard methods.
The reasoning chain is: logistic loss measures prediction quality, ERM averages that loss over the training set, and convexity with respect to w makes the resulting minimization efficiently solvable using standard methods.
Reading the Signed Prediction
The logistic loss for one labeled example is log(1 + exp(−y〈w, x〉)). Here, y is the true label, w is the parameter vector, and 〈w, x〉 is the model's score for the example. The product y〈w, x〉 evaluates that score relative to the actual label.
The signed quantity y〈w, x〉 is the central signal inside the loss. When it is positive, the prediction supports the true label. When it is negative, the prediction disagrees with the true label and the loss is higher. Its magnitude also matters: a strongly positive signed quantity represents stronger support for the true label, while a strongly negative signed quantity represents stronger contradiction.
Comparing Three Signed Predictions
Compare the logistic loss when the signed quantity y〈w, x〉 is strongly positive, near zero, or strongly negative.
- 1Strongly positive
The model's score supports the actual label. The prediction is favorable, so the logistic loss is low relative to the other cases.
- 2Near zero
The score provides little directional support for either agreement or contradiction. The loss is not as favorable as in the strongly positive case.
- 3Strongly negative
The score contradicts the actual label. The logistic loss increases because the prediction works against the true label.
As y〈w, x〉 becomes less favorable and eventually negative, logistic loss increases.
From Individual Loss to ERM
The logistic loss evaluates one prediction on one labeled example. Empirical Risk Minimization changes the scale of the problem: instead of considering one example in isolation, ERM computes the average logistic loss over all training examples and seeks parameters w that minimize that average.
A Training Set's Average Loss
Suppose a training set contains several labeled examples. Each example receives its own logistic loss. What quantity does ERM use?
- 1Evaluate each example
For every training example, evaluate how well the prediction agrees with its true label using the logistic loss.
- 2Collect the losses
Keep the individual loss values for all examples in the training set.
- 3Average the losses
Compute the average of those individual logistic losses.
- 4Choose parameters
ERM seeks the parameter vector w that minimizes this average.
ERM minimizes average logistic loss, connecting the quality of individual predictions to the overall training objective.
Convexity and Efficient Minimization
The logistic loss function is convex with respect to w. This is the central mathematical property in the topic. Because the loss has this convexity property, the ERM problem for logistic regression can be minimized efficiently using standard methods.
Tracing the ERM Reasoning
Explain why logistic regression's ERM problem has an efficient solution path according to the source.
- 1Start with the loss
Logistic loss measures how bad each prediction is for its true label.
- 2Aggregate over data
ERM forms the average logistic loss across the training examples.
- 3Use the shape
The logistic loss is convex with respect to w, which is the property relevant to minimizing the ERM objective.
- 4State the conclusion precisely
Standard methods can minimize the ERM problem efficiently. The conclusion does not specify one particular algorithm.
Convexity supplies the link between the logistic regression objective and efficient standard-method minimization.
Common Interpretation Errors
Common mistake Inspecting 〈w, x〉 without considering the label
What it looks likeTreating the score 〈w, x〉 by itself as the complete indicator of prediction quality.
Why it failsThe loss evaluates the score relative to the true label through y〈w, x〉.
Do this insteadInterpret the signed product first. Its sign tells you whether the score supports or contradicts the actual label.
Common mistake Thinking a negative y〈w, x〉 means a low loss
What it looks likeAssuming that a negative signed quantity is favorable because negative values appear inside the expression.
Why it failsNegative values indicate disagreement with the label and lead to higher logistic loss.
Do this insteadRead the entire signed prediction semantically: negative means contradiction, not a good prediction.
Common mistake Confusing individual loss with empirical risk
What it looks likeDescribing the loss on one example as the complete ERM objective.
Why it failsERM uses the average logistic loss over all training examples.
Do this insteadSeparate the local quantity for one example from the average quantity used by ERM.
Common mistake Turning the convexity claim into an algorithm claim
What it looks likeSaying that the source specifically requires one named optimization algorithm.
Why it failsThe source establishes efficient minimization using standard methods, not a particular algorithm.
Do this insteadState only that convexity gives the ERM problem an efficient standard-method solution path.
Check Your Understanding
A labeled example has a strongly negative value of y〈w, x〉. Explain what this says about the prediction and what happens to its logistic loss. Then explain how this individual loss contributes to the ERM objective.
Hints
- Use the sign of y〈w, x〉 to compare the prediction with its true label.
- Remember that ERM averages individual logistic losses across the training set.
- State the convexity result separately from the interpretation of this one example.
What do you think happens?
As y〈w, x〉 changes from strongly positive to strongly negative, what happens to logistic loss?
Reveal answer
It generally increases.
Strongly positive values support the true label, while negative values indicate disagreement. The source identifies increasing loss as the signed prediction becomes less favorable and negative.
Key Takeaways
- 1Logistic loss measures how bad a logistic regression prediction is for its true label.
- 2Its expression is log(1 + exp(−y〈w, x〉)), so the signed quantity y〈w, x〉 is essential to interpretation.
- 3Positive signed values support the true label, while negative values indicate disagreement and produce higher loss.
- 4ERM minimizes the average logistic loss over the training set.
- 5Logistic loss is convex with respect to w, which allows the ERM problem for logistic regression to be solved efficiently using standard methods without specifying a particular algorithm.
At a glance
- Logistic loss evaluates prediction quality relative to the true label.
- The sign and magnitude of y〈w, x〉 indicate whether a prediction supports or contradicts that label.
- ERM averages the individual logistic losses across the training set.
- Convexity with respect to w makes logistic regression's ERM problem efficiently solvable using standard methods.
- This convexity conclusion does not identify a specific optimization algorithm.