Classification with Logistic Regression
The central idea is a two-stage transformation: a linear function first produces a real-valued score, and the logistic sigmoid function then transforms that score into a value in [0, 1].
Try it: Logistic Regression
How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.
How it works
- The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
- Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
- ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
- Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
- Finally the threshold t: predict +1 when , i.e. when ; count the training errors.
Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇LS(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).
Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.
Loading the simulation…
From Input to Probability
Logistic regression is used for classification tasks, but its hypothesis does not produce a class label directly as its first operation. Instead, it transforms an input into a value in the interval [0, 1]. For an input x, this value h(x) can be interpreted as the probability that x has label 1.
The central idea is a two-stage transformation: a linear function first produces a real-valued score, and the logistic sigmoid function then transforms that score into a value in [0, 1].
What do you think happens?
Which stage should receive the original input x first?
Reveal answer
The linear stage
The linear function maps the input from Rd to a real number. The sigmoid function accepts that real-valued result and maps it into [0, 1], so the order is linear first and sigmoid second.
The Composed Hypothesis
The hypothesis class in logistic regression is formed by composing the logistic sigmoid function over the class of linear functions L_d. In practical terms, choose a linear function, apply it to x, and then pass the resulting real number to the sigmoid function. The complete composition is the logistic regression hypothesis.
Tracing One Input Through the Hypothesis
Describe what happens to an input x as it passes through a logistic regression hypothesis.
- 1Start with x
The input x belongs to the input space Rd.
- 2Apply a linear function
The linear stage maps x from Rd to a real number. This real-valued result is the score produced by the first stage.
- 3Apply the sigmoid function
The sigmoid stage takes the real-valued score as its input and maps it into the interval [0, 1].
- 4Interpret h(x)
The final value h(x) is interpreted as the probability that x has label 1.
The composed hypothesis turns an input in R^d into a value in [0, 1] with a class-1 probability interpretation.
Changing Spaces at Each Stage
| Stage | Input | Output | Role |
|---|---|---|---|
| Linear stage | An input x from Rd | A real number | Produces the score |
| Sigmoid stage | A real number | A value in [0, 1] | Produces the final hypothesis value |
The input and output spaces of the two composed stages
Keeping these spaces separate prevents a common confusion. The linear function does not directly produce the final probability interpretation. It produces a real number. The sigmoid function then takes that real number and converts it into the bounded interval [0, 1].
Two Function Classes
| Function class | Receives | Produces | Place in the composition |
|---|---|---|---|
| Linear functions L_d | An input from Rd | A real number | First |
| Logistic sigmoid function | A real number | A value in [0, 1] | Second |
The two function classes have different input and output roles
Reading the Final Value
The final hypothesis value h(x) is not described merely as an arbitrary number. Its intended interpretation is the probability that the input x has label 1. The sigmoid stage is what places the result in [0, 1], making this probability interpretation possible.
Interpreting an Abstract Output
Suppose a logistic regression hypothesis produces h(x) for an input x. What should you ask about this value?
- 1Check the range
Because the sigmoid stage maps its input into [0, 1], h(x) belongs to that interval.
- 2Identify the meaning
The value h(x) can be interpreted as the probability that x has label 1.
- 3Keep the stages distinct
The value is produced after the linear score has been passed through the sigmoid function; it is not the raw output of the linear stage.
Read h(x) as a class-1 probability, while remembering that it comes from a linear stage followed by a sigmoid stage.
Mistakes in the Two-Stage Model
Common mistake Treating the linear output as the final probability.
What it looks likeDescribing the real number produced by the linear stage as h(x).
Why it failsThe linear stage produces a real-valued score, while the sigmoid stage produces the value in [0, 1] used for the class-1 probability interpretation.
Do this insteadTrace both stages: first obtain the real-valued score, then apply the sigmoid function to obtain h(x).
Common mistake Reversing the order of the functions.
What it looks likeApplying the sigmoid function to x before applying a linear function.
Why it failsThe source composition places the linear function first. The sigmoid function receives the real number produced by that linear function.
Do this insteadUse the order linear stage, then sigmoid stage.
Common mistake Saying that logistic regression immediately outputs a class label.
What it looks likeDescribing the first output of the hypothesis as label 1 or another class label.
Why it failsThe hypothesis first produces a value in [0, 1], which is interpreted as the probability that x has label 1.
Do this insteadDescribe the immediate output as h(x), a value in [0, 1] with a class-1 probability interpretation.
Common mistake Blending the two mappings into one unexplained step.
What it looks likeSaying only that x becomes a probability without identifying the intermediate score.
Why it failsThe linear and sigmoid functions have different input and output spaces and distinct roles.
Do this insteadName the intermediate real-valued score before describing the final value in [0, 1].
Trace It Yourself
For an input x from Rd, write a four-step trace of a logistic regression hypothesis. Your trace should name the input space, the output of the linear stage, the input and output of the sigmoid stage, and the interpretation of h(x).
Hints
- The linear stage comes first.
- The linear stage produces a real number.
- The sigmoid stage maps that real number into [0, 1].
- Interpret the final value in relation to label 1.
Key Takeaways
- Logistic regression is used for classification tasks.
- Its hypothesis class composes the logistic sigmoid function over the class of linear functions L_d.
- The linear stage maps an input from Rd to a real number.
- The sigmoid stage maps that real number into [0, 1].
- The final value h(x) can be interpreted as the probability that x has label 1.