Linear Functions

The central mechanism is a composition: a linear function acts first, and the logistic sigmoid function acts second.

Programming

Interactive lab

Try it: Logistic Regression

How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.

How it works

The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
Finally the threshold t: predict +1 when , i.e. when ; count the training errors.

Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇L_{_S}(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).

Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.

Educational simulation

Loading the simulation…

From Input to Probability

Logistic regression is used for classification tasks, but its hypothesis does not produce a class label directly as its first operation. Instead, it transforms an input into a value in the interval [0, 1]. For an input x, this final value h(x) can be interpreted as the probability that x has label 1.

Key point

The central mechanism is a composition: a linear function acts first, and the logistic sigmoid function acts second.

How does an input move through the linear function first and then through the sigmoid function to produce the final hypothesis value?

Tracing the Two Stages

The first stage belongs to the class of linear functions L_d. It receives an input from R^d and maps that input to a real number. This real-valued result is the intermediate score produced by the linear stage.

The second stage is the logistic sigmoid function. It receives the real number produced by the linear function and maps that number into the interval [0, 1]. The final result is the hypothesis value h(x), which can be interpreted as the probability that the original input x has label 1.

What does each stage receive and return, and how does the scalar output of the linear function become the input to the sigmoid function?

What do you think happens?

Which stage must come first if the sigmoid function accepts a real number as its input?

The sigmoid function
The linear function
Either stage; order does not matter

Reveal answer

Answer:

The linear function

The linear stage maps the input from R^{^d} to a real number. That real-valued result is then suitable as the input to the sigmoid stage, which maps it into [0, 1].

A Composed Hypothesis

Definition

A composed hypothesis is formed when the output of one function becomes the input to another function. In logistic regression, the hypothesis class is formed by composing the logistic sigmoid function over the class of linear functions L_d.

Worked example

Following One Input Through the Composition

Trace a single input x through the two stages of a logistic regression hypothesis.

1
Start with x
The input x belongs to the input space R^{^d}.
2
Apply the linear stage
A linear function from L_d receives x and returns one real number.
3
Pass the intermediate result onward
The real number returned by the linear stage becomes the input to the logistic sigmoid function.
4
Apply the sigmoid stage
The sigmoid function maps that real number into the interval [0, 1].
5
Interpret h(x)
The resulting value h(x) is interpreted as the probability that x has label 1.

Result

The complete hypothesis is a two-stage mapping from x in R^{^d} to a value h(x) in [0, 1].

What is the difference between the linear function's real-valued result and the sigmoid function's bounded probability output?

Reading the Final Value

The final value is not the original input and it is not the intermediate real number from the linear stage. It is the output of the sigmoid stage. Because the sigmoid maps into [0, 1], this output can be interpreted as a probability: h(x) is the probability that x has label 1.

Worked example

Interpreting h(x)

A composed hypothesis receives an input x and produces h(x) in [0, 1]. What should the learner understand from that output?

1
Identify the stage
The value h(x) is produced after the sigmoid stage, not directly after the linear stage.
2
Identify the range
The sigmoid output lies in [0, 1].
3
Apply the interpretation
The value is interpreted as the probability that the input x has label 1.

Result

The hypothesis value h(x) is a probability interpretation for label 1, produced by applying the sigmoid to the linear stage's real-valued output.

Why does the sigmoid output represent the probability that the input belongs to label 1?

Mistakes in the Two-Stage Model

Common mistake
Treating the linear function as the complete logistic regression hypothesis.
What it looks like
Describing the real-valued output of the linear stage as the final probability.
Why it fails
The linear stage only maps an input from R^{^d} to a real number. The probability interpretation comes after the sigmoid stage.
Do this instead
Trace the input through both stages: linear function first, sigmoid function second.
Common mistake
Reversing the order of the functions.
What it looks like
Applying the sigmoid function first and then the linear function.
Why it fails
The sigmoid stage receives a real number produced by the linear stage.
Do this instead
Remember that the linear output is the sigmoid input.
Common mistake
Saying that the hypothesis immediately returns a class label.
What it looks like
Describing h(x) as label 0 or label 1 before discussing its numerical value.
Why it fails
The hypothesis first produces a value in [0, 1], which can be interpreted as the probability that x has label 1.
Do this instead
```
Describe h(x) first as a probability-valued output.
```
Common mistake
Confusing the input and output spaces of the stages.
What it looks like
Claiming that the linear function maps directly from R^d to [0, 1].
Why it fails
The linear stage maps from R^{^d} to a real number, while the sigmoid stage maps from a real number to [0, 1].
Do this instead
Name the spaces separately for each stage.

Check Your Understanding

EASY

Explain the complete path of an input x through a logistic regression hypothesis. Your explanation must name the input space of x, the output of the linear stage, the input and output of the sigmoid stage, and the interpretation of h(x).

Hints

Start with x in R^{^d}.
The linear stage returns a real number.
That real number becomes the sigmoid input.
The final value lies in [0, 1] and is interpreted as the probability of label 1.

Your attempt (not saved)

MEDIUM

A learner says, "Logistic regression uses a sigmoid function to map an input from R^{^d} directly to a probability." Rewrite the statement so that it correctly describes the two-stage composition.