Concepts / Sigmoid Function

Sigmoid Function

The linear function comes first because the sigmoid function accepts a real number as input.

  • Programming
Interactive lab

Try it: Logistic Regression

How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.

How it works

  1. The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
  2. Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
  3. ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
  4. Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
  5. Finally the threshold t: predict +1 when , i.e. when ; count the training errors.

Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇LS(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).

Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.

Educational simulation

Loading the simulation…

From Input to Probability

Logistic regression is used for classification tasks, but its hypothesis does not begin by producing a class label directly. Instead, it transforms an input into a value in the interval [0, 1]. For an input x, this value h(x) can be interpreted as the probability that x has label 1.

Input xR^d Linear functionL_dReal-valued scorea real numberSigmoid functioninput: real numberh(x)[0, 1]entersmaps toentersmaps to
How does an input move through the linear function and then the sigmoid function to produce the final hypothesis value?

The Two Function Stages

The hypothesis class in logistic regression is formed by composing the logistic sigmoid function over the class of linear functions L_d. This means that logistic regression uses two function classes in a fixed order: a linear function first and the sigmoid function second.

StageInput spaceOutput spaceRole
Linear functionRdThe real numbersMaps the input to a real-valued score
Sigmoid functionThe real numbers[0, 1]Maps the score to a bounded value

The input and output spaces of the two stages

R^d input spaceReal numberlinear output[0, 1]sigmoid outputlinear functionsigmoid function
What type of value enters and leaves the linear stage, and how does the sigmoid stage transform that value into a number between 0 and 1?

Tracing a Composed Hypothesis

Worked example

Following One Input Through Both Functions

Trace an input x through the two stages of a logistic regression hypothesis.

  1. 1
    Start with the input

    The input x belongs to the input space Rd.

  2. 2
    Apply the linear function

    The linear stage maps x to a real number. This intermediate result is a real-valued score, not yet a probability-like output.

  3. 3
    Apply the sigmoid function

    The sigmoid stage receives that real number and maps it into the interval [0, 1].

  4. 4
    Interpret h(x)

    The final value h(x) is interpreted as the probability that x has label 1.

Result

The composed hypothesis first produces a real-valued score and then converts that score into a value in [0, 1] with a probability interpretation.

Linear functionoutput: real numberReal-valued scoreintermediate resultSigmoid functionoutput: [0, 1]h(x)probability of label 1producesbecomes input toproduces
What is the difference between the linear function's score and the sigmoid function's probability-like output, and how are they connected?

Reading the Final Value

The final hypothesis value is not described as a direct class label. It is a value in [0, 1], and for an input x, h(x) can be interpreted as the probability that x has label 1. The probability interpretation belongs to the completed composition, after the real-valued result from the linear stage has passed through the sigmoid stage.

In the real world

Generated example: If a completed logistic regression hypothesis produces h(x) = 0.8 for an input x, the value is interpreted as an 0.8 probability that x has label 1. The number 0.8 is being used as an illustration of a value in [0, 1]; the important point is the interpretation of the final hypothesis value, not the particular number.

h(x)value in [0, 1]Probability of label1interpretation for input xinterpreted as
How should the final sigmoid output be interpreted as the probability that the input belongs to label 1?

Common Interpretation Errors

  • Common mistake

    Treating the linear-stage output as the final probability.

    What it looks like

    The linear function maps x to a real number, and that real number is immediately called the probability.

    Why it fails

    The linear stage produces an intermediate real-valued score. The probability interpretation belongs to the value after the sigmoid stage.

    Do this instead

    Trace both stages: first the linear mapping, then the sigmoid mapping into [0, 1].

  • Common mistake

    Putting the sigmoid function before the linear function.

    What it looks like

    The input is sent through the sigmoid function and its output is then sent into the linear function.

    Why it fails

    The stated composition uses the linear function first. The sigmoid function accepts the real-valued result of that first stage.

    Do this instead

    Use the order input from R^d, linear function to a real number, then sigmoid function to [0, 1].

  • Common mistake

    Assuming the hypothesis directly produces a class label as its first operation.

    What it looks like

    The hypothesis immediately returns label 0 or label 1.

    Why it fails

    A logistic regression hypothesis first produces a value in [0, 1]. That value can be interpreted as the probability that the input has label 1.

    Do this instead

    Distinguish the final probability-like hypothesis value from a direct class label.

  • Common mistake

    Blending the two mappings into one unexplained step.

    What it looks like

    The model turns x into a probability without identifying the intermediate real-valued score.

    Why it fails

    This hides the separate roles of the linear function and the sigmoid function.

    Do this instead

    Name the two mappings separately: R^d to the real numbers, followed by the real numbers to [0, 1].

Check Your Understanding

EASY

An input x is passed through a logistic regression hypothesis. Describe the input and output of each stage, state which stage comes first, and explain how the final value h(x) should be interpreted.

Hints
  • Start with the input space Rd.
  • The first stage produces a real number.
  • The second stage produces a value in [0, 1].
  • The final value is interpreted with respect to label 1.

What do you think happens?

Before reading the explanation, predict what kind of value the linear stage produces and what kind of value the sigmoid stage produces.

  • The linear stage produces a real number; the sigmoid stage produces a value in [0, 1].

  • The linear stage produces a value in [0, 1]; the sigmoid stage produces an input in Rd.

  • Both stages directly produce class labels.

Reveal answer
Answer:

The linear stage produces a real number; the sigmoid stage produces a value in [0, 1].

The linear stage maps the input from Rd to a real number. The sigmoid stage then maps that real number into [0, 1], where the final value can be interpreted as the probability that the input has label 1.

Key Takeaways

  1. 1
    Logistic regression is used for classification tasks.
  2. 2
    Its hypothesis class composes the logistic sigmoid function over the class of linear functions L_d.
  3. 3
    The linear stage maps an input from Rd to a real number.
  4. 4
    The sigmoid stage maps that real number into [0, 1].
  5. 5
    The final value h(x) can be interpreted as the probability that x has label 1.

At a glance

  • Logistic regression combines a linear function with a logistic sigmoid function.
  • The linear function comes first and maps Rd to a real number.
  • The sigmoid function comes second and maps that real number into [0, 1].
  • The completed hypothesis value h(x) can be interpreted as the probability that x has label 1.

Related Topics

Related concepts

More concepts starting with S