Halfspace Hypotheses
The same dot product ⟨w, x⟩ can be interpreted probabilistically by logistic regression or deterministically by a halfspace.
Try it: Logistic Regression
How logistic regression turns the linear score ⟨w,x⟩ + b into a probability , how gradient descent on the average logistic loss moves the boundary, and how the decision threshold decides which points count as training errors.
How it works
- The linear stage computes a real score ⟨w,x⟩ + b for every point; the sigmoid stage maps it into [0, 1] as , read as the probability that the label is +1.
- Each labelled point (y ∈ {±1}) costs log(1 + exp(−y(⟨w,x⟩ + b))): small when y·score is large and positive, large when the score contradicts the label.
- ERM averages these losses into ; gradient descent starts at (every , every loss ) and repeats .
- Or place the boundary yourself: the line through handles A and B, with steepness ‖w‖, and read off every probability and loss.
- Finally the threshold t: predict +1 when , i.e. when ; count the training errors.
Default run (57 steps): Start at w = 0, b = 0: every score is 0, so every h_w(x) = σ(0) = 0.5 and every logistic loss is ln 2 ≈ 0.693. Gradient descent now steps w ← w − η∇LS(w) with η = 0.5. … Trained: w = (0.34, 0.478), b = -0.521, average logistic loss 0.51. Threshold: predict +1 when h_w(x) ≥ 0.5 ⇔ ⟨w,x⟩ + b ≥ ln(0.5/0.5) = 0. That gives 2 training errors (p_6, p_12).
Simplified: Two features on a half-unit grid, at most 14 points, full-batch gradient descent with a fixed step size and no regularisation. On separable data the weights keep growing, so training simply stops after the chosen number of iterations (or when the gradient is below 1e-9). Some intermediate iterations are grouped into one step when there are more than 60.
Loading the simulation…
One Score, Two Interpretations
Both logistic regression and a halfspace hypothesis begin with the same quantity: the dot product ⟨w, x⟩. The difference is what happens next. A halfspace uses the sign of this value to choose a label directly, while logistic regression passes the value through the sigmoid function to produce a probability.
From Dot Product to Probability
Logistic regression treats ⟨w, x⟩ as an input to the sigmoid function. The sigmoid converts that input into a probability between the two extremes of 0 and 1. When the dot product is very large, the resulting probability is near one extreme. When the dot product is very small, meaning very negative, the resulting probability is near the opposite extreme. When the dot product is close to zero, the logistic output is near 0.5.
In this comparison, a logistic prediction is probabilistic because its output communicates how close the model is to one probability extreme or the other. A value near 0.5 indicates uncertainty rather than a strong preference for either extreme.
Reading Extreme and Central Scores
Three dot-product situations
Interpret the logistic output when ⟨w, x⟩ is very positive, close to zero, or very negative.
- 1Very positive
A very large dot product is sent through the sigmoid and produces a probability near one extreme, near 1.
- 2Close to zero
A dot product close to zero produces an output near 0.5. The model is least certain because the result is near the middle rather than near either probability extreme.
- 3Very negative
A very small dot product, meaning a very negative one, is sent through the sigmoid and produces a probability near the opposite extreme, near 0.
The magnitude and sign of ⟨w, x⟩ determine where the logistic probability lies: near 1 for very positive values, near 0.5 for values close to zero, and near 0 for very negative values.
The important pattern is not a particular numerical calculation. It is the interpretation of location. Large magnitude means the logistic output is near an extreme, while magnitude close to zero means the output is near the middle.
Logistic Versus Halfspace Predictions
| Method | Use of ⟨w, x⟩ | Type of output | Treatment of uncertainty |
|---|---|---|---|
| Logistic regression | Passes the dot product through the sigmoid | A probability | Can indicate uncertainty with an output near 0.5 |
| Halfspace hypothesis | Uses the sign of the dot product | A label of 1 or -1 | Does not express uncertainty |
The starting quantity is the same, but the prediction rule is different.
Suppose the shared dot product is close to zero. Logistic regression preserves that lack of a strong direction by producing a probability near 0.5. The halfspace does something different: it still uses the sign and returns either 1 or -1. Therefore, the halfspace gives a definite label even in the region where logistic regression communicates the greatest uncertainty.
The Decision Boundary and Uncertainty
The value |⟨w, x⟩| measures how far the dot product is from zero in magnitude. When this magnitude is close to zero, the dot product is near the central transition between negative and positive values. Logistic regression responds with an output near 0.5, so it is least certain. A halfspace does not show this gradation: it uses the sign and returns 1 or -1.
Common Interpretation Mistakes
Common mistake Treating the dot product itself as the final logistic probability.
What it looks likeReading ⟨w, x⟩ as if it were already a value between 0 and 1.
Why it failsLogistic regression first passes the dot product through the sigmoid function.
Do this insteadSeparate the shared score ⟨w, x⟩ from the sigmoid output, which is the probability.
Common mistake Assuming that a halfspace reports uncertainty.
What it looks likeInterpreting the halfspace label -1 as a weak prediction because the dot product is near zero.
Why it failsA halfspace always returns 1 or -1 and does not express uncertainty.
Do this insteadUse the logistic output near 0.5 to identify uncertainty; treat the halfspace output as a deterministic label.
Common mistake Thinking that an output near 0.5 means high certainty.
What it looks likeCalling a probability near 0.5 a strong positive prediction.
Why it failsThe source identifies an output near 0.5 as uncertainty when the magnitude of the dot product is close to zero.
Do this insteadRecognize that probabilities near the middle signal less certainty than probabilities near the extremes.
Check Your Interpretation
For each situation, state whether logistic regression should be near a probability extreme or near 0.5, and state whether a halfspace would express uncertainty: a very positive dot product, a very negative dot product, and a dot product close to zero.
Hints
- Very positive and very negative dot products lead to opposite probability extremes.
- A dot product close to zero leads to a logistic output near 0.5.
- The halfspace returns 1 or -1 in every case.
- 1A correct response should identify very positive values with a probability near one extreme, very negative values with the opposite extreme, and values close to zero with a probability near 0.5. Only the logistic prediction communicates the uncertainty near the center; the halfspace still returns 1 or -1.
Key Takeaways
- Logistic regression and a halfspace hypothesis start with the same dot product ⟨w, x⟩.
- Logistic regression sends the dot product through the sigmoid function to produce a probability.
- Very large and very small dot products produce probabilities near opposite extremes, while a dot product close to zero produces an output near 0.5.
- A halfspace uses the sign of the dot product to return 1 or -1 and does not express uncertainty.
- Logistic regression is least certain when |⟨w, x⟩| is close to zero because its output is near the middle value, 0.5.