Linear Decision Boundaries

A halfspace hypothesis provides a rule for making that binary decision.

Programming

One Space, Two Labels

Imagine a feature space in which every instance must receive one of two labels. A halfspace hypothesis provides a rule for making that binary decision. It separates the space into regions associated with the labels -1 and +1. The separating boundary is controlled by a weight vector and a bias term.

Definition

A halfspace hypothesis assigns one of two labels to an input vector. Its weight vector and bias parameter determine how the input space is divided between those labels.

How does a halfspace hypothesis map an input vector to one of two labels based on the separating rule?

Tracing the Boundary

In two-dimensional feature space, the separating hyperplane is a line. A boundary can be represented by an equation such as w·x + b = 0, where x is an input vector, w is the weight vector, and b is the bias term. The points that satisfy the boundary equation lie on the line. The rest of the feature space lies on one side or the other.

Worked example

Reading a Two-Dimensional Boundary

Suppose a two-dimensional feature space is divided by the boundary w·x + b = 0. Determine what the boundary represents and how an input is classified geometrically.

1
Locate the boundary
The equation w·x + b = 0 identifies the line that separates the two regions of the feature space.
2
Identify the regions
The line has two sides. Each side is associated with one of the two possible classification labels.
3
Classify the input
An input vector is assigned the label associated with the side of the line on which it lies.

Result

The halfspace hypothesis uses the line as a decision boundary and assigns an input to one of the two labels according to its side of the line.

How does an equation such as w·x + b = 0 create a line that divides a two-dimensional feature space?

Weights, Bias, and Orientation

The weight vector and bias parameter determine the hypothesis. In two dimensions, the corresponding hyperplane is a line perpendicular to the weight vector. This gives the weight vector a geometric role: it determines the direction that is perpendicular to the boundary and therefore controls the boundary's orientation.

The bias term is part of the parameter set that determines the hypothesis and its boundary. Changing the parameters changes the hypothesis and therefore changes the boundary or its orientation. In a geometric sketch, changing the weight vector changes the boundary's orientation, while changing the bias changes where the boundary is positioned in the feature space.

How does changing the weight vector rotate the boundary, and how does changing the bias shift it?

Positive and Negative Sides

The hyperplane is the separating boundary, not one of the two regions. The regions on its sides are the halfspaces associated with the possible classifications. In the two-label setting described here, one side is associated with the positive label +1 and the other side with the negative label -1.

How do the two sides of the decision boundary correspond to positive and negative class labels?

In the real world

Consider any application in which each instance must be assigned to one of two categories. A halfspace hypothesis places the instances in a feature space, establishes a separating hyperplane, and associates one side with +1 and the other side with -1. The labels come from the regions defined by the boundary; the words used for the categories depend on the application.

Common Misunderstandings

Common mistake
Treating the hyperplane as one of the classes
What it looks like
Calling the boundary itself the positive region or the negative region.
Why it fails
The hyperplane is the separating boundary. The two regions are the parts of feature space associated with the two classifications.
Do this instead
Distinguish the line in two dimensions from the two sides that it separates.
Common mistake
Assuming above and below always mean physical vertical position
What it looks like
Interpreting the positive class as physically higher in every application.
Why it fails
Above and below refer to the geometric relationship with the hyperplane and the direction of the weight vector.
Do this instead
Use the hyperplane and weight-vector relationship to interpret the two sides.
Common mistake
Ignoring the parameters that define the hypothesis
What it looks like
Describing a boundary without considering the weight vector or bias term.
Why it fails
The weight vector and bias parameter determine the hypothesis. Changing parameters changes the hypothesis and can change the boundary or its orientation.
Do this instead
When analyzing a boundary, identify both the weight vector and the bias term.
Common mistake
Thinking a two-dimensional hyperplane is a higher-dimensional object in the picture
What it looks like
Looking for a surface instead of a line in a two-dimensional feature space.
Why it fails
In two dimensions, the corresponding hyperplane is a line.
Do this instead
Use the dimension of the feature space: for the two-dimensional case, visualize a line separating two regions.

Practice and Summary

MEDIUM

Explain, in your own words, what changes when the weight vector changes and what changes when the bias term changes. Then describe the roles of the hyperplane, the positive halfspace, and the negative halfspace in a two-dimensional classification problem.

Hints

Start with the relationship between the weight vector and the line's orientation.
Then explain how the parameters determine the hypothesis and its boundary.
Separate the boundary itself from the two regions on its sides.

Your attempt (not saved)

1
A halfspace hypothesis assigns one of two labels, commonly -1 and +1, to an input vector.
2
The weight vector and bias term determine the hypothesis.
3
In two-dimensional feature space, the hyperplane is a line perpendicular to the weight vector.
4
The line separates feature space into two regions, associated with positive and negative labels.
5
Above and below describe geometric relationships to the hyperplane and weight-vector direction, not necessarily physical vertical position.

At a glance

A halfspace hypothesis creates a binary decision rule by dividing feature space into two label-associated regions.
The weight vector and bias parameter determine the decision hypothesis.
In two dimensions, the separating hyperplane is a line perpendicular to the weight vector.
The two sides of the line correspond to positive and negative labels.
Changing the parameters changes the boundary or its orientation.

More concepts starting with L