Support Vector Machines for Large Margin Classification
Do not calculate the margin by averaging distances. A training set may contain many distant points and one very close point; the close point still determines the margin.
Try it: Support Vector Machine: Maximum Margin
How hard-SVM picks, among all separating lines, the one whose closest training point is farthest away, which points become support vectors, and how soft-SVM trades margin for violations with λ.
How it works
- Hard-SVM: minimise ‖w‖ subject to y(⟨w,x⟩ + b) ≥ 1 for every point; the margin is 1/‖w‖.
- In 2-D the optimum is fixed by one point of each class (their perpendicular bisector) or by two points of one class and one of the other.
- Check every such candidate against all constraints and keep the feasible one with the largest margin; the points on the margin lines are the support vectors.
- Compare with the Perceptron's separator, which also separates but usually with a smaller margin.
- Soft-SVM: minimise λ‖(w, b)‖² + average hinge loss by SGD (step 1/(2λt)), outputting the average iterate.
Default run (3 steps): Hard-SVM: among all (w, b) with y(⟨w,x⟩ + b) ≥ 1 for every point, find the smallest ‖w‖ — equivalently the separator whose closest point is farthest away (margin 1/‖w‖). … Max margin: w = (0.435, 0.696), b = -1.783, ‖w‖ = 0.82; margin 1/‖w‖ = 1.219 (band width 2.438). Support vectors p_5, p_6, p_7 sit exactly on y(⟨w,x⟩ + b) = 1. Checked 64 candidate sets. The Perceptron's line also separates, but its closest point is only 0.271 away.
Simplified: 2-D, at most 12 points, exact candidate enumeration instead of a general QP solver. The soft-SVM SGD regularises the bias too (x is extended with a constant 1) and runs a fixed number of seeded steps.
Loading the simulation…
Why Separation Alone Is Not Enough
A hyperplane can separate a training set, but that fact alone does not tell us how much room exists between the training points and the separating boundary. That room matters. If the instances change slightly, a boundary with very little room may no longer separate them, while a boundary with more room can remain effective. Large-margin classification focuses on measuring this room.
Finding the Smallest Distance
The margin of a hyperplane with respect to a training set is the smallest distance from any training point to that hyperplane. To determine it, inspect every training point, consider its distance to the boundary, and select the smallest distance. The closest training point determines the margin; distances from the other points cannot make the margin larger.
Selecting the margin
Suppose three training points have distances 5, 2, and 3 from a separating hyperplane. Which distance determines the margin?
- 1List the distances
The distances from the training points to the hyperplane are 5, 2, and 3.
- 2Find the smallest distance
Among these distances, 2 is the smallest.
- 3Interpret the result
The point at distance 2 is the nearest training point, so its distance determines the margin.
The margin is determined by the smallest distance, which is 2 in this generated example.
Support Vectors at the Boundary of Safety
The training points that are closest to the hyperplane are the points that determine the available room. They are often called support vectors in the context of support vector machines because they support the margin: moving the boundary's closest training point farther away would change the minimum distance, while points already farther away do not determine that minimum.
Do not calculate the margin by averaging distances. A training set may contain many distant points and one very close point; the close point still determines the margin.
How Margin Creates Robustness
A large margin means that even the closest training point is relatively far from the hyperplane. That distance provides room for the point to move slightly while the same hyperplane continues to separate the training set. By contrast, when the margin is small, a small perturbation can move the closest point across the boundary or otherwise interfere with separation.
Training point has substantial distance from the hyperplane.
Large available margin
Training point moves slightly toward the hyperplane but remains separated.
Separation preserved after a slight perturbation
Margin and True Error
The margin of a halfspace does not directly report the exact true classification error. Instead, the true error can be bounded using the margin. The important relationship is that increasing the margin is associated with a smaller bound on true error. This relationship does not depend on the Euclidean dimension.
| Quantity | What it tells us | What it does not tell us |
|---|---|---|
| Margin | The smallest distance from a training point to the hyperplane | The exact true classification error |
| True error bound | A bound whose size is related to the margin | A guarantee that the exact error equals the bound |
| Larger margin | More room for slight perturbations and an associated smaller error bound | Perfect performance under every possible change |
Keep the geometric margin separate from the true error and from the bound on true error.
Mistakes in Margin Reasoning
Common mistake Using the average distance instead of the smallest distance
What it looks likeDistances from three training points are 5, 2, and 3.
Why it failsThe margin is controlled by the nearest training point, not by a typical or average point.
Do this insteadChoose the smallest distance. In this example, the margin is determined by 2.
Common mistake Treating separation as proof of robustness
What it looks likeA hyperplane separates every current training point but leaves almost no room around the closest point.
Why it failsA slight perturbation may affect separation when the closest point is very near the hyperplane.
Do this insteadInspect the minimum distance and use it to judge how much room the separation has.
Common mistake Calling the margin the exact true error
What it looks likeThe margin is large, so the true error must be zero.
Why it failsThe margin provides a quantity in terms of which true error can be bounded; it does not directly report the exact true error.
Do this insteadState that a larger margin is associated with a smaller bound on true error.
Common mistake Assuming the margin relationship depends on Euclidean dimension
What it looks likeA margin-based relationship is treated as useful only in a particular number of dimensions.
Why it failsThe source concept states that the relationship between increasing margin and a smaller true-error bound does not depend on the Euclidean dimension.
Do this insteadSeparate the geometric measurement from assumptions about a specific dimension.
Check Your Understanding
A hyperplane separates a training set. Four training points have distances 4, 7, 1, and 5 from the hyperplane. Identify which point determines the margin, state what happens to the margin if that point moves farther away while all other distances stay fixed, and explain why a larger margin can make the separation more robust to slight perturbations.
Hints
- Compare all four distances.
- The margin is controlled by the smallest distance.
- Use the idea of available room, not a claim that the margin directly equals true error.
- 1The margin is the smallest distance from any training point to the separating hyperplane. The nearest training point determines it, so averaging distances gives the wrong result. A larger margin leaves more room for slight perturbations while preserving separation. For a halfspace, the margin is related to true error through a bound: increasing the margin is associated with a smaller bound, not necessarily a directly known exact error.
Key Takeaways
- The margin measures the smallest distance from a training point to a hyperplane.
- The nearest training point determines the margin; distant points do not increase it.
- A large margin provides more room for slight perturbations while preserving separation.
- The margin does not directly equal true error, but increasing it is associated with a smaller bound on true error.
- The relationship between increasing margin and a smaller true-error bound does not depend on Euclidean dimension.