Hard-Margin SVM Optimization

Soft-SVM preserves the margin constraint while allowing controlled violations.

Programming

Interactive lab

Try it: Support Vector Machine: Maximum Margin

How hard-SVM picks, among all separating lines, the one whose closest training point is farthest away, which points become support vectors, and how soft-SVM trades margin for violations with λ.

How it works

Hard-SVM: minimise ‖w‖ subject to y(⟨w,x⟩ + b) ≥ 1 for every point; the margin is 1/‖w‖.
In 2-D the optimum is fixed by one point of each class (their perpendicular bisector) or by two points of one class and one of the other.
Check every such candidate against all constraints and keep the feasible one with the largest margin; the points on the margin lines are the support vectors.
Compare with the Perceptron's separator, which also separates but usually with a smaller margin.
Soft-SVM: minimise λ‖(w, b)‖² + average hinge loss by SGD (step 1/(2λt)), outputting the average iterate.

Default run (3 steps): Hard-SVM: among all (w, b) with y(⟨w,x⟩ + b) ≥ 1 for every point, find the smallest ‖w‖ — equivalently the separator whose closest point is farthest away (margin 1/‖w‖). … Max margin: w = (0.435, 0.696), b = -1.783, ‖w‖ = 0.82; margin 1/‖w‖ = 1.219 (band width 2.438). Support vectors p_5, p_6, p_7 sit exactly on y(⟨w,x⟩ + b) = 1. Checked 64 candidate sets. The Perceptron's line also separates, but its closest point is only 0.271 away.

Simplified: 2-D, at most 12 points, exact candidate enumeration instead of a general QP solver. The soft-SVM SGD regularises the bias too (x is extended with a constant 1) and runs a fixed number of seeded steps.

Educational simulation

Loading the simulation…

When Perfect Separation Is Too Strict

Hard-margin SVM requires every training example to satisfy the margin constraint yi(<w, xi> + b) ≥ 1. That requirement can be too strict in practice: some examples may fail to meet it. Soft-SVM keeps the original margin idea, but it does not discard examples that violate the requirement. Instead, it allows controlled violations and assigns a cost to allowing them.

What changes when every example must satisfy the margin constraint versus when violations are explicitly allowed?

Tracing One Training Example

For a particular training example, first evaluate the original margin expression yi(<w, xi> + b). If it already meets the requirement of being at least 1, that example needs no relaxation. If it falls short, Soft-SVM introduces a corresponding nonnegative slack variable ξi. This variable records how much the original requirement is being relaxed for that example.

For one training example, how does the required relaxation change as the example moves from outside the margin to inside the margin or beyond the classification boundary?

Worked example

Following ξi for Three Cases

Compare the role of ξi when a training example satisfies the desired margin, falls inside the margin, or is misclassified.

1
Outside the margin
If yi(<w, xi> + b) is at least 1, the original requirement is satisfied, so no relaxation is needed for that example. Its slack can be zero.
2
Inside the margin
If the margin expression is below 1 but the example is not described as a wrong-side example, the original requirement is violated. A positive ξi records the required relaxation.
3
Misclassified
A wrong-side example also fails the desired margin condition. Its ξi records a larger or more substantial relaxation than the case that merely falls short of the margin.

Result

ξi belongs to one example and measures that example's relaxation of the original margin requirement; it is not a replacement for w or b.

Building the Soft-SVM Objective

The Soft-SVM optimization balances two goals. The first is the norm-related term involving w, which represents the margin-related part of the optimization. The second is the average slack term, which represents the cost of permitting constraint violations. Soft-SVM does not simply minimize violations without regard to w, and it does not simply optimize the norm while ignoring violations. It balances both.

Formula, step by step

1
The norm-related term evaluates the part of the objective associated with w and the margin.
2
The average slack term aggregates the relaxations assigned to the training examples.
3
λ weights the average slack term relative to the norm-related term.
4
The optimizer therefore seeks a balance between the size of the norm-related term and the amount of permitted violation.

What role does each part of the Soft-SVM objective play?

Changing the λ Balance

λ controls how strongly the optimization reacts to the average slack term. A larger relative weight on slack makes constraint violations more costly in the objective, so the balance shifts toward accepting fewer or smaller violations, even if that affects the norm-related part. A smaller relative weight on slack makes violations less costly, so the optimization can place more emphasis on the norm-related term. The important idea is not that λ changes the meaning of ξi; ξi still measures relaxation for its own example. λ changes how heavily those relaxations count in the overall objective.

How does changing λ shift the balance between the norm-related term and constraint violations?

From Slack Constraints to Hinge Loss

The constrained formulation and the regularized-loss formulation describe the same Soft-SVM idea from two perspectives. In the explicit-constraint view, every training example has its own nonnegative ξi, and the constraint is relaxed when necessary. In the loss-minimization view, the norm-related part acts as a regularization term, while the loss part measures failure to satisfy the desired margin condition. That loss is the hinge-loss perspective on the accumulated margin violations.

How are explicit slack constraints connected to regularized loss minimization?

Worked example

Recognizing the Equivalent View

A Soft-SVM explanation mentions a norm-related term and an accumulated penalty for examples that fail the desired margin. Identify the corresponding parts of the regularized-loss interpretation.

1
Find the regularization part
The norm-related term is interpreted as the regularization term in the loss-minimization view.
2
Find the loss part
The penalty associated with slack records the loss caused by failing to satisfy the desired margin condition.
3
Connect the views
The explicit ξi formulation and the regularized hinge-loss formulation are two ways to describe the same balance between margin-related preference and constraint violations.

Result

Soft-SVM can be understood either as optimization with explicit slack constraints or as regularized loss minimization with hinge loss.

Mistakes in Reading Soft-SVM

Common mistake
Treating ξi as a replacement for w or b.
What it looks like
Thinking that the slack variables define the classifier instead of recording individual constraint relaxations.
Why it fails
ξi is attached to a particular training example. It records the amount by which that example's original requirement is relaxed.
Do this instead
Keep the roles separate: w and b are classifier parameters, while each ξi describes one example's relaxation.
Common mistake
Assuming Soft-SVM ignores the margin constraint.
What it looks like
Describing Soft-SVM as if it simply accepts any classification result.
Why it fails
Soft-SVM preserves the margin constraint as the desired requirement and permits controlled violations by adding slack and a cost.
Do this instead
Say that the constraint is relaxed, not removed.
Common mistake
Confusing ξi with the tradeoff parameter λ.
What it looks like
Using λ to describe how much one particular example violates the margin.
Why it fails
ξi belongs to one training example, whereas λ controls the relative balance between the norm-related term and the average slack term.
Do this instead
Use ξi for an example-level relaxation and λ for the objective-level tradeoff.
Common mistake
Looking only at the slack term.
What it looks like
Assuming the objective should minimize violations without considering the norm-related term.
Why it fails
The Soft-SVM objective balances both the norm-related term and the average slack term.
Do this instead
Interpret the objective as a two-goal balance.
Common mistake
Missing the regularized-loss interpretation.
What it looks like
Treating explicit slack constraints and hinge-loss minimization as unrelated methods.
Why it fails
The constrained problem can be rewritten as regularized loss minimization with hinge loss.
Do this instead
Translate between the two views: norm-related term becomes regularization, and margin failures become hinge-loss penalties.

Check Your Interpretation

MEDIUM

A training example fails the original requirement yi(<w, xi> + b) ≥ 1. Explain what ξi records, identify which part of the objective charges for this failure, and state what changing λ affects.

Hints

Start with the fact that ξi belongs to this particular training example.
Separate the norm-related term from the average slack term.
λ changes the relative weight of the slack term; it does not change the meaning of ξi.

Your attempt (not saved)

What do you think happens?

Which statement best describes a larger relative weight on the slack term?

Constraint violations count more heavily in the objective.
Each ξi is replaced by w.
The margin constraint is discarded entirely.
The norm-related term disappears.

Reveal answer

Answer:

Constraint violations count more heavily in the objective.

λ controls the balance between the norm-related term and the average slack term. Increasing the relative weight on slack makes permitted violations more costly, while ξi remains an example-specific relaxation.

Key Takeaways

1
Soft-SVM keeps the desired margin constraint but permits controlled violations when the hard requirement is too strict.
2
Each nonnegative ξi belongs to one training example and measures that example's constraint relaxation.
3
The objective balances a norm-related term against an average slack term.
4
λ controls how strongly constraint violations influence that balance.
5
The explicit-slack formulation can also be viewed as regularized loss minimization with hinge loss.

At a glance

Soft-SVM replaces an inflexible requirement that every example satisfy the margin with relaxed constraints and a cost for violations.
The slack variable ξi is specific to one training example and records how much that example's original requirement is relaxed.
The objective contains a norm-related term and an average slack term, with λ controlling their tradeoff.
The same optimization can be understood as regularized loss minimization using hinge loss.

Related concepts

More concepts starting with H