Soft-SVM and Regularization
Soft-SVM combines labeled training pairs and a positive λ parameter in an optimization problem.
Try it: Support Vector Machine: Maximum Margin
How hard-SVM picks, among all separating lines, the one whose closest training point is farthest away, which points become support vectors, and how soft-SVM trades margin for violations with λ.
How it works
- Hard-SVM: minimise ‖w‖ subject to y(⟨w,x⟩ + b) ≥ 1 for every point; the margin is 1/‖w‖.
- In 2-D the optimum is fixed by one point of each class (their perpendicular bisector) or by two points of one class and one of the other.
- Check every such candidate against all constraints and keep the feasible one with the largest margin; the points on the margin lines are the support vectors.
- Compare with the Perceptron's separator, which also separates but usually with a smaller margin.
- Soft-SVM: minimise λ‖(w, b)‖² + average hinge loss by SGD (step 1/(2λt)), outputting the average iterate.
Default run (3 steps): Hard-SVM: among all (w, b) with y(⟨w,x⟩ + b) ≥ 1 for every point, find the smallest ‖w‖ — equivalently the separator whose closest point is farthest away (margin 1/‖w‖). … Max margin: w = (0.435, 0.696), b = -1.783, ‖w‖ = 0.82; margin 1/‖w‖ = 1.219 (band width 2.438). Support vectors p_5, p_6, p_7 sit exactly on y(⟨w,x⟩ + b) = 1. Checked 64 candidate sets. The Perceptron's line also separates, but its closest point is only 0.271 away.
Simplified: 2-D, at most 12 points, exact candidate enumeration instead of a general QP solver. The soft-SVM SGD regularises the bias too (x is extended with a constant 1) and runs a fixed number of seeded steps.
Loading the simulation…
From Training Data to an Optimization Problem
Soft-SVM does not take labeled examples and calculate w and b by a direct one-step rule. Instead, the labeled training pairs and a positive parameter λ define an optimization problem. The optimization chooses w, b, and slack variables ξ so that the objective is minimized while the margin constraints are satisfied as far as the slack variables allow. The procedure returns w and b.
What the Slack Variables Record
The margin constraints describe the desired relationship between each labeled training pair and the separating parameters w and b. Soft-SVM includes a slack variable ξ for the constraints. Each slack variable is required to be nonnegative, and it allows the corresponding constraint to be satisfied less strictly when necessary. Thus, the formulation does not treat every training pair as an all-or-nothing constraint: the slack variables provide a way to represent constraint violations while still charging for them in the objective.
Balancing Norm and Violation Cost
The Soft-SVM objective has two competing parts. One part is λ ‖w‖², which involves the norm of w and is weighted by the positive parameter λ. The other part is the average slack cost. Minimizing the objective therefore requires the optimization to consider both the size-related term involving w and the amount of constraint violation recorded by the slack variables.
Reading a Soft-SVM Setup
Suppose a Soft-SVM problem is supplied with labeled training pairs and a positive λ. Identify what is input, what is optimized, and what is returned.
- 1Identify the inputs
The labeled training pairs and the positive parameter λ are the inputs that define the optimization problem.
- 2Identify the optimization variables
The optimization chooses w, b, and the slack variables ξ. The ξ variables are constrained to be nonnegative.
- 3Identify the objective terms
The objective balances λ ‖w‖² with the average slack cost, so both the norm-related term and the constraint-violation term matter.
- 4Identify the output
After the optimization problem is solved, the procedure lists w and b as its output.
Soft-SVM is an optimization-based procedure: labeled pairs and positive λ define the problem, w, b, and ξ are chosen within it, and w and b are returned.
Following Constraints into the Objective
The formulation connects each training pair to the objective through its margin constraint. First, the constraint is checked relative to the chosen w and b. If the constraint cannot be fully met, the corresponding nonnegative slack variable permits the formulation to account for that shortfall. Those slack values then contribute to the average slack cost. The optimization therefore cannot choose w and b while ignoring ξ: the constraints determine what slack is needed, and the objective penalizes the resulting average slack cost.
When reading the Soft-SVM formulation, separate three questions: which objects are supplied, which quantities are optimized, and which quantities are returned. This prevents the common mistake of treating the slack variables as outputs in the same sense as w and b. They participate in the optimization, while the stated procedure outputs w and b.
Corollary 15.7 Setup
The setup of Corollary 15.7 begins with a distribution D over X × {0, 1}. The set X is defined as the set of x satisfying ‖x‖ ≤ ρ. A training set S is then sampled according to Dᵐ, and A(S) denotes the solution of Soft-SVM on S.
| Statement | Role in the setup |
|---|---|
| D is a distribution over X × {0, 1} | Assumption defining the data distribution |
| X = {x satisfying ‖x‖ ≤ ρ} | Definition of the input domain |
| S is sampled according to Dᵐ | Assumption describing how the training set is sampled |
| A(S) is the solution of Soft-SVM on S | Notation for the procedure applied to the sampled training set |
| The corollary's conclusion | Not supplied here and should not be inferred from the setup |
Separate the stated assumptions and notation from the omitted conclusion.
Mistakes in Reading Soft-SVM
Common mistake Treating Soft-SVM as a direct calculation from the training pairs.
What it looks likeInput pairs are supplied, so w and b are assumed to be computed immediately.
Why it failsThe inputs and λ define an optimization problem that also contains w, b, and ξ.
Do this insteadDescribe Soft-SVM as an optimization procedure whose solution supplies w and b.
Common mistake Ignoring the slack variables after mentioning the margin constraints.
What it looks likeThe explanation lists the margin constraints but leaves ξ out of the objective.
Why it failsThe constraints and the objective both refer to the slack variables, and the nonnegative slack constraints are essential.
Do this insteadTrack ξ in both places: it permits the constraints to be relaxed and contributes to the average slack cost.
Common mistake Describing λ as an extra training example or as an output.
What it looks likeThe procedure is said to return the training pairs and λ.
Why it failsλ is a positive parameter used in the objective, while the stated outputs are w and b.
Do this insteadClassify λ as an input parameter and w and b as the outputs.
Common mistake Treating the setup of Corollary 15.7 as if it included its conclusion.
What it looks likeThe distribution and sampling assumptions are followed by a claimed guarantee that is not stated in the available material.
Why it failsThe supplied description specifies only the setup and the notation A(S).
Do this insteadReport the assumptions and notation without inventing or inferring the omitted conclusion.
Check Your Understanding
A description says: “Given labeled pairs and positive λ, Soft-SVM returns w, b, and ξ.” Identify the inaccurate part and rewrite the description using the correct roles for the inputs, optimization variables, and outputs.
Hints
- Separate quantities chosen inside the optimization from quantities listed as the procedure's output.
- Remember that ξ participates in the constraints and objective.
For Corollary 15.7, classify each item as part of the setup or as an omitted conclusion: D, X, the sampling of S according to Dᵐ, A(S), and the corollary's final result.
Hints
- The setup names a distribution, defines a domain, describes sampling, and introduces notation.
- The available material does not state the final result.
Essential Takeaways
- 1Soft-SVM takes labeled training pairs and a positive λ parameter, then defines an optimization problem rather than directly calculating w and b.
- 2The optimization variables are w, b, and nonnegative slack variables ξ; the stated outputs are w and b.
- 3The objective balances λ ‖w‖² with the average slack cost.
- 4Margin constraints and nonnegative slack constraints work together: slack variables allow the formulation to account for constraint violations and are penalized through the objective.
- 5Corollary 15.7 assumes a distribution D over X × {0, 1}, uses X defined by ‖x‖ ≤ ρ, samples S according to Dᵐ, and denotes the Soft-SVM solution by A(S); its conclusion is not supplied here.
At a glance
- Soft-SVM is an optimization problem defined by labeled training pairs and a positive λ parameter.
- The optimization chooses w, b, and nonnegative slack variables ξ, while the procedure returns w and b.
- The objective combines λ ‖w‖² with the average slack cost, linking regularization to constraint violations.
- Corollary 15.7 describes a distribution, bounded domain, sampling process, and notation for the Soft-SVM solution; the available setup does not state its conclusion.