Summary
Support vector machines (Vapnik, 1990s) were the dominant paradigm for a decade. The kernel trick let linear classifiers learn nonlinear decision boundaries; the theoretical guarantees were the strongest in ML. SVMs held the field until deep learning ate their applications in the early 2010s.
Objectives
- 01Explain the kernel trick without formalism.
- 02State the maximum-margin principle.
- 03Explain what SVMs were displaced by, and why.
The Lesson
The theory
An SVM finds the hyperplane that maximally separates the classes. The kernel trick maps inputs to a higher-dimensional space where such a plane exists without ever computing the mapping explicitly.
The displacement
SVMs excelled on small, clean datasets with hand-crafted features. When features became learnable (deep nets), the feature-engineering advantage disappeared. SVMs remain useful on small tabular problems but no longer define the frontier.
Key Ideas
- SVMs are the theoretical high-water mark of pre-deep-learning ML.
- Feature learning killed feature engineering as a competitive edge.
References
- Cortes & Vapnik, 'Support-Vector Networks,' Machine Learning (1995)