Summary
Neural networks survived the winters as a research minority — Hinton, LeCun, Bengio kept the work alive through the SVM era. The revival began with 2006 papers on deep belief nets, gathered force with better activations (ReLU) and initialization, and broke through with AlexNet in 2012.
Objectives
- 01Name the researchers who kept connectionism alive.
- 02Explain why the 2006-2012 revival was possible when the 1980s wave was not.
- 03State the technical enablers that let deep networks train.
The Lesson
The keepers of the flame
Hinton (Toronto), LeCun (NYU), Bengio (Montreal), and Schmidhuber (IDSIA) continued neural-network research when the field had moved to kernels and Bayes. The 2018 Turing Award recognized the first three.
What made 2012 possible
Larger datasets (ImageNet), faster hardware (GPUs), and small technical fixes (ReLU activations, dropout, better initialization) let networks with tens of layers actually train.
Key Ideas
- The revival was preceded by two decades of unfashionable research.
- Deep learning needed data, hardware, and a handful of small fixes — all at once.