Summary
Backpropagation (Rumelhart, Hinton, Williams, 1986) is the algorithm that lets a network learn — apply the chain rule to compute how much each weight contributed to the error. It was known before 1986 and worked without ceremony after 2006. What changed was everything else around it.
Objectives
- 01State backpropagation in one sentence.
- 02Explain why backprop had to wait for the deep-learning era.
- 03Distinguish the algorithm from the implementations that made it fast.
The Lesson
The algorithm
Backpropagation computes gradients by applying the chain rule backward through the computation graph. Given a loss and a network, it produces the direction each weight should move to reduce the loss.
Why it waited
Backprop needed differentiable networks, non-saturating activations, good initialization, and lots of labeled data. Without those, gradients vanished or exploded. The algorithm was correct; the conditions took thirty years.
Key Ideas
- Backprop is the chain rule applied to computation graphs.
- The algorithm was ready in 1986; the ecosystem was ready in 2012.