Era 1 · Origins · 1970 / 1974

5 Backprop prehistory

Reverse-mode AD (Linnainmaa, 1970) · applied to neural networks (Werbos, 1974)
🟦 this write-up is enough~20–30 minoriginal ↗
The gist in 20 seconds. Backprop is not 1986 magic but reverse-mode automatic differentiation, discovered by Linnainmaa in 1970 (as an analysis of rounding errors!) and applied to networks by Werbos in 1974. It computes the gradient with respect to all parameters for the cost of a single pass — the computational core of all of deep learning.

Context

To train a network by gradient descent you need the derivative of the error with respect to every weight. Computing it naively (one weight at a time) is hopelessly expensive for a network with millions of parameters. The solution came not from AI but from numerical analysis.

The idea and the mechanism

Any computation is a graph: nodes are operations, edges are values. The derivative comes from the chain rule; the question is in what order you apply it. The adjoint (the sensitivity of the output to a node) accumulates from that node's children:

v = ∂L∂v = Σc ∈ children(v) c · ∂c∂v

Linnainmaa described this in 1970 as a way of tracking rounding errors — with no connection to neural networks at all (a master's thesis, in Finnish). Werbos (1974, Harvard) was the first to propose applying the trick to training networks.

calculus Forward-mode vs reverse-mode: why reverse wins for ML

Let the network be a composition f = fL ∘ … ∘ f1. By the chain rule the gradient is a product of Jacobians JL · … · J1. The cost depends on the order in which you multiply the matrices:

  • Forward-mode multiplies from the input side (producing Jacobian-vector products). Cost ∝ the number of inputs n.
  • Reverse-mode multiplies from the output side (vector-Jacobian products). Cost ∝ the number of outputs m.

In machine learning the output is a single scalar loss (m = 1), while there are millions of parameters (n is huge). Which means:

reverse-mode: all n partial derivatives in ≈ 1 pass  vs  forward-mode: ≈ n passes

Hence the efficiency of backprop: one backward pass gives the gradient with respect to every weight at roughly the cost of one forward pass. The price is that you have to keep the intermediate values (memory ∝ the size of the graph).

Python A backward pass by hand on a small expression 
# f(a,b) = a*b + a;  find ∂f/∂a and ∂f/∂b with a backward pass
a, b = 3.0, 4.0
u = a * b                 # forward pass
f = u + a

df = 1.0                  # adjoints from the output back to the inputs:
du = df * 1.0             # f = u + a  →  ∂f/∂u = 1
da = df * 1.0             #            →  contribution to ∂f/∂a = 1
da += du * b              # u = a*b   →  ∂u/∂a = b
db = du * a               #            →  ∂u/∂b = a
print(da, db)             # → 5.0, 3.0   (∂f/∂a = b+1, ∂f/∂b = a)
x ·w +b σ L forward pass: values → ← backward pass: adjoints (gradients)
The computation graph. The forward pass computes values; the backward pass propagates the gradient right to left, yielding ∂L with respect to every parameter in a single pass.
Analogy. A post-mortem after a failure. Instead of asking each person in turn "how much of this was you?" (forward-mode — expensive with a big team), the manager walks the chain backwards from the outcome once and apportions responsibility in proportion to each link's influence. Reverse-mode is responsibility, sent back down the chain.

Why it matters

Reverse-mode AD is the computational engine of all deep learning. When PyTorch calls loss.backward(), it runs exactly this algorithm. It is also a textbook case of repeated independent discovery: the algorithm appeared 12–16 years before the famous 1986 paper (#7), which merely made it well known.

Connections

→ popularized in7. Backpropagation

The same reverse-mode idea, but stated loudly and with a demonstration: the 1986 paper shows that multi-layer networks really can be trained with it and that hidden layers learn features. The invention is here; the fame is there.

← motivated by3. Perceptron

Training a single layer is easy. The moment people wanted hidden layers (to get past XOR), they needed an efficient way to push the gradient through them — and that is what reverse mode provides.

↔ complemented by23. Adam

Backprop gives you the gradient; the optimizer decides what step to take along it. Separating "how to compute the derivative" (AD) from "how to move along it" (SGD/Adam) gives you two independent layers on which all of training rests.

Questions worth asking

If reverse-mode is so efficient, what is forward-mode for at all?

It wins in the opposite situation: few inputs, many outputs (then the cost ∝ the number of inputs is small). Forward-mode gives Jacobian-vector products, is handy for sensitivity analysis, and does not require storing the whole graph — its memory is constant. In ML there is one output (the scalar loss) and an enormous number of input parameters, so reverse rules almost everywhere.

Reverse-mode has to store the activations of the whole graph — isn't that expensive in memory?

It is, and it is a real problem when training large models: memory ∝ the size of the graph (every intermediate activation). The treatment is gradient checkpointing — some activations are not stored but recomputed on the backward pass, trading memory for extra compute. The classic compute↔memory trade-off.

Why then does 1986 get the credit rather than Linnainmaa?

Because science rewards not only the discovery but also the delivery of it to a community. Nobody connected Linnainmaa's Finnish-language thesis about rounding errors with neural networks; the 1986 Nature paper showed the payoff and landed at the right moment. Jürgen Schmidhuber has insisted for years that priority belongs to Linnainmaa — and formally he is right.

What to read in the original

The primary sources are niche and hard to get hold of. Understanding the reverse-mode idea and the history of repeated discovery is enough — that is the best this work gives the canon.