Русская версия →

The ML Canon — deep dives

59 landmark machine learning papers, 1943 → 2025, worked through properly: context, the mechanism with its math, diagrams, analogies and the links between topics. Read in order or jump by interest. Short summaries and a progress tracker live in the interactive dashboard (artifact ml-canon).

🟥 read the original in full🟧 read selectively🟦 this write-up is enough
0 / 59 read
1943–1986

1. Origins

Neuron, perceptron, backprop: the idea of a learning network is born — and very nearly dies under criticism.
1. McCulloch–Pitts neuron1943
write-up
2. Hebbian learning1949
write-up
3. Perceptron1958
selective
4. Perceptrons (the critique)1969
write-up
5. Backprop prehistory1970
write-up
6. Hopfield network1982
selective
7. Backpropagation1986
in full
1989–2009

2. Foundations

CNNs, LSTM, SVM, embeddings: the building blocks, and the long winter before the thaw. Data (ImageNet) walks on stage.
8. Universal approximation1989
write-up
9. LeNet / CNN1998
selective
10. SVM1995
selective
11. LSTM1997
selective
12. Random Forests2001
selective
13. Neural language model2003
selective
14. Deep Belief Nets2006
write-up
15. ImageNet2009
write-up
2012–2014

3. The deep learning explosion

GPUs plus big data: AlexNet, word2vec, GANs, attention — neural nets suddenly start working.
16. AlexNet2012
in full
17. Word2Vec2013
selective
18. DQN (Atari)2013
in full
19. Dropout2014
selective
20. GAN2014
in full
21. Seq2Seq2014
selective
22. Attention (Bahdanau)2014
in full
23. Adam2014
selective
24. GloVe2014
selective
25. VGG2014
write-up
2015–2017

4. Architectures and scale

ResNet makes networks deep, AlphaGo beats a human, and the Transformer throws recurrence away.
26. Batch Normalization2015
selective
27. ResNet2015
in full
28. Knowledge Distillation2015
selective
29. AlphaGo2016
selective
30. WaveNet2016
write-up
31. Mixture-of-Experts2017
selective
32. Transformer ★2017
in full
33. PPO2017
in full
2018–2022

5. The LLM era

BERT → GPT-3 → RLHF: pre-training, scale, alignment. ChatGPT is already at the door.
34. BERT2018
selective
35. GPT-12018
write-up
36. GPT-22019
selective
37. Scaling Laws2020
selective
38. GPT-32020
selective
39. Vision Transformer2020
selective
40. CLIP2021
selective
41. LoRA2021
selective
42. Chinchilla2022
selective
43. Chain-of-Thought2022
in full
44. InstructGPT (RLHF)2022
in full
45. Constitutional AI2022
selective
2020–2025

6. Generative models and systems

A parallel line: diffusion draws pictures, FlashAttention and vLLM make inference cheap, open weights break the monopoly.
46. DDPM (Diffusion)2020
in full
47. Stable Diffusion2022
selective
48. FlashAttention2022
selective
49. vLLM / PagedAttention2023
selective
50. LLaMA2023
selective
51. DPO2023
in full
52. Mixtral2024
selective
53. DeepSeek V3 / R12025
selective
54. Prefix caching → CacheBlend2024
selective
55. Structured Output2024
selective
56. GQA / MQA2023
selective
57. Speculative decoding2023
selective
58. o1 / test-time compute2024
in full
59. GRPO2024
selective
Epilogue · Where all this is heading →
the threads running through the canon, and where the frontier is moving