Era 3 · The deep learning explosion · 2014

20 GAN

Generative Adversarial Nets · Goodfellow, Pouget-Abadie, Mirza et al. · NeurIPS
🟥 read in full~45 minoriginal ↗
The gist in 20 seconds. A generative model as a game between two networks: the generator moulds fakes out of noise, the discriminator tells them from the real thing. They learn by fighting each other, and at equilibrium the generator reproduces the data distribution. No explicit densities, no integrals — training through competition.

Context

Generative models (ones that produce new data) are hard to train: normally you have to specify an explicit probabilistic model and evaluate intractable normalization integrals. Ian Goodfellow (2014) sidesteps all of that — he replaces "compute a probability" with "fool a critic".

The idea and the mechanism

Two networks play a zero-sum game. The generator G(z) turns random noise z into a sample (a picture, say). The discriminator D(x) outputs the probability that x is real (from the data) rather than a fake. The objective is a minimax:

minG maxD  V(D,G) = Ex∼pdata[log D(x)] + Ez[log(1 − D(G(z)))]

The discriminator maximizes its accuracy (labeling real and fake correctly), the generator minimizes it (making fakes indistinguishable). Training alternates steps on D and on G. At perfect equilibrium the generator reproduces the data and the discriminator is helpless — it outputs ½ for everything.

probability Why the game drives pg to pdata

Step 1. The optimal discriminator. Fix G. Rewrite the objective in terms of the generator's distribution pg (replacing the integral over the noise):

V = ∫ pdata(x) log D(x) + pg(x) log(1 − D(x)) dx

The integrand is a function of the form a log y + b log(1−y). Its maximum over y ∈ (0,1) follows from the derivative a/y − b/(1−y) = 0, giving y* = a/(a+b). So the optimal discriminator is:

D*(x) = pdata(x)pdata(x) + pg(x)

Step 2. What G then minimizes. Substituting D* back into V and rearranging, we get (where JSD is the Jensen–Shannon divergence):

V(D*, G) = −log 4 + 2·JSD(pdata ‖ pg)

Since JSD ≥ 0, and is zero if and only if pg = pdata, the global minimum of the game (−log 4) is reached exactly when the generator has reproduced the data. ∎ So the contest formally forces the generator's distribution to converge to the data distribution.

PyTorch One step of adversarial training 
import torch
import torch.nn.functional as F

def gan_step(real, G, D, optD, optG, z_dim):
    z = torch.randn(len(real), z_dim)
    fake = G(z)
    ones, zeros = torch.ones(len(real), 1), torch.zeros(len(real), 1)
    # discriminator: real → 1, fake → 0
    lossD = F.binary_cross_entropy(D(real), ones) + F.binary_cross_entropy(D(fake.detach()), zeros)
    optD.zero_grad(); lossD.backward(); optD.step()
    # generator: wants to fool D (D(fake) → 1)
    lossG = F.binary_cross_entropy(D(fake), ones)
    optG.zero_grad(); lossG.backward(); optG.step()
noise z G fake real x D real / fake the gradient from D teaches G to be more convincing
G turns noise into a fake; D judges the fake against real data; the gradient from D teaches G to deceive better. The loop runs until the forgeries are indistinguishable.
Analogy. A counterfeiter against a detective. The counterfeiter (G) prints fake notes, the detective (D) learns to spot them. Each forces the other to improve: the detective gets fussier, the forgeries get more convincing — until the fakes are indistinguishable from real money. At that point the counterfeiter has "learned" the true distribution of banknotes.

Why it matters

GANs set off a wave of photorealistic generation (faces, super-resolution, style transfer) that ran for years; Yann LeCun called it "the coolest idea in ML in 20 years". Conceptually it is a new way to train generative models: not through an explicit probability, but through an adversarial signal.

The ailments: unstable training (the two networks have to grow in step) and mode collapse (the generator produces little diversity). Later, in image generation, GANs will be pushed aside by diffusion models (#46) — steadier, and covering all the modes.

Connections

← trained via7. Backpropagation

Both networks are ordinary neural networks trained by backprop; the only non-trivial part is that they optimize opposing objectives at the same time. The discriminator's gradient with respect to the fake flows back into the generator, telling it how to become more convincing — backprop through two networks at once.

↔ contrast / successor46. DDPM (Diffusion)

The same task (image generation), the opposite approach: instead of two networks competing, one network is trained to gradually denoise data. Diffusion cures the GAN's signature ailments (instability, mode collapse) and by the early 2020s displaces it as the dominant paradigm for generating pictures.

→ leads to47. Stable Diffusion

GANs opened the era of the deep generative model and taught the field to think of generation as a learned mapping from noise. The "text-to-image for everyone" line will carry on with diffusion — but it was the GAN that made the very idea of photorealistic neural generation mainstream.

What to read in the original

The paper is short and elegant — worth reading in full. It pays to understand the minimax formulation and the derivation of the optimal discriminator (see the math box above) — it is a model example of reducing a game-theoretic setup to minimizing a statistical divergence.