Why Discriminative Learning Dominated First: The Pragmatic Path to Modern AI
This post provides historical context for the paradigm shift discussed in Machine Learning Paradigms: Learning Distributions vs Approximating Functions.
The Pragmatic Reality
From the 1980s through the early 2010s, machine learning was almost synonymous with discriminative learning—teaching computers to classify, predict, and make decisions. Generative modeling existed in theory, but remained largely academic curiosity rather than practical tool.
Why?
The answer isn’t mysterious: Discriminative learning was simply easier, faster, and good enough for the problems people cared about solving.
Three Fundamental Obstacles to Generative Modeling
1. The Normalization Constant Problem
The mathematical barrier: To model a distribution $p(x)$, you need to compute:
\[p(x) = \frac{\tilde{p}(x)}{Z} \quad \text{where} \quad Z = \int \tilde{p}(x) \, dx\]This integral $Z$ is intractable for high-dimensional data. For images with $256 \times 256 \times 3 \approx 200{,}000$ dimensions, this integral is computationally impossible to evaluate.
Discriminative learning sidesteps this entirely: You never need $Z$ to compute $p(Y \mid X)$—just the ratio of unnormalized probabilities.
Example: For classification:
\[p(Y=k \mid X=x) = \frac{e^{f\_\theta^{(k)}(x)}}{\sum\_j e^{f\_\theta^{(j)}(x)}}\]The denominator is a finite sum over classes, not an intractable integral over data space!
For a deep dive into this fundamental problem, see The Normalization Constant Problem: Why Computing Z Is So Hard.
2. The Sampling Challenge
Even if you could evaluate $p(x)$, how do you sample from it?
The problem: For complex distributions, direct sampling requires either:
- Inverting the CDF: $F^{-1}(u)$ where $u \sim \text{Uniform}(0,1)$ → analytically intractable
- MCMC methods: Metropolis-Hastings, Gibbs sampling → slow convergence for high dimensions
- Rejection sampling: Requires tight proposal bounds → exponentially inefficient in high dimensions
For a detailed analysis, see Why Direct Sampling from PDFs Is Hard.
Discriminative learning doesn’t need sampling: Just one forward pass $x \to \hat{y}$, done.
3. The “Curse of Dimensionality”
Volume explodes: In $d$ dimensions, if you want to cover 10% of each dimension, you need to evaluate $0.1^d$ of the space:
- $d=2$: 1% of space
- $d=10$: $10^{-10}$ of space
- $d=1000$ (image): effectively zero coverage
Data becomes sparse: To adequately represent $p(x)$ in high dimensions, you need exponentially more data.
Discriminative learning is dimension-efficient: You only model the decision boundary, not the entire data manifold. This requires far less data because you’re solving a simpler problem.
For an intuitive exploration of why high dimensions are so challenging, see The Curse of Dimensionality in Machine Learning.
Why Discriminative Learning Thrived
It Solved Real Problems
The 1980s-2000s AI agenda:
- Handwritten digit recognition (MNIST)
- Face detection
- Speech recognition
- Spam filtering
- Credit scoring
All of these are prediction tasks, not generation tasks. They needed $f: X \to Y$, not $p(X)$.
Result: The field optimized for what it needed.
It Had Clear Evaluation
Discriminative: Accuracy, precision, recall, F1-score, mean squared error—objectively measurable.
Generative: How do you evaluate $p(x)$?
- Log-likelihood requires knowing $Z$ (intractable)
- Sample quality is subjective
- No clear “ground truth” for generation
Before GANs and modern metrics (FID, IS), evaluating generative models was genuinely hard.
It Scaled with Compute
Key insight: More compute → train deeper networks → better predictions
This virtuous cycle:
- GPUs enable deeper networks (2012: AlexNet)
- Deeper networks → better ImageNet accuracy
- Better accuracy → more funding → more GPUs
- Repeat
This loop worked for discriminative learning because the objective was clear: minimize classification error.
Generative modeling couldn’t ride this wave because training objectives were unstable (early GANs) or intractable (likelihood-based models).
What Changed? The Generative Renaissance
1. Algorithmic Breakthroughs (2014-2020)
GANs (2014): Sidestep normalization by adversarial training
- Generator: $G(z) \to x$
- Discriminator: $D(x) \to \text{real/fake}$
- Never compute $Z$, just train via game dynamics
VAEs (2014): Variational approximation to avoid intractable integrals
- Approximate posterior: $q_\phi(z \mid x) \approx p(z \mid x)$
- Optimize tractable lower bound (ELBO)
Normalizing Flows (2015): Exact likelihoods via invertible transformations
- Change of variables: $\log p(x) = \log p(z) - \log \left\lvert \det \frac{\partial f}{\partial z} \right\rvert$
- Requires special architectures but enables exact inference
Diffusion Models (2020): Iterative denoising as implicit generative process
- Forward: gradually add noise
- Reverse: learn to denoise
- Stable training, high-quality samples
2. Computational Scale
GPUs became powerful enough to:
- Train models with billions of parameters
- Iterate over massive datasets (LAION-5B for Stable Diffusion)
- Run inference with iterative refinement (diffusion requires ~50-1000 steps)
Cloud infrastructure made this accessible beyond academic labs.
3. Massive Datasets
Internet-scale data became available:
- ImageNet (2009): 14M images
- LAION-5B (2022): 5 billion image-text pairs
- Common Crawl: petabytes of text
Generative models need far more data than discriminative models (must model entire distribution, not just boundaries). This data finally became available.
4. New Applications Demanded Generation
The 2010s shift: From “predict” to “create”
- Content creation: art, music, text
- Data augmentation for privacy/fairness
- Drug discovery: generate novel molecules
- Game development: procedural content generation
Demand drove investment, which funded research, which enabled breakthroughs.
The Philosophical Shift
Early AI: Prediction as Intelligence
Dominant view: “If we can predict accurately, we understand.”
- Speech recognition → predict words from audio
- Computer vision → predict labels from images
- NLP → predict next word from context
This view worked for narrow, task-specific AI.
Modern AI: Understanding Through Generation
New view: “If we can generate, we truly understand.”
- Language models (GPT): generate coherent text → demonstrate understanding of language structure
- Diffusion models: generate realistic images → capture visual world’s distribution
- AlphaFold: generate protein structures → understand biological constraints
Generation became a proxy for understanding.
The Technical Turning Point
Before 2014
Generative models were:
- Restricted Boltzmann Machines (RBMs): hard to train, shallow
- Gaussian Mixture Models: parametric, limited expressiveness
- Hidden Markov Models: good for sequences, not images
- Bayesian Networks: exact inference intractable
All suffered from:
- Training instability
- Poor sample quality
- Computational intractability
- Limited scalability
After 2014
Generative models became:
- Deep, expressive (billions of parameters)
- Stable to train (better objectives, architectures)
- Scalable (GPU-friendly, parallelizable)
- High-quality (photorealistic images, coherent text)
The key: Problems that seemed fundamental (normalization, sampling, training) were algorithmic, not inherent.
The Convergence
Modern AI blurs the line:
GPT models are:
- Generative: Model $p(\text{next token})$
- Used discriminatively: Classification, Q&A via prompting
Diffusion models are:
- Generative: Model $p(\text{image})$
- Used discriminatively: Image embeddings for retrieval
Foundation models are trained generatively (self-supervised on raw data) then fine-tuned discriminatively (task-specific).
The paradigm shift: Generative pre-training → discriminative fine-tuning
This combines the data efficiency of generative learning (unsupervised) with the task performance of discriminative learning (supervised).
Key Takeaways
-
Discriminative learning dominated because it was easier (no normalization, no sampling, clear metrics) and sufficient for prediction tasks.
- Three obstacles delayed generative modeling:
- Intractable normalization constants
- Difficult sampling in high dimensions
- Curse of dimensionality requiring massive data
- The 2014-2020 breakthrough came from:
- Algorithmic innovations (GANs, VAEs, flows, diffusion)
- Computational scale (GPUs, cloud)
- Internet-scale datasets
- The shift from prediction to generation reflects:
- New applications demanding content creation
- Understanding that modeling distributions captures deeper structure
- Foundation models trained generatively, used discriminatively
- The future is hybrid: Train on distributions, adapt to tasks.
Historical Irony
The irony: Generative modeling is arguably more fundamental—if you know $p(X, Y)$, you can derive $p(Y \mid X)$ via Bayes’ rule:
\[p(Y \mid X) = \frac{p(X, Y)}{p(X)} = \frac{p(X, Y)}{\sum\_y p(X, Y=y)}\]But pragmatically, the direct path $X \to Y$ was easier.
Now we’ve come full circle: Model $p(X)$ first (self-supervised pre-training), then specialize to $p(Y \mid X)$ (supervised fine-tuning).
The lesson: Sometimes the theoretically elegant path (generative) is impractical until computational and algorithmic infrastructure catches up.
Further Reading
History:
- Hinton (2007), “Learning Multiple Layers of Representation” - deep learning foundations
- Krizhevsky et al. (2012), “ImageNet Classification with Deep CNNs” - AlexNet moment
- Goodfellow et al. (2014), “Generative Adversarial Nets” - GAN breakthrough
Technical Challenges:
- Murphy, Machine Learning: A Probabilistic Perspective (Chapter 27: latent variable models)
- The Normalization Constant Problem: Why Computing Z Is So Hard - deep dive into partition functions
- The Curse of Dimensionality in Machine Learning - intuitive exploration of high-dimensional spaces
- Why Direct Sampling from PDFs Is Hard - sampling challenges
Modern Context:
- Machine Learning Paradigms: Learning Distributions vs Approximating Functions
- Brownian Motion and Modern Generative Models
The bottom line: Discriminative learning dominated not because it was theoretically superior, but because it was practically tractable with 1990s-2000s compute and algorithms. The generative renaissance came when we finally built the tools—GANs, VAEs, diffusion models—that made distribution modeling as practical as function approximation. We didn’t take the “easy” path by choice; we took the only path that was computationally possible at the time.