Probabilistic Generative Models Overview

This post is the first and introductory post in the series: From Probabilistic Modeling to Generative Modeling. It is my attempt to reproduce the knowledge I gained from learning about probabilistic generative models. In this entry, I break down the general concepts needed to understand five probabilistic generative models: Gaussian Mixture Model (GMM), Variational Autoencoders (VAE), Normalizing Flows (NF), Generative Adversarial Networks (GAN), and Diffusion Models (DM), which will be explained in the next posts. In this series, these models will be presented in a logical order that I find more intuitive, rather than following a strict categorization. Before starting, it is necessary to understand why generative modeling is useful.

1 – Why Probabilistic Generative Modeling?

To answer this question, we need to understand the difference between discriminative and generative models. The classifier predicts probabilities of labels given images, meaning it learns conditional probability.

Classification algorithms summarize complex inputs into simple outputs, discarding most of the original information.

2 – Why Probability?

Probability is the mathematical framework used to quantify uncertainty and reason about data.

2.1 – Bayesian vs Frequentist

The frequentist approach focuses on long-run frequencies, while the Bayesian approach focuses on belief and uncertainty.

2.2 – Key Rules

Sum Rule

\[ P(X) = P(X, Y) + P(X, \bar{Y}) \]

Marginalization

\[ P(X) = \sum P(X, y) \]

Conditional Probability

\[ P(Y|X) = \frac{P(X,Y)}{P(X)} \]

Bayes Theorem

\[ P(Y|X) = \frac{P(X|Y)P(Y)}{P(X)} \]

3 – Probabilistic Modeling

Maximum Likelihood Estimation

\[ \hat{\theta} = \arg \max \log p(D | \theta) \]

5 – Latent Generative Models

Latent Variables

Latent variables represent hidden structures in data.

6 – Hands-On: Generating Images

Images can be treated as samples drawn from a probability distribution.

8 – Sources

• Deep Learning Book
• Bishop Book
• Probabilistic Machine Learning