# Generative Models

Generative models offer more approaches for classification problems that can get great results when logistic regression breaks down (like with large class separation).

Given y=k, model x distribution then flip with Bayes.

# Overview

y has K >= 2 classes

pi_k prior that a random observation comes from kth class

for kth class, f_k(x) = P(X | y=k)

then P(y=k|X=x) = pi_k * f_k(x) / sum(l=1, K) pi_l * f_l(x)

= P_k(x) (posterior)

Finding which one has the highest posterior probability.

# Linear Discriminate Analysis (LDA)

Assume that f_k(x) is normal.

Take a log of the full expression and maximize that.

d_k(x) = x * M_k/s^2 - M_k^2/2s^2 + log(pi_k)

LDA can also use estimators for mean, standard deviation, and pi sub k. The general estimators are modified to account for each class.

# Quadratic Discriminate Analysis (QDA)

Assumes each class has its own co-variance matrix.

Same setup as LDA, assign observation to class that maximizes:

d_k(x) = -1/2 * x^T * Sigma_k^-1 * x + x^T * Sigma_k^-1 * M_k - 1/2 * M_k^T *
Sigma_k^-1 * M_k - 1/2 * log|Sigma_k| + log(pi_k)

QDA is a more flexible model than LDA.

# Naive Bayes

Assume predictors are completely independent of each other.