Statistics

Probability Basics

Probability Rules:

$$\begin{align} P(A \cup B) &= P(A) + P(B) - P(A \cap B) \\ P(A \cap B) &= P(A|B)P(B) = P(B|A)P(A) \end{align}$$

Conditional Probability:

$$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

Independence: $P(A \cap B) = P(A)P(B)$

Bayes' Theorem

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$$

Extended Form:

$$P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|\neg A)P(\neg A)}$$

ML Context (Parameter Estimation):

In machine learning, we want to estimate parameters $\theta$ given observed data $D$:

$$p(\theta|D) = \frac{p(D|\theta) \cdot p(\theta)}{p(D)}$$

where:

• $p(\theta|D)$ is the posterior (probability of parameters given data)
• $p(D|\theta)$ is the likelihood (probability of observing data given parameters)
• $p(\theta)$ is the prior (prior belief about parameters before seeing data)
• $p(D)$ is the evidence or marginal likelihood (normalizing constant)

Law of Total Probability:

$$p(D) = \int p(D|\theta) \cdot p(\theta) \, d\theta$$

Maximum A Posteriori (MAP)

MAP estimation finds the most likely parameter values given the data:

$$\hat{\theta}_{MAP} = \arg\max_\theta \, p(\theta|D) = \arg\max_\theta \, \frac{p(D|\theta) \cdot p(\theta)}{p(D)}$$

Since $p(D)$ doesn't depend on $\theta$, this simplifies to:

$$\hat{\theta}_{MAP} = \arg\max_\theta \, p(D|\theta) \cdot p(\theta)$$

Taking logarithm (for numerical stability):

$$\hat{\theta}_{MAP} = \arg\max_\theta \, [\log p(D|\theta) + \log p(\theta)]$$

Comparison with MLE:

• MLE: $\hat{\theta}_{MLE} = \arg\max_\theta \, p(D|\theta)$ (ignores prior)
• MAP: $\hat{\theta}_{MAP} = \arg\max_\theta \, p(D|\theta) \cdot p(\theta)$ (incorporates prior)
• If prior is uniform, MAP = MLE

Application to Naive Bayes Classification

For classification with Naive Bayes, we want to find:

$$\hat{y} = \arg\max_{y \in \{0,1\}} \, p(y|x) = \arg\max_{y \in \{0,1\}} \, \frac{p(x|y) \cdot p(y)}{p(x)}$$

Since $p(x)$ is constant for all classes, this becomes:

$$\hat{y} = \arg\max_{y \in \{0,1\}} \, p(x|y) \cdot p(y)$$

This is MAP estimation where:

• $y$ is the parameter we're estimating (the class label)
• $p(y)$ is the prior (class prior, e.g., $p(y=1) = \phi$)
• $p(x|y)$ is the likelihood (computed using Naive Bayes assumption: $p(x|y) = \prod_{j=1}^d p(x_j|y)$)
• We pick the class that maximizes the posterior

Why MAP works for Naive Bayes:

1. We treat the class label $y$ as the "parameter" to estimate
2. The prior $p(y)$ represents class frequencies in training data
3. The likelihood $p(x|y)$ comes from the Naive Bayes assumption
4. We choose the class with maximum posterior probability

Extended Form (binary classification):

$$p(y=1|x) = \frac{p(x|y=1) \cdot p(y=1)}{p(x|y=1) \cdot p(y=1) + p(x|y=0) \cdot p(y=0)}$$

where $p(y=1) = \phi$ and $p(y=0) = 1-\phi$.

Expected Value and Variance

Expected Value (Mean):

$$\mathbb{E}[X] = \mu = \sum_{i} x_i P(x_i) \quad \text{(discrete)}$$

$$\mathbb{E}[X] = \int x f(x) dx \quad \text{(continuous)}$$

Properties:

• $\mathbb{E}[aX + b] = a\mathbb{E}[X] + b$
• $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$
• Jensen's inequality: $\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])$ for convex function $f$.

Variance:

$$\text{Var}(X) = \sigma^2 = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$$

Properties:

• $\text{Var}(aX + b) = a^2\text{Var}(X)$
• If $X$ and $Y$ are independent: $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$
• Variance is always non-negative: $\text{Var}(X) = \mathbb{E}[X^{2}] - (\mathbb{E}[X])^{2} \geq 0$.

Standard Deviation: $\sigma = \sqrt{\text{Var}(X)}$

Covariance:

$$\text{Cov}(X, Y) = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)] = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]$$

Correlation:

$$\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$$

where $-1 \leq \rho \leq 1$.

Common Probability Distributions

Distribution	PDF or PMF	Support / Values	Mean	Variance
Bernoulli$(p)$	$\begin{cases} p, & x=1 \\ 1-p, & x=0 \end{cases}$	$x \in \{0, 1\}$	$p$	$p(1-p)$
Binomial$(n, p)$	$\displaystyle \binom{n}{k} p^k (1-p)^{n-k}$	$k = 0, 1, ..., n$	$np$	$np(1-p)$
Geometric$(p)$	$(1-p)^{k-1}p$	$k = 1, 2, ...$	$\dfrac{1}{p}$	$\dfrac{1-p}{p^2}$
Poisson$(\lambda)$	$\dfrac{e^{-\lambda}\lambda^k}{k!}$	$k = 0, 1, ...$	$\lambda$	$\lambda$
Uniform$(a, b)$	$\dfrac{1}{b-a}$	$x \in [a, b]$	$\dfrac{a+b}{2}$	$\dfrac{(b-a)^2}{12}$
Gaussian$(\mu, \sigma^2)$	$\dfrac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$	$x \in (-\infty, \infty)$	$\mu$	$\sigma^2$
Exponential$(\lambda)$	$\lambda e^{-\lambda x}$	$x \geq 0$	$\dfrac{1}{\lambda}$	$\dfrac{1}{\lambda^2}$

Notes:

• The Geometric distribution here uses the convention "number of trials until first success," so $k$ starts at 1.
• The Poisson distribution models counts of events in a fixed interval.
• The Uniform distribution models a constant density over $[a, b]$.
• The Gaussian is also called the normal distribution.
• The Exponential distribution is often used for modeling waiting times.

For multivariate Gaussian:

• $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$
• PDF: $f(\mathbf{x}) = \dfrac{1}{(2\pi)^{n/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\tfrac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\mathbf{x} - \boldsymbol{\mu})\right)$
• Mean: $\boldsymbol{\mu}$, Covariance: $\boldsymbol{\Sigma}$

Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation is a method for estimating parameters of a statistical model by finding the parameter values that maximize the probability of observing the given data.

Intuition

Given data $D = \{x_1, ..., x_n\}$, MLE asks: "Which parameter values $\theta$ make this observed data most likely?"

For example, if you flip a coin 10 times and get 7 heads, MLE would estimate $p = 0.7$ because that makes your observed outcome most probable.

Mathematical Formulation

Likelihood Function:

The likelihood of parameters $\theta$ given data $D$ (assuming i.i.d. observations):

$$L(\theta) = P(D|\theta) = \prod_{i=1}^n P(x_i|\theta)$$

Log-Likelihood (easier to work with):

Taking logarithm converts products to sums and is monotonic (doesn't change the argmax):

$$\ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log P(x_i|\theta)$$

MLE Estimate:

Find $\theta$ that maximizes the log-likelihood:

$$\hat{\theta}_{MLE} = \arg\max_\theta \ell(\theta)$$

Finding MLE

Method 1: Set derivative to zero

$$\frac{\partial \ell(\theta)}{\partial \theta} = 0$$

Solve for $\theta$ (if closed-form solution exists).

Method 2: Numerical optimization

Use gradient descent or other optimization algorithms when no closed-form solution exists.

Example: Gaussian Distribution

For data from $\mathcal{N}(\mu, \sigma^2)$, the MLE estimates are:

$$\hat{\mu}_{MLE} = \frac{1}{n}\sum_{i=1}^n x_i \quad \text{(sample mean)}$$

$$\hat{\sigma}^2_{MLE} = \frac{1}{n}\sum_{i=1}^n (x_i - \hat{\mu})^2 \quad \text{(sample variance)}$$

Properties of MLE

✅ Consistent: Converges to true parameter as $n \to \infty$
✅ Asymptotically normal: Distribution approaches Gaussian for large $n$
✅ Asymptotically efficient: Achieves lowest possible variance (Cramér-Rao bound)
✅ Invariant: If $\hat{\theta}_{MLE}$ is MLE for $\theta$, then $g(\hat{\theta}_{MLE})$ is MLE for $g(\theta)$

MLE vs MAP

Aspect	MLE	MAP
Formula	$\arg\max_\theta \, p(D\|\theta)$	$\arg\max_\theta \, p(D\|\theta) \cdot p(\theta)$
Prior	No prior (uniform prior)	Incorporates prior $p(\theta)$
Interpretation	Most likely parameters given data	Most likely parameters given data and prior belief
Regularization	No regularization	Prior acts as regularization
Special case	MAP with uniform prior = MLE	Includes MLE as special case

Connection: MLE is equivalent to MAP with a uniform (non-informative) prior on $\theta$.