Statistical Inference

1. Frequentist vs. Bayesian Frameworks

Statistical inference is the process of drawing conclusions about a population or data-generating process from a sample. Two dominant philosophical frameworks govern how we interpret probability and what "inference" means.

Frequentist Framework

In the frequentist view, parameters are fixed (unknown) constants. Probability is defined as the limiting relative frequency of an event in an infinite sequence of independent repetitions of an experiment. A procedure is evaluated by its long-run behavior across all possible datasets that could have been observed.

Key concepts: estimators, sampling distributions, confidence intervals (which trap the true parameter in 95% of repeated experiments), significance levels, and p-values. Inference statements are about the procedure, not about the probability of specific parameter values.

Bayesian Framework

In the Bayesian view, parameters are treated as random variables with a prior distribution pi(theta) encoding beliefs before observing data. After observing data x, the posterior is computed via Bayes' theorem:

The posterior pi(theta | x) summarizes all inference about theta given x. Point estimates use the posterior mean, median, or mode. Interval estimates are credible intervals with direct probabilistic interpretations.

2. Point Estimation

A point estimator is a statistic T(X1, ..., Xn) used to estimate a scalar or vector parameter theta. Key properties of estimators include unbiasedness, consistency, efficiency, and sufficiency.

Bias, Variance, and MSE

The bias of an estimator T is Bias(T) = E[T] - theta. An estimator is unbiased if its bias is zero for all theta. The mean squared error decomposes as:

A biased estimator can have lower MSE than an unbiased one if its variance is sufficiently smaller — this is the bias-variance tradeoff. The sample variance S^2 = (1/(n-1)) * sum(Xi - X-bar)^2 is unbiased for sigma^2, while (1/n) times the same sum is the MLE but biased.

Maximum Likelihood Estimation (MLE)

For i.i.d. data x1, ..., xn with density f(x | theta), the likelihood function is:

The MLE theta-hat maximizes log L(theta). Under regularity conditions the MLE is: (1) consistent, (2) asymptotically normal with variance 1/[n*I(theta)], and (3) asymptotically efficient.

Method of Moments (MOM)

The method of moments sets theoretical moments equal to sample moments. The k-th population moment is mu_k = E[X^k]. The k-th sample moment is m_k = (1/n)*sum(Xi^k). Solve the system mu_k(theta) = m_k for theta. MOM estimators are consistent but generally less efficient than MLEs.

3. Sufficiency, Completeness, and the Rao-Blackwell Theorem

Sufficient Statistics

A statistic T(X) is sufficient for theta if the conditional distribution of X given T(X) does not depend on theta. Intuitively, T captures all information about theta contained in the data. The Factorization Theorem (Neyman-Fisher) states T is sufficient if and only if the joint density factors as:

For Exponential(lambda) data: the joint density factors as lambda^n * exp(-lambda * sum(xi)), so T = sum(Xi) is sufficient for lambda. For Normal(mu, sigma^2) data with both unknown, T = (sum Xi, sum Xi^2) or equivalently (X-bar, S^2) is a sufficient statistic.

Minimal Sufficient Statistics

A sufficient statistic T is minimal sufficient if it is a function of every other sufficient statistic — it achieves the maximum data reduction consistent with sufficiency. The ratio f(x|theta)/f(y|theta) being free of theta for some pair (x, y) is the Lehmann-Scheffé criterion for minimal sufficiency.

Completeness

A statistic T is complete for a family of distributions if: whenever E[g(T)] = 0 for all theta implies g(T) = 0 almost surely for all theta. Completeness eliminates the possibility of unbiased estimators of zero (other than zero itself). In exponential families, the natural sufficient statistic is complete.

Rao-Blackwell Theorem

If W is an unbiased estimator of theta and T is sufficient for theta, then the conditional expectation phi(T) = E[W | T] satisfies:

This means conditioning on a sufficient statistic never increases variance. Combined with completeness (Lehmann-Scheffé theorem), if T is a complete sufficient statistic and phi(T) is unbiased, then phi(T) is the unique minimum variance unbiased estimator (UMVUE).

4. Fisher Information and the Cramer-Rao Lower Bound

Fisher Information

The Fisher information I(theta) measures the amount of information that an observable random variable X carries about the unknown parameter theta. For a single observation:

The score function s(theta; x) = d/dtheta log f(x|theta) has E[s(theta;X)] = 0 under regularity conditions. The Fisher information is the variance of the score.

Cramer-Rao Lower Bound (CRLB)

For any unbiased estimator T(X) of theta based on n i.i.d. observations:

An unbiased estimator achieving equality is called efficient. Under regularity conditions, the MLE is asymptotically efficient: its asymptotic variance equals 1/[n*I(theta)].

5. Interval Estimation

Confidence Intervals via Pivots

A pivot is a function Q(X, theta) whose distribution does not depend on theta (or any other unknown parameter). We find a pivot, compute tail probabilities, and invert the inequalities to get bounds for theta.

When sigma^2 is unknown, replace sigma with S and N(0,1) with t_(n-1), giving the t-interval: X-bar +/- t_(n-1, alpha/2) * S/sqrt(n).

Interpreting Confidence Intervals Correctly

Likelihood Ratio Confidence Intervals

The likelihood ratio confidence region inverts the likelihood ratio test. Define:

Likelihood ratio intervals have several advantages over Wald intervals (based on normal approximation): they are invariant to reparameterization, tend to be more accurate in small samples, and can naturally handle bounded parameters.

Common Confidence Intervals

6. Hypothesis Testing Foundations

Hypothesis testing provides a formal decision framework for evaluating evidence against a null hypothesis H0. A test partitions the sample space into a rejection region C (reject H0 when the test statistic falls in C) and its complement.

Types of Error

Power Function

The power function beta(theta) = P_(theta)(reject H0) gives the probability of rejection as a function of the true parameter value. For theta in H0, we want beta(theta) <= alpha (size constraint). For theta in H1, we want beta(theta) to be large (high power).

P-values

The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming H0 is true:

A p-value is NOT the probability that H0 is true, and it is NOT the probability of making a wrong decision. It measures only how surprising the data are under H0. The p-value is a uniform(0,1) random variable under the null, so thresholding at alpha gives a valid test of size alpha.

7. Neyman-Pearson Lemma and UMP Tests

Neyman-Pearson Lemma

For testing H0: theta = theta_0 against H1: theta = theta_1 (both simple), the most powerful level-alpha test rejects H0 when:

For continuous distributions there is no randomization needed: the critical value k corresponds directly to a quantile of the null distribution of the test statistic.

Uniformly Most Powerful (UMP) Tests

A test is uniformly most powerful (UMP) at level alpha if it is most powerful against every specific alternative theta_1 in H1. UMP tests exist for one-sided hypotheses in one-parameter exponential families.

For exponential families with natural parameter eta(theta) monotone in theta and sufficient statistic T(x), the UMP test for H0: theta <= theta_0 vs H1: theta > theta_0 rejects when T(x) > c, where c is chosen to achieve size alpha. The key property is that the rejection region does not depend on the specific theta_1 chosen.

UMP tests generally do not exist for two-sided alternatives H1: theta ≠ theta_0. The best achievable in that case is a UMP unbiased (UMPU) test — one that is unbiased (power >= alpha for all theta in H1) and most powerful within the class of unbiased tests.

8. Likelihood Ratio Tests

For composite hypotheses, the generalized likelihood ratio test (GLRT) statistic is:

Wilks' Theorem

Under H0 and regularity conditions, as n approaches infinity:

This allows us to find critical values from chi-squared tables without knowing the exact distribution of the test statistic. The LRT is asymptotically equivalent to the Wald test and the score test (Rao test) — all three converge to chi^2_r under H0.

9. Chi-Squared, t, and F Distributions

Chi-Squared Distribution

If Z1, ..., Zk are i.i.d. N(0,1), then V = Z1^2 + ... + Zk^2 follows a chi-squared distribution with k degrees of freedom, written V ~ chi^2_k.

Student's t Distribution

If Z ~ N(0,1) and V ~ chi^2_k independently, then T = Z / sqrt(V/k) follows a t-distribution with k degrees of freedom, t_k.

F Distribution

If U ~ chi^2_m and V ~ chi^2_n independently, then F = (U/m) / (V/n) follows an F-distribution with (m, n) degrees of freedom, F_(m,n).

10. Analysis of Variance (ANOVA)

ANOVA tests whether the means of three or more groups are equal by decomposing total variation into between-group and within-group components.

One-Way ANOVA

The one-way model has k groups with n_i observations each (total N = sum n_i):

Two-Way ANOVA

Two-way ANOVA includes two factors A (a levels) and B (b levels) plus their interaction. The model (balanced, one observation per cell) is:

A significant interaction means the effect of factor A depends on the level of factor B and vice versa. If the interaction is significant, main effects must be interpreted cautiously — examining the cell means (profile plots) is essential.

Post-Hoc Testing

A significant ANOVA F-test only indicates that some means differ. Post-hoc tests identify which pairs differ while controlling error rates. Common methods include Tukey's HSD (controls familywise error for all pairwise comparisons), Bonferroni correction (conservative, adjusts alpha for the number of comparisons), and Scheffe's method (valid for all linear contrasts, most conservative).

11. Regression Inference

In linear regression Y = X*beta + epsilon with epsilon ~ N(0, sigma^2 * I_n), the OLS estimator is beta-hat = (X'X)^(-1) X'Y. Under normality, exact inference is available.

Sampling Distribution of beta-hat

Confidence Bands for the Regression Line

At a specific predictor value x_0, the confidence interval for the mean response E[Y | X = x_0] = x_0' * beta is:

This is the confidence interval for the mean of Y at x_0 — it is narrowest at the mean of X and widens as x_0 moves away. Pointwise confidence bands join these intervals for all x_0; simultaneous Working-Hotelling bands use an F-critical value to cover all x_0 simultaneously.

Prediction Intervals

A prediction interval for a new observation Y_new at x_0 is wider than the confidence interval for the mean because it must also account for the individual error epsilon_new:

Overall F-Test for Regression

12. Nonparametric Tests

Nonparametric tests make minimal distributional assumptions. They are based on ranks or order statistics rather than the actual values, making them robust to outliers and valid when normality cannot be assumed.

Wilcoxon Signed-Rank Test

The Wilcoxon signed-rank test is the nonparametric alternative to the one-sample or paired t-test. It tests whether the median of a symmetric distribution equals a hypothesized value mu_0.

Mann-Whitney U Test (Wilcoxon Rank-Sum)

The Mann-Whitney U test is the nonparametric alternative to the two-sample t-test. It tests whether two populations have the same distribution (location shift).

Kolmogorov-Smirnov Test

The KS test compares a sample empirical CDF Fn(x) to either a theoretical CDF F0(x) (one-sample) or another empirical CDF (two-sample). The test statistic is the maximum absolute deviation:

The KS test is distribution-free under H0 (when F0 is continuous and fully specified). Its power is often lower than parametric tests when the parametric assumptions hold, but it is more broadly applicable.

Kruskal-Wallis Test

The Kruskal-Wallis test is the nonparametric analogue of one-way ANOVA, testing whether k groups have the same distribution. Rank all N observations jointly, compute mean rank per group, and compute:

13. Bootstrap Methods

The bootstrap approximates the sampling distribution of a statistic by resampling from the observed data. It is invaluable when analytical formulas for standard errors or confidence intervals are unavailable or unreliable.

Nonparametric Bootstrap

Bootstrap Confidence Intervals

Several bootstrap CI methods exist with different accuracy levels:

Parametric Bootstrap

In the parametric bootstrap, instead of resampling from the data, we fit a parametric model with estimated parameters theta-hat and then draw bootstrap samples from this fitted distribution. This is more efficient when the model is correct and is used to calibrate test statistics or assess goodness of fit.

14. Multiple Testing Corrections

When conducting m simultaneous hypothesis tests each at level alpha, the probability of at least one false rejection (familywise error rate, FWER) inflates dramatically. For m independent tests: FWER = 1 - (1-alpha)^m. For m = 100, alpha = 0.05: FWER = 1 - 0.95^100 = 0.994. Multiple testing corrections address this.

Bonferroni Correction

The Bonferroni correction controls FWER at level alpha by testing each of m hypotheses at level alpha/m. It is valid for any dependence structure:

Holm-Bonferroni (Step-Down) Procedure

Holm's procedure also controls FWER but is uniformly more powerful than Bonferroni:

False Discovery Rate: Benjamini-Hochberg

The FDR is the expected proportion of rejected hypotheses that are false discoveries. Controlling FDR is less stringent than controlling FWER but typically offers much higher power — it is the standard approach in genomics, neuroimaging, and other high-dimensional settings.

15. Bayesian Inference

Posterior and Point Estimates

The posterior distribution combines the likelihood and prior:

Credible Intervals

A (1-alpha)*100% credible interval [a,b] satisfies P(a <= theta <= b | x) = 1-alpha. Two standard forms:

Conjugate Priors

A prior is conjugate for a likelihood if the posterior is in the same family as the prior. This enables closed-form Bayesian updating.

Normal-Normal Conjugate Analysis

For X1, ..., Xn ~ N(mu, sigma^2) with sigma^2 known and prior mu ~ N(mu_0, tau^2):

Markov Chain Monte Carlo (MCMC)

When conjugate priors are unavailable or the model is complex, analytical posteriors are intractable. MCMC methods generate samples from the posterior by constructing a Markov chain whose stationary distribution equals the posterior.

Metropolis-Hastings Algorithm

Gibbs Sampling

Gibbs sampling is a special case of MH for multivariate parameters where we cycle through each component, sampling from its full conditional distribution given all other components. It requires no acceptance-rejection step:

MCMC diagnostics include trace plots (to assess mixing and convergence), autocorrelation plots, Gelman-Rubin R-hat statistic (compares between-chain and within-chain variance; values near 1.0 indicate convergence), and effective sample size (accounts for autocorrelation among draws).

16. Frequently Asked Questions

Related Math Topics