Least Squares as Maximum Likelihood
~8 min read · Tier toggle changes depth (T1 plain · T2 standard · T3 rigour)
You should be comfortable with the definition of a likelihood function and the procedure for finding its maximum by setting the derivative of the log-likelihood to zero. Familiarity with the probability density function of the normal (Gaussian) distribution is also assumed.
Why this matters
This lecture bridges the gap between statistical theory and the practical fitting algorithms used in every physics laboratory.
For centuries, physicists used the method of least squares to fit lines to data, often justifying it with geometric intuition about minimizing distances. However, this approach lacks a rigorous statistical foundation. By demonstrating that least squares is a direct consequence of maximum likelihood estimation under Gaussian error assumptions, we gain a powerful tool: we can derive the optimal fitting procedure for any noise distribution, not just the Gaussian. This unification allows us to understand exactly when least squares is valid and how to modify it when our experimental errors are non-Gaussian, providing a robust framework for data analysis in experimental physics.
The Intuition: Why Squares?
Imagine you are measuring the position of a particle, but your detector has some inherent noise. If you assume this noise is random and symmetric, with most errors being small and large errors being rare, the most probable distribution of these errors is the Gaussian (normal) distribution. The likelihood of observing your specific set of data points, given a certain model, is the product of the probabilities of each individual point. Because the Gaussian probability density involves the exponential of the negative square of the error, maximizing the product of these probabilities is mathematically equivalent to minimizing the sum of the squared errors. In plain language, least squares works because it is the most probable way to explain your data if your measurement errors are Gaussian. It is not an arbitrary geometric choice; it is a statistical necessity for this specific type of noise.
From Likelihood to Chi-Squared
Consider a dataset of \( N \) independent measurements \( y_i \) taken at known input values \( x_i \). We propose a model \( f(x_i; \theta) \) with parameters \( \theta \). We assume that each measurement \( y_i \) is scattered about the true model value by a Gaussian error with standard deviation \( \sigma_i \). The probability density for a single measurement is given by the normal distribution:
Since the measurements are independent, the joint likelihood \( L(\theta) \) is the product of the individual densities:
To find the maximum likelihood estimator \( \hat{\theta}_{\text{MLE}} \), we maximize \( L(\theta) \). It is standard practice to maximize the log-likelihood \( \ell(\theta) = \ln L(\theta) \), which converts the product into a sum:
The first term is a constant with respect to \( \theta \) and does not affect the location of the maximum. Therefore, maximizing \( \ell(\theta) \) is equivalent to minimizing the second term. We define the weighted sum of squared residuals, \( \chi^2(\theta) \), as:
Thus, the condition for the maximum likelihood estimator is identical to the condition for the minimum of \( \chi^2 \):
The Role of Weights
A crucial insight from this derivation is the appearance of the term \( 1/\sigma_i^2 \) in the sum. This term acts as a weight for each data point. If a measurement has a small uncertainty \( \sigma_i \), its weight is large, meaning it contributes significantly to the fit. Conversely, if a measurement has a large uncertainty, its weight is small, and it has less influence on the final parameter estimate. This is physically intuitive: we should trust precise measurements more than imprecise ones. In the special case where all uncertainties are equal, \( \sigma_i = \sigma \), the weights are constant and can be factored out, reducing the problem to the unweighted least squares method: minimizing \( \sum (y_i - f(x_i; \theta))^2 \). However, in real experimental physics, uncertainties often vary from point to point (e.g., due to counting statistics or varying signal-to-noise ratios), making the weighted form essential for a statistically optimal fit.
Generalization to Other Distributions
The power of the maximum likelihood framework is that it is not limited to Gaussian errors. The derivation above relied specifically on the Gaussian form of the probability density. If the experimental errors follow a different distribution, the likelihood function changes, and consequently, the optimal fitting criterion changes. For example, if the errors follow a Laplacian (double exponential) distribution, which has heavier tails than the Gaussian, maximizing the likelihood leads to minimizing the sum of absolute residuals, \( \sum |y_i - f(x_i; \theta)| \), rather than the sum of squares. This is known as the method of least absolute deviations. It is more robust to outliers because a single large error does not dominate the fit as severely as it does in the least squares case. Therefore, the choice of fitting method should always be guided by the underlying statistical model of the noise, not just by habit.
The derivations
Least Squares from Gaussian Likelihood
Given N independent measurements yi at known inputs xi, each scattered about a model prediction f(xi; θ) by a Gaussian error of standard deviation σi, the parameters θ that maximise the joint likelihood are exactly those that minimise the weighted sum of squared residuals χ² = Σi (yi − f(xi; θ))² / σi². Maximum likelihood and least squares coincide.
The Method of Maximum Likelihood
Given a dataset x = (x1, …, xn) modelled as independent draws from a density (or mass function) f(x; θ) indexed by a parameter θ, the likelihood is the joint density read as a function of θ with the data held fixed, L(θ) = ∏i f(xi; θ). The maximum-likelihood estimator θ̂ maximises L, equivalently the log-likelihood ℓ(θ) = ∑i ln f(xi; θ), and for a regular interior maximum it solves the estimating (score) equation ∑i ∂ ln f(xi; θ)/∂θ = 0.
A common misconception is that the value of \( \chi^2_{\text{min}} \) on its own shows how good a fit is. Its size only means something relative to the number of degrees of freedom. A small \( \chi^2_{\text{min}} \) does not automatically mean the model is correct; it only means that the parameters \( \hat{\theta} \) are the best fit for that specific model. To assess goodness of fit, one must compare \( \chi^2_{\text{min}} \) to the expected value, which is approximately equal to the number of degrees of freedom \( N - k \) (where \( k \) is the number of fitted parameters), and check if it falls within the appropriate confidence interval. C
Worked example
Problem. A student measures the voltage \( V \) across a resistor at three different current settings \( I \). The measurements are: \( (I_1, V_1) = (1.0\ \mathrm{A}, 5.1\ \mathrm{V}) \), \( (I_2, V_2) = (2.0\ \mathrm{A}, 10.2\ \mathrm{V}) \), and \( (I_3, V_3) = (3.0\ \mathrm{A}, 14.8\ \mathrm{V}) \). The standard uncertainties in the voltage measurements are \( \sigma_1 = 0.2\ \mathrm{V} \), \( \sigma_2 = 0.3\ \mathrm{V} \), and \( \sigma_3 = 0.4\ \mathrm{V} \). The current settings are known exactly. Assume the relationship is linear, \( V = R I \), where \( R \) is the resistance. Find the maximum likelihood estimate of \( R \) and the corresponding \( \chi^2 \) value.
- Define the model: \( f(I; R) = R I \). The residuals are \( r_i = V_i - R I_i \).
- Write the weighted chi-squared function: \( \chi^2(R) = \sum_{i=1}^{3} \frac{(V_i - R I_i)^2}{\sigma_i^2} \).
- Substitute the values: \( \chi^2(R) = \frac{(5.1 - R)^2}{0.2^2} + \frac{(10.2 - 2R)^2}{0.3^2} + \frac{(14.8 - 3R)^2}{0.4^2} \).
- Simplify the coefficients: \( \frac{1}{0.04} = 25 \), \( \frac{1}{0.09} \approx 11.11 \), \( \frac{1}{0.16} = 6.25 \).
- Expand the terms: \( \chi^2(R) = 25(5.1 - R)^2 + 11.11(10.2 - 2R)^2 + 6.25(14.8 - 3R)^2 \).
- Differentiate with respect to \( R \) and set to zero: \( \frac{d\chi^2}{dR} = -50(5.1 - R) - 44.44(10.2 - 2R) - 37.5(14.8 - 3R) = 0 \).
- Collect terms for \( R \): \( R(50 + 88.88 + 112.5) = 255 + 453.28 + 555 \).
- Solve for \( R \): \( 251.38 R = 1263.28 \implies R \approx 5.025\ \Omega \).
- Calculate \( \chi^2 \) at \( R = 5.025 \): \( \chi^2 \approx 25(0.075)^2 + 11.11(0.15)^2 + 6.25(-0.275)^2 \approx 0.14 + 0.25 + 0.47 \approx 0.86 \).
Units check. The term \( (V_i - R I_i)^2 / \sigma_i^2 \) has units \( (\mathrm{V})^2 / (\mathrm{V})^2 = 1 \), which is dimensionless, as required for \( \chi^2 \). The resistance \( R \) has units \( \mathrm{V/A} = \Omega \), which is consistent.
Three quick tests
- Why does maximizing the log-likelihood for Gaussian errors lead to minimizing the sum of squared residuals?
Answer
The log of the Gaussian PDF contains a term proportional to \( -\frac{1}{2\sigma^2}(y_i - f_i)^2 \). Summing these terms and dropping constants leaves the sum of squared residuals. - How does the weighting factor \( 1/\sigma_i^2 \) affect the fit if one data point has a much larger uncertainty than the others?
Answer
The point with the larger uncertainty receives a smaller weight, so it has less influence on the final parameter estimate, preventing it from dominating the fit. - If the experimental errors are known to follow a Laplacian distribution, what fitting criterion should be used instead of least squares?
Answer
Least absolute deviations, which minimizes the sum of the absolute values of the residuals.
Problem set for PU-106 →
Each derivation in this unit ends with five graded problems with full solutions.
Lecture notes drafted with a local model and checked line by line by the editor; every number in the worked example was recomputed.
Derivations cited: D-120 · D-119 · Threads: chance