Uncertainties on Fitted Parameters
~8 min read · Tier toggle changes depth (T1 plain · T2 standard · T3 rigour)
You should be comfortable with matrix inversion, the concept of a covariance matrix, and the basic setup of weighted least-squares fitting from the previous lecture.
Why this matters
A fitted number without an error bar is just a guess.
In experimental physics, the value of a parameter is meaningless unless we know how much it could fluctuate due to random noise in the data. This lecture bridges the gap between finding the "best" fit and quantifying the reliability of that fit. We will see that the uncertainty in our parameters is not arbitrary but is mathematically determined by the curvature of the likelihood function and the structure of our data. This provides the rigorous foundation for every error bar you will ever report in a lab report or publication.
The Intuition: Curvature and Confidence
Imagine you are trying to find the bottom of a valley. If the valley is very steep and narrow, you can pinpoint the bottom location quite accurately; a small step in either direction makes you climb high up the walls. If the valley is shallow and wide, it is much harder to know exactly where the bottom is; you could be standing slightly to the left or right and still be near the lowest point. In data analysis, the "valley" is the error function (or negative log-likelihood) as a function of your parameters. The "steepness" of this valley at the minimum tells us how sensitive the fit is to changes in the parameters. A steep, narrow valley implies small uncertainty, while a shallow, wide valley implies large uncertainty. This geometric intuition is the core of parameter error estimation.
From Least Squares to Covariance
Recall from the previous lecture that for a linear model \( y = A p + \epsilon \), the best-fit parameters \( \hat{p} \) are found by minimizing the chi-squared statistic \( \chi^2 \). The solution is given by the normal equations. However, minimizing \( \chi^2 \) only gives us the central value. To find the uncertainty, we look at how \( \chi^2 \) changes when we perturb the parameters slightly away from the minimum.
For a Gaussian noise model, the probability of observing the data \( y \) given parameters \( p \) is proportional to \( \exp(-\chi^2/2) \). The variance of the parameter estimates is related to the second derivative of the log-likelihood (or equivalently, the second derivative of \( \chi^2 \)) with respect to the parameters. In matrix form, the covariance matrix of the fitted parameters, denoted \( \mathrm{Cov}(\hat{p}) \), is given by the inverse of the matrix that defines the quadratic form of \( \chi^2 \).
Here, \( W \) is the weight matrix, which is typically the inverse of the covariance matrix of the noise. If the noise is uncorrelated with constant variance \( \sigma^2 \), then \( W = \frac{1}{\sigma^2} I \), and the covariance matrix simplifies to \( \sigma^2 (A^T A)^{-1} \). The diagonal elements of this matrix give the variances of the individual parameters, and the square roots of these diagonal elements are the standard errors.
The General Case: Fisher Information
While the least-squares approach works perfectly for linear models with Gaussian noise, real-world data often involves non-linear models or non-Gaussian noise. The general statistical framework uses the concept of Fisher Information. The Fisher Information Matrix \( I(\theta) \) measures the amount of information that an observable random variable carries about an unknown parameter \( \theta \). Mathematically, it is defined as the expected value of the negative second derivative of the log-likelihood function.
where \( \ell(\theta) = \ln P(\text{data} | \theta) \) is the log-likelihood. A key result in statistical theory, the Cramér–Rao bound, states that the covariance matrix of any unbiased estimator \( \hat{\theta} \) must be greater than or equal to the inverse of the Fisher Information Matrix in the matrix sense (Löwner order).
For large sample sizes, the Maximum Likelihood Estimator (MLE) achieves this bound, meaning its covariance is asymptotically equal to \( I(\theta)^{-1} \). This provides a universal method for estimating uncertainties: calculate the curvature of the log-likelihood at the maximum, and invert it. In the special case of Gaussian noise, the Fisher Information Matrix is directly related to the Hessian of the chi-squared function, linking the general theory back to the familiar least-squares results.
Correlations and Degeneracy
One of the most critical insights from the covariance matrix is that parameter uncertainties are not always independent. The off-diagonal elements of \( \mathrm{Cov}(\hat{p}) \) represent the covariances between different parameters. If two parameters are correlated, a change in one is compensated by a change in the other to maintain a good fit. This is often visualized as an elongated ellipse in the parameter space rather than a circle.
High correlation can lead to a situation where individual parameters have large uncertainties, even if the combination of those parameters is well-determined. This is known as parameter degeneracy. For example, in fitting a line \( y = mx + c \), if the data points are clustered far from \( x=0 \), the slope \( m \) and intercept \( c \) are highly correlated (if they are centred on \( x = 0 \), the correlation vanishes). A small change in slope can be offset by a change in intercept, making it difficult to determine either one precisely from the data alone. Always check the correlation matrix (the normalized covariance matrix) to understand the structure of your uncertainties.
The derivations
Parameter Uncertainties and the Cramer-Rao Bound
For a statistical model with log-likelihood ℓ(θ) depending on a parameter vector θ = (θ1, …, θp), we derive that (i) the maximum-likelihood estimator θ̂ has, asymptotically, covariance equal to the inverse of the Fisher information matrix I(θ), where Ijk = −E[∂²ℓ/∂θj∂θk] is the expected curvature of the log-likelihood; and (ii) for any unbiased estimator the covariance is bounded below by I−1 in the matrix (Löwner) sense — the Cramér–Rao bound cov(θ̂) ⪰ I(θ)−1.
Linear Least Squares and the Normal Equations
For a model that is linear in its parameters, y = Ap + noise, the weighted least-squares estimate p̂ that minimises χ2 = (y − Ap)TW(y − Ap) satisfies the normal equations (ATWA)p̂ = ATWy, where A is the N×M design matrix, y is the vector of N observations, and W is a symmetric positive-definite weight matrix.
A common pitfall is assuming that the standard error from the covariance matrix is the final answer. This is only true if the model is correct and the noise is Gaussian. If the model is misspecified, the "best fit" may be biased, and the covariance matrix may underestimate the true uncertainty. Always perform a goodness-of-fit test (e.g., checking if \( \chi^2 \) is consistent with the degrees of freedom) before trusting the error bars. C
Worked example
Problem. The position \( y \) of a particle is measured at three times: \( (t, y) = (1\ \mathrm{s}, 10\ \mathrm{m}) \), \( (2\ \mathrm{s}, 20\ \mathrm{m}) \) and \( (3\ \mathrm{s}, 31\ \mathrm{m}) \), each with uncertainty \( \sigma = 1\ \mathrm{m} \). Fit \( y = v t + x_0 \) and find \( v \), \( x_0 \), their standard errors and their correlation.
- Design matrix (columns \( t \) and \( 1 \)) and weights: \( A = \begin{pmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{pmatrix} \), \( W = I/\sigma^2 \). Because \( W \) is a multiple of \( I \), it cancels in \( \hat{p} \) and \( \mathrm{Cov}(\hat{p}) = \sigma^2 (A^T A)^{-1} \).
- \( A^T A = \begin{pmatrix} 14 & 6 \\ 6 & 3 \end{pmatrix} \) and \( A^T y = \begin{pmatrix} 10 + 40 + 93 \\ 10 + 20 + 31 \end{pmatrix} = \begin{pmatrix} 143 \\ 61 \end{pmatrix} \).
- Determinant \( 14 \cdot 3 - 6 \cdot 6 = 6 \), so \( (A^T A)^{-1} = \frac{1}{6} \begin{pmatrix} 3 & -6 \\ -6 & 14 \end{pmatrix} = \begin{pmatrix} 0.5 & -1 \\ -1 & 2.333 \end{pmatrix} \).
- Best fit: \( v = \frac{1}{6}(3 \cdot 143 - 6 \cdot 61) = \frac{63}{6} = 10.5\ \mathrm{m/s} \) and \( x_0 = \frac{1}{6}(-6 \cdot 143 + 14 \cdot 61) = -\frac{4}{6} \approx -0.67\ \mathrm{m} \).
- Standard errors: \( \sigma_v = \sqrt{0.5}\,\sigma \approx 0.71\ \mathrm{m/s} \) and \( \sigma_{x_0} = \sqrt{2.333}\,\sigma \approx 1.53\ \mathrm{m} \).
- Correlation: \( \rho = \frac{-1}{\sqrt{0.5 \times 2.333}} \approx -0.93 \). The intercept is extrapolated to \( t = 0 \), well outside the data, so a steeper line must start lower: the two parameters are strongly anticorrelated.
Units check. \( (A^T A)^{-1} \) has units \( \mathrm{s^{-2}} \) in its first diagonal element and is dimensionless in its second, so \( \sigma_v \) is in \( \mathrm{m/s} \) and \( \sigma_{x_0} \) in \( \mathrm{m} \), matching \( v \) and \( x_0 \); \( \rho \) is dimensionless.
Three quick tests
- If the noise variance \( \sigma^2 \) doubles, what happens to the standard error of the fitted parameters?
Answer
It increases by a factor of \( \sqrt{2} \), because the covariance matrix is proportional to \( \sigma^2 \). - What does a large off-diagonal element in the covariance matrix indicate?
Answer
It indicates a strong correlation between the two corresponding parameters, meaning they are not independently determined by the data. - Why is the inverse of the Fisher Information Matrix a lower bound on the variance of unbiased estimators?
Answer
This is the Cramér–Rao bound, a fundamental result in statistical theory stating that no unbiased estimator can have a variance smaller than the inverse of the Fisher Information.
Problem set for PU-106 →
Each derivation in this unit ends with five graded problems with full solutions.
Lecture notes drafted with a local model and checked line by line by the editor; every number in the worked example was recomputed.
Derivations cited: D-122 · D-121 · Threads: chance