Confidence Intervals and Hypothesis Testing
~7 min read · Tier toggle changes depth (T1 plain · T2 standard · T3 rigour)
You should be comfortable with the normal distribution, the concept of a probability density function, and the definition of the \(\chi^2\) distribution with \(\nu\) degrees of freedom as established in the previous lecture.
Why this matters
A single number without an uncertainty is merely a guess; confidence intervals and hypothesis testing provide the rigorous framework to quantify how much we can trust that number.
In experimental physics, we rarely measure a true parameter directly. Instead, we obtain a set of noisy data points and use statistical methods to estimate parameters like a slope or an intercept. However, because our data is finite and noisy, our estimate is only one possible outcome of a random process. To communicate the reliability of our result, we must define a range of values within which the true parameter is likely to reside, given our data. This lecture bridges the gap between a point estimate and a defensible claim, introducing the concepts of confidence levels, p-values, and the decision-making process inherent in hypothesis testing.
The Logic of Uncertainty
Imagine you are measuring the mass of a particle. You perform ten measurements and calculate an average. This average is your "point estimate." However, if you repeated the experiment tomorrow, you would likely get a slightly different average. The distribution of these possible averages is what we call the sampling distribution. A confidence interval is a range constructed around your point estimate such that, if you were to repeat the experiment many times, a certain percentage of these intervals (for example, 95%) would contain the true, fixed value of the mass. It is crucial to understand that the true value does not move; the interval does. The "95% confidence" refers to the long-run success rate of the method used to generate the interval, not the probability that the true value lies within the specific interval you have calculated. This distinction is the foundation of frequentist statistics, which is the standard framework in experimental physics.
Constructing Confidence Intervals
For a single measurement \(x\) with a known standard deviation \(\sigma\), the probability that an interval \(x \pm k\sigma\), built in this way, covers the true value \(\mu\) is given by the cumulative distribution function (CDF) of the normal distribution. If we desire a confidence level \(C\), we find the critical value \(z_{\alpha/2}\) such that the area under the normal curve between \(-z_{\alpha/2}\) and \(+z_{\alpha/2}\) is equal to \(C\). For a 95% confidence interval, \(\alpha = 0.05\), and \(z_{0.025} \approx 1.96\). Thus, the interval is \(x \pm 1.96\sigma\).
When estimating a mean from \(N\) samples, the uncertainty of the mean is the standard error, \(\sigma_{\bar{x}} = \sigma / \sqrt{N}\). If \(\sigma\) is unknown and estimated from the data as \(s\), we use the Student's t-distribution instead of the normal distribution, especially for small \(N\). The interval becomes \(\bar{x} \pm t_{\alpha/2, N-1} \frac{s}{\sqrt{N}}\), where \(t_{\alpha/2, N-1}\) is the critical value from the t-distribution with \(N-1\) degrees of freedom. As \(N\) increases, the t-distribution converges to the normal distribution.
Hypothesis Testing and P-values
Hypothesis testing allows us to make decisions about a claim. We start with a null hypothesis \(H_0\), which is typically the statement of "no effect" or "no difference" (e.g., the measured value equals the theoretical prediction). We calculate a test statistic, such as a z-score or t-statistic, which measures how far our observed data is from what we would expect under \(H_0\). The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the one calculated, assuming \(H_0\) is true.
A small p-value indicates that the observed data is unlikely under the null hypothesis. We compare the p-value to a pre-defined significance level \(\alpha\) (commonly 0.05). If \(p \lt \alpha\), we reject \(H_0\). It is vital to note that rejecting \(H_0\) does not prove the alternative hypothesis is true; it only suggests that the data is inconsistent with \(H_0\) at the chosen confidence level. Failing to reject \(H_0\) does not prove it is true either; it simply means we do not have sufficient evidence to discard it.
Connecting to Goodness of Fit
In the context of fitting a model to data, as discussed in the previous lecture, the \(\chi^2\) statistic serves as our test statistic. We compare the minimised \(\chi^2_{\min}\) to the expected value \(N - M\). The p-value for the fit is the probability of obtaining a \(\chi^2\) value greater than or equal to our observed \(\chi^2_{\min}\) under the assumption that the model is correct and the errors are Gaussian. This is given by the survival function of the \(\chi^2\) distribution with \(\nu = N - M\) degrees of freedom.
If the p-value is very small (e.g., \(p \lt 0.01\)), it suggests that the model is a poor description of the data, or that the assigned errors are underestimated. If the p-value is very large (e.g., \(p \gt 0.99\)), it may suggest that the errors are overestimated. A "good" fit typically yields a p-value in the middle range, indicating that the deviations between data and model are consistent with random noise.
The derivation
Goodness of Fit and Degrees of Freedom
For N data points with independent Gaussian errors, fitted to a model with M adjustable parameters by minimising χ², the minimised statistic obeys E[χ²min] = N − M and, for a linear model, is distributed as χ² with ν = N − M degrees of freedom. Each fitted parameter therefore removes exactly one degree of freedom, and the reduced statistic χ²ν = χ²min / (N − M) has expectation one, which is the working goodness-of-fit test.
A common misconception is that a p-value of 0.05 means there is a 5% chance that the null hypothesis is true. This is incorrect. The p-value is the probability of the data given the null hypothesis, \(P(\text{Data} | H_0)\), not the probability of the null hypothesis given the data, \(P(H_0 | \text{Data})\). Confusing these two conditional probabilities is known as the prosecutor's fallacy and can lead to erroneous conclusions in experimental analysis. C
Worked example
Problem. A student measures the half-life of a radioactive isotope 10 times. The sample mean is \(\bar{t} = 15.2\ \mathrm{s}\) and the sample standard deviation is \(s = 0.8\ \mathrm{s}\). The theoretical prediction is \(t_0 = 15.0\ \mathrm{s}\). Calculate the 95% confidence interval for the half-life and determine if the result is consistent with the theory at the 5% significance level.
- Determine the standard error of the mean: \(\mathrm{SE} = s / \sqrt{N} = 0.8 / \sqrt{10} \approx 0.253\ \mathrm{s}\).
- Find the critical t-value for 95% confidence with \(\nu = N - 1 = 9\) degrees of freedom. From tables, \(t_{0.025, 9} \approx 2.262\).
- Calculate the margin of error: \(ME = t \times \mathrm{SE} = 2.262 \times 0.253 \approx 0.572\ \mathrm{s}\).
- Construct the confidence interval: \(15.2 \pm 0.572\), which is \([14.628, 15.772]\ \mathrm{s}\).
- Check consistency: The theoretical value \(15.0\ \mathrm{s}\) lies within the interval \([14.628, 15.772]\ \mathrm{s}\). Alternatively, calculate the t-statistic: \(t = (15.2 - 15.0) / 0.253 \approx 0.79\). Since \(|t| \lt 2.262\), we fail to reject the null hypothesis.
Units check. The standard deviation and mean are in seconds, so the standard error and margin of error are also in seconds. The final interval is in seconds, consistent with the quantity measured.
Three quick tests
- If a 95% confidence interval is calculated for a parameter, does this mean there is a 95% probability that the true value is inside that specific interval?
Answer
No. It means that 95% of intervals calculated by this method from repeated experiments will contain the true value. The true value is fixed; the interval is random. - You perform a hypothesis test and obtain a p-value of 0.03. If your significance level is \(\alpha = 0.05\), what is your decision?
Answer
Reject the null hypothesis, because \(p \lt \alpha\). The data is unlikely to occur if the null hypothesis were true. - In a \(\chi^2\) goodness-of-fit test, if the p-value is 0.99, what does this suggest about the error bars assigned to the data points?
Answer
It suggests the errors are likely overestimated, as the data fits the model "too well" for the assumed level of noise.
Problem set for PU-106 →
Each derivation in this unit ends with five graded problems with full solutions.
Lecture notes drafted with a local model and checked line by line by the editor; every number in the worked example was recomputed.
Derivations cited: D-124 · Threads: chance