Goodness of Fit and Degrees of Freedom

The Intuition of Degrees of Freedom

Imagine you have a set of data points and you want to draw a straight line through them. If you have only two points, there is exactly one straight line that passes through both of them perfectly. The "error" between the data and the model is zero. However, this does not mean the model is "good" in a statistical sense; it simply means you have used up all your freedom to adjust the line. You have no "leftover" freedom to check if the line is consistent with the noise.

Now imagine you have ten points. You can still adjust the slope and the intercept (two parameters) to minimize the distance between the line and the points. However, because you have many more points than parameters, the line cannot pass through every single point. The deviations that remain are the "residuals." These residuals represent the information that was not used to define the line. Statistically, each parameter you fit "consumes" one degree of freedom. If you have \( N \) data points and fit \( M \) parameters, you are left with \( N - M \) independent pieces of information that can be used to assess the fit. This number, \( \nu = N - M \), is the degrees of freedom. It is the effective number of independent errors that remain after the model has been tuned to the data.

The Chi-Squared Statistic in Fitting

When we fit a model \( f(x; \theta) \) to data \( y_i \) with known uncertainties \( \sigma_i \), we define the chi-squared statistic as:

Our goal is to find the parameters \( \theta \) that minimize this quantity. Let \( \chi^2_{\min} \) be the minimum value achieved. If the model is correct and the errors are Gaussian, the minimized statistic is not just a single number; it follows a specific probability distribution. According to the derivation cited in D-124, for a linear model with independent Gaussian errors, the distribution of \( \chi^2_{\min} \) is the chi-squared distribution with \( \nu = N - M \) degrees of freedom.

This result is profound because it tells us that the "best" fit is not expected to have \( \chi^2 = 0 \). Instead, the expected value of the minimized statistic is equal to the degrees of freedom:

If you find a fit where \( \chi^2_{\min} \) is significantly smaller than \( N - M \), it suggests that the errors have been overestimated or that the model has enough freedom to follow the noise. If \( \chi^2_{\min} \) is significantly larger, the model is likely incorrect or the errors are underestimated.

The Reduced Chi-Squared

To make the goodness-of-fit test independent of the size of the dataset, we define the reduced chi-squared, often denoted as \( \chi^2_\nu \) or \( \bar{\chi}^2 \):

By dividing by the degrees of freedom, we normalize the statistic. For a good fit, the expected value of this reduced statistic is exactly one:

This provides a simple rule of thumb. If \( \chi^2_\nu \approx 1 \), the fit is consistent with the data within the expected statistical fluctuations. If \( \chi^2_\nu \gg 1 \), the model is poor. If \( \chi^2_\nu \ll 1 \), the errors may be overestimated. This normalization is crucial because it allows us to compare the quality of fits from experiments with vastly different numbers of data points. For example, 100 points with two fitted parameters and \( \chi^2_{\min} = 95 \) give \( \chi^2_\nu \approx 0.97 \), and 10 points with one parameter and \( \chi^2_{\min} = 9 \) give \( \chi^2_\nu = 1.0 \): both are consistent. The larger data set tests the model more sharply, because the spread of \( \chi^2_\nu \) about 1 is \( \sqrt{2/\nu} \).

Connection to the Chi-Squared Distribution

The result that \( \chi^2_{\min} \) follows a chi-squared distribution with \( N - M \) degrees of freedom relies on the properties of the chi-squared distribution itself, derived in D-123. Recall that if \( Z_1, \dots, Z_k \) are independent standard normal variables, then \( Q = \sum_{i=1}^k Z_i^2 \) follows a chi-squared distribution with \( k \) degrees of freedom, with mean \( E[Q] = k \) and variance \( \text{Var}(Q) = 2k \).

In the context of fitting, the residuals \( r_i = y_i - f(x_i; \hat{\theta}) \) are not independent in the same way as the raw errors because they are constrained by the fitting conditions (the derivatives of \( \chi^2 \) with respect to the parameters are zero at the minimum). These constraints reduce the effective number of independent variables from \( N \) to \( N - M \). Therefore, the sum of the squared standardized residuals behaves like a sum of \( N - M \) independent squared standard normal variables. This is why the degrees of freedom are \( N - M \) rather than \( N \). The variance of the minimized chi-squared statistic is therefore \( 2(N - M) \), which is essential for calculating the probability of obtaining a fit as good or worse than the observed one.