Propagation of Uncertainty II: Correlated Errors

The Intuition Behind Correlated Errors

Imagine measuring the length and width of a rectangle to find its area. If your ruler is consistently too short, both the length and width measurements will be biased in the same direction. This is a positive correlation. If you calculate the area, the errors in length and width reinforce each other, making the area error larger than if the errors were random and independent. Conversely, if measuring one quantity tends to make another smaller (a negative correlation), the errors might partially cancel out in the derived quantity. The key insight is that the "cross-terms" in the variance calculation represent these interactions. When variables are independent, these cross-terms vanish. When they are correlated, they become crucial components of the total uncertainty budget.

Variance of Linear Combinations

Let us start with the simplest case: a linear combination of random variables. Suppose we have random variables \( X_1, \dots, X_n \) with finite variances and fixed real constants \( a_1, \dots, a_n \). We define the linear combination \( S = \sum_{i} a_i X_i \). The variance of this sum is not simply the sum of the variances unless the variables are uncorrelated.

According to the general rule for linear combinations, the variance is given by the double sum of the covariances weighted by the coefficients:

If the variables are pairwise uncorrelated, then \( \mathrm{Cov}(X_i, X_j) = 0 \) for \( i \neq j \), and the expression collapses to the familiar weighted sum of variances:

However, when correlations exist, the terms where \( i \neq j \) contribute significantly. Note that \( \mathrm{Cov}(X_i, X_j) = \mathrm{Cov}(X_j, X_i) \), so the double sum effectively counts each pair twice (once as \( i,j \) and once as \( j,i \)).

The Covariance Matrix Formalism

For general differentiable functions, we use the Jacobian matrix approach. Let \( \mathbf{x} \in \mathbb{R}^n \) be a random input vector with mean \( \boldsymbol{\mu} \) and covariance matrix \( \boldsymbol{\Sigma}_x \). Let \( \mathbf{y} = \mathbf{f}(\mathbf{x}) \) be a differentiable vector-valued map from \( \mathbb{R}^n \) to \( \mathbb{R}^m \). The covariance of the output vector \( \mathbf{y} \) to first order is determined by the Jacobian matrix \( \mathbf{J} \).

The Jacobian \( \mathbf{J} \) is an \( m \times n \) matrix with elements \( J_{ij} = \frac{\partial f_i}{\partial x_j} \) evaluated at the mean \( \boldsymbol{\mu} \). The propagation of uncertainty is then given by the matrix equation:

This compact form encapsulates all the cross-terms. The diagonal elements of \( \boldsymbol{\Sigma}_y \) give the variances of the output components, while the off-diagonal elements give the covariances between different output components. This method is powerful because it handles any number of inputs and outputs simultaneously, provided the function is differentiable at the mean.

Interpreting the Cross-Terms

Consider a simple case with two inputs \( x_1 \) and \( x_2 \) and one output \( y = f(x_1, x_2) \). The Jacobian is a row vector \( \mathbf{J} = \left[ \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2} \right] \). The covariance matrix \( \boldsymbol{\Sigma}_x \) is a \( 2 \times 2 \) matrix:

where \( \rho \) is the correlation coefficient. The variance of \( y \) is:

The last term is the correlation term. If \( \rho > 0 \) and the partial derivatives have the same sign, this term increases the total variance. If \( \rho < 0 \) or the derivatives have opposite signs, it decreases the variance. This demonstrates that correlation can either amplify or dampen the uncertainty in the result.