Covariance and Correlation

1. Covariance

Covariance measures how two random variables vary together. For example, height and weight of giraffes have positive covariance because when one is large, the other tends to be large too.

a) Definition

For random variables $X$ and $Y$ with means $\mu_X$ and $\mu_Y$, the covariance is:

$$\operatorname{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)].$$

b) Key Properties of Covariance

Zero covariance means no linear relationship, but variables can still be dependent in other ways.

c) Computing Covariance: Sums and Integrals

• Discrete case: For joint pmf $p(x_i, y_j)$,

$$\operatorname{Cov}(X, Y) = \sum_{i=1}^n \sum_{j=1}^m p(x_i, y_j)(x_i - \mu_X)(y_j - \mu_Y) = \left(\sum_{i=1}^n \sum_{j=1}^m p(x_i, y_j) x_i y_j\right) - \mu_X \mu_Y.$$

• Continuous case: For joint pdf $f(x,y)$ over $[a,b]\times[c,d]$,

$$\operatorname{Cov}(X, Y) = \int_c^d \int_a^b (x - \mu_X)(y - \mu_Y) f(x,y) \, dx \, dy = \left(\int_c^d \int_a^b x y f(x,y) \, dx \, dy\right) - \mu_X \mu_Y.$$

d) Example (Summary)

• Flip a fair coin 3 times.
• Define $X$ = number of heads in first 2 flips, $Y$ = number of heads in last 2 flips.
• Compute $\operatorname{Cov}(X, Y)$ using joint pmf and properties.
• Marginal means: $E[X] = E[Y] = 1$.
• Use definition or properties to find covariance.

To remember: Covariance quantifies joint variability; zero covariance means no linear association but not necessarily independence. Use the formula $\operatorname{Cov}(X, Y) = E[XY] - \mu_X \mu_Y$ for calculations.

1. Computation of Covariance

• Covariance definition:
  $\operatorname{Cov}(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - \mu_X \mu_Y.$

• Example calculation:
  Given joint probabilities,
  $\operatorname{Cov}(X,Y) = \frac{1}{8}(0-1)(0-1) + \frac{1}{8}(2-1)(2-1) = \frac{1}{4}.$

• Using expectation of product:
  $E[XY] = 1 \cdot \frac{2}{8} + 2 \cdot \frac{1}{8} + 2 \cdot \frac{1}{8} + 4 \cdot \frac{1}{8} = \frac{5}{4}.$

  Then,
  $\operatorname{Cov}(X,Y) = E[XY] - \mu_X \mu_Y = \frac{5}{4} - 1 = \frac{1}{4}.$

2. Covariance via Properties of Covariance

• If $X = X_1 + X_2$ and $Y = X_2 + X_3$, where $X_i \sim \text{Bernoulli}(0.5)$ are independent tosses, then:

  $$\operatorname{Cov}(X,Y) = \operatorname{Cov}(X_1 + X_2, X_2 + X_3) = \operatorname{Cov}(X_1,X_2) + \operatorname{Cov}(X_1,X_3) + \operatorname{Cov}(X_2,X_2) + \operatorname{Cov}(X_2,X_3).$$

• Independence implies zero covariance for distinct tosses:
  $\operatorname{Cov}(X_1,X_2) = \operatorname{Cov}(X_1,X_3) = \operatorname{Cov}(X_2,X_3) = 0.$

• Only non-zero term is variance of $X_2$:
  $\operatorname{Cov}(X,Y) = \operatorname{Var}(X_2) = \frac{1}{4}.$

3. Important Example: Zero Covariance Does Not Imply Independence

• Let $X$ take values $-2, -1, 0, 1, 2$ each with probability $1/5$, and define $Y = X^2$.

• Means:
  $E[X] = 0, \quad E[Y] = 2.$

• Not independent:
  For example,
  $P(X=-2, Y=0) = 0 \neq P(X=-2) \cdot P(Y=0) = \frac{1}{5} \times \frac{1}{5} = \frac{1}{25}.$

• Covariance calculation:

  $$\operatorname{Cov}(X,Y) = \frac{1}{5}(-8 - 1 + 0 + 1 + 8) - 0 \times 2 = 0.$$

4. Key Takeaways

Covariance measures only linear dependence; zero covariance does not guarantee independence.

1. Properties of correlation

• $\rho$ is the covariance of the standardized variables:
  $\rho = \text{Cov}\left(\frac{X - \mu_X}{\sigma_X}, \frac{Y - \mu_Y}{\sigma_Y}\right)$

• Range:
  $-1 \leq \rho \leq 1$

• Interpretation:

  • $\rho = 1$ means perfect positive linear relationship
  • $\rho = -1$ means perfect negative linear relationship
  • $\rho = 0$ means no linear relationship (but variables may still be dependent)

• Scale invariance:
  Correlation does not change if $X$ or $Y$ are scaled or shifted by constants.

To remember: Correlation is the normalized covariance, measuring linear association independent of units.

1. Bivariate Normal Distributions

• The joint density of a bivariate normal $(X,Y)$ with means $\mu_X, \mu_Y$, standard deviations $\sigma_X, \sigma_Y$, and correlation $\rho$ is:

  $$f(x,y) = \frac{1}{2\pi \sigma_X \sigma_Y \sqrt{1-\rho^2}} \exp\left(-\frac{1}{2(1-\rho^2)} \left[ \frac{(x-\mu_X)^2}{\sigma_X^2} + \frac{(y-\mu_Y)^2}{\sigma_Y^2} - \frac{2\rho(x-\mu_X)(y-\mu_Y)}{\sigma_X \sigma_Y} \right] \right)$$

• Marginals: $X$ and $Y$ are individually normal with parameters $(\mu_X, \sigma_X^2)$ and $(\mu_Y, \sigma_Y^2)$ respectively.

• The correlation between $X$ and $Y$ is exactly $\rho$.

• Geometric interpretation:

  • Data points tend to cluster around the line $y = \rho \frac{\sigma_Y}{\sigma_X} (x - \mu_X) + \mu_Y$.
  • When $\rho \to \pm 1$, points lie almost perfectly on a line.
  • Positive $\rho$ → points concentrate in 1st and 3rd quadrants relative to means.
  • Negative $\rho$ → points concentrate in 2nd and 4th quadrants.

To remember: The correlation coefficient $\rho$ quantifies the strength and direction of a linear relationship between two variables, bounded between -1 and 1, with $\pm 1$ indicating perfect linear dependence.

2. Visualizing Correlation in Bivariate Normal Samples

• Scatter plots of simulated bivariate normal data with varying $\rho$ show:
  • For $\rho = 0$, points are scattered without linear pattern.
  • For $\rho > 0$, points cluster along an increasing line.
  • For $\rho < 0$, points cluster along a decreasing line.
  • As $|\rho|$ approaches 1, points align more tightly along a straight line.

3. Summary Table: Correlation Properties

4. Note on Limitations

• Correlation can fail to detect nonlinear relationships.
• Example: variables related by a quadratic or other nonlinear function may have zero correlation despite strong dependence.

1. Proofs of Covariance Properties

• Covariance linearity and expectation properties imply:

  1. $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$ (symmetry)
  2. $\operatorname{Cov}(aX + b, Y) = a \operatorname{Cov}(X, Y)$ for constants $a,b$

• Variance as covariance with itself:

  $$\operatorname{Cov}(X, X) = E[(X - \mu_X)^2] = \operatorname{Var}(X)$$

• Covariance expressed via expectation:

  $$\operatorname{Cov}(X, Y) = E[XY] - \mu_X \mu_Y$$

• Variance of sum:

  $$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2 \operatorname{Cov}(X, Y)$$

• Covariance of independent variables:

  If $X$ and $Y$ are independent, then

  $$\operatorname{Cov}(X, Y) = 0$$

2. Proof of Correlation Bounds

• Define correlation coefficient $\rho = \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \sigma_Y}$.

• Using variance positivity:

  $$0 \leq \operatorname{Var}\left(\frac{X}{\sigma_X} - \frac{Y}{\sigma_Y}\right) = 2 - 2\rho \implies \rho \leq 1$$

• Similarly,

  $$0 \leq \operatorname{Var}\left(\frac{X}{\sigma_X} + \frac{Y}{\sigma_Y}\right) \implies -1 \leq \rho$$

• Conclusion: Correlation coefficient satisfies

  $$-1 \leq \rho \leq 1$$

• If $\rho = 1$, then

  $$\frac{X}{\sigma_X} - \frac{Y}{\sigma_Y} = c \quad \text{(constant almost surely)}$$