06: Goodness-of-Fit and Contingency Tables

From Continuous to Categorical: A Different Kind of Test

Everything so far has focused on continuous data (means, variances).

But many business questions involve categorical data:

• Does customer preference differ across regions?
• Is the industry distribution of firms uniform?
• Are two classification variables independent?

The chi-square (\(\chi^2\)) test family provides the tools for these questions.

The Chi-Square Distribution

If \(Z_1, Z_2, \ldots, Z_k\) are independent standard normal random variables, then:

\[X = \sum_{i=1}^k Z_i^2 \sim \chi^2_k\]

Key properties:

The parameter \(k\) (degrees of freedom) controls the distribution’s center and spread.

Degrees of Freedom: Intuition

Degrees of freedom (df) = number of values free to vary given constraints.

Example: If 5 category frequencies must sum to \(n = 100\):

• You can freely choose the first 4 frequencies
• The 5th is determined: \(f_5 = 100 - f_1 - f_2 - f_3 - f_4\)
• df = \(5 - 1 = 4\)

General formulas:

Chi-Square Goodness-of-Fit Test

Question: Does an observed frequency distribution match a hypothesized one?

\[\chi^2 = \sum_{i=1}^{k} \frac{(O_i - E_i)^2}{E_i}\]

where \(O_i\) = observed frequency, \(E_i\) = expected frequency under \(H_0\).

Mathematical essence: Each term \((O_i - E_i)/\sqrt{E_i}\) is approximately standard normal, so the sum of squares follows \(\chi^2\).

Critical condition: All \(E_i \geq 5\) (otherwise the normal approximation breaks down).

Six-Step Testing Procedure

Always verify: All \(E_i \geq 5\). If not, merge adjacent categories.

Case: Industry Distribution in the YRD

Question: Are the top 5 industries equally represented among YRD listed companies?

\(\chi^2 = 63.5\), \(p < 0.001\)

Conclusion: The distribution is far from uniform — Computer & Communication is significantly over-represented.

Dirty Work: The Art of Binning

Binning (how you define categories) directly affects results.

The problem: Different binning schemes → different p-values → potential manipulation.

Three golden rules:

Defense against binning hacking: Report results under multiple binning schemes (Multiverse Analysis).

Dirty Work: The Curse of Large N

With large samples, everything becomes statistically significant.

The paradox:

• \(\chi^2\) statistic scales linearly with \(n\)
• Even trivial deviations from the null become “significant”
• A perfectly functioning roulette wheel will “fail” the uniformity test with enough spins

Solution: Always report effect sizes

Contingency Tables and Independence

A contingency table cross-tabulates two categorical variables.

Expected frequency under independence:

\[E_{ij} = \frac{R_i \times C_j}{n}\]

where \(R_i\) = row total, \(C_j\) = column total, \(n\) = grand total.

Test statistic:

\[\chi^2 = \sum_i \sum_j \frac{(O_{ij} - E_{ij})^2}{E_{ij}}, \quad df = (r-1)(c-1)\]

Intuition: If the variables are independent, knowing the row shouldn’t help predict the column.

Case: Industry vs. Region in the YRD

Question: Is the industry distribution independent of province (Shanghai, Jiangsu, Zhejiang)?

Testing the top 4 industries across 3 provinces:

• \(\chi^2\) statistic computed from \(4 \times 3\) table
• \(df = (4-1)(3-1) = 6\)
• Result: Cannot reject independence (\(p > 0.05\))

Interpretation: The three YRD provinces have surprisingly homogeneous industry structures — the tech and manufacturing mix is similar across Shanghai, Jiangsu, and Zhejiang.

Cramér’s V confirms: the association strength is negligible.

Homogeneity vs. Independence: What’s the Difference?

They use the same test statistic but answer conceptually different questions.

McNemar Test: Paired Categorical Data

For before/after or matched-pair binary outcomes:

Only the discordant pairs (\(b\) and \(c\)) carry information:

\[\chi^2 = \frac{(b - c)^2}{b + c}\]

Use case: Did a training program change employees’ views? (Before vs. after survey)

Fisher’s Exact Test: When Chi-Square Fails

Problem: Chi-square requires \(E_i \geq 5\). With small samples, this fails.

Solution: Fisher’s exact test computes the exact probability using the hypergeometric distribution.

Case: Risk Control Model Comparison (Securities firm, 50 flagged transactions)

• Odds Ratio = 6.0
• Fisher exact \(p = 0.009\)
• Conclusion: Model A significantly outperforms Model B

Heuristic: Is the Lottery Random?

Test: Simulate 100 draws of China’s “Double Color Ball” (红球) lottery and test whether the 33 numbers appear with equal frequency.

• Under \(H_0\): each number has probability \(1/33\)
• Apply chi-square goodness-of-fit test
• Result: \(p = 0.265\) → Cannot reject randomness

Key insight: Even with a fair lottery, some numbers WILL appear more often than others due to sampling variability. The test tells us whether the deviation is beyond what chance alone would produce.

Heuristic: Are Zodiac Signs and Success Related?

Test: Among 100 wealthy individuals, test whether their zodiac sign distribution differs from population proportions (weighted by days per sign).

• \(H_0\): Zodiac distribution matches calendar proportions
• \(\chi^2\) test with 11 df
• Result: \(p = 0.536\) → No evidence of association

P-Hacking warning: If you tested 12 individual signs, one might appear “significant” at \(\alpha = 0.05\) by pure chance (\(12 \times 0.05 = 0.6\) expected false positives).

Chapter Summary

Always report Cramér’s V alongside \(\chi^2\) to guard against the curse of large \(N\).