Resources

A college chemistry department conducts an educational study comparing two laboratory teaching formats on final exam performance. Group A has nA=16n_A = 16 students and achieves an average exam score advantage of xˉA−xˉ0=7.5\bar{x}_A - \bar{x}_0 = 7.5 points with p=0.082p = 0.082. Group B represents a massive statewide online rollout with nB=64,000n_B = 64{,}000 students that achieves an average exam score advantage of xˉB−xˉ0=0.3\bar{x}_B - \bar{x}_0 = 0.3 points with p<0.0001p < 0.0001. A student asserts: 'Group B provides undeniable scientific proof of a transformative, superior teaching format because its pp-value is vastly smaller than that of Group A, whereas Group A failed to achieve anything meaningful.' (a) Explain the conceptual flaw in equating a smaller pp-value with a larger or more meaningful real-world effect. (b) Calculate the standard error of the mean difference for Group B assuming σ=15.0\sigma = 15.0 points to demonstrate why its pp-value is extraordinarily small. (c) Classify both groups in terms of statistical significance (at α=0.05\alpha = 0.05) versus practical significance, assuming an exam advantage of ≥5.0\ge 5.0 points is required for practical educational impact.

Answer and steps

Final answer

(a) A pp-value measures the probability of observing data at least as extreme under the null hypothesis, which depends heavily on sample size nn; it does NOT measure the magnitude or practical importance of an effect. A tiny effect can yield an infinitesimal pp-value if nn is massive, while a large effect can fail to reach statistical significance if nn is small (low power). (b) For Group B, SE=σnB=15.064,000=15.0252.98≈0.0593SE = \frac{\sigma}{\sqrt{n_B}} = \frac{15.0}{\sqrt{64{,}000}} = \frac{15.0}{252.98} \approx 0.0593 points; with xˉB−xˉ0=0.3\bar{x}_B - \bar{x}_0 = 0.3, z=0.30.0593≈5.06z = \frac{0.3}{0.0593} \approx 5.06, yielding p<0.0001p < 0.0001. (c) Group A has practical significance (7.5≥5.07.5 \ge 5.0 points) but lacks statistical significance at α=0.05\alpha = 0.05 (p=0.082>0.05p = 0.082 > 0.05, underpowered sample); Group B has statistical significance (p<0.0001<0.05p < 0.0001 < 0.05) but completely lacks practical significance (0.3≪5.00.3 \ll 5.0 points).

Steps

Part (a): Conceptual Flaw in Equating pp-Value with Effect Size

A pp-value is defined as the probability, assuming the null hypothesis H0H_0 is true, of observing a test statistic at least as extreme as the one calculated from the sample.

The test statistic for a difference in means follows the general form: z=Observed Difference−0SE=xˉ−xˉ0σ/nz = \frac{\text{Observed Difference} - 0}{SE} = \frac{\bar{x} - \bar{x}_0}{\sigma / \sqrt{n}}

Because sample size nn appears in the denominator of the standard error (SE=σ/nSE = \sigma / \sqrt{n}), the test statistic scales directly with n\sqrt{n}:

  • Massive Sample Size (n→∞n \to \infty): The standard error shrinks toward zero. Consequently, even an infinitesimal, trivial difference in the real world will produce an enormous test statistic and an extremely small pp-value.
  • Small Sample Size: The standard error is large, leading to low statistical power. A large, meaningful educational gain may fail to reach statistical significance simply because the sample is too small to rule out random chance.

Therefore, a pp-value reflects evidence against the null hypothesis given the sample size, not the magnitude, importance, or practical utility of the treatment.


Part (b): Standard Error and Test Statistic for Group B

Given:

  • Sample size: nB=64,000n_B = 64{,}000
  • Population standard deviation: σ=15.0\sigma = 15.0 points
  • Observed advantage: xˉB−xˉ0=0.3\bar{x}_B - \bar{x}_0 = 0.3 points
  1. Calculate the standard error (SESE): SE=σnB=15.064,000=15.0252.982≈0.0593 pointsSE = \frac{\sigma}{\sqrt{n_B}} = \frac{15.0}{\sqrt{64{,}000}} = \frac{15.0}{252.982} \approx 0.0593 \text{ points}
  2. Compute the test statistic (zz): z=xˉB−xˉ0SE=0.30.0593≈5.06z = \frac{\bar{x}_B - \bar{x}_0}{SE} = \frac{0.3}{0.0593} \approx 5.06

Because an observation 5.065.06 standard errors above the mean is exceptionally rare under a standard normal distribution (tail probability well below 0.00010.0001), the pp-value is p<0.0001p < 0.0001. This shows that the tiny pp-value is entirely an artifact of the enormous sample size reducing SESE to under 0.060.06 points, rather than a large exam score improvement.


Part (c): Statistical vs. Practical Significance

Using the significance threshold α=0.05\alpha = 0.05 and the practical threshold of an exam score advantage ≥5.0\ge 5.0 points:

GroupSample Size (nn)Observed Advantagepp-valueStatistical Significance (α=0.05\alpha = 0.05)Practical Significance (≥5.0\ge 5.0 pts)
Group A16167.57.5 points0.0820.082No (p=0.082>0.05p = 0.082 > 0.05)Yes (7.5≥5.07.5 \ge 5.0 points)
Group B64,00064{,}0000.30.3 points<0.0001< 0.0001Yes (p<0.0001≤0.05p < 0.0001 \le 0.05)No (0.3≪5.00.3 \ll 5.0 points)
  • Group A demonstrates practical significance without statistical significance: an advantage of 7.57.5 points is educationally meaningful, but the small sample size (n=16n = 16) yields insufficient statistical power to reject H0H_0 at the 5%5\% level.
  • Group B demonstrates statistical significance without practical significance: an advantage of 0.30.3 points is virtually imperceptible on an exam, but the vast sample size makes this tiny shift statistically detectable.

Typical errors

OverviewACEpath Teacher's Diagnosis

01 The student conflates statistical significance with practical significance, concluding that Group B has practical importance while Group A does not.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    A smaller p-value proves a stronger effect because lower probability of chance means the educational impact is more real and substantial.

    Correct

    A pp-value indicates how incompatible the observed sample data are with the null hypothesis given sample size nn; it does not measure the magnitude or real-world importance of the effect. With an enormous sample size, even a trivial effect yields a tiny pp-value.

    You affirmed the student's assertion instead of identifying the flaw. A pp-value depends directly on sample size nn through the standard error; a microscopic pp-value only indicates that a nonzero difference is unlikely to be pure random noise, not that the difference is large or educationally meaningful.

  2. Step 2

    SEB=15.064000=15.0252.98≈0.0593\text{SE}_B = \frac{15.0}{\sqrt{64000}} = \frac{15.0}{252.98} \approx 0.0593

  3. Step 3

    Group A: Not statistically significant because p=0.082>0.05p = 0.082 > 0.05; not practically significant because it failed to achieve significance. ⟹ Group B: Statistically significant because p<0.0001<0.05p < 0.0001 < 0.05; practically significant because ithe extremelly small p-value establishes real-world superiority.

    Correct

    Group A: Statistically insignificant (p=0.082>0.05p = 0.082 > 0.05), but practically significant because 7.5≥5.07.5 \ge 5.0 points. ⟹ Group B: Statistically significant (p<0.0001<0.05p < 0.0001 < 0.05), but not practically significant because 0.3<5.00.3 < 5.0 points.

    You accurately checked the pp-values against α=0.05\alpha = 0.05 for statistical significance, but you evaluated practical significance using the pp-value instead of the observed point gain. Group A gained 7.57.5 points (exceeding the 5.05.0-point benchmark), while Group B gained only 0.30.3 points (far below 5.05.0 points).

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly identified the standard error formula and accurately computed SEB≈0.0593\text{SE}_B \approx 0.0593.

You correctly applied the significance threshold α=0.05\alpha = 0.05 to classify both pp-values for statistical significance.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Conflating statistical significance with effect size/practical significance

You treated practical significance as a downstream consequence of achieving a small pp-value rather than an independent check of the observed effect size against the 5.05.0-point threshold.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You viewed the pp-value as an indicator of effect magnitude and real-world value, missing that pp-values shrink toward zero when sample sizes become massive regardless of how small the underlying effect is.

REASONING PATH
Your ACEpath teacher’s suggested approach
Define what a $p$-value actually measures
  • A pp-value quantifies evidence against H0H_0 conditional on sample size; it does not measure effect size.

  • Always separate the test of chance (pp-value vs. α\alpha) from the evaluation of real-world impact (point estimate vs. practical threshold).

Recognize the sample size amplifier
  • Because SE=σn\text{SE} = \frac{\sigma}{\sqrt{n}}, very large nn produces tiny standard errors, making even trivial differences yield p<0.0001p < 0.0001.

  • Whenever you see a massive sample size, check the estimated difference xˉ−xˉ0\bar{x} - \bar{x}_0 directly before deciding if the result matters in practice.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Compare Group A's observed gain of 7.57.5 points to the required impact threshold of 5.05.0 points: since 7.5≥5.07.5 \ge 5.0, Group A has practical significance.

  2. Compare Group B's observed gain of 0.30.3 points to the threshold: since 0.3<5.00.3 < 5.0, Group B lacks practical significance.

  3. Combine both criteria: Group A has practical significance without statistical significance (p=0.082>0.05p = 0.082 > 0.05 due to small nn), whereas Group B has statistical significance without practical significance (p<0.0001p < 0.0001 driven by n=64,000n = 64{,}000).

Final answer

Group A is practically significant but not statistically significant; Group B is statistically significant but not practically significant.

NEXT STEPS
Final Takeaways from Your Teacher
  • Review the distinction between test statistics (zz or tt) and effect sizes (such as Cohen's dd or mean differences).

  • Practice identifying 'overpowered' studies where massive sample sizes detect negligible differences and 'underpowered' studies where large differences fail to reach α=0.05\alpha = 0.05.

OverviewACEpath Teacher's Diagnosis

02 The student forgets to take the square root of the sample size in the standard error formula, dividing σ by n directly.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    A small p-value only indicates strong evidence that the effect is non-zero, not that the effect is large. Large sample sizes reduce standard error, producing small p-values even for tiny effect sizes.

  2. Step 2

    SEB=15.064000=0.000234 pointsSE_B = \frac{15.0}{64000} = 0.000234\text{ points}

    Correct

    SEB=σnB=15.064,000=15.0252.98≈0.0593 pointsSE_B = \frac{\sigma}{\sqrt{n_B}} = \frac{15.0}{\sqrt{64{,}000}} = \frac{15.0}{252.98} \approx 0.0593\text{ points}

    You divided σ\sigma directly by nn rather than by n\sqrt{n}. The standard error of the sample mean is given by SE=σnSE = \frac{\sigma}{\sqrt{n}}.

  3. Step 3

    Group A: Practically significant because 7.5≥5.07.5 \ge 5.0, but not statistically significant because p=0.082>0.05p = 0.082 > 0.05. ⟹ Group B: Statistically significant because p<0.0001≤0.05p < 0.0001 \le 0.05, but not practically significant because 0.3<5.00.3 < 5.0.

WHAT YOU DID WELL
Compliments from Your Teacher

You accurately articulated why a small pp-value reflects evidence against the null rather than the magnitude of practical impact.

You correctly identified that Group A has practical significance but lacks statistical significance due to small sample size.

You correctly identified that Group B has statistical significance but lacks practical significance.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Omitted square root in standard error formula

In part (b), you divided the population standard deviation directly by nn (64,00064{,}000) instead of n\sqrt{n} (64,000\sqrt{64{,}000}).

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You confused the formula for standard error of the mean, SE=σnSE = \frac{\sigma}{\sqrt{n}}, with dividing variability directly by sample size, forgetting that sample variance scales as σ2n\frac{\sigma^2}{n} while standard deviation scales as σn\frac{\sigma}{\sqrt{n}}.

REASONING PATH
Your ACEpath teacher’s suggested approach
Square root of sample size
  • The standard error of a sample mean is SE=σnSE = \frac{\sigma}{\sqrt{n}}, not σn\frac{\sigma}{n}.

  • Always verify that the radical n\sqrt{n} is written in the denominator before dividing.

Order of magnitude check
  • Since 64,000≈253\sqrt{64{,}000} \approx 253, the denominator should be roughly 250250, yielding SE≈0.06SE \approx 0.06, not 0.00020.0002.

  • Perform a quick mental estimate of n\sqrt{n} to catch omitted square roots.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Substitute σ=15.0\sigma = 15.0 and nB=64,000n_B = 64{,}000: SEB=15.064,000SE_B = \frac{15.0}{\sqrt{64{,}000}}

  2. Compute the square root of the sample size: 64,000≈252.982\sqrt{64{,}000} \approx 252.982

  3. Divide to find the standard error: SEB=15.0252.982≈0.0593 pointsSE_B = \frac{15.0}{252.982} \approx 0.0593\text{ points}

  4. Calculate the test statistic: z=0.3−00.0593≈5.06z = \frac{0.3 - 0}{0.0593} \approx 5.06, which confirms p<0.0001p < 0.0001 due to extreme sample size rather than a large gain.

Final answer

SEB≈0.0593 pointsSE_B \approx 0.0593\text{ points} (yielding z≈5.06z \approx 5.06 and p<0.0001p < 0.0001)

NEXT STEPS
Final Takeaways from Your Teacher
  • Practice writing down SE=σnSE = \frac{\sigma}{\sqrt{n}} explicitly with the square root symbol before substituting numbers.

  • Check your computed SESE by multiplying it back: SE×nSE \times \sqrt{n} must equal σ\sigma (0.0593×253≈15.00.0593 \times 253 \approx 15.0).

  • Solve the follow-up drills below to solidify computing SESE and interpreting practical vs. statistical significance.

OverviewACEpath Teacher's Diagnosis

03 The student uses the wrong threshold direction for statistical significance, incorrectly believing that p > α indicates statistical significance.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    A small p-value simply measures evidence against the null hypothesis rather than the magnitude of effect; an enormous sample size can make trivial differences statistically detectable.

  2. Step 2

    SEB=15.0/64000=15.0/252.982=0.0593SE_B = 15.0 / \sqrt{64000} = 15.0 / 252.982 = 0.0593

  3. Step 3

    Group A: Statistically significant because p=0.082>0.05p = 0.082 > 0.05; practically significant because 7.5≥5.07.5 \ge 5.0.

    Correct

    Group A: Not statistically significant because p=0.082>0.05p = 0.082 > 0.05; practically significant because 7.5≥5.07.5 \ge 5.0.

    You judged practical significance correctly, but statistical significance requires p≤αp \le \alpha. Because p=0.082>0.05p = 0.082 > 0.05, Group A is not statistically significant at the 5%5\% level.

  4. Step 4

    Group B: Not statistically significant because p<0.0001<0.05p < 0.0001 < 0.05; not practically significant because 0.3<5.00.3 < 5.0.

    Correct

    Group B: Statistically significant because p<0.0001≤0.05p < 0.0001 \le 0.05; not practically significant because 0.3<5.00.3 < 5.0.

    You correctly determined that an advantage of 0.30.3 points lacks practical significance, but because p<0.0001<0.05p < 0.0001 < 0.05, Group B is statistically significant.

WHAT YOU DID WELL
Compliments from Your Teacher

You clearly explained how an enormous sample size can make trivial differences statistically detectable.

You calculated the standard error for Group B with complete numerical accuracy.

You correctly applied the practical benchmark threshold to both groups.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Reversed significance decision rule

no

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You reversed the direction of the decision rule for hypothesis testing: a result is statistically significant when the pp-value is less than or equal to the significance level α\alpha (p≤αp \le \alpha), not when p>αp > \alpha.

REASONING PATH
Your ACEpath teacher’s suggested approach
Decision Rule for Statistical Significance
  • If p≤αp \le \alpha, reject H0H_0 (statistically significant). If p>αp > \alpha, fail to reject H0H_0 (not statistically significant).

  • Always compare pp to α\alpha by asking whether the observed data is sufficiently improbable under the null hypothesis (p≤αp \le \alpha) to claim significance.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Compare each pp-value to the threshold α=0.05\alpha = 0.05: reject H0H_0 if p≤0.05p \le 0.05.

  2. For Group A: p=0.082>0.05p = 0.082 > 0.05, so fail to reject H0H_0 (not statistically significant). Because 7.5≥5.07.5 \ge 5.0, it is practically significant.

  3. For Group B: p<0.0001≤0.05p < 0.0001 \le 0.05, so reject H0H_0 (statistically significant). Because 0.3<5.00.3 < 5.0, it is not practically significant.

Final answer

Group A has practical significance but not statistical significance (p=0.082>0.05p = 0.082 > 0.05). Group B has statistical significance (p<0.0001≤0.05p < 0.0001 \le 0.05) but not practical significance (0.3<5.00.3 < 5.0).

NEXT STEPS
Final Takeaways from Your Teacher
  • Memorize the rhyme 'If pp is low, the null must go' to remember that small pp-values correspond to statistical significance.

  • Practice separating test decisions (p≤αp \le \alpha) from effect size evaluations (∣xˉ−μ0∣≥threshold|\bar{x} - \mu_0| \ge \text{threshold}) in two distinct verification steps.

Lock it in

Similar question

01

A software company tests a new algorithm on n=1,000,000n = 1{,}000{,}000 user sessions. Page load time improves by 1.2 ms1.2\text{ ms} with p=0.0002p = 0.0002. The product team requires at least a 50 ms50\text{ ms} reduction for users to notice. Classify this outcome at α=0.05\alpha = 0.05 in terms of statistical and practical significance.

Answer

Statistical significance: Yes, because p=0.0002<0.05p = 0.0002 < 0.05. Practical significance: No, because 1.2 ms<50 ms1.2\text{ ms} < 50\text{ ms}. Key takeaway: The massive sample size detects a sub-perceptual difference.

02

An athlete trains with a new sprinting technique (n=12n = 12 trials) and shaves 0.42 seconds0.42\text{ seconds} off their 100 m100\text{ m} time (p=0.071p = 0.071, σ=0.8 s\sigma = 0.8\text{ s}). In elite sprinting, an improvement of ≥0.10 seconds\ge 0.10\text{ seconds} is game-changing. Classify the result at α=0.05\alpha = 0.05 and explain why the pp-value was not significant despite the huge gain.

Answer

Classification: Practically significant (0.42 s≥0.10 s0.42\text{ s} \ge 0.10\text{ s}), but not statistically significant at α=0.05\alpha = 0.05 (p=0.071>0.05p = 0.071 > 0.05). Reason: The sample size (n=12n = 12) is too small, yielding a large standard error (SE=0.812≈0.231 s\text{SE} = \frac{0.8}{\sqrt{12}} \approx 0.231\text{ s}) and low statistical power.

03

Study X has n=25n = 25 and observed difference dˉ=8.0\bar{d} = 8.0 with SE=4.5\text{SE} = 4.5. Study Y has n=10,000n = 10{,}000 and observed difference dˉ=0.5\bar{d} = 0.5 with SE=0.1\text{SE} = 0.1. Both assume H0:μd=0H_0: \mu_d = 0. (a) Calculate the zz-statistic for both studies. (b) Determine which study has stronger statistical evidence against H0H_0 and which study shows a larger real-world effect.

Answer

(a) Study X: z=8.04.5≈1.78z = \frac{8.0}{4.5} \approx 1.78 (p≈0.075p \approx 0.075). Study Y: z=0.50.1=5.0z = \frac{0.5}{0.1} = 5.0 (p<0.0001p < 0.0001). (b) Study Y provides much stronger statistical evidence against H0H_0 (z=5.0z = 5.0 vs. 1.781.78), but Study X demonstrates a vastly larger real-world effect size (8.08.0 units vs. 0.50.5 units).

Loading workspace…