Resources

A school district implements a new computerized algebra diagnostic program. A random sample of n=25,000n = 25{,}000 high school freshmen is tested before and after the semester. The mean score increase is xˉd=0.45\bar{x}_d = 0.45 points on a 100100-point scale, with a sample standard deviation of differences sd=12.50s_d = 12.50 points. (a) Conduct a paired tt-test at the α=0.01\alpha = 0.01 level to determine whether the score gain is statistically significant, reporting the test statistic tt and state whether to reject or fail to reject H0:μd=0H_0: \mu_d = 0 in favor of Ha:μd>0H_a: \mu_d > 0. (b) Calculate Cohen's dd effect size for this score gain. (c) Explain whether the program demonstrates practical significance if district policy requires a meaningful improvement to have at least a small-to-moderate effect size (d≥0.20d \ge 0.20) or an average gain of at least 3.03.0 points.

Answer and steps

Final answer

(a) t=0.45−012.5025,000=0.450.07906≈5.69t = \frac{0.45 - 0}{\frac{12.50}{\sqrt{25{,}000}}} = \frac{0.45}{0.07906} \approx 5.69; with p<0.0001<0.01p < 0.0001 < 0.01, reject H0H_0; the score gain is statistically significant. (b) Cohen's d=xˉdsd=0.4512.50=0.0360d = \frac{\bar{x}_d}{s_d} = \frac{0.45}{12.50} = 0.0360. (c) The program does not demonstrate practical significance because d=0.0360<0.20d = 0.0360 < 0.20 and the mean gain of 0.450.45 points is far below the district's minimum meaningful threshold of 3.03.0 points; the statistical significance is merely an artifact of the very large sample size (n=25,000n = 25{,}000).

Steps

Part (a): Paired tt-Test for Statistical Significance

  1. State the Hypotheses:

    • Null hypothesis: H0:μd=0H_0: \mu_d = 0 (the mean score difference is zero).
    • Alternative hypothesis: Ha:μd>0H_a: \mu_d > 0 (the mean score difference is greater than zero).
  2. Compute the Standard Error (SESE): The standard error of the mean difference is given by: SE=sdn=12.5025,000=12.50158.1139≈0.07906SE = \frac{s_d}{\sqrt{n}} = \frac{12.50}{\sqrt{25{,}000}} = \frac{12.50}{158.1139} \approx 0.07906
  3. Compute the Test Statistic tt: t=xˉd−μ0SE=0.45−00.07906≈5.69t = \frac{\bar{x}_d - \mu_0}{SE} = \frac{0.45 - 0}{0.07906} \approx 5.69
  4. Make the Decision:

    • For degrees of freedom df=n−1=24,999df = n - 1 = 24{,}999, the critical value for a right-tailed test at α=0.01\alpha = 0.01 is approximately z0.01≈2.326z_{0.01} \approx 2.326.
    • Since t≈5.69>2.326t \approx 5.69 > 2.326 (corresponding to a pp-value p<0.0001<0.01p < 0.0001 < 0.01), we reject H0H_0.
    • Conclusion: The score gain is statistically significant at the α=0.01\alpha = 0.01 level.

Part (b): Cohen's dd Effect Size

Cohen's dd quantifies the magnitude of the difference independent of sample size: d=xˉdsd=0.4512.50=0.0360d = \frac{\bar{x}_d}{s_d} = \frac{0.45}{12.50} = 0.0360


Part (c): Practical Significance vs. Statistical Significance

  • Comparison to District Criteria:

    • By effect size: d=0.0360d = 0.0360, which is far below the benchmark for a small effect (d=0.20d = 0.20).
    • By absolute score gain: xˉd=0.45\bar{x}_d = 0.45 points on a 100100-point scale, which is well below the district's minimum meaningful threshold of 3.03.0 points.
  • Explanation: The program does not demonstrate practical significance. Although the result is highly statistically significant, this statistical significance is solely an artifact of the exceptionally large sample size (n=25,000n = 25{,}000). With such a large nn, the standard error becomes tiny (SE≈0.079SE \approx 0.079), enabling the test to detect even trivial, inconsequential differences as statistically significant.

Typical errors

OverviewACEpath Teacher's Diagnosis

01 The student forgets to divide the standard deviation by the square root of n when computing the test statistic, confusing the individual difference distribution with the sampling distribution.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    t=xˉd−0sd  ⟹  t=0.4512.50=0.036t = \frac{\bar{x}_d - 0}{s_d} \implies t = \frac{0.45}{12.50} = 0.036

    Correct

    t=xˉd−μ0sdn=0.45−012.5025,000≈5.69t = \frac{\bar{x}_d - \mu_0}{\frac{s_d}{\sqrt{n}}} = \frac{0.45 - 0}{\frac{12.50}{\sqrt{25{,}000}}} \approx 5.69

    You divided the mean difference directly by the sample standard deviation sds_d instead of the standard error SE=sdnSE = \frac{s_d}{\sqrt{n}}. A test statistic measures how many standard errors the sample mean is from the hypothesized mean.

  2. Step 2

    Critical value for α=0.01\alpha = 0.01 is 2.326  ⟹  2.326 \implies Since 0.036<2.3260.036 < 2.326, fail to reject H0H_0

    Correct

    Since t≈5.69>2.326t \approx 5.69 > 2.326, reject H0H_0 at the α=0.01\alpha = 0.01 level.

    Your comparison logic and critical value of 2.3262.326 were sound, but because your calculated tt-statistic was incorrect from Step 1, your decision to fail to reject H0H_0 was a downstream error.

  3. Step 3

    d=xˉdsd=0.4512.50=0.036d = \frac{\bar{x}_d}{s_d} = \frac{0.45}{12.50} = 0.036

  4. Step 4

    The program does not show practical significance because d=0.036<0.20d = 0.036 < 0.20 and the gain of 0.450.45 is less than 3.03.0 points.

WHAT YOU DID WELL
Compliments from Your Teacher

You identified the correct one-tailed critical value of 2.3262.326 for α=0.01\alpha = 0.01.

You correctly applied the formula for Cohen's dd effect size.

You accurately evaluated practical significance against the stated district standards.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Sampling Distribution Error

You used the formula for Cohen's dd in place of the formula for the tt test statistic.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You confused the variability of individual differences (sds_d) with the variability of the sample mean difference (SE=sd/nSE = s_d / \sqrt{n}). Test statistics evaluate sampling distributions, which shrink by a factor of n\sqrt{n} as sample size increases.

REASONING PATH
Your ACEpath teacher’s suggested approach
Standard Error vs. Standard Deviation
  • For hypothesis testing on sample means, the denominator is the standard error SE=snSE = \frac{s}{\sqrt{n}}, not the sample standard deviation ss.

  • Always write down SE=sdnSE = \frac{s_d}{\sqrt{n}} before assembling the test statistic.

Test Statistic vs. Effect Size
  • Cohen's d=xˉdsdd = \frac{\bar{x}_d}{s_d} depends only on the distribution spread, whereas t=xˉdsd/n=dnt = \frac{\bar{x}_d}{s_d / \sqrt{n}} = d\sqrt{n} scales directly with sample size.

  • Check whether your calculated test statistic incorporates the sample size nn.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Compute the standard error of the mean difference: SE=sdn=12.5025,000=12.50158.1139≈0.07906SE = \frac{s_d}{\sqrt{n}} = \frac{12.50}{\sqrt{25{,}000}} = \frac{12.50}{158.1139} \approx 0.07906.

  2. Compute the test statistic: t=xˉd−0SE=0.450.07906≈5.69t = \frac{\bar{x}_d - 0}{SE} = \frac{0.45}{0.07906} \approx 5.69.

  3. Compare tt to the critical value: since t=5.69>2.326t = 5.69 > 2.326 (with p<0.0001p < 0.0001), reject H0H_0 in favor of HaH_a.

Final answer

(a) t≈5.69t \approx 5.69, reject H0H_0; the score gain is statistically significant. (b) Cohen's d=0.036d = 0.036. (c) The program does not demonstrate practical significance because d=0.036<0.20d = 0.036 < 0.20 and the mean gain of 0.450.45 points is below the minimum threshold of 3.03.0 points.

NEXT STEPS
Final Takeaways from Your Teacher
  • Practice identifying whether a question asks for a test statistic (tt or zz, which divides by SESE) or an effect size (Cohen's dd, which divides by ss).

  • Solve problems involving very large sample sizes to observe how small effect sizes can produce large, statistically significant tt-values.

OverviewACEpath Teacher's Diagnosis

02 The student miscalculates Cohen's d by dividing by the standard error instead of the sample standard deviation, confusing effect size with the test statistic.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    SE=12.5025000=0.0791  ⟹  t=0.450.0791=5.69  ⟹  t=5.69>2.326SE = \frac{12.50}{\sqrt{25000}} = 0.0791 \implies t = \frac{0.45}{0.0791} = 5.69 \implies t = 5.69 > 2.326, so reject H0H_0

  2. Step 2

    d=xˉdSE=0.450.0791=5.69d = \frac{\bar{x}_d}{SE} = \frac{0.45}{0.0791} = 5.69

    Correct

    d=xˉdsd=0.4512.50=0.0360d = \frac{\bar{x}_d}{s_d} = \frac{0.45}{12.50} = 0.0360

    Cohen's dd standardizes the mean difference using the sample standard deviation of differences (sds_d), not the standard error (SESE). Dividing by SE=sdnSE = \frac{s_d}{\sqrt{n}} simply reproduces the test statistic tt rather than calculating the sample-size-independent effect size.

  3. Step 3

    The program demonstrates practical significance because d=5.69d = 5.69 is much greater than 0.200.20.

    Correct

    The program does not demonstrate practical significance because d=0.0360<0.20d = 0.0360 < 0.20 and the average gain of 0.450.45 points is far below the 3.03.0-point policy threshold.

    Because your calculated dd was inflated from using SESE, you concluded that the program had practical significance. Evaluating the true effect size (d=0.0360d = 0.0360) and comparing the mean gain of 0.450.45 points against the required 3.03.0 points reveals that the gain is practically negligible despite being statistically significant.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly calculated the standard error of the mean differences.

You computed the correct paired tt-statistic and properly compared it to the critical value to reject the null hypothesis.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Confusing Standard Error with Standard Deviation in Effect Size

You substituted SESE into the denominator of Cohen's dd, repeating the formula for the test statistic tt.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

Effect size is designed to measure the real-world magnitude of an effect independently of sample size, which requires dividing by the standard deviation of individual observations (sds_d). Dividing by the standard error (SE=sd/nSE = s_d / \sqrt{n}) re-introduces n\sqrt{n} into the calculation, turning the effect size into a sample-size-dependent test statistic.

REASONING PATH
Your ACEpath teacher’s suggested approach
Denominator of Cohen's d
  • Cohen's d=xˉdsdd = \frac{\bar{x}_d}{s_d}, dividing the mean change by the standard deviation of the scores, never by the standard error.

  • Always check that the denominator of an effect size contains standard deviation (ss), not standard error (SE=s/nSE = s / \sqrt{n}).

Large Sample Fallacy
  • With very large sample sizes (such as n=25,000n = 25{,}000), tiny and practically meaningless differences can produce huge tt-values and extreme statistical significance.

  • Separate statistical significance (is p<αp < \alpha?) from practical significance (is d≥0.20d \ge 0.20 or is the raw gain meaningful?).

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Calculate Cohen's dd using the sample standard deviation: d=xˉdsd=0.4512.50=0.0360d = \frac{\bar{x}_d}{s_d} = \frac{0.45}{12.50} = 0.0360.

  2. Compare the calculated effect size to the district benchmark: d=0.0360<0.20d = 0.0360 < 0.20, which represents a trivial, negligible effect.

  3. Compare the mean score increase to the district point threshold: xˉd=0.45<3.0\bar{x}_d = 0.45 < 3.0 points.

  4. Conclude that although the score gain is statistically significant due to the massive sample size (n=25,000n = 25{,}000), it does not demonstrate practical significance.

Final answer

(b) d=0.0360d = 0.0360; (c) The program does not demonstrate practical significance because d=0.0360<0.20d = 0.0360 < 0.20 and the average gain of 0.450.45 points is far below the required 3.03.0-point threshold.

NEXT STEPS
Final Takeaways from Your Teacher
  • Review the distinction between t=xˉdsd/nt = \frac{\bar{x}_d}{s_d / \sqrt{n}} and d=xˉdsdd = \frac{\bar{x}_d}{s_d}.

  • Practice interpreting practical significance benchmarks alongside pp-values when working with large datasets.

OverviewACEpath Teacher's Diagnosis

03 The student equates statistical significance with practical significance, concluding the program is practically meaningful solely because the null hypothesis was rejected.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    SE=12.5025,000=0.0791  ⟹  t=0.450.0791=5.69  ⟹  Since t=5.69>2.326, reject H0SE = \frac{12.50}{\sqrt{25{,}000}} = 0.0791 \implies t = \frac{0.45}{0.0791} = 5.69 \implies \text{Since } t = 5.69 > 2.326\text{, reject } H_0

  2. Step 2

    d=0.4512.50=0.036d = \frac{0.45}{12.50} = 0.036

  3. Step 3

    Rejecting H0 proves the improvement is real and statistically significant, so the program demonstrates practical significance.\text{Rejecting } H_0 \text{ proves the improvement is real and statistically significant, so the program demonstrates practical significance.}

    Correct

    The program does not demonstrate practical significance because d=0.036<0.20d = 0.036 < 0.20 and the mean gain of 0.450.45 points is far below the 3.03.0-point policy benchmark. The statistical significance is merely driven by the massive sample size (n=25,000n = 25{,}000).

    You equated statistical significance with practical significance. A tiny pp-value only indicates that the gain is unlikely to be zero by chance, not that the magnitude is meaningful in the real world.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly calculated the standard error and test statistic for the paired tt-test.

You properly evaluated the rejection rule against the critical value to reject the null hypothesis.

You accurately calculated Cohen's dd effect size.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Equating Statistical and Practical Significance

No slip was made in your numerical calculations.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You conflated rejecting the null hypothesis with real-world meaningfulness. With very large sample sizes (n=25,000n = 25{,}000), standard error shrinks close to zero, causing even negligible differences (like a 0.450.45-point shift on a 100100-point scale) to achieve extreme statistical significance (p<0.0001p < 0.0001). Practical significance must be judged by effect size (dd) and domain benchmarks, independent of sample size.

REASONING PATH
Your ACEpath teacher’s suggested approach
Statistical vs. Practical Significance
  • Statistical significance (p<αp < \alpha) only means xˉd≠0\bar{x}_d \neq 0 is unlikely due to chance; practical significance evaluates whether the magnitude of the difference matters in practice.

  • Compare the observed difference and Cohen's dd against real-world policy thresholds rather than the hypothesis test decision.

Sample Size Inflation of Test Statistics
  • Because t=xˉd−0sd/n=d⋅nt = \frac{\bar{x}_d - 0}{s_d / \sqrt{n}} = d \cdot \sqrt{n}, an enormous nn guarantees a large tt even when effect size dd is tiny.

  • When sample sizes are very large, always check effect size (dd) and raw units to evaluate whether the effect has any real-world value.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Check the calculated effect size: d=0.036d = 0.036, which is far below the district's minimum small-to-moderate threshold of d=0.20d = 0.20.

  2. Check the raw mean gain: xˉd=0.45\bar{x}_d = 0.45 points on a 100100-point scale, which is well below the required gain of at least 3.03.0 points.

  3. Acknowledge that rejecting H0H_0 occurred solely because n=25,000n = 25{,}000 reduces the standard error to 0.07910.0791, magnifying a trivial difference into a large tt-statistic.

Final answer

The program does not demonstrate practical significance because both the effect size (d=0.036<0.20d = 0.036 < 0.20) and the raw score gain (0.45<3.00.45 < 3.0 points) fail to meet the district's thresholds. Statistical significance was achieved only because of the huge sample size (n=25,000n = 25{,}000).

NEXT STEPS
Final Takeaways from Your Teacher
  • Always separate the question 'Is the effect non-zero?' (statistical significance) from 'Does the effect matter?' (practical significance).

  • When interpreting statistical significance with large nn, inspect Cohen's dd and contextual benchmark units before drawing practical conclusions.

Lock it in

Similar question

01

A reading intervention is evaluated on n=400n = 400 students. The mean difference is xˉd=1.6\bar{x}_d = 1.6 points with sd=8.0s_d = 8.0 points. Calculate (a) the standard error of the mean difference, (b) the paired tt-statistic for H0:μd=0H_0: \mu_d = 0, and (c) Cohen's dd.

Answer

(a) SE=sdn=8.0400=8.020=0.40SE = \frac{s_d}{\sqrt{n}} = \frac{8.0}{\sqrt{400}} = \frac{8.0}{20} = 0.40. (b) t=1.6−00.40=4.00t = \frac{1.6 - 0}{0.40} = 4.00. (c) d=1.68.0=0.20d = \frac{1.6}{8.0} = 0.20.

02

A statewide software trial of n=100,000n = 100{,}000 users reports an average increase of xˉd=0.12\bar{x}_d = 0.12 minutes on task with sd=15.0s_d = 15.0 minutes. (a) Compute the tt-statistic testing H0:μd=0H_0: \mu_d = 0. (b) Compute Cohen's dd. (c) State whether the result is statistically significant at α=0.05\alpha = 0.05 (critical value z=1.645z = 1.645) and whether it shows practical significance under Cohen's benchmark (d≥0.20d \ge 0.20).

Answer

(a) SE=15.0100,000≈0.04743  ⟹  t=0.120.04743≈2.53SE = \frac{15.0}{\sqrt{100{,}000}} \approx 0.04743 \implies t = \frac{0.12}{0.04743} \approx 2.53. (b) d=0.1215.0=0.008d = \frac{0.12}{15.0} = 0.008. (c) Statistically significant (t=2.53>1.645t = 2.53 > 1.645, reject H0H_0), but not practically significant (d=0.008≪0.20d = 0.008 \ll 0.20).

03

A company wants to detect a tiny effect size of Cohen's d=0.05d = 0.05 at the α=0.05\alpha = 0.05 significance level for a one-tailed test (requiring t≥1.645t \ge 1.645). Using the relationship t=dnt = d\sqrt{n}, what minimum sample size nn is required for the test statistic to reach statistical significance?

Answer

Using t=dn≥1.645t = d\sqrt{n} \ge 1.645: 0.05n≥1.645  ⟹  n≥1.6450.05=32.9  ⟹  n≥32.92≈1,082.410.05\sqrt{n} \ge 1.645 \implies \sqrt{n} \ge \frac{1.645}{0.05} = 32.9 \implies n \ge 32.9^2 \approx 1{,}082.41. Thus, a minimum sample size of n=1,083n = 1{,}083 is required.

Loading workspace…