Resources
A school district implements a new computerized algebra diagnostic program. A random sample of high school freshmen is tested before and after the semester. The mean score increase is points on a -point scale, with a sample standard deviation of differences points. (a) Conduct a paired -test at the level to determine whether the score gain is statistically significant, reporting the test statistic and state whether to reject or fail to reject in favor of . (b) Calculate Cohen's effect size for this score gain. (c) Explain whether the program demonstrates practical significance if district policy requires a meaningful improvement to have at least a small-to-moderate effect size () or an average gain of at least points.
Answer and steps
Final answer
(a) ; with , reject ; the score gain is statistically significant. (b) Cohen's . (c) The program does not demonstrate practical significance because and the mean gain of points is far below the district's minimum meaningful threshold of points; the statistical significance is merely an artifact of the very large sample size ().
Steps
Part (a): Paired -Test for Statistical Significance
-
State the Hypotheses:
- Null hypothesis: (the mean score difference is zero).
- Alternative hypothesis: (the mean score difference is greater than zero).
- Compute the Standard Error (): The standard error of the mean difference is given by:
- Compute the Test Statistic :
-
Make the Decision:
- For degrees of freedom , the critical value for a right-tailed test at is approximately .
- Since (corresponding to a -value ), we reject .
- Conclusion: The score gain is statistically significant at the level.
Part (b): Cohen's Effect Size
Cohen's quantifies the magnitude of the difference independent of sample size:
Part (c): Practical Significance vs. Statistical Significance
-
Comparison to District Criteria:
- By effect size: , which is far below the benchmark for a small effect ().
- By absolute score gain: points on a -point scale, which is well below the district's minimum meaningful threshold of points.
- Explanation: The program does not demonstrate practical significance. Although the result is highly statistically significant, this statistical significance is solely an artifact of the exceptionally large sample size (). With such a large , the standard error becomes tiny (), enabling the test to detect even trivial, inconsequential differences as statistically significant.
Typical errors
01 The student forgets to divide the standard deviation by the square root of n when computing the test statistic, confusing the individual difference distribution with the sampling distribution.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1Correct
You divided the mean difference directly by the sample standard deviation instead of the standard error . A test statistic measures how many standard errors the sample mean is from the hypothesized mean.
- Step 2
Critical value for is Since , fail to reject
CorrectSince , reject at the level.
Your comparison logic and critical value of were sound, but because your calculated -statistic was incorrect from Step 1, your decision to fail to reject was a downstream error.
- Step 3
- Step 4
The program does not show practical significance because and the gain of is less than points.
WHAT YOU DID WELL
Compliments from Your TeacherYou identified the correct one-tailed critical value of for .
You correctly applied the formula for Cohen's effect size.
You accurately evaluated practical significance against the stated district standards.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou used the formula for Cohen's in place of the formula for the test statistic.
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou confused the variability of individual differences () with the variability of the sample mean difference (). Test statistics evaluate sampling distributions, which shrink by a factor of as sample size increases.
REASONING PATH
Your ACEpath teacher’s suggested approach- Standard Error vs. Standard Deviation
For hypothesis testing on sample means, the denominator is the standard error , not the sample standard deviation .
Always write down before assembling the test statistic.
- Test Statistic vs. Effect Size
Cohen's depends only on the distribution spread, whereas scales directly with sample size.
Check whether your calculated test statistic incorporates the sample size .
CORRECTED PATH
ACEpath Teacher's DemonstrationCompute the standard error of the mean difference: .
Compute the test statistic: .
Compare to the critical value: since (with ), reject in favor of .
(a) , reject ; the score gain is statistically significant. (b) Cohen's . (c) The program does not demonstrate practical significance because and the mean gain of points is below the minimum threshold of points.
02 The student miscalculates Cohen's d by dividing by the standard error instead of the sample standard deviation, confusing effect size with the test statistic.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
, so reject
- Step 2Correct
Cohen's standardizes the mean difference using the sample standard deviation of differences (), not the standard error (). Dividing by simply reproduces the test statistic rather than calculating the sample-size-independent effect size.
- Step 3
The program demonstrates practical significance because is much greater than .
CorrectThe program does not demonstrate practical significance because and the average gain of points is far below the -point policy threshold.
Because your calculated was inflated from using , you concluded that the program had practical significance. Evaluating the true effect size () and comparing the mean gain of points against the required points reveals that the gain is practically negligible despite being statistically significant.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly calculated the standard error of the mean differences.
You computed the correct paired -statistic and properly compared it to the critical value to reject the null hypothesis.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou substituted into the denominator of Cohen's , repeating the formula for the test statistic .
WHY IT HAPPENED
ACEpath Teacher's DiagnosisEffect size is designed to measure the real-world magnitude of an effect independently of sample size, which requires dividing by the standard deviation of individual observations (). Dividing by the standard error () re-introduces into the calculation, turning the effect size into a sample-size-dependent test statistic.
REASONING PATH
Your ACEpath teacher’s suggested approach- Denominator of Cohen's d
Cohen's , dividing the mean change by the standard deviation of the scores, never by the standard error.
Always check that the denominator of an effect size contains standard deviation (), not standard error ().
- Large Sample Fallacy
With very large sample sizes (such as ), tiny and practically meaningless differences can produce huge -values and extreme statistical significance.
Separate statistical significance (is ?) from practical significance (is or is the raw gain meaningful?).
CORRECTED PATH
ACEpath Teacher's DemonstrationCalculate Cohen's using the sample standard deviation: .
Compare the calculated effect size to the district benchmark: , which represents a trivial, negligible effect.
Compare the mean score increase to the district point threshold: points.
Conclude that although the score gain is statistically significant due to the massive sample size (), it does not demonstrate practical significance.
(b) ; (c) The program does not demonstrate practical significance because and the average gain of points is far below the required -point threshold.
03 The student equates statistical significance with practical significance, concluding the program is practically meaningful solely because the null hypothesis was rejected.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
- Step 2
- Step 3Correct
The program does not demonstrate practical significance because and the mean gain of points is far below the -point policy benchmark. The statistical significance is merely driven by the massive sample size ().
You equated statistical significance with practical significance. A tiny -value only indicates that the gain is unlikely to be zero by chance, not that the magnitude is meaningful in the real world.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly calculated the standard error and test statistic for the paired -test.
You properly evaluated the rejection rule against the critical value to reject the null hypothesis.
You accurately calculated Cohen's effect size.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeNo slip was made in your numerical calculations.
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou conflated rejecting the null hypothesis with real-world meaningfulness. With very large sample sizes (), standard error shrinks close to zero, causing even negligible differences (like a -point shift on a -point scale) to achieve extreme statistical significance (). Practical significance must be judged by effect size () and domain benchmarks, independent of sample size.
REASONING PATH
Your ACEpath teacher’s suggested approach- Statistical vs. Practical Significance
Statistical significance () only means is unlikely due to chance; practical significance evaluates whether the magnitude of the difference matters in practice.
Compare the observed difference and Cohen's against real-world policy thresholds rather than the hypothesis test decision.
- Sample Size Inflation of Test Statistics
Because , an enormous guarantees a large even when effect size is tiny.
When sample sizes are very large, always check effect size () and raw units to evaluate whether the effect has any real-world value.
CORRECTED PATH
ACEpath Teacher's DemonstrationCheck the calculated effect size: , which is far below the district's minimum small-to-moderate threshold of .
Check the raw mean gain: points on a -point scale, which is well below the required gain of at least points.
Acknowledge that rejecting occurred solely because reduces the standard error to , magnifying a trivial difference into a large -statistic.
The program does not demonstrate practical significance because both the effect size () and the raw score gain ( points) fail to meet the district's thresholds. Statistical significance was achieved only because of the huge sample size ().
Lock it in
What is the key difference between the denominator of a paired -test statistic and the denominator of Cohen's ?
The -test denominator is the standard error , which shrinks with . Cohen's uses the sample standard deviation , which does not depend on sample size.
Similar question
A reading intervention is evaluated on students. The mean difference is points with points. Calculate (a) the standard error of the mean difference, (b) the paired -statistic for , and (c) Cohen's .
Answer
(a) . (b) . (c) .
A statewide software trial of users reports an average increase of minutes on task with minutes. (a) Compute the -statistic testing . (b) Compute Cohen's . (c) State whether the result is statistically significant at (critical value ) and whether it shows practical significance under Cohen's benchmark ().
Answer
(a) . (b) . (c) Statistically significant (, reject ), but not practically significant ().
A company wants to detect a tiny effect size of Cohen's at the significance level for a one-tailed test (requiring ). Using the relationship , what minimum sample size is required for the test statistic to reach statistical significance?
Answer
Using : . Thus, a minimum sample size of is required.