Resources
Consider a one-sample -test of versus with fixed true mean discrepancy , known population standard deviation , and significance level (). (a) Express the test statistic and Cohen's as functions of the sample size . (b) Determine , the resulting limit of the -value as , and the limit of Cohen's as . (c) Explain the asymptotic behavior of statistical hypothesis testing: why does any trivially small, non-zero difference inevitably become statistically significant as , and why does this necessitate reporting effect size point estimates and confidence intervals for practical interpretation?
Answer and steps
Final answer
(a) ; Cohen's (independent of ). (b) ; consequently, ; while . (c) As , the standard error , driving the test statistic to infinity and the -value to zero for any non-zero difference , no matter how microscopic. Therefore, statistical significance alone cannot assess whether an effect matters; reporting effect sizes (such as Cohen's ) and confidence intervals is mandatory to evaluate practical significance.
Steps
Part (a): Expressing and Cohen's
The one-sample -test statistic tests the null hypothesis against . When the true population mean is , the expected value of the sample mean is .
The expected value of the test statistic as a function of sample size is given by:
Substituting the given values and :
Cohen's is a standardized measure of effect size that quantifies the magnitude of the difference relative to the population standard deviation:
Because Cohen's depends solely on the population parameters and , it is completely independent of the sample size .
Part (b): Evaluating the Limits as
- Limit of the test statistic :
- Limit of the -value: For an upper-tailed test, the -value corresponds to the probability under the standard normal distribution in the upper tail: Taking the limit as :
- Limit of Cohen's : Since Cohen's is constant with respect to :
Part (c): Statistical vs. Practical Significance
As , the standard error of the mean, , approaches zero:
Because the standard error shrinks toward zero, any non-zero difference —no matter how minuscule or trivial—is divided by an arbitrarily small standard error. This forces the test statistic toward infinity and drives the -value toward zero (). Consequently, with a sufficiently large sample size, any microscopic discrepancy is virtually guaranteed to be rejected as "statistically significant."
However, statistical significance only answers the binary question of whether the observed data are consistent with the null hypothesis (). It provides no information about the practical importance or magnitude of that difference. Here, Cohen's represents an extremely negligible effect (conventionally, is considered small, medium, and large).
Therefore, reporting effect sizes and confidence intervals alongside -values is essential:
- Effect sizes quantify the magnitude of the real-world impact independently of sample size.
- Confidence intervals bound the true parameter value in real-world units, allowing researchers to see whether the entire plausible range of values lies within a region of practical equivalence or negligible clinical/engineering importance.
Typical errors
01 The student confuses the standard error with the population standard deviation in the denominator, missing the factor of √n in the test statistic.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1Correct
A sample mean has standard error , not . Dividing by computes a single observation's -score rather than the test statistic for a sample mean of size .
- Step 2
- Step 3Correct
Because your omitted , your limits were held constant at rather than diverging to for the test statistic and converging to for the -value.
- Step 4
- Step 5
As , the test statistic stays at and does not exceed , so is never rejected; effect sizes and CIs are needed because test statistics cannot detect small differences.
CorrectAs , , driving and . Any non-zero difference inevitably becomes statistically significant; effect sizes and CIs are needed because -values cannot assess practical importance.
Large samples do not fail to detect small differences; rather, they overpower tests so that even trivial differences inevitably become statistically significant (). Effect sizes and confidence intervals are needed to judge whether a detectable difference is practically meaningful.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly computed Cohen's from the population parameters.
You correctly identified that Cohen's is independent of sample size .
You accurately computed the upper-tail probability for your stated -value.
THE ERROR
Where Your ACEpath Teacher Spotted the Mistakeno
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou confused the standard deviation of individual observations () with the standard error of the sample mean () in the denominator of the test statistic. This caused you to miss the scaling factor, leading to the inverted misconception that test statistics cannot detect small differences with large samples.
REASONING PATH
Your ACEpath teacher’s suggested approach- Denominator of a Sample Mean Test Statistic
, so .
Always divide the mean difference by the standard error , ensuring the test statistic scales with .
- Large-Sample Asymptotic Behavior
For any fixed , and .
Remember that statistical significance tests whether , not whether is large; sufficiently large samples will detect any non-zero difference.
CORRECTED PATH
ACEpath Teacher's DemonstrationExpress the test statistic with standard error in the denominator: .
Evaluate the asymptotic limits: , which yields .
Explain practical significance: because , inevitably exceeds any critical value () for any , making trivial effects statistically significant. Reporting effect size () and confidence intervals reveals whether the detected effect has real-world importance.
(a) ; Cohen's . (b) , , . (c) As , the standard error approaches , driving and for any non-zero difference; effect sizes and confidence intervals are essential to assess practical importance.
02 The student incorrectly scales Cohen's d by √n, conflating standardized effect size with the test statistic.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
- Step 2Correct
d = = = 0.050
Cohen's is a population effect size metric defined as , which does not depend on the sample size . You incorrectly multiplied it by , conflating the standardized effect size with the test statistic.
- Step 3
- Step 4Correct
Because you defined Cohen's as a function of in Step 2, your limit evaluated to instead of remaining constant at .
- Step 5
As grows, both the test statistic and effect size become infinitely large, which forces the -value to zero. Confidence intervals are needed because infinite effect sizes in large samples require bounds on practical significance.
CorrectAs , the standard error , forcing and for any , even when the true difference is negligible. Because statistical significance only indicates , reporting effect sizes (here ) and confidence intervals is necessary to assess whether the magnitude of the difference has real-world practical importance.
Because you assumed effect size increases with sample size, your explanation missed the fundamental contrast between statistical significance (which depends heavily on ) and practical significance (which measures the true, sample-size-invariant effect size).
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly derived the formula for the test statistic .
You correctly evaluated the asymptotic limits of the test statistic and the -value as .
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeMultiplying Cohen's by .
WHY IT HAPPENED
ACEpath Teacher's DiagnosisConflating sample-dependent test statistics with population effect sizes. The test statistic scales with because the standard error shrinks as sample size grows. In contrast, Cohen's divides the mean difference by the population standard deviation, making it purely a property of the population distributions and completely invariant to .
REASONING PATH
Your ACEpath teacher’s suggested approach- Effect Size vs. Test Statistic Relationship
(e.g., )
Keep Cohen's defined strictly as ; do not include sample size in effect size definitions.
- Practical Significance Interpretation
Large sample sizes can make minuscule effects statistically significant () even when practical effect size is near zero.
Use effect sizes () and confidence intervals to quantify the real-world magnitude of a difference independently of sample size.
CORRECTED PATH
ACEpath Teacher's DemonstrationState the definition of Cohen's : .
Note that does not contain , so .
Explain that because , and for any non-zero . However, remains tiny, demonstrating that statistical significance does not imply a meaningful real-world effect.
(a) ; Cohen's . (b) , , . (c) Shrinking standard error forces statistical significance () even for trivial differences; effect sizes and confidence intervals reveal whether the true effect has practical importance.
03 The student divides by √n instead of multiplying by √n when simplifying the complex fraction for z(n).
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
and
Correctand
You correctly set up the formula for and computed Cohen's . However, dividing by the standard error means multiplying by the reciprocal: , not .
- Step 2Correct
Your limit evaluation logically followed your expression from Part (a), and your limit for Cohen's is correct. Because actually grows without bound as , the -value approaches , not .
- Step 3
"As approaches infinity, the test statistic shrinks to 0, which prevents the test from ever reaching statistical significance. Reporting confidence intervals is necessary to determine the real difference because large samples eliminate the test statistic."
CorrectAs , , which forces and the for any non-zero difference . Thus, any trivially small difference inevitably becomes statistically significant, necessitating effect sizes and confidence intervals to evaluate practical importance.
Because of the earlier reciprocal slip, your interpretation was inverted. Large sample sizes do not eliminate the test statistic; they shrink the standard error to zero, making even a negligible effect statistically significant.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly identified the definition and formula for Cohen's .
You recognized that Cohen's is a population parameter independent of sample size .
You correctly related the upper-tail -value to the cumulative standard normal distribution.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou placed in the denominator instead of the numerator when simplifying .
WHY IT HAPPENED
ACEpath Teacher's DiagnosisDividing by a fraction is equivalent to multiplying by its reciprocal, . Forgetting to invert the denominator fraction caused the test statistic to decay with sample size rather than scale with .
REASONING PATH
Your ACEpath teacher’s suggested approach- Simplifying Standard Errors in Test Statistics
Always rewrite division by a fraction as multiplication by its reciprocal: .
- Large-Sample Behavior of $p$-values
As sample size increases, standard error shrinks toward 0, causing the test statistic for any real difference to grow toward and .
Sanity check asymptotic behavior: larger samples always provide more power to detect differences, never less.
CORRECTED PATH
ACEpath Teacher's DemonstrationSimplify the test statistic correctly by inverting the denominator: .
Evaluate the limits: , which yields .
Explain the core concept: because , any non-zero difference will achieve as . Therefore, statistical significance alone cannot tell you if an effect is meaningful; you must report effect sizes (like ) and confidence intervals to judge practical significance.
(a) , ; (b) , , ; (c) As , , forcing and , which guarantees statistical significance for any regardless of practical importance.
Lock it in
What is the denominator of the one-sample -test statistic for a sample mean , and how does it depend on ?
The denominator is the standard error , which shrinks toward as increases, scaling by .
Similar question
A researcher tests versus with known . If the true mean is , calculate: (1) Cohen's , (2) the test statistic when , and (3) the test statistic when .
Answer
(1) . (2) For , , so (, not significant at ). (3) For , , so (, highly significant).
In an educational study with students, an online intervention raises exam scores by points on a -point scale with known points. (a) Compute and state whether the result is statistically significant at (). (b) Compute Cohen's and evaluate whether the intervention has practical significance.
Answer
(a) . Thus . The result is highly statistically significant (). (b) . Because is far below the conventional threshold for a small effect () and represents only of a standard deviation, the intervention lacks practical significance despite its overwhelming statistical significance.
Two studies evaluate the same drug ( vs. , ). Study A has and observes . Study B has and observes . For each study, find the confidence interval for and the test statistic . Explain which study shows a clinically meaningful effect and which shows statistical significance.
Answer
For Study A: . Test statistic (, significant at ). CI: . Effect size (medium, clinically meaningful), but the estimate has high uncertainty. For Study B: . Test statistic (, highly significant). CI: . Effect size (negligible). Study B is statistically significant but practically trivial, while Study A shows a clinically meaningful effect that warrants replication with a narrower interval.