Resources
An industrial web platform conducts an A/B test with an extremely large sample size of user sessions per variant to detect changes in click-through rate ( versus ). The observed conversion rates are and . The pooled standard error is . (a) Compute the standardized test statistic to two decimal places. (b) Determine the two-tailed -value to four decimal places. (c) Contrast the statistical significance at with the practical significance of the observed difference in conversion rates ( percentage points).
Answer and steps
Final answer
(a) Test statistic: ; (b) -value: ; (c) The result is statistically significant at (), so is rejected; however, the practical effect size is an absolute gain of only (), demonstrating that massive sample sizes can drive -values to near zero for trivial real-world differences.
Steps
Part (a): Compute the Standardized Test Statistic
The test statistic for comparing two independent population proportions under the null hypothesis is given by:
- Calculate the difference between sample proportions:
- Divide by the pooled standard error:
Rounding to two decimal places gives:
Part (b): Determine the Two-Tailed -Value
Because the alternative hypothesis is two-sided (), the -value is the probability of observing a test statistic at least as extreme as in either tail under the standard normal distribution :
- Using the standard normal cumulative distribution table, find the tail area:
- Double this probability for a two-tailed test:
Part (c): Statistical Significance vs. Practical Significance
- Statistical Significance: At a significance level of , the test is statistically significant because the -value is strictly less than (). Consequently, we reject the null hypothesis and conclude that there is a non-zero difference in conversion rates.
- Practical Significance: The absolute gain in conversion rate is only , or percentage points. In practical business contexts, an effect this minuscule is often negligible and may not justify the engineering or business costs of rolling out the change.
Key Pitfall: Do not equate statistical significance with practical importance. Standard error scales inversely with the square root of sample size (). With extremely large sample sizes (here, ), the standard error becomes vanishingly small, allowing hypothesis tests to detect trivially small differences that have virtually no practical or operational impact.
Typical errors
01 The student calculates a one-tailed P-value instead of a two-tailed P-value despite the two-sided alternative hypothesis.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
- Step 2Correct
Because the alternative hypothesis is two-sided (), extreme outcomes occur in both directions. You must multiply the one-tail area by to get the two-tailed -value.
- Step 3
Because , the result is statistically significant, but the tiny difference of means it lacks practical significance.
CorrectBecause , the result is statistically significant, but the tiny difference of means it lacks practical significance.
Your reasoning comparing statistical significance to practical significance is excellent and fully sound, but the sentence quotes your un-doubled -value from the previous step.
WHAT YOU DID WELL
Compliments from Your TeacherYou accurately substituted the sample proportions and standard error into the -statistic formula.
You correctly found the upper-tail area using standard normal probabilities.
You articulated a clear and correct interpretation of why a massive sample size can produce statistical significance for an operationally negligible difference.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou found only the upper-tail area without multiplying by for the two-sided alternative hypothesis.
WHY IT HAPPENED
ACEpath Teacher's DiagnosisOverlooking the two-sided structure of when translating the standardized test statistic into a -value, treating an extreme positive difference as the only rejection region.
REASONING PATH
Your ACEpath teacher’s suggested approach- Hypothesis Directionality Check
If the alternative hypothesis uses , the test is two-tailed, so . If it uses or , the test is one-tailed.
Look at the sign in before writing the -value integral or probability expression to decide whether to multiply by .
CORRECTED PATH
ACEpath Teacher's DemonstrationIdentify the alternative hypothesis: , which requires considering deviations in both tails: .
Multiply the single tail area by : .
(a) ; (b) ; (c) Statistically significant (), but practically negligible due to the tiny effect size ().
02 The student conflates statistical significance with practical importance, wrongly concluding that a statistically significant result must have substantial real-world impact.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
- Step 2
- Step 3
Because the test is statistically significant at , the observed difference of is proven to be practically significant and highly impactful for the platform.
CorrectWhile the result is statistically significant (), an absolute gain of ( percentage points) is practically negligible; large sample sizes can produce statistical significance even for trivial real-world differences.
Statistical significance only indicates that the observed difference is unlikely to be due to chance alone under the null hypothesis. It does not measure the magnitude or real-world importance of the effect.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly computed the difference in proportions and evaluated the standardized test statistic .
You correctly calculated the two-tailed -value as using the standard normal distribution.
You correctly identified that constitutes statistical significance.
THE ERROR
Where Your ACEpath Teacher Spotted the Mistakenone
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou treated statistical significance () as proof of meaningful real-world impact, overlooking that extremely large sample sizes shrink standard errors and make practically negligible effect sizes statistically significant.
REASONING PATH
Your ACEpath teacher’s suggested approach- Separating Significance from Effect Size
Statistical significance () tests whether an effect differs from zero, whereas practical significance evaluates whether the magnitude of the effect matters in real-world application.
Always evaluate the observed effect size (e.g., ) independently of the -value before declaring real-world impact.
- The Large Sample Size Trap
Because , massive sample sizes can drive -values below for tiny, trivial differences.
When is very large, inspect confidence intervals and practical utility rather than relying solely on the reject/fail-to-reject decision.
CORRECTED PATH
ACEpath Teacher's DemonstrationNote that at , , so the null hypothesis is rejected, making the result statistically significant.
Evaluate the effect size: the observed difference is only , or percentage points.
Explain that with , standard error is tiny (), allowing detection of miniscule differences that have negligible real-world or commercial impact.
The result is statistically significant (), but it lacks practical significance because an absolute difference of percentage points is far too small to have meaningful real-world impact.
03 The student misinterprets the 0.04% percentage point difference as a relative increase rather than an absolute difference, computing the test statistic using percentage values instead of proportions.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1Correct
You expressed the conversion rates as percentages () in the numerator while leaving the standard error in decimal proportion units () in the denominator. Both must be on the same scale, so the numerator should be .
- Step 2Correct
Your two-tailed setup is completely correct, but your numerical answer was affected by the inflated -score from step 1.
- Step 3
The result is statistically significant since P-value < 0.05, but practically negligible because an absolute shift of 0.04 percentage points is trivial.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly used the two-tailed probability structure to calculate the -value.
You clearly articulated why a statistically significant result can still be practically negligible.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou converted the proportions to percentages in the numerator ( and ) without converting the standard error to the same percentage scale.
WHY IT HAPPENED
ACEpath Teacher's DiagnosisMixing proportion scales (decimals) and percentage scales within the same standardized ratio.
REASONING PATH
Your ACEpath teacher’s suggested approach- Standard Error Scale Alignment
The numerator and denominator must be expressed in the exact same units (both as decimals or both as percentages).
Always keep sample proportions in decimal form unless is explicitly given in percentage points.
CORRECTED PATH
ACEpath Teacher's DemonstrationCompute the proportion difference: .
Divide by the decimal standard error: .
Evaluate the two-tailed -value: .
(a) ; (b) ; (c) Statistically significant (), but practically negligible ( point difference).
Lock it in
How does the alternative hypothesis symbol determine whether you multiply the tail probability by 2?
If has '≠' (two-sided), you double the tail probability: . If has '>' or '<' (one-sided), you keep the single tail area.
Similar question
A quality engineer tests vs. . The test statistic is calculated as . Given that , compute the -value and state whether is rejected at .
Answer
Since has , the test is two-tailed: . Because , reject .
In an A/B test with users per group, variants and have click rates of and . The test statistic for is , with . (a) State the -value. (b) Explain why this test is statistically significant at yet may lack practical significance.
Answer
(a) . (b) With , the difference is statistically significant. However, the absolute difference is only ( percentage points); huge sample sizes make standard error so small that negligible real-world differences become statistically detectable.
A researcher tests vs. and finds (). A colleague re-analyzes the identical data using . Compare the -values of the two researchers and evaluate whether both reject at .
Answer
For the one-tailed test (), , so is rejected. For the two-tailed test (), , so is NOT rejected. The choice of alternative hypothesis directly changes the significance conclusion.