Resources

An industrial web platform conducts an A/B test with an extremely large sample size of n=1,000,000n = 1{,}000{,}000 user sessions per variant to detect changes in click-through rate (H0:p1−p2=0H_0: p_1 - p_2 = 0 versus Ha:p1−p2≠0H_a: p_1 - p_2 \neq 0). The observed conversion rates are p^1=0.05040\hat{p}_1 = 0.05040 and p^2=0.05000\hat{p}_2 = 0.05000. The pooled standard error is SE=0.0001554\text{SE} = 0.0001554. (a) Compute the standardized test statistic zz to two decimal places. (b) Determine the two-tailed PP-value to four decimal places. (c) Contrast the statistical significance at α=0.05\alpha = 0.05 with the practical significance of the observed difference in conversion rates (0.04%0.04\% percentage points).

Answer and steps

Final answer

(a) Test statistic: z=2.57z = 2.57; (b) PP-value: P=2×P(Z≥2.57)=2×0.0051=0.0102P = 2 \times P(Z \ge 2.57) = 2 \times 0.0051 = 0.0102; (c) The result is statistically significant at α=0.05\alpha = 0.05 (P=0.0102<0.05P = 0.0102 < 0.05), so H0H_0 is rejected; however, the practical effect size is an absolute gain of only 0.00040.0004 (0.04%0.04\%), demonstrating that massive sample sizes can drive PP-values to near zero for trivial real-world differences.

Steps

Part (a): Compute the Standardized Test Statistic zz

The test statistic for comparing two independent population proportions under the null hypothesis H0:p1−p2=0H_0: p_1 - p_2 = 0 is given by:

z=(p^1−p^2)−0SEz = \frac{(\hat{p}_1 - \hat{p}_2) - 0}{\text{SE}}

  1. Calculate the difference between sample proportions: p^1−p^2=0.05040−0.05000=0.00040\hat{p}_1 - \hat{p}_2 = 0.05040 - 0.05000 = 0.00040
  2. Divide by the pooled standard error: z=0.000400.0001554≈2.5740z = \frac{0.00040}{0.0001554} \approx 2.5740

Rounding to two decimal places gives: z=2.57z = 2.57


Part (b): Determine the Two-Tailed PP-Value

Because the alternative hypothesis is two-sided (Ha:p1−p2≠0H_a: p_1 - p_2 \neq 0), the PP-value is the probability of observing a test statistic at least as extreme as z=2.57z = 2.57 in either tail under the standard normal distribution Z∼N(0,1)Z \sim N(0, 1):

P=2⋅P(Z≥2.57)P = 2 \cdot P(Z \ge 2.57)

  1. Using the standard normal cumulative distribution table, find the tail area: P(Z≥2.57)=1−Φ(2.57)=1−0.9949=0.0051P(Z \ge 2.57) = 1 - \Phi(2.57) = 1 - 0.9949 = 0.0051
  2. Double this probability for a two-tailed test: P=2⋅0.0051=0.0102P = 2 \cdot 0.0051 = 0.0102

Part (c): Statistical Significance vs. Practical Significance

  • Statistical Significance: At a significance level of α=0.05\alpha = 0.05, the test is statistically significant because the PP-value is strictly less than α\alpha (0.0102<0.050.0102 < 0.05). Consequently, we reject the null hypothesis H0H_0 and conclude that there is a non-zero difference in conversion rates.
  • Practical Significance: The absolute gain in conversion rate is only p^1−p^2=0.0004\hat{p}_1 - \hat{p}_2 = 0.0004, or 0.040.04 percentage points. In practical business contexts, an effect this minuscule is often negligible and may not justify the engineering or business costs of rolling out the change.

Key Pitfall: Do not equate statistical significance with practical importance. Standard error scales inversely with the square root of sample size (SE∝1n\text{SE} \propto \frac{1}{\sqrt{n}}). With extremely large sample sizes (here, n=1,000,000n = 1{,}000{,}000), the standard error becomes vanishingly small, allowing hypothesis tests to detect trivially small differences that have virtually no practical or operational impact.

Typical errors

OverviewACEpath Teacher's Diagnosis

01 The student calculates a one-tailed P-value instead of a two-tailed P-value despite the two-sided alternative hypothesis.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    z=0.05040−0.050000.0001554  ⟹  z=0.000400.0001554=2.57z = \frac{0.05040 - 0.05000}{0.0001554} \implies z = \frac{0.00040}{0.0001554} = 2.57

  2. Step 2

    P-value=P(Z≥2.57)=1−0.9949=0.0051\text{P-value} = P(Z \ge 2.57) = 1 - 0.9949 = 0.0051

    Correct

    P-value=2⋅P(Z≥2.57)=2⋅(1−0.9949)=0.0102\text{P-value} = 2 \cdot P(Z \ge 2.57) = 2 \cdot (1 - 0.9949) = 0.0102

    Because the alternative hypothesis is two-sided (Ha:p1−p2≠0H_a: p_1 - p_2 \neq 0), extreme outcomes occur in both directions. You must multiply the one-tail area by 22 to get the two-tailed PP-value.

  3. Step 3

    Because P-value=0.0051<0.05\text{P-value} = 0.0051 < 0.05, the result is statistically significant, but the tiny difference of 0.04%0.04\% means it lacks practical significance.

    Correct

    Because P-value=0.0102<0.05\text{P-value} = 0.0102 < 0.05, the result is statistically significant, but the tiny difference of 0.04%0.04\% means it lacks practical significance.

    Your reasoning comparing statistical significance to practical significance is excellent and fully sound, but the sentence quotes your un-doubled PP-value from the previous step.

WHAT YOU DID WELL
Compliments from Your Teacher

You accurately substituted the sample proportions and standard error into the zz-statistic formula.

You correctly found the upper-tail area 1−Φ(2.57)=0.00511 - \Phi(2.57) = 0.0051 using standard normal probabilities.

You articulated a clear and correct interpretation of why a massive sample size can produce statistical significance for an operationally negligible difference.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
One-tailed vs. Two-tailed P-value Omission

You found only the upper-tail area P(Z≥2.57)P(Z \ge 2.57) without multiplying by 22 for the two-sided alternative hypothesis.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

Overlooking the two-sided structure of Ha:p1−p2≠0H_a: p_1 - p_2 \neq 0 when translating the standardized test statistic into a PP-value, treating an extreme positive difference as the only rejection region.

REASONING PATH
Your ACEpath teacher’s suggested approach
Hypothesis Directionality Check
  • If the alternative hypothesis uses ≠\neq, the test is two-tailed, so P-value=2⋅P(Z≥∣z∣)\text{P-value} = 2 \cdot P(Z \ge |z|). If it uses >> or <<, the test is one-tailed.

  • Look at the sign in HaH_a before writing the PP-value integral or probability expression to decide whether to multiply by 22.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Identify the alternative hypothesis: Ha:p1−p2≠0H_a: p_1 - p_2 \neq 0, which requires considering deviations in both tails: {Z≥2.57}∪{Z≤−2.57}\{Z \ge 2.57\} \cup \{Z \le -2.57\}.

  2. Multiply the single tail area by 22: P-value=2×P(Z≥2.57)=2×0.0051=0.0102\text{P-value} = 2 \times P(Z \ge 2.57) = 2 \times 0.0051 = 0.0102.

Final answer

(a) z=2.57z = 2.57; (b) P-value=0.0102P\text{-value} = 0.0102; (c) Statistically significant (P=0.0102<0.05P = 0.0102 < 0.05), but practically negligible due to the tiny effect size (0.04%0.04\%).

NEXT STEPS
Final Takeaways from Your Teacher
  • Double-check the inequality sign in HaH_a every time you compute a PP-value.

  • Practice sketching the standard normal curve with both tails shaded whenever HaH_a contains ≠\neq.

OverviewACEpath Teacher's Diagnosis

02 The student conflates statistical significance with practical importance, wrongly concluding that a statistically significant result must have substantial real-world impact.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    z=0.05040−0.050000.0001554  ⟹  z=0.000400.0001554=2.57z = \frac{0.05040 - 0.05000}{0.0001554} \implies z = \frac{0.00040}{0.0001554} = 2.57

  2. Step 2

    P-value=2×(1−0.9949)=2×0.0051=0.0102\text{P-value} = 2 \times (1 - 0.9949) = 2 \times 0.0051 = 0.0102

  3. Step 3

    Because the test is statistically significant at α=0.05\alpha = 0.05, the observed difference of 0.04%0.04\% is proven to be practically significant and highly impactful for the platform.

    Correct

    While the result is statistically significant (P=0.0102<0.05P = 0.0102 < 0.05), an absolute gain of 0.00040.0004 (0.040.04 percentage points) is practically negligible; large sample sizes can produce statistical significance even for trivial real-world differences.

    Statistical significance only indicates that the observed difference is unlikely to be due to chance alone under the null hypothesis. It does not measure the magnitude or real-world importance of the effect.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly computed the difference in proportions and evaluated the standardized test statistic z=2.57z = 2.57.

You correctly calculated the two-tailed PP-value as 0.01020.0102 using the standard normal distribution.

You correctly identified that P=0.0102<0.05P = 0.0102 < 0.05 constitutes statistical significance.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Conflating Statistical Significance with Practical Significance

none

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You treated statistical significance (P<αP < \alpha) as proof of meaningful real-world impact, overlooking that extremely large sample sizes shrink standard errors and make practically negligible effect sizes statistically significant.

REASONING PATH
Your ACEpath teacher’s suggested approach
Separating Significance from Effect Size
  • Statistical significance (P<αP < \alpha) tests whether an effect differs from zero, whereas practical significance evaluates whether the magnitude of the effect matters in real-world application.

  • Always evaluate the observed effect size (e.g., p^1−p^2=0.0004\hat{p}_1 - \hat{p}_2 = 0.0004) independently of the PP-value before declaring real-world impact.

The Large Sample Size Trap
  • Because SE∝1n\text{SE} \propto \frac{1}{\sqrt{n}}, massive sample sizes can drive PP-values below α\alpha for tiny, trivial differences.

  • When nn is very large, inspect confidence intervals and practical utility rather than relying solely on the reject/fail-to-reject decision.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Note that at α=0.05\alpha = 0.05, P=0.0102<0.05P = 0.0102 < 0.05, so the null hypothesis H0H_0 is rejected, making the result statistically significant.

  2. Evaluate the effect size: the observed difference is only p^1−p^2=0.0004\hat{p}_1 - \hat{p}_2 = 0.0004, or 0.040.04 percentage points.

  3. Explain that with n=1,000,000n = 1{,}000{,}000, standard error is tiny (extSE≈0.0001554 ext{SE} \approx 0.0001554), allowing detection of miniscule differences that have negligible real-world or commercial impact.

Final answer

The result is statistically significant (P=0.0102<0.05P = 0.0102 < 0.05), but it lacks practical significance because an absolute difference of 0.040.04 percentage points is far too small to have meaningful real-world impact.

NEXT STEPS
Final Takeaways from Your Teacher
  • Review the distinction between PP-values (evidence against chance) and effect sizes (magnitude of difference).

  • Practice interpreting A/B test results where sample sizes are in the millions.

  • Calculate confidence intervals for difference in proportions to see how narrow ranges around near-zero effects highlight lack of practical impact.

OverviewACEpath Teacher's Diagnosis

03 The student misinterprets the 0.04% percentage point difference as a relative increase rather than an absolute difference, computing the test statistic using percentage values instead of proportions.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    z=5.04−5.000.0001554  ⟹  z=0.040.0001554=257.40z = \frac{5.04 - 5.00}{0.0001554} \implies z = \frac{0.04}{0.0001554} = 257.40

    Correct

    z=0.05040−0.050000.0001554=0.000400.0001554≈2.57z = \frac{0.05040 - 0.05000}{0.0001554} = \frac{0.00040}{0.0001554} \approx 2.57

    You expressed the conversion rates as percentages (5.04−5.00=0.045.04 - 5.00 = 0.04) in the numerator while leaving the standard error in decimal proportion units (0.00015540.0001554) in the denominator. Both must be on the same scale, so the numerator should be 0.05040−0.05000=0.000400.05040 - 0.05000 = 0.00040.

  2. Step 2

    P-value=2×P(Z≥257.40)≈0.0000\text{P-value} = 2 \times P(Z \ge 257.40) \approx 0.0000

    Correct

    P-value=2×P(Z≥2.57)=2×0.0051=0.0102\text{P-value} = 2 \times P(Z \ge 2.57) = 2 \times 0.0051 = 0.0102

    Your two-tailed setup 2×P(Z≥z)2 \times P(Z \ge z) is completely correct, but your numerical answer was affected by the inflated zz-score from step 1.

  3. Step 3

    The result is statistically significant since P-value < 0.05, but practically negligible because an absolute shift of 0.04 percentage points is trivial.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly used the two-tailed probability structure 2×P(Z≥∣z∣)2 \times P(Z \ge |z|) to calculate the PP-value.

You clearly articulated why a statistically significant result can still be practically negligible.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Units and scale mismatch

You converted the proportions to percentages in the numerator (5.045.04 and 5.005.00) without converting the standard error to the same percentage scale.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

Mixing proportion scales (decimals) and percentage scales within the same standardized ratio.

REASONING PATH
Your ACEpath teacher’s suggested approach
Standard Error Scale Alignment
  • The numerator (p^1−p^2)(\hat{p}_1 - \hat{p}_2) and denominator SE\text{SE} must be expressed in the exact same units (both as decimals or both as percentages).

  • Always keep sample proportions in decimal form unless SE\text{SE} is explicitly given in percentage points.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Compute the proportion difference: p^1−p^2=0.05040−0.05000=0.00040\hat{p}_1 - \hat{p}_2 = 0.05040 - 0.05000 = 0.00040.

  2. Divide by the decimal standard error: z=0.000400.0001554≈2.57z = \frac{0.00040}{0.0001554} \approx 2.57.

  3. Evaluate the two-tailed PP-value: P=2×[1−Φ(2.57)]=2×0.0051=0.0102P = 2 \times [1 - \Phi(2.57)] = 2 \times 0.0051 = 0.0102.

Final answer

(a) z=2.57z = 2.57; (b) P=0.0102P = 0.0102; (c) Statistically significant (P<0.05P < 0.05), but practically negligible (0.04%0.04\% point difference).

NEXT STEPS
Final Takeaways from Your Teacher
  • Check that numerator and denominator units match before evaluating any test statistic.

  • Practice converting between proportions and percentage points without dropping factor-of-100 scaling in formulas.

Lock it in

Similar question

01

A quality engineer tests H0:μ=50H_0: \mu = 50 vs. Ha:μ≠50H_a: \mu \neq 50. The test statistic is calculated as z=2.15z = 2.15. Given that P(Z≥2.15)=0.0158P(Z \ge 2.15) = 0.0158, compute the PP-value and state whether H0H_0 is rejected at α=0.05\alpha = 0.05.

Answer

Since HaH_a has ≠\neq, the test is two-tailed: P=2×0.0158=0.0316P = 2 \times 0.0158 = 0.0316. Because 0.0316<0.050.0316 < 0.05, reject H0H_0.

02

In an A/B test with n=500,000n = 500{,}000 users per group, variants AA and BB have click rates of 0.02000.0200 and 0.02030.0203. The test statistic for Ha:pA≠pBH_a: p_A \neq p_B is z=2.14z = 2.14, with P(Z≥2.14)=0.0162P(Z \ge 2.14) = 0.0162. (a) State the PP-value. (b) Explain why this test is statistically significant at α=0.05\alpha = 0.05 yet may lack practical significance.

Answer

(a) P=2×0.0162=0.0324P = 2 \times 0.0162 = 0.0324. (b) With P=0.0324<0.05P = 0.0324 < 0.05, the difference is statistically significant. However, the absolute difference is only 0.00030.0003 (0.030.03 percentage points); huge sample sizes make standard error so small that negligible real-world differences become statistically detectable.

03

A researcher tests H0:p=0.40H_0: p = 0.40 vs. Ha:p>0.40H_a: p > 0.40 and finds z=1.80z = 1.80 (P(Z≥1.80)=0.0359P(Z \ge 1.80) = 0.0359). A colleague re-analyzes the identical data using Ha:p≠0.40H_a: p \neq 0.40. Compare the PP-values of the two researchers and evaluate whether both reject H0H_0 at α=0.05\alpha = 0.05.

Answer

For the one-tailed test (p>0.40p > 0.40), P=0.0359<0.05P = 0.0359 < 0.05, so H0H_0 is rejected. For the two-tailed test (p≠0.40p \neq 0.40), P=2×0.0359=0.0718>0.05P = 2 \times 0.0359 = 0.0718 > 0.05, so H0H_0 is NOT rejected. The choice of alternative hypothesis directly changes the significance conclusion.

Loading workspace…