Resources

An operations research team analyzes website load times before and after a caching redesign. Across n=10,000n = 10{,}000 web page requests, the mean load time decreased by xˉ=15.0 ms\bar{x} = 15.0\text{ ms} with a standard deviation s=200.0 mss = 200.0\text{ ms}. Simultaneously, financial modeling shows that the redesign requires an ongoing server cost of $12,000\text{\char36}12{,}000 per year, while each 1.0 ms1.0\text{ ms} reduction in page load time produces an estimated $500\text{\char36}500 in annual revenue gains. (a) Test whether the decrease in load time is statistically significant at α=0.001\alpha = 0.001 using a one-sample zz-test (z∗=3.090z^* = 3.090). (b) Compute Cohen's dd and evaluate practical significance from a human perception standpoint, given that the human visual perceptual threshold for webpage latency differences is 100 ms100\text{ ms}. (c) Couple this with the economic cost-benefit principle: calculate the net annual monetary gain/loss, and determine whether the intervention possesses practical economic significance.

Answer and steps

Final answer

(a) z=15.0−0200.010,000=15.02.0=7.50z = \frac{15.0 - 0}{\frac{200.0}{\sqrt{10{,}000}}} = \frac{15.0}{2.0} = 7.50. Since z=7.50>3.090z = 7.50 > 3.090 (p<10−13<0.001p < 10^{-13} < 0.001), reject H0H_0; the reduction is highly statistically significant. (b) Cohen's d=15.0200.0=0.075d = \frac{15.0}{200.0} = 0.075; because d=0.075<0.20d = 0.075 < 0.20 and the absolute decrease (15.0 ms15.0\text{ ms}) is well below the 100 ms100\text{ ms} human sensory perception threshold, the result lacks practical behavioral significance for users. (c) Annual revenue gain =15.0 ms×$500/ms=$7,500= 15.0\text{ ms} \times \text{\char36}500/\text{ms} = \text{\char36}7{,}500; Net annual balance =$7,500−$12,000=−$4,500= \text{\char36}7{,}500 - \text{\char36}12{,}000 = -\text{\char36}4{,}500 (a net loss of $4,500\text{\char36}4{,}500 per year). Therefore, the project also lacks practical economic significance, demonstrating that a statistically significant finding can still be economically detrimental.

Steps

Part (a): Hypothesis Test for Statistical Significance

  1. State the hypotheses and parameters: Let μ\mu represent the true mean reduction in webpage load time (in ms\text{ms}).

    • Null hypothesis: H0:μ=0H_0: \mu = 0 (the caching redesign has no effect on load time).
    • Alternative hypothesis: Ha:μ>0H_a: \mu > 0 (the caching redesign reduces load time).
  2. Compute the Standard Error (SESE): Given sample size n=10,000n = 10{,}000 and sample standard deviation s=200.0 mss = 200.0\text{ ms}: SE=sn=200.010,000=200.0100=2.0 msSE = \frac{s}{\sqrt{n}} = \frac{200.0}{\sqrt{10{,}000}} = \frac{200.0}{100} = 2.0\text{ ms}
  3. Calculate the test statistic (zz): z=xˉ−μ0SE=15.0−02.0=7.50z = \frac{\bar{x} - \mu_0}{SE} = \frac{15.0 - 0}{2.0} = 7.50
  4. Compare with the critical value: For a one-tailed test at significance level α=0.001\alpha = 0.001, the critical value is z∗=3.090z^* = 3.090. Since z=7.50>3.090z = 7.50 > 3.090 (which corresponds to p<10−13<0.001p < 10^{-13} < 0.001), we reject H0H_0. The observed reduction in webpage load time is statistically significant.

Part (b): Practical Significance and Effect Size

  1. Calculate Cohen's dd: Cohen's dd measures the standardized mean difference: d=xˉs=15.0 ms200.0 ms=0.075d = \frac{\bar{x}}{s} = \frac{15.0\text{ ms}}{200.0\text{ ms}} = 0.075
  2. Evaluate the effect size and practical relevance:

    • By Cohen's conventional thresholds (0.200.20 is small, 0.500.50 is medium, and 0.800.80 is large), a value of d=0.075<0.20d = 0.075 < 0.20 indicates an extremely small or negligible effect size.
    • In domain terms, the physical reduction of 15.0 ms15.0\text{ ms} is far below the human visual perceptual threshold of 100 ms100\text{ ms}. Users cannot perceive a 15 ms15\text{ ms} difference, meaning the redesign has no discernible effect on user experience. Therefore, it lacks practical behavioral significance.

Part (c): Economic Cost-Benefit Analysis

  1. Calculate the annual revenue gain: Annual Revenue Gain=15.0 ms×$500/ms=$7,500\text{Annual Revenue Gain} = 15.0\text{ ms} \times \text{\char36}500/\text{ms} = \text{\char36}7{,}500
  2. Calculate the net annual monetary outcome: Net Annual Balance=Revenue Gain−Server Cost=$7,500−$12,000=−$4,500\text{Net Annual Balance} = \text{Revenue Gain} - \text{Server Cost} = \text{\char36}7{,}500 - \text{\char36}12{,}000 = -\text{\char36}4{,}500
  3. Determine practical economic significance: The redesign results in a net loss of $4,500\text{\char36}4{,}500 per year. Because it produces an unfavorable return on investment, the intervention lacks practical economic significance.

Key Takeaway and Common Pitfall

  • Pitfall: Confusing statistical significance with practical significance.
  • With a very large sample size (n=10,000n = 10{,}000), the standard error becomes tiny (SE=2.0 msSE = 2.0\text{ ms}), causing even minute, trivial differences to generate an enormous test statistic (z=7.50z = 7.50). While the data convincingly shows the difference is not zero (statistical significance), it remains practically unnoticeable to humans and financially detrimental to the business.

Typical errors

OverviewACEpath Teacher's Diagnosis

01 The student calculates Cohen's d using the standard error rather than the sample standard deviation, greatly overstating the standardized effect size.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    SE=200.010000=2.0  ⟹  z=15.02.0=7.50  ⟹  7.50>3.090SE = \frac{200.0}{\sqrt{10000}} = 2.0 \implies z = \frac{15.0}{2.0} = 7.50 \implies 7.50 > 3.090, so reject H0H_0 and the decrease is statistically significant.

  2. Step 2

    d=xˉSE=15.02.0=7.50d = \frac{\bar{x}}{SE} = \frac{15.0}{2.0} = 7.50

    Correct

    d=xˉs=15.0 ms200.0 ms=0.075d = \frac{\bar{x}}{s} = \frac{15.0\text{ ms}}{200.0\text{ ms}} = 0.075

    Cohen's dd measures the standardized effect size in units of the individual variability (standard deviation ss), not the sample mean variability (standard error SESE). Dividing by SESE mistakenly computes the test statistic zz rather than effect size dd.

  3. Step 3

    Since d=7.50d = 7.50 is extremely large and 15 ms15\text{ ms} is noticeable, it has practical significance.

    Correct

    Because d=0.075<0.20d = 0.075 < 0.20 and the 15.0 ms15.0\text{ ms} reduction is well below the 100 ms100\text{ ms} human sensory threshold, the redesign lacks practical significance.

    Because your calculation of dd was inflated and the problem noted that human visual latency detection requires at least 100 ms100\text{ ms}, the true difference of 15.0 ms15.0\text{ ms} is imperceptible to users and lacks practical significance.

  4. Step 4

    Revenue=15.0×500=7500  ⟹  Net=7500−12000=−4500  ⟹  \text{Revenue} = 15.0 \times 500 = 7500 \implies \text{Net} = 7500 - 12000 = -4500 \implies There is an annual loss of $4,500\text{\char36}4{,}500, so it lacks practical economic significance.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly computed the standard error SE=2.0 msSE = 2.0\text{ ms} and the test statistic z=7.50z = 7.50.

You properly compared the test statistic with the critical value z∗=3.090z^* = 3.090 to reject the null hypothesis.

You accurately calculated the annual revenue gain and net loss of −$4,500-\text{\char36}4{,}500 to evaluate practical economic significance.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Formula confusion between standard deviation and standard error

no

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You conflated the test statistic z=xˉSEz = \frac{\bar{x}}{SE} with the effect size Cohen's d=xˉsd = \frac{\bar{x}}{s}. With large sample sizes, SE=snSE = \frac{s}{\sqrt{n}} becomes very small, making zz large even when the physical effect size dd is negligible.

REASONING PATH
Your ACEpath teacher’s suggested approach
Effect Size vs. Test Statistic
  • Cohen's d=xˉ−μ0sd = \frac{\bar{x} - \mu_0}{s}, whereas z=xˉ−μ0s/n=d×nz = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} = d \times \sqrt{n}.

  • Always divide by the standard deviation ss when calculating Cohen's dd to quantify practical magnitude independent of sample size.

Perceptual and Domain Thresholds
  • Statistical significance indicates only that an effect is unlikely due to random chance (p<αp < \alpha), not that it is meaningful in the real world.

  • Compare the raw difference (ar{x} = 15.0\text{ ms}) directly against domain-specific baselines (such as the given 100 ms100\text{ ms} threshold).

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Calculate Cohen's dd using the sample standard deviation: d=xˉs=15.0 ms200.0 ms=0.075d = \frac{\bar{x}}{s} = \frac{15.0\text{ ms}}{200.0\text{ ms}} = 0.075.

  2. Evaluate Cohen's dd against standard benchmarks: since d=0.075<0.20d = 0.075 < 0.20, the standardized effect size is considered negligible.

  3. Compare the physical magnitude against the perceptual threshold: the reduction of 15.0 ms15.0\text{ ms} is far below the 100 ms100\text{ ms} human detection limit.

  4. Conclude that the caching redesign has no practical significance for user experience.

Final answer

(a) z=7.50>3.090z = 7.50 > 3.090, reject H0H_0; the decrease is statistically significant. (b) Cohen's d=0.075<0.20d = 0.075 < 0.20 and 15.0 ms<100 ms15.0\text{ ms} < 100\text{ ms}; the redesign lacks practical significance. (c) Net annual balance is −$4,500-\text{\char36}4{,}500; the redesign lacks practical economic significance.

NEXT STEPS
Final Takeaways from Your Teacher
  • Memorize the structural relationship: z=d⋅nz = d \cdot \sqrt{n}, reminding you that dd divides by ss, while zz divides by SESE.

  • Practice separating statistical evidence (relying on sample size nn) from real-world utility (benchmarked against thresholds or costs).

OverviewACEpath Teacher's Diagnosis

02 The student omits dividing the standard deviation by the square root of n when computing the test statistic, leading to an incorrect failure to reject the null hypothesis.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    z=15.0200.0=0.075  ⟹  0.075<3.090z = \frac{15.0}{200.0} = 0.075 \implies 0.075 < 3.090, so fail to reject H0H_0; the decrease is not statistically significant.

    Correct

    SE=sn=200.010,000=2.0 ms  ⟹  z=15.0−02.0=7.50SE = \frac{s}{\sqrt{n}} = \frac{200.0}{\sqrt{10{,}000}} = 2.0\text{ ms} \implies z = \frac{15.0 - 0}{2.0} = 7.50. Since z=7.50>3.090z = 7.50 > 3.090, reject H0H_0; the decrease is statistically significant.

    When testing a sample mean xˉ\bar{x}, the denominator of the test statistic is the standard error SE=snSE = \frac{s}{\sqrt{n}}, not the sample standard deviation ss. With n=10,000n = 10{,}000, n=100\sqrt{n} = 100, giving SE=2.0 msSE = 2.0\text{ ms} and z=7.50z = 7.50.

  2. Step 2

    d=15.0200.0=0.075  ⟹  dd = \frac{15.0}{200.0} = 0.075 \implies d is very small and 15 ms<100 ms15\text{ ms} < 100\text{ ms} threshold, so it lacks practical perceptual significance.

  3. Step 3

    Revenue =15.0×500=7500  ⟹  = 15.0 \times 500 = 7500 \implies Net =7500−12000=−4500  ⟹  = 7500 - 12000 = -4500 \implies The project loses $4,500\text{\char36}4{,}500 per year, so it has no practical economic significance.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly computed Cohen's dd to assess the standardized effect size.

You correctly compared the absolute reduction against the domain perceptual threshold of 100 ms100\text{ ms}.

You correctly carried out the annual cost-benefit calculation to evaluate practical economic significance.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Standard Error vs Standard Deviation Confusion

no

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

You confused the formula for the test statistic zz with the formula for standardized effect size (Cohen's dd). A test statistic evaluates sampling error and requires dividing by the standard error of the mean SE=snSE = \frac{s}{\sqrt{n}}, whereas Cohen's dd measures population spread using ss.

REASONING PATH
Your ACEpath teacher’s suggested approach
Standard Error for Sampling Distribution of the Mean
  • SE=snSE = \frac{s}{\sqrt{n}} measures variability of sample means, whereas ss measures variability among individual observations.

  • Before calculating zz, always divide the standard deviation ss by n\sqrt{n} when testing a sample mean.

Test Statistic vs. Effect Size
  • z=xˉ−μ0s/nz = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} grows with n\sqrt{n}, while Cohen's d=xˉ−μ0sd = \frac{\bar{x} - \mu_0}{s} is independent of nn.

  • Check your denominator: use SE=snSE = \frac{s}{\sqrt{n}} for inferential hypothesis tests, and use ss for Cohen's dd effect sizes.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Calculate the standard error of the mean: SE=sn=200.010,000=200.0100=2.0 msSE = \frac{s}{\sqrt{n}} = \frac{200.0}{\sqrt{10{,}000}} = \frac{200.0}{100} = 2.0\text{ ms}.

  2. Compute the zz-test statistic: z=xˉ−μ0SE=15.0−02.0=7.50z = \frac{\bar{x} - \mu_0}{SE} = \frac{15.0 - 0}{2.0} = 7.50.

  3. Compare the test statistic to the critical value: because z=7.50>3.090z = 7.50 > 3.090 (p<10−13<0.001p < 10^{-13} < 0.001), reject H0H_0.

Final answer

(a) The decrease in load time is statistically significant (z=7.50>3.090z = 7.50 > 3.090, reject H0H_0). (b) Lacks practical perceptual significance (d=0.075<0.20d = 0.075 < 0.20 and 15 ms<100 ms15\text{ ms} < 100\text{ ms}). (c) Lacks practical economic significance (net annual loss of $4,500\text{\char36}4{,}500).

NEXT STEPS
Final Takeaways from Your Teacher
  • Review the distinction between the standard deviation of individual observations (ss) and the standard error of the sample mean (s/ns/\sqrt{n}).

  • Practice calculating zz-statistics and Cohen's dd side-by-side to reinforce when to divide by n\sqrt{n}.

OverviewACEpath Teacher's Diagnosis

03 The student ignores the ongoing server cost and evaluates economic significance based solely on gross revenue gains.

Sample Student Work

Reviewed by ACEpath Teacher
Step-by-Step Review
Walk Through It with Your Teacher
Partially correct
  1. Step 1

    SE=200.010000=2.0  ⟹  z=15.02.0=7.50  ⟹  7.50>3.090SE = \frac{200.0}{\sqrt{10000}} = 2.0 \implies z = \frac{15.0}{2.0} = 7.50 \implies 7.50 > 3.090, so reject H0H_0; the reduction is statistically significant.

  2. Step 2

    d=15.0200.0=0.075  ⟹  dd = \frac{15.0}{200.0} = 0.075 \implies d is small and 15 ms15\text{ ms} is below the 100 ms100\text{ ms} threshold, so it has no practical perceptual significance.

  3. Step 3

    Net gain=15.0×$500=$7,500  ⟹  \text{Net gain} = 15.0 \times \text{\char36}500 = \text{\char36}7{,}500 \implies Because it generates a positive monetary gain of $7,500\text{\char36}7{,}500 per year, it possesses practical economic significance.

    Correct

    Annual Revenue=15.0×$500=$7,500  ⟹  Net Annual Balance=$7,500−$12,000=−$4,500\text{Annual Revenue} = 15.0 \times \text{\char36}500 = \text{\char36}7{,}500 \implies \text{Net Annual Balance} = \text{\char36}7{,}500 - \text{\char36}12{,}000 = -\text{\char36}4{,}500. Because the project yields a net annual loss of $4,500\text{\char36}4{,}500, it lacks practical economic significance.

    You calculated the gross revenue gain of $7,500\text{\char36}7{,}500 correctly, but labeled it as net gain without subtracting the ongoing annual server cost of $12,000\text{\char36}12{,}000. Because expenses exceed revenue gains by $4,500\text{\char36}4{,}500 per year, the change is economically unviable.

WHAT YOU DID WELL
Compliments from Your Teacher

You correctly calculated the standard error and the test statistic for the hypothesis test.

You appropriately compared the test statistic to the critical value and drew the correct inference.

You accurately computed Cohen's dd and contextualized it with the human visual threshold.

THE ERROR
Where Your ACEpath Teacher Spotted the Mistake
Omission of Operating Costs in Cost-Benefit Analysis

You calculated gross revenue gain rather than net gain by omitting the $12,000\text{\char36}12{,}000 annual server cost.

WHY IT HAPPENED
ACEpath Teacher's Diagnosis

Confusing gross revenue gains with net profit when determining practical economic significance. An intervention only possesses positive economic significance if total financial benefits exceed total implementation and maintenance costs.

REASONING PATH
Your ACEpath teacher’s suggested approach
Evaluate Net Benefit Rather Than Gross Gain
  • Net Economic Outcome=Total Financial Benefits−Total Ongoing Costs\text{Net Economic Outcome} = \text{Total Financial Benefits} - \text{Total Ongoing Costs}

  • Check whether any ongoing operational, infrastructure, or maintenance costs must be subtracted from the projected revenue before assessing economic viability.

CORRECTED PATH
ACEpath Teacher's Demonstration
  1. Calculate the gross annual revenue gain: Revenue=15.0 ms×$500/ms=$7,500\text{Revenue} = 15.0\text{ ms} \times \text{\char36}500/\text{ms} = \text{\char36}7{,}500.

  2. Subtract the ongoing annual server operating cost: Net Balance=$7,500−$12,000=−$4,500\text{Net Balance} = \text{\char36}7{,}500 - \text{\char36}12{,}000 = -\text{\char36}4{,}500.

  3. Evaluate economic significance based on the net balance: because the intervention produces a net annual deficit of $4,500\text{\char36}4{,}500, the caching redesign lacks practical economic significance.

Final answer

(a) z=7.50>3.090z = 7.50 > 3.090, reject H0H_0; statistically significant. (b) Cohen's d=0.075<0.20d = 0.075 < 0.20 and 15.0 ms<100 ms15.0\text{ ms} < 100\text{ ms}; lacks practical perceptual significance. (c) Net annual balance =$7,500−$12,000=−$4,500= \text{\char36}7{,}500 - \text{\char36}12{,}000 = -\text{\char36}4{,}500; the redesign incurs an annual net loss of $4,500\text{\char36}4{,}500 and therefore lacks practical economic significance.

NEXT STEPS
Final Takeaways from Your Teacher
  • In multi-step business statistics problems, distinguish between gross revenue and net profit by identifying all recurring expenses mentioned in the prompt.

  • Remember that very large sample sizes can yield extreme statistical significance (p<0.001p < 0.001) even when practical effect size and economic returns are negligible or negative.

Lock it in

Similar question

01

A clinical study of n=2,500n = 2{,}500 patients finds that a new headache medication reduces pain duration by an average of xˉ=4.0 minutes\bar{x} = 4.0\text{ minutes} with s=50.0 minutess = 50.0\text{ minutes}. (a) Calculate the test statistic zz. (b) Calculate Cohen's dd. (c) Is the result statistically significant at α=0.05\alpha = 0.05 (z∗=1.96z^* = 1.96), and does it have practical significance if patients only perceive relief for differences ≥15 minutes\ge 15\text{ minutes}?

Answer

(a) SE=50.02,500=1.0 minSE = \frac{50.0}{\sqrt{2{,}500}} = 1.0\text{ min}, so z=4.01.0=4.00z = \frac{4.0}{1.0} = 4.00. (b) d=4.050.0=0.08d = \frac{4.0}{50.0} = 0.08. (c) Because z=4.00>1.96z = 4.00 > 1.96, it is statistically significant (p<0.001p < 0.001). However, with d=0.08<0.20d = 0.08 < 0.20 and 4.0 min<15 min4.0\text{ min} < 15\text{ min}, it lacks practical significance.

02

An e-commerce company tests a checkout button redesign on n=40,000n = 40{,}000 users. The average order completion time drops by xˉ=0.60 seconds\bar{x} = 0.60\text{ seconds} with s=30.0 secondss = 30.0\text{ seconds}. Each 1 second1\text{ second} reduction yields $10,000\text{\char36}10{,}000 in annual conversions, but maintaining the redesigned system costs $15,000\text{\char36}15{,}000 per year. Determine whether the intervention achieves: (a) statistical significance at α=0.01\alpha = 0.01 (z∗=2.326z^* = 2.326), and (b) practical economic significance.

Answer

(a) SE=30.040,000=30.0200=0.15 sSE = \frac{30.0}{\sqrt{40{,}000}} = \frac{30.0}{200} = 0.15\text{ s}; z=0.600.15=4.00>2.326z = \frac{0.60}{0.15} = 4.00 > 2.326, so it is statistically significant. (b) Annual gain =0.60×$10,000=$6,000= 0.60 \times \text{\char36}10{,}000 = \text{\char36}6{,}000; Net =$6,000−$15,000=−$9,000= \text{\char36}6{,}000 - \text{\char36}15{,}000 = -\text{\char36}9{,}000. It results in a net financial loss, so it lacks practical economic significance.

03

A software engineer argues: 'Our new algorithm achieved p<0.0001p < 0.0001 with z=6.2z = 6.2 over n=100,000n = 100{,}000 runs, which proves the effect is huge.' If the sample standard deviation is s=50 mss = 50\text{ ms}, calculate the raw effect size xˉ\bar{x} and Cohen's dd. Explain mathematically why the engineer's claim is flawed.

Answer

SE=50100,000≈0.1581 msSE = \frac{50}{\sqrt{100{,}000}} \approx 0.1581\text{ ms}. Raw difference xˉ=z×SE=6.2×0.1581≈0.98 ms\bar{x} = z \times SE = 6.2 \times 0.1581 \approx 0.98\text{ ms}. Cohen's d=xˉs=0.9850≈0.0196d = \frac{\bar{x}}{s} = \frac{0.98}{50} \approx 0.0196. The engineer conflates statistical significance with effect size: since z=dnz = d\sqrt{n}, a massive sample size (n=100,000n = 100{,}000) inflates zz even when the real-world effect (d≈0.02d \approx 0.02, xˉ<1 ms\bar{x} < 1\text{ ms}) is virtually nonexistent.

Loading workspace…