Resources
An operations research team analyzes website load times before and after a caching redesign. Across web page requests, the mean load time decreased by with a standard deviation . Simultaneously, financial modeling shows that the redesign requires an ongoing server cost of per year, while each reduction in page load time produces an estimated in annual revenue gains. (a) Test whether the decrease in load time is statistically significant at using a one-sample -test (). (b) Compute Cohen's and evaluate practical significance from a human perception standpoint, given that the human visual perceptual threshold for webpage latency differences is . (c) Couple this with the economic cost-benefit principle: calculate the net annual monetary gain/loss, and determine whether the intervention possesses practical economic significance.
Answer and steps
Final answer
(a) . Since (), reject ; the reduction is highly statistically significant. (b) Cohen's ; because and the absolute decrease () is well below the human sensory perception threshold, the result lacks practical behavioral significance for users. (c) Annual revenue gain ; Net annual balance (a net loss of per year). Therefore, the project also lacks practical economic significance, demonstrating that a statistically significant finding can still be economically detrimental.
Steps
Part (a): Hypothesis Test for Statistical Significance
-
State the hypotheses and parameters: Let represent the true mean reduction in webpage load time (in ).
- Null hypothesis: (the caching redesign has no effect on load time).
- Alternative hypothesis: (the caching redesign reduces load time).
- Compute the Standard Error (): Given sample size and sample standard deviation :
- Calculate the test statistic ():
- Compare with the critical value: For a one-tailed test at significance level , the critical value is . Since (which corresponds to ), we reject . The observed reduction in webpage load time is statistically significant.
Part (b): Practical Significance and Effect Size
- Calculate Cohen's : Cohen's measures the standardized mean difference:
-
Evaluate the effect size and practical relevance:
- By Cohen's conventional thresholds ( is small, is medium, and is large), a value of indicates an extremely small or negligible effect size.
- In domain terms, the physical reduction of is far below the human visual perceptual threshold of . Users cannot perceive a difference, meaning the redesign has no discernible effect on user experience. Therefore, it lacks practical behavioral significance.
Part (c): Economic Cost-Benefit Analysis
- Calculate the annual revenue gain:
- Calculate the net annual monetary outcome:
- Determine practical economic significance: The redesign results in a net loss of per year. Because it produces an unfavorable return on investment, the intervention lacks practical economic significance.
Key Takeaway and Common Pitfall
- Pitfall: Confusing statistical significance with practical significance.
- With a very large sample size (), the standard error becomes tiny (), causing even minute, trivial differences to generate an enormous test statistic (). While the data convincingly shows the difference is not zero (statistical significance), it remains practically unnoticeable to humans and financially detrimental to the business.
Typical errors
01 The student calculates Cohen's d using the standard error rather than the sample standard deviation, greatly overstating the standardized effect size.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
, so reject and the decrease is statistically significant.
- Step 2Correct
Cohen's measures the standardized effect size in units of the individual variability (standard deviation ), not the sample mean variability (standard error ). Dividing by mistakenly computes the test statistic rather than effect size .
- Step 3
Since is extremely large and is noticeable, it has practical significance.
CorrectBecause and the reduction is well below the human sensory threshold, the redesign lacks practical significance.
Because your calculation of was inflated and the problem noted that human visual latency detection requires at least , the true difference of is imperceptible to users and lacks practical significance.
- Step 4
There is an annual loss of , so it lacks practical economic significance.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly computed the standard error and the test statistic .
You properly compared the test statistic with the critical value to reject the null hypothesis.
You accurately calculated the annual revenue gain and net loss of to evaluate practical economic significance.
THE ERROR
Where Your ACEpath Teacher Spotted the Mistakeno
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou conflated the test statistic with the effect size Cohen's . With large sample sizes, becomes very small, making large even when the physical effect size is negligible.
REASONING PATH
Your ACEpath teacher’s suggested approach- Effect Size vs. Test Statistic
Cohen's , whereas .
Always divide by the standard deviation when calculating Cohen's to quantify practical magnitude independent of sample size.
- Perceptual and Domain Thresholds
Statistical significance indicates only that an effect is unlikely due to random chance (), not that it is meaningful in the real world.
Compare the raw difference (ar{x} = 15.0\text{ ms}) directly against domain-specific baselines (such as the given threshold).
CORRECTED PATH
ACEpath Teacher's DemonstrationCalculate Cohen's using the sample standard deviation: .
Evaluate Cohen's against standard benchmarks: since , the standardized effect size is considered negligible.
Compare the physical magnitude against the perceptual threshold: the reduction of is far below the human detection limit.
Conclude that the caching redesign has no practical significance for user experience.
(a) , reject ; the decrease is statistically significant. (b) Cohen's and ; the redesign lacks practical significance. (c) Net annual balance is ; the redesign lacks practical economic significance.
02 The student omits dividing the standard deviation by the square root of n when computing the test statistic, leading to an incorrect failure to reject the null hypothesis.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
, so fail to reject ; the decrease is not statistically significant.
Correct. Since , reject ; the decrease is statistically significant.
When testing a sample mean , the denominator of the test statistic is the standard error , not the sample standard deviation . With , , giving and .
- Step 2
is very small and threshold, so it lacks practical perceptual significance.
- Step 3
Revenue Net The project loses per year, so it has no practical economic significance.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly computed Cohen's to assess the standardized effect size.
You correctly compared the absolute reduction against the domain perceptual threshold of .
You correctly carried out the annual cost-benefit calculation to evaluate practical economic significance.
THE ERROR
Where Your ACEpath Teacher Spotted the Mistakeno
WHY IT HAPPENED
ACEpath Teacher's DiagnosisYou confused the formula for the test statistic with the formula for standardized effect size (Cohen's ). A test statistic evaluates sampling error and requires dividing by the standard error of the mean , whereas Cohen's measures population spread using .
REASONING PATH
Your ACEpath teacher’s suggested approach- Standard Error for Sampling Distribution of the Mean
measures variability of sample means, whereas measures variability among individual observations.
Before calculating , always divide the standard deviation by when testing a sample mean.
- Test Statistic vs. Effect Size
grows with , while Cohen's is independent of .
Check your denominator: use for inferential hypothesis tests, and use for Cohen's effect sizes.
CORRECTED PATH
ACEpath Teacher's DemonstrationCalculate the standard error of the mean: .
Compute the -test statistic: .
Compare the test statistic to the critical value: because (), reject .
(a) The decrease in load time is statistically significant (, reject ). (b) Lacks practical perceptual significance ( and ). (c) Lacks practical economic significance (net annual loss of ).
03 The student ignores the ongoing server cost and evaluates economic significance based solely on gross revenue gains.
Sample Student Work
Reviewed by ACEpath TeacherStep-by-Step Review
Walk Through It with Your Teacher- Step 1
, so reject ; the reduction is statistically significant.
- Step 2
is small and is below the threshold, so it has no practical perceptual significance.
- Step 3
Because it generates a positive monetary gain of per year, it possesses practical economic significance.
Correct. Because the project yields a net annual loss of , it lacks practical economic significance.
You calculated the gross revenue gain of correctly, but labeled it as net gain without subtracting the ongoing annual server cost of . Because expenses exceed revenue gains by per year, the change is economically unviable.
WHAT YOU DID WELL
Compliments from Your TeacherYou correctly calculated the standard error and the test statistic for the hypothesis test.
You appropriately compared the test statistic to the critical value and drew the correct inference.
You accurately computed Cohen's and contextualized it with the human visual threshold.
THE ERROR
Where Your ACEpath Teacher Spotted the MistakeYou calculated gross revenue gain rather than net gain by omitting the annual server cost.
WHY IT HAPPENED
ACEpath Teacher's DiagnosisConfusing gross revenue gains with net profit when determining practical economic significance. An intervention only possesses positive economic significance if total financial benefits exceed total implementation and maintenance costs.
REASONING PATH
Your ACEpath teacher’s suggested approach- Evaluate Net Benefit Rather Than Gross Gain
Check whether any ongoing operational, infrastructure, or maintenance costs must be subtracted from the projected revenue before assessing economic viability.
CORRECTED PATH
ACEpath Teacher's DemonstrationCalculate the gross annual revenue gain: .
Subtract the ongoing annual server operating cost: .
Evaluate economic significance based on the net balance: because the intervention produces a net annual deficit of , the caching redesign lacks practical economic significance.
(a) , reject ; statistically significant. (b) Cohen's and ; lacks practical perceptual significance. (c) Net annual balance ; the redesign incurs an annual net loss of and therefore lacks practical economic significance.
Lock it in
What is the defining formula for Cohen's in a one-sample context, and which spread metric does its denominator use?
. The denominator is the sample standard deviation (), never the standard error ().
Similar question
A clinical study of patients finds that a new headache medication reduces pain duration by an average of with . (a) Calculate the test statistic . (b) Calculate Cohen's . (c) Is the result statistically significant at (), and does it have practical significance if patients only perceive relief for differences ?
Answer
(a) , so . (b) . (c) Because , it is statistically significant (). However, with and , it lacks practical significance.
An e-commerce company tests a checkout button redesign on users. The average order completion time drops by with . Each reduction yields in annual conversions, but maintaining the redesigned system costs per year. Determine whether the intervention achieves: (a) statistical significance at (), and (b) practical economic significance.
Answer
(a) ; , so it is statistically significant. (b) Annual gain ; Net . It results in a net financial loss, so it lacks practical economic significance.
A software engineer argues: 'Our new algorithm achieved with over runs, which proves the effect is huge.' If the sample standard deviation is , calculate the raw effect size and Cohen's . Explain mathematically why the engineer's claim is flawed.
Answer
. Raw difference . Cohen's . The engineer conflates statistical significance with effect size: since , a massive sample size () inflates even when the real-world effect (, ) is virtually nonexistent.