What p-value should I use for A/B tests?

p < 0.05 (95% confidence) is the standard threshold for most product experiments. Financial or medical decisions warrant p < 0.01. Using a looser threshold increases the risk of shipping ineffective changes.

How long should I run an A/B test?

Until you reach the pre-calculated sample size for your desired statistical power (typically 80%). Stopping early when results look positive — called 'peeking' — inflates false positives significantly.

What is the difference between statistical and practical significance?

Statistical significance means the result is unlikely due to chance. Practical significance means the effect size is large enough to matter to your business. A test can be statistically significant but practically irrelevant if the lift is 0.01%.

A/B Test Calculator | DevTools Surf

DevTools Surf

Example: A/B Test Calculator

A worked example, rendered from real sample data. Sign in to run the tool on your own input.

Input

450, 1000
520, 1050

Output

═══ Verdict ═══
✓ Variant A beats Control: 10.05% lift, two-tailed p = 0.040307 < α = 0.0500
ℹ Single comparison at α = 0.050
⚠ Current power for the observed effect is only 53% — this test is underpowered, so a null result is not evidence of no effect
✓ Traffic split looks consistent with an even allocation

═══ Groups ═══
Control: 450/1,000 = 45.000% (CI 41.94%–48.10%)
Variant A: 520/1,050 = 49.524% (CI 46.51%–52.54%)
Intervals are Wilson score intervals, which stay valid at low conversion counts.

═══ Comparisons ═══
─── Variant A vs Control ───
Absolute difference: 4.524% (CI 0.205% to 8.843%)
Relative lift: 10.05% (CI 0.40% to 20.63%)
z: 2.0506
p-value (two-tailed): 0.040307
Significant at α 0.0500: yes

═══ Practical Significance ═══
ℹ Variant A: plausible true lift runs from 0.40% to 20.63% — decide whether the low end is still worth the change
Statistical significance says the difference is probably not zero. Practical significance asks whether it is big enough to act on. They are not the same question.

═══ Sample Size and Power ═══
Variant A: 1,911 per group needed for 80% power at this effect; you have 1,000 (52%), current power 53%
Required n is computed for the effect you actually observed, which is itself noisy — treat it as a rough guide.

═══ Stopping Rules and Peeking ═══
⚠ Checking the dashboard repeatedly and stopping when p first drops below 0.05 inflates the false-positive rate well past 5% — often to 20-30%.
Decide the sample size in advance, run to it, then look once.
If you must monitor mid-flight, use a sequential method (alpha spending, group seque
…

About A/B Test Calculator

A/B Test Calculator preview - Statistics tool

Calculate A/B test results and statistical significance. Part of the DevTools Surf developer suite. Browse more tools in the Statistics collection.

Use Cases

Determine whether a landing page headline change lifts conversions
Validate checkout flow redesigns before full rollout
Measure statistical significance of email subject line experiments
Evaluate feature flag impact on retention metrics

Tips

Enter raw visitor and conversion counts — not rates — for the most accurate significance calculation
Check the confidence level selector: 95% is the industry standard, but safety-critical tests should use 99%
Run the test until the recommended sample size is reached before reading the result; peeking early inflates false positives

Fun Facts

A/B testing at scale was pioneered by Google in 2000 when they tested 41 shades of blue to optimize ad click-through rates — a decision now famous in UX lore.
The statistical test underlying most A/B tools is a two-proportion z-test, first described by Karl Pearson in the late 1800s — long before the web existed.
Microsoft's experimentation platform runs over 10,000 A/B tests simultaneously across its products, making it one of the largest continuous testing operations in the world.

FAQ

What p-value should I use for A/B tests?: p < 0.05 (95% confidence) is the standard threshold for most product experiments. Financial or medical decisions warrant p < 0.01. Using a looser threshold increases the risk of shipping ineffective changes.
How long should I run an A/B test?: Until you reach the pre-calculated sample size for your desired statistical power (typically 80%). Stopping early when results look positive — called 'peeking' — inflates false positives significantly.
What is the difference between statistical and practical significance?: Statistical significance means the result is unlikely due to chance. Practical significance means the effect size is large enough to matter to your business. A test can be statistically significant but practically irrelevant if the lift is 0.01%.

Related Statistics Tools

Statistical Significance Tester Confidence Interval Calculator Sample Size Calculator Chi-Square Tester T-Test Calculator Correlation Analyzer Regression Analyzer Distribution Analyzer