Message Test BenchAbout
Menu

Free comparison tool

Compare two message response rates.

Enter the outcomes and audience size for two independent variants. See the absolute difference, relative lift, uncertainty interval, and two-sided significance test in one place.

Version A
For example: accurate answers, signups, clicks, or purchases.
Version B
Use the same outcome definition and measurement window as A.

The observed difference is statistically distinguishable at the 95% level under this model.

Version A rate55.0%
Version B rate63.0%
Difference (B − A)+8.0 pts
Relative lift+14.5%
95% interval+1.2 pts to +14.8 pts
Two-sided p-value0.021
Interpretation guardrails
  • A statistically distinguishable difference does not establish practical importance, representativeness, or a valid causal effect without a sound design.

Model: two independent proportions; an unpooled normal interval for the percentage-point difference and a pooled two-sided z-test for equal rates. Use predeclared outcomes and genuinely independent, preferably randomized groups.

Inputs stay in this browser tab and are not submitted or stored by the application.

Start with the percentage-point difference

If version A produces a 55% outcome rate and version B produces 63%, the absolute difference is eight percentage points. Relative lift expresses the same change against A’s baseline, but it can look dramatic when that baseline is small. Report both and keep the raw counts visible.

Use the interval, not the p-value alone

The interval shows the range of differences reasonably compatible with this normal-approximation model. The p-value asks how surprising the observed difference would be if the two underlying rates were equal. Neither tells you whether the difference is valuable enough to act on.

Design determines what you may claim

Random assignment supports a cleaner causal comparison between the tested variants. It does not make the recruited sample representative of a broader population. Define the outcome, analysis, exclusions, stopping rule, and minimum difference worth acting on before results are visible.

Method references

Statsmodels: two-sample proportions z-test — implementation reference for the simple pooled normal test.

NIST/SEMATECH: confidence intervals for proportions — notes the limitations of symmetric normal intervals for small samples or rare outcomes.