Message Test BenchAbout
Menu

Free comparison tool

Compare two message response rates.

Enter the outcomes and audience size for two independent variants. See the absolute difference and relative lift, with an uncertainty interval and two-sided test when counts support the normal approximation.

Version A
For example: accurate answers, signups, clicks, or purchases.
Version B
Use the same outcome definition and measurement window as A.

The observed difference is statistically distinguishable at the 95% level under this model.

Version A rate55.0%
Version B rate63.0%
Difference (B − A)+8.0 pts
Relative lift+14.5%
95% interval+1.2 pts to +14.8 pts
Two-sided p-value0.021

Does this clear your worthwhile-improvement target?

Your target remains unresolved: the interval includes differences at or below the target and above it.

Specify this target before examining results. Changing it after seeing results is exploratory. This check compares your target with the interval; the equal-rates p-value above does not test this target. It is not an adoption recommendation—study validity, guardrails and costs still matter.

Review summary before sharing

Copying writes the summary to your device clipboard only when you choose it. Review it before sharing; inputs are not added to the page URL or sent to our servers.

Interpretation guardrails
  • A statistically distinguishable difference does not establish practical importance, representativeness, or a valid causal effect without a sound design.

Model: two independent proportions; an unpooled normal interval for the percentage-point difference and a pooled two-sided z-test for equal rates. The interval and test are withheld when either group has fewer than 10 outcomes or 10 non-outcomes. This count guardrail does not validate the study design. Use predeclared outcomes and genuinely independent, preferably randomized groups.

Inputs stay in this browser tab and are not submitted or stored by the application.

Start with the percentage-point difference

If version A produces a 55% outcome rate and version B produces 63%, the absolute difference is eight percentage points. Relative lift expresses the same change against A’s baseline, but it can look dramatic when that baseline is small. Report both and keep the raw counts visible.

Use the interval, not the p-value alone

The interval shows the range of differences reasonably compatible with this normal-approximation model. The two-sided p-value is the probability of a test statistic at least as extreme as the observed one, in either direction, assuming the two underlying rates are equal and the model applies. It is not the probability that a message works. Neither measure tells you whether the difference is valuable enough to act on.

If either group has fewer than 10 outcomes or 10 non-outcomes, this calculator withholds the normal interval and test. The descriptive rates remain useful, but missing inference does not mean zero uncertainty or equal underlying rates. A different method suited to sparse counts and the study design is needed for inference.

Follow the fictional 220/400 versus 252/400 walkthrough to see why a statistically distinguishable +8-point result can still fail a predeclared adoption rule. The filled brief and calculation worksheet also show a separate guardrail outcome; no fieldwork occurred.

Design determines what you may claim

Random assignment supports a cleaner causal comparison between the tested variants. It does not make the recruited sample representative of a broader population. Define the outcome, analysis, exclusions, stopping rule, and minimum difference worth acting on before results are visible.

Method references

Statsmodels: two-sample proportions z-test — implementation reference for the simple pooled normal test.

NIST/SEMATECH: confidence intervals for proportions — notes the limitations of symmetric normal intervals for small samples or rare outcomes.