Write the decision brief before the message test

Record eleven fields before recruitment

A workable brief names: decision owner; decision date; available actions; target audience; exposure context; exact variants; primary contrast; primary outcome; guardrail outcomes; smallest difference that would change the decision; and evidence that would make the study inconclusive or unsafe to act on.

Add the recruitment source, assignment method, analysis population, exclusion rules, sample plan, stopping rule, and implementation constraints as attachments. The brief should be short enough to use in a review meeting and precise enough that a different analyst could tell whether the result answers the original question.

Separate the action threshold from statistical significance

A small difference can be statistically distinguishable yet too minor to justify production cost, legal review, or customer disruption. A large observed difference can remain too uncertain to act on. Define the smallest practically useful difference and the acceptable uncertainty before choosing a sample size.

Do not make a p-value the sole decision rule. Report estimates, uncertainty, assumptions, guardrails, data quality, and operational consequences together. If several outcomes or subgroups could become the headline, designate the primary one and describe how the rest will be treated.

Fictional worked example: define the decision first

This is a constructed teaching example, not an MTB study, survey finding, customer result, or completed field test. All counts, messages, costs, and decision rules below are fictional. The filled brief and calculation worksheet let you reproduce the reasoning without suggesting that people were recruited.

Imagine a workflow product that organizes inputs into a draft weekly report; a person must check and send it. Message A reads: “Organize your team’s weekly updates in one place.” Message B reads: “Turn team updates into a draft weekly report for you to check and send.” The illustration concerns English-speaking U.S. small-business operations managers only. A Canadian or French-language decision would require its own justified design; this example supplies neither.

The primary outcome is an accurate unaided restatement: the answer must identify both assembling a draft from updates and the remaining human check/send step. The question is “In your own words, what would this product do, and what would you still need to do?” A written codebook, masked condition labels, and a disagreement-resolution process belong in the real protocol. The separate guardrail flags an answer asserting that the product finishes and sends reports without a person.

Before looking at any outcomes, the fictional owner sets a conservative adoption rule: the lower bound of the two-sided 95% interval for B minus A must exceed +5 percentage points on accurate restatement, and the upper bound for additional false-automation beliefs must remain below +3 points. Both conditions and the operational checks must pass. These illustrative tolerances are business choices, not universal research standards. Failing them means retain A provisionally and revise or investigate B; it does not prove A is better or that either message is acceptable without factual review.

Bridge the brief to the field-size planner

For an operational illustration, enter 2 variants, 1 market, 1 mutually exclusive segment, and 400 usable completions per cell in the field-size and cost planner. Enter a 12% expected quality-exclusion rate, 65% incidence, 40% screener-start rate, 80% completion rate, and US$2.50 per field completion. The outputs are 2 cells, 800 usable completions, a 910-completion field target, 4,375 invitations, and US$2,275 indicative field cost.

The arithmetic is 800 ÷ 0.88, rounded up to 910; then 910 ÷ (0.40 × 0.65 × 0.80) = 4,375. Cost is 910 × US$2.50. This assumes that all field completions, including later quality exclusions, are billed. Recruitment, screening, translations, coding, taxes, and platform fees are not included. Vendor charging rules may differ.

The 400-per-arm base is chosen to make the worked arithmetic reproducible. It is not a validated sample-size recommendation or evidence that the +5-point adoption rule has adequate power. Before real recruitment, justify the sample and stopping rule against expected rates, the decision threshold, required power, and guardrail precision. Do not extend a study after seeing a near-miss simply to get a favorable result.

Reproduce the comparison: 220 of 400 versus 252 of 400

Enter 220 outcomes out of 400 for A and 252 out of 400 for B in the comparison calculator, with 95% confidence. A is 55%; B is 63%. B minus A is +8 percentage points. Relative lift is 8 ÷ 55 × 100, approximately +14.5%; it is not an additional 14.5 percentage points.

Using the disclosed independent-group normal approximation, the difference interval is approximately +1.2 to +14.8 points and the pooled two-sided test of equal rates gives p≈0.021. The interval’s lower bound exceeds zero, but it does not exceed the predeclared +5-point practical threshold. This example therefore does not establish a gain larger than +5 points under the chosen rule. The p-value tests equality of rates; it does not test whether the gain is large enough to adopt B.

Now check the fictional guardrail separately: 40 of 400 answers for A and 44 of 400 for B imply 10% versus 11%, or +1 point. Enter those counts in the same calculator. Its 95% difference interval is approximately −3.2 to +5.2 points. The upper bound exceeds the +3-point tolerance, so the illustration also fails to establish acceptable guardrail risk. A small observed increase alone is insufficient. These are separate outcome intervals, not a joint 95% guarantee.

Decision: do not adopt B on this evidence. Record that the practical-gain and guardrail requirements were not met; retain A provisionally while reviewing the false-automation problem and planning any follow-up before new data are collected. Do not write “B won,” “no difference,” or “A is proven safe.” Statistical uncertainty is only one limitation: a real opt-in sample would not establish population representativeness merely because assignment to A or B was randomized.

Carry the decision into an honest report

A suitable caption for this teaching material is: “Fictional calculation example; no respondents or fieldwork. Constructed accurate-restatement counts of 220/400 for A and 252/400 for B imply a +8-point difference, with an approximate 95% normal interval of +1.2 to +14.8 points. The predeclared illustrative adoption rule requires the lower bound to exceed +5; it does not. The separate false-automation guardrail is also unresolved. These figures explain a decision rule and are not empirical findings.”

The filled brief records the before-results plan. The companion calculation worksheet records the synthetic inputs, formulas, interpretations, and intentional missing operational evidence. Keep these roles separate rather than inserting a favorable result into the original plan. For real research, the reporting checklist additionally requires actual field dates, participant flow, exclusions, assignment checks, exact instrument, coding evidence, sponsor disclosure, and limitations; this illustration cannot supply those records.

Pretest the brief and the instrument

Ask a reviewer who was not involved in drafting to explain the planned decision from the brief. If they cannot identify which result changes which action, revise it. Then pretest the stimulus, questions, response options, logic, device presentation, and data export with people in scope.

Operational verification is separate from respondent pretesting. Confirm assignment ratios, event labels, timestamps, exclusion flags, and exports with synthetic records before live collection. A clear research question can still fail if the implementation cannot reproduce the planned contrast.

Failure modes and scope

A decision brief is not a guarantee against judgment errors. It makes them visible. Stop and revise when any of these patterns appears:

• The only available action is “launch the preferred version,” regardless of the outcome.
• The primary metric is chosen after viewing the strongest difference.
• The practical threshold is backfilled from the observed result.
• A guardrail is reported but cannot change the action.
• Variants, eligibility, or exclusions change without a dated version break.
• The owner treats an inconclusive result as evidence of equivalence.

Sources and scope

1. Two-sample proportions z-testStatsmodels

   Method reference for the pooled normal-approximation equality test used in the fictional walkthrough; it does not turn the practical threshold into a tested equality claim.

2. Standards and Guidelines for Statistical SurveysU.S. Office of Management and Budget

   Planning reference for defining objectives, decisions, variables, analyses, required precision, and pretesting before data collection.

3. Statistical Quality Standard A2U.S. Census Bureau

   Official framework for instrument requirements, pretesting, documentation, and functional verification.

4. Controlled Experiments on the Web: Survey and Practical GuideData Mining and Knowledge Discovery

   Peer-reviewed guidance on trustworthy experiments, defined evaluation criteria, guardrails, and practical interpretation.

5. Disclosure StandardsAmerican Association for Public Opinion Research

   Professional reporting reference for pre-specified methods, sample, measurement, weighting, and deviations.