Write the decision brief before the message test

Record eleven fields before recruitment

A workable brief names: decision owner; decision date; available actions; target audience; exposure context; exact variants; primary contrast; primary outcome; guardrail outcomes; smallest difference that would change the decision; and evidence that would make the study inconclusive or unsafe to act on.

Add the recruitment source, assignment method, analysis population, exclusion rules, sample plan, stopping rule, and implementation constraints as attachments. The brief should be short enough to use in a review meeting and precise enough that a different analyst could tell whether the result answers the original question.

Separate the action threshold from statistical significance

A small difference can be statistically distinguishable yet too minor to justify production cost, legal review, or customer disruption. A large observed difference can remain too uncertain to act on. Define the smallest practically useful difference and the acceptable uncertainty before choosing a sample size.

Do not make a p-value the sole decision rule. Report estimates, uncertainty, assumptions, guardrails, data quality, and operational consequences together. If several outcomes or subgroups could become the headline, designate the primary one and describe how the rest will be treated.

Worked example: clarity versus false automation

A fictional workflow product must choose between message A and message B for a landing page aimed at U.S. and Canadian small-business operations managers. The owner can keep A, adopt B, revise both, or run follow-up research. The primary outcome is accurate unaided restatement of the main function; the guardrail is the false belief that the product completes reporting automatically.

The brief says B will replace A only if its accurate-restatement estimate improves by at least the predeclared practical threshold, the interval is sufficiently informative for the decision, the false-automation estimate does not worsen beyond its guardrail, and no material data-quality failure occurs. A favorable click-intent rating cannot override a failed comprehension guardrail.

If the primary result is uncertain, the team records “no decision from this study” instead of selecting the numerically higher message. If both messages create the false takeaway, both are revised. These outcomes are useful because the action paths existed before results were visible.

Pretest the brief and the instrument

Ask a reviewer who was not involved in drafting to explain the planned decision from the brief. If they cannot identify which result changes which action, revise it. Then pretest the stimulus, questions, response options, logic, device presentation, and data export with people in scope.

Operational verification is separate from respondent pretesting. Confirm assignment ratios, event labels, timestamps, exclusion flags, and exports with synthetic records before live collection. A clear research question can still fail if the implementation cannot reproduce the planned contrast.

Failure modes and scope

A decision brief is not a guarantee against judgment errors. It makes them visible. Stop and revise when any of these patterns appears:

• The only available action is “launch the preferred version,” regardless of the outcome.
• The primary metric is chosen after viewing the strongest difference.
• The practical threshold is backfilled from the observed result.
• A guardrail is reported but cannot change the action.
• Variants, eligibility, or exclusions change without a dated version break.
• The owner treats an inconclusive result as evidence of equivalence.

Sources and scope

1. Standards and Guidelines for Statistical SurveysU.S. Office of Management and Budget

   Planning reference for defining objectives, decisions, variables, analyses, required precision, and pretesting before data collection.

2. Statistical Quality Standard A2U.S. Census Bureau

   Official framework for instrument requirements, pretesting, documentation, and functional verification.

3. Controlled Experiments on the Web: Survey and Practical GuideData Mining and Knowledge Discovery

   Peer-reviewed guidance on trustworthy experiments, defined evaluation criteria, guardrails, and practical interpretation.

4. Disclosure StandardsAmerican Association for Public Opinion Research

   Professional reporting reference for pre-specified methods, sample, measurement, weighting, and deviations.

Sources support the specific statements described above; they do not validate this publication’s rubric, guarantee a compliant execution, or replace context-specific professional advice.