Pilot Testing Protocol

Origin. Clinical-trial design; A/B testing; staged rollout.

Mechanism. Buys information about an intervention at reduced scale before committing to full scale. Its validity rests on the pilot population and conditions being exchangeable with the target ones — which staged rollouts systematically violate, since pilots run on enthusiasts.

Procedure. Define the decision the pilot will settle and the threshold that settles it, before running. Define the stopping rule. Randomize if possible; if not, name the confounders and state what will be done about them. Pre-register the analysis.

Applies to. Reversible interventions with a measurable proximate outcome.

Limitations. Without a pre-declared threshold, the pilot produces data that confirms whatever the sponsor intended. Pilot-scale success is also not evidence of scale success where the mechanism is capacity- or network-dependent.