The statistics engine
Most experimentation dashboards quietly encourage the two classic sins: peeking at fixed-horizon p-values, and trusting a broken split. ShiftZero's engine is built so the honest reading is the default one.
Attribution first
Before any math, exposures and conversions are joined with explicit rules:
- First exposure wins — a visitor's arm is the first one they were served
- Exposure-before-conversion ordering — conversions can't precede the exposure they credit
- A configurable attribution window (default 30 days)
- Experiment scoping — conversions stamped by the track bridge with the visitor’s assignments are credited only to those experiments, eliminating cross-experiment contamination
Two inference regimes, labelled
Live experiments are scored with always-valid confidence sequences (asymptotic, Waudby-Smith & Ramdas) — intervals that hold uniformly over time, so you can check after every visitor and stop the moment the interval excludes zero. Settled experiments get the fixed-horizon analysis (two-proportion z-test for rates, Welch t for counts and values), which is the correct final read. The scoreboard always names the regime it’s showing — a peek-safe badge on live views, fixed-horizon on final ones.
Precision & robustness
| Technique | What it buys you |
|---|---|
| CUPED | Each user's pre-exposure behaviour is regressed out (pooled θ). Same effect estimate, meaningfully less variance — experiments conclude sooner. On by default for count/value metrics. |
| Winsorization | Value metrics capped at a pooled percentile (default p99.5) so one whale order can't flip a revenue test. Raw totals stay visible. |
| Holm correction | A/B/n: adjusted p-values across treatment arms keep the promised family-wise error rate. |
| SRM guardrail | A chi-square check of observed vs configured split. On mismatch the whole readout is flagged before any winner claim — a broken randomiser invalidates everything downstream. |
| Bayesian companion | Beta/normal posteriors → probability-to-beat-control and expected relative loss if shipped. Decision-friendly framing; the frequentist verdict stays the formal one. |
Segmentation
Results slice by device, country, and path — each slice is a full re-analysis with its own SRM check and intervals, not a filtered summary. Slices are defined by the visitor’s first exposure, keeping them stable as users move around.