Measurement

The product is the instrument.

Every email tool will tell you what customers spent after you emailed them. Unlapse also tells you what they would have spent anyway, because five in every hundred never get its emails. It reports the difference whatever it says, including when the answer is nothing.

The holdout

Five of every hundred never get our emails.

Chosen by a stable hash of the customer id, so the same people are held back every night, forever. Both arms are recorded and both enter the same 21 day cooldown, so nothing about the comparison drifts. Lift is then a subtraction, not a story: what the emailed group spent, minus what the held-back group spent anyway, with a confidence interval, on your own orders.

5.03%

of send records were the held-back control on the first brand

3,929

held-back records out of 78,079, 6 Jul to 21 Sep 2026

43,151

distinct customers over 78 send days

customers re-sent inside 21 days since the send guards shipped on 7 Sep 2026

One hundred dots, five of them grey: the customers who never get our emails.

What the holdout says on the first brand, today

The measured lift is not settled, and Unlapse says so. Every read at 14 days and longer since August has been not distinguishable from zero in aggregate: the engaged base mostly buys anyway, and until a deconfliction segment went live in September the control was contaminated by the brand's own campaigns. A clean read restarted on 14 Sep 2026. We quote no lift figure as a result. The claim is the instrument, and that it reports “nothing” when the answer is nothing.

Calibrated probabilities

The highest score anywhere in the system is 0.598. Nothing reads 99 percent.

Every campaign score is a calibrated probability of that campaign's action within about 90 days, so a 0.30 reorder and a 0.30 subscription mean the same thing and can be ranked against each other. Before calibration the subscription score read 0.99; the honest rate was about 0.056. Where the data runs out the score is clipped to the highest observed rate, never extrapolated, and a category with fewer than 300 samples or 30 positives gets a flat base rate instead of a fitted curve.

Campaign	Held-out ECE	Customers
Dormant	0.0005	296,438
Reactivation	0.0016	65,135
Win-back	0.0063	28,040
Resubscribe	0.0099	12,404
Second order	0.0205	7,169

Expected calibration error on a held-out set, from the calibration fit of July 2026 on an 89 day holdout. Lower is better.

Campaign	Highest calibrated score
Second order	0.598
Resubscribe	0.439
Reorder	0.404
Win-back	0.311
Cross-sell	0.186
Reactivation	0.164
Subscription	0.089
Dormant	0.058

The best customer in each campaign on 21 Sep 2026, as a probability within about 90 days.

Walk-forward grading, every night

Each model is graded against the rule it replaced. The table records the losses.

The model being measured trains on an earlier window and is scored on a later one it never saw. Two of the three beat their baselines by 38x and 27x. The cross-sell model currently loses to the co-purchase rule at the top of the list, and the engine reports that every night.

Model	ROC-AUC	Customers	Base rate	Top 1,000	Baseline, top 1,000
Reorder, buys within 60 days	0.7835	302,671	0.84%	32.5%	0.84%
Subscription, starts within 90 days	0.7008	302,169	0.51%	8.2%	0.3% (heuristic)
Cross-sell, new product within 90 days	0.5849	50,767	14.0%	15.5%loses	20.9% (co-purchase rule)

21 Sep 2026; the figures move a little every night. Further down the lists: reorder top 5,000 16.5% and top 10,000 10.4%; subscription top 500 8.8% against 0.4%; cross-sell top 10,000 18.6% against 18.7%, a tie.

The models are given a budget and mostly refuse to spend it. The reorder model may nominate 15,000 candidates a night and the subscription model 5,000; on 21 Sep 2026 648 customers were assigned the reorder campaign and 379 the subscription one, because every nominee still has to clear the segment, cadence and product gates.

The rule-based lapsed tiers are graded separately, on an 89 day held-out window rebuilt from order history alone: win-back, reactivation and dormant customers came back at 5.37%, 2.10% and 0.81%.

Attribution, labelled

Attribution at 2, 7 and 14 days, beside the holdout.

Every send is attributed nightly at 2, 7 and 14 day windows, giveaway-only orders excluded, each window marked matured or still open. It is the number you can compare with your other tools, so we compute it. It is labelled “attributed” everywhere, never “generated”, because attribution flatters: on the first brand, a subscription-heavy one, roughly three quarters of 14-day attributed revenue came from subscription renewals that process in a nightly batch, and the held-back group renewed at the same rate per person.

Attributed

Orders placed within the window after a send. The industry-standard number. Not causation.

Measured against the holdout

What the emailed group did minus what the held-back group did anyway, with a confidence interval. The only number that says what the emails caused.

Head to head

Against the incumbent retention tool, including the two comparisons that were not wins.

Graded leak-free on the first brand, June to July 2026, on orders that happened after both systems made their calls. Publishing only the wins would be marketing.

Comparison	Unlapse	Incumbent	Result
Reorder, ROC-AUC	0.6517	0.4645	Win
Reorder, precision at top 500	54.8%	29.0%	Win
Reorder, median predicted-date error (days)	12.2	29.3	Win
Win-back, ROC-AUC	0.7374	0.4766	Win
Win-back, precision at top 1,000	39.0%	18.2%	Win
Cross-sell, field comparison	0.451	0.481	Tie
Subscription ranking	21 starts	21 starts	Inconclusive

The reorder difference is 0.187 with a bootstrapped 95 percent interval of 0.167 to 0.208, which excludes zero. The incumbent's reorder score was worse than a coin flip on the same customers. The subscription comparison rested on 21 starts, too few to call.

about 1.5x

Order rate of the engine's un-emailed picks (13.0 / 12.7 / 12.0 / 11.5 percent on four dates) against the incumbent's actually emailed list (8.8 / 8.4 / 8.0 / 8.1 percent). Matched budget, subscribers stripped.

56%

of the incumbent's "upgrade to subscription" list were already active subscribers. Only about 23% were genuine candidates.

0% vs 31%

of offers that were free-sample products: the engine's against the incumbent's.

All figures are from the first brand the engine runs for, a subscription-driven DTC brand. Model figures are one night's snapshot, 21 Sep 2026, unless another date is given.

Tell us about your store.

Ask us to show you the holdout report on your own orders after the audit.

Request a demo or book a call