Test budgets, breakeven and the kill rule — Learn

What this lesson teaches

Set a test budget from breakeven CPA rather than from what you can spare
Compute the probability of seeing no sale when a product is actually fine
Choose a stopping rule and state its error rate rather than pretending it has none

A test budget answers one question: how much am I willing to pay for this answer? Set it from the economics, in advance, and stop when it is spent — not when it feels like enough.

The common rule, stated properly

A widely used stopping rule is: spend up to a fraction of the breakeven acquisition cost; if no sale has occurred by then, stop. With a breakeven CPA of £18.82 and a fraction of two-thirds, that is £12.55 of spend on a single product-audience pair before a decision. The rule is sound as cost control. It is important to be precise about what it proves, because it is often described as though it proved more.

What a no-sale result actually implies

Assume orders arrive independently (Poisson) and that the true
CPA equals breakeven, £18.82 — i.e. the product is exactly
marginal, not bad.

Spend £12.55  → expected orders = 12.55/18.82 = 0.667
  P(no sale)  = e^−0.667 = 51.3%

Spend £18.82  → expected orders = 1.00
  P(no sale)  = e^−1.00  = 36.8%

Spend £37.64  → expected orders = 2.00
  P(no sale)  = e^−2.00  = 13.5%

So a no-sale outcome at two-thirds of breakeven happens about half the time even when the product is exactly at breakeven. The rule is therefore not a significance test; it is a budget cap with roughly a coin-flip false-kill rate at the margin. That is a defensible choice — when candidates are plentiful and cheap to test, killing marginal ones quickly is correct even at a 50% error rate, because the cost of a wrong kill is one lost marginal product and the cost of hesitation is spread across every other candidate you did not get to. It is indefensible when candidates are scarce, when each one costs real money to set up, or when the same rule is then reported as evidence the product does not work.

Choosing a fraction on purpose

False-kill probability for a product whose true CPA equals breakeven.
Spend before kill	Expected orders	P(no sale) — false kill
⅓ of breakeven CPA	0.33	71.7%
⅔ of breakeven CPA	0.67	51.3%
1 × breakeven CPA	1.00	36.8%
2 × breakeven CPA	2.00	13.5%
3 × breakeven CPA	3.00	5.0%

Three times breakeven CPA is where a no-sale result reaches the conventional 5% threshold. Whether that is worth £56.46 on a single pair depends entirely on how many pairs you intend to test and what each setup costs. The point is to choose the row deliberately and know its error rate, rather than to inherit a fraction from someone whose situation was not yours.

One variable per test

If a test changes the creative, the audience and the landing page at once, a failure identifies nothing and a success cannot be repeated. Hold everything but one thing still — and accept that this makes testing slower, because the alternative is that it makes testing meaningless.

Figures in this lesson are illustrative inputs chosen so the arithmetic can be checked. They are not measurements. What FlowFinds Solutions actually measures is published, with its artifacts, in research.