A/B Decision Engine

Most A/B testing work stops at one question: is the p-value below 0.05? That's the obvious tool to build, and plenty of calculators already do it.

That question comes too late. Experiments mislead teams in three ways before significance even matters. A broken redirect skews the traffic split and quietly invalidates the test. Teams check the dashboard every day and stop at the first good-looking result. And an average can hide a change that helps one group while hurting another.

How strong is this? Each failure is well documented in experimentation research, and each one was reproduced here with a simulator whose true effect is known, so every method could be checked against the right answer.

An engine that runs four checks and returns one verdict: ship, kill or hold.

It detects a mismatched traffic split and finds which platform the imbalance comes from. It uses sequential testing, which stays valid however often you peek. It uses CUPED, which borrows pre-experiment data to cut noise. And an uplift model shows who the change actually worked for. It all runs behind a FastAPI service with a small dashboard.

• A single p-value verdictIt's the answer people expect, and the one that misleads them. The verdict only comes after the trust checks pass.
• A dashboard before the maths was provenThe first eight steps were statistics, each validated against simulated ground truth. The API and dashboard came last, so there was nothing to make look good until it was right.
• A full Qini curveA simpler ranking check confirms the uplift model orders users sensibly. Publication-grade evaluation would have taken longer than it would have changed any decision.

• Sequential testing came in conservative, at 1% rather than exactly 5%. That's safer but less sensitive, and a tuning parameter can trade some of it back.
• CUPED helps much less on yes-or-no outcomes like conversion than on revenue. That's a property of the method, so the tool says so rather than overselling it.

Put it in front of a team running real tests and track one thing: how many "ship" decisions it changes to "hold". That's the number that says it saved someone from a bad launch.