A/B testing is a lie detector, not a decision engine.
Five things a senior PM should be able to explain in a launch review without opening a stats textbook.
This week, the piece I would want to hand a PM the first time they own an experiment on something real.
Every PM I know has shipped a change based on a test that did not actually prove what they thought it proved. I have done it. I have watched a team celebrate a two percent lift that vanished the next quarter. I have sat in a review where a variant got killed because the presenter could not answer one question about sample size. I have approved something based on a test that peeked five times before it ended, which meant the “significant” result had roughly the same evidentiary weight as a coin flip.
I run experiments on a platform used by Microsoft employees, customers, partners, and technical communities. A lot of what we ship touches labs, assessments, content workflows, and the tooling that keeps a large catalog alive. The traffic is real, the stakes are real, and the temptation to declare victory on a clean chart is real. This is the field guide I wish somebody had given me at early levels. Five things a senior PM should be able to explain in a launch review without opening a stats textbook.
1. What you are actually buying when you run a test
The reason to run an experiment is not to prove your idea works. It is to rule out the alternative explanations for why the number moved.
Say you are running a test on a labs discovery page. You change the layout on Tuesday, and by Friday, lab starts are up three percent. You want to say the new layout caused the lift. You cannot. You just changed the layout the same week a partner ran a big skilling push, the same week a competing platform had an outage, and the same week a top acquisition surface shifted its mix. Any of those could be the story. Correlation is easy to see. Causation is the thing you have to earn.
An A/B test earns causation by holding everything else equal. Randomize users into two groups, show one the new layout and the other the old one, run both at the same time to the same audience. Now the partner push, the outage, the acquisition shift, all of it hits both groups equally. If the treatment group starts more labs, the treatment is the most defensible explanation left standing.
That is the whole product. Everything else in this issue is about the ways that machine breaks. Three risks show up most often. Confounders: something other than your change is correlated with the treatment (you shipped the flow to one region first, so you tested geography, not the flow). Selection bias: users in each bucket are not comparable (you let people opt in, so you tested motivation, not the feature). Bad randomization: the assignment logic is broken in a way you did not check.
If you cannot name which of these three risks your test is protecting against, you are not running an experiment. You are running a slower rollout with a chart on top.
2. Hypothesis, metric, guardrail. In that order.
Before the code goes anywhere near the assignment logic, write three lines. Hypothesis: if we change X, we expect Y will move in direction Z, because of mechanism M. Primary metric: the single number that decides the test. One. Not three. Guardrails: the numbers that must not get worse for the launch to be a good idea even if the primary moves.
The guardrails are where the discipline slips. Say you are testing a change to an AI-assisted authoring step in a content workflow. Your primary metric is “time to draft ready.” It drops thirty percent. Wonderful. Meanwhile the downstream editorial rework rate quietly climbs, because the drafts you shipped faster were half-baked. You did not win. You moved a problem downstream.
Pick the primary metric that most tightly matches the causal claim. If the mechanism is “reduce friction at the first-lab-start step,” measure lab starts, not total learning hours. Total hours are downstream of a hundred things you are not testing. Choose a metric that a competent teammate could not argue moved for the wrong reason. That is the whole game.
3. The statistics you cannot skip
You do not need to be a statistician. You do need to explain four ideas without hedging.
P-value. The chance you would see a result at least this large if the treatment did nothing. A p-value of 0.03 means “if the feature had zero effect, we would still see a lift this big about three times in a hundred.” That is not “there is a three percent chance the feature does not work.” Never say that in a review.
Statistical significance. A pre-committed threshold, usually 0.05. It is a convention, not a physical law. You choose it in advance. You do not shop for a threshold that makes your result significant after the fact.
Statistical power. The probability the test detects a real effect if one is there. Standard target is 80 percent. If the test only has a 40 percent chance of detecting a real two percent lift, and the result comes back “not significant,” you learned nothing. The absence of evidence was baked in before you started.
Sample size. A function of the smallest effect you would care about (the minimum detectable effect, MDE), the baseline variance of your metric, and the power you want. Run the calculation before the test. If the required duration is longer than your patience, negotiate the MDE up or pick a less noisy metric. Do not shorten the test to fit the calendar.
4. Multiple variants, multiple metrics
The moment a test has more than one variant, or you look at more than one metric, the p-value stops meaning what you think it means. Run twenty independent tests where nothing is happening, and about one will come back significant at p=0.05 anyway. If you A/B/C/D test four variants of a lab landing page and pick the winner by “which one is significant,” you have quietly increased the odds of a false positive.
The fixes are boring and they work. Reduce the number of comparisons. Correct the alpha before you start (Bonferroni: divide alpha by the number of tests, so four comparisons means 0.0125, not 0.05). Split primary and secondary metrics: primary decides, secondaries teach, do not promote a secondary because it looked good.
If the team pushes back that this is too conservative, ask “if we make the wrong call on this test, what is the cost.” Small UI tweak with a cheap rollback: be aggressive. Change to how a large catalog is scored, ranked, or surfaced to millions of learners: be conservative. Match the rigor to the reversibility of the decision, and to how many people it touches.
5. The four mistakes that make tests unreliable
These are the ones I have watched hurt real launches. Learn to name them and refuse to sign off when they are present.
Peeking. You look at the p-value every day and stop the test as soon as it crosses 0.05. This inflates the false positive rate dramatically. The p-value is only valid at the sample size you committed to. Pre-commit to a sample size or a duration, and only read the result there. If you must monitor, use a sequential testing method designed for it.
Sample Ratio Mismatch (SRM). You expected a fifty-fifty split. You got fifty-two, forty-eight. This sounds close. It is not. If the assignment mechanism is broken, whatever caused the imbalance is very likely correlated with the outcome, and every downstream number is contaminated. Check SRM before you look at the result. If the split is off by more than random chance would predict, throw the test out and fix the pipeline. This bites hardest on platforms with complex assignment logic (bot filters, region gates, feature entitlements), because there are so many places the split can quietly break.
Novelty and primacy. Users click on the new thing because it is new, and the effect decays. Or existing users resist the change, and the treatment stabilizes later. Both bias the result. Run long enough for the effect to stabilize (usually more than two weeks on established products), and look at the trend inside the window, not just the final average.
Interference. The behavior of users in the treatment group affects users in the control group. On a shared platform this is everywhere: a discovery ranking change where treated users’ behavior reshapes the ranking that control users then see, a workflow where treated and control authors share a review queue, a capacity-constrained resource like lab environments where both groups draw from the same pool. Fix it with cluster-based randomization (whole tenants, regions, or catalog areas get one variant), switchback tests, or by treating the measured effect as a lower bound. Do not pretend interference does not exist because it is inconvenient.
Sidebar: what I now do differently
The list I would tape to a monitor if I could.
Write the hypothesis, primary metric, and guardrails before the ticket goes to engineering. Not after.
Run the sample size calculation. If the required duration is unrealistic, negotiate the MDE or the metric, not the duration.
Check the split ratio on day one and day three. If SRM is present, stop, fix, rerun.
Do not peek. Read the result once, at the pre-committed sample size or date.
If there are more than two variants, correct the alpha before you start.
Pair every primary metric with a guardrail. A launch that moves the primary while wrecking a guardrail is not a launch, it is a bill coming due next quarter.
For long-running features, plan a holdout group that stays on control for at least a month post-launch. Novelty effects show themselves there.
Ask “what would falsify this result” before signing off. If the answer is nothing, the test was not designed to decide anything.
The senior PM job on experimentation is not to run the math. Your data science partner runs the math. The job is to make sure the test is designed to answer the question you actually care about, and to refuse to draw a conclusion the test cannot support.
Most launch reviews I have been in do not fail on the analysis. They fail on the setup. The chart is not the decision. The chart is the argument. And the argument is only as good as the machine that produced it.
See you next Tuesday.
Luiz
If you run experiments on a platform, or you are the PM who has to defend the result in a launch review, I want to hear which of these five mistakes bit you hardest. Reply or drop it in the comments.
Views are my own and do not represent Microsoft.