Read a finished A/B test with Claude in one session: significance sanity, segment-level lift, novelty effect, a 1-page learnings note, and the next 2-4 tests already queued. 15-25 minutes per test.

I run analysis as a layer most CRO programs skip. It is the part where you turn "test went green" into "here is what we actually learned, and here is what we test next." It is the cheapest compounding asset in the whole program, and almost nobody does it properly.

This is the recipe pulled straight from how we run it across 1,000+ tests with $50M-$300M DTC Shopify brands. You give Claude three things. It hands back five.

You give it:

1. The raw test results (from your testing platform over MCP, or a CSV export)
2. The hypothesis the test was set up to validate
3. Segment-level data if you can get it (device, traffic source, returning vs new, geo)

Claude returns:

• A statistical sanity report (was the test actually significant, was the design valid)
• A segment-level breakdown (where the lift actually came from)
• A novelty / learning effect check (whether the lift is fading over time)
• A 1-page learnings note in plain English
• 2-4 next-test recommendations grounded in what you just learned

The point: every test produces compounding insight, not a single yes/no. After 50 tests analysed this way you have a stronger playbook than after 200 tests where only the headline got read.

The 30-second version

Why analysis is the most under-invested phase of CRO

Most CRO programs we audit pour their effort into design, dev, and launch, then spend almost nothing on reading the result. The output is a Slack message. "Test X won at +6%. Shipping it." Test Y was inconclusive, so move on.