PostHog’s Post

View organization page for PostHog, graphic

15,573 followers

We've added relative deltas and credible intervals to A/B testing in PostHog. Here's what that means... A relative delta is the percentage change in conversion rate between the control and test variants. So, a bigger delta means a bigger impact for an experiment. The credible interval is more nuanced. Basically, an experiment measures a certain value (like a conversion rate) and the true value isn't *actually* the result that's displayed – it's just an approximation because an experiment only measures a small sample of the population. The credible interval gives you a better look at the true data by showing a likely range for the results, as well as a probability percentage that reflects certainty. Relative deltas are useful for a lot of situations where you want to understand the broad improvement. Credible interval is a more advanced metric that's useful for getting into the nitty-gritty of statistical significance. We've also made it easier to ship the winning variant when your experiment reaches a significant result, via a shiny new 'Make decision' modal. Snazzy!

  • No alternative text description for this image
Lezginᅠ B.

CT0 @Qred | AWS Community builder | Sec-t(.org) | AWS Stockholm meetup organizer

2mo

Thanks for an awesome product,I just got internal feedback on this when I shared it, "This is great! I was calculating the relative delta manually every time"

Like
Reply

To view or add a comment, sign in

Explore topics