PostHog’s Post

❔ What if the observed effect in an A/B test is smaller than the minimum detectable effect (MDE) used in planning it? Is it cause for concern? Does it make a test less trustworthy? Should you require both statistical significance and an observed effect higher than the MDE for a valid experiment? These and other questions surrounding the MDE and its relationship with the observed effect from a test are discussed in detail in the second installment of my three-part series on observed power 👉 https://lnkd.in/dy64RA_i Does one of the issues pointed out in the article stand out to you as the biggest problems for practitioners or clients? Do you think the term itself may be to blame for its multiple misinterpretations? #abtesting #statistics #power #pvalue #hypothesistesting

This article by John V. Kane on Medium provides an interesting but often overlooked contrast between Correlation analysis (specifically, Pearson’s pairwise correlation) and regression analysis (specifically, bivariate ordinary least squares (OLS). Major point I drew from the article is that: while correlation and regression coefficients are often seen as being similar, it is possible for two variables to show a strong correlation but actually have a weak overall effect on each other or regression. Have you encountered situations where correlation and regression analysis yielded different insights? How did you handle it? If you would like to know more about this, give it a read by following the link to this article attached to this post. https://lnkd.in/dRekMTiD

Given the significant advantages of the Cox PH model, particularly in handling continuous variables, adjusting for covariates, and providing more accurate P-values, it is difficult to justify the continued reliance on the log-rank test. The medical and epidemiological communities should reconsider the widespread use of the log-rank test and prioritize more robust methods like the Cox PH model, which can offer more reliable insights into survival data.

One-way ANOVA is a statistical test used to determine if there are significant differences between the means of three or more independent groups. It assesses the impact of a single categorical independent variable on a continuous dependent variable. This test is commonly used in research to compare group means and identify any statistically significant variation.

We have deployed test of goodness of probabilistic prediction (https://lnkd.in/e4Cm9H5D). For example, using 6-month forecast period, in Jan we forecast July, in Feb we forecast Aug, etc. Based on one-year record, NVDA turns out to be mostly going up, the realised probability level mostly centres 0.5-1. (https://lnkd.in/eWmrZxet) MMM turns out to be going down, the realised probability level mostly centres 0-0.7. (https://lnkd.in/euhzCH_m) The realised probability level (https://lnkd.in/e9xWfApD) measures how much probability mass in the predicted distribution is below the actual observed value. The prediction was always made 6-month before the actual value is observed, with 6-month forecast period.

The Welch test is a statistical test used in the context of Analysis of Variance (ANOVA) when the assumption of equal variances (homogeneity of variances) across groups is violated. Traditional ANOVA assumes that all groups have the same variance, but when this assumption is not met, the Welch test provides a robust alternative. When to Use the Welch Test? Violation of Homogeneity of Variances: When a preliminary test (such as Levene's test or Bartlett's test) indicates that the variances across groups are significantly different. Unequal Sample Sizes: When the groups being compared have different sample sizes, which can exacerbate the impact of unequal variances. Robustness to Assumption Violations: When there is concern that the assumptions underlying the traditional ANOVA are not fully met, and a more robust test is desired.

Which statistical test is used to compare two or more categorical and continuous variables?

The chi-square test is a statistical method used to determine if there is a significant association between categorical variables. It compares the observed frequencies in each category to the expected frequencies, which are calculated under the assumption that there is no association. This test is commonly used in hypothesis testing to evaluate the independence of variables in contingency tables and to assess goodness of fit.

𝗪𝗵𝗮𝘁 𝗜𝘀 ‘𝗖𝗼𝘃𝗮𝗿𝗶𝗮𝘁𝗲 𝗔𝗱𝗷𝘂𝘀𝘁𝗺𝗲𝗻𝘁’ 𝗜𝗻 𝗔/𝗕 𝗧𝗲𝘀𝘁𝗶𝗻𝗴? Covariate adjustment in A/B testing is a statistical technique used to account for pre-existing differences between groups by incorporating additional variables (covariates) into the analysis. This method helps to improve the precision and accuracy of the estimated treatment effects, thereby enhancing the reliability of the test results.