How can you explain the significance of p-values to a non-statistical audience?

Let's illustrate this with a coin-flip experiment. Suppose we get 60 heads and 40 tails in 100 flips. - Question: Is the coin biased or is this result due to chance? - The Hypothesis: Assume the coin is fair (i.e., null hypothesis). - The p-Value: Measures how likely it is to get a result as extreme as 60 heads if the coin is fair. Now, to interpret the p-Value, - High p-Value (e.g., 0.5): This means getting 60 heads is quite likely if the coin is fair.The result could easily occur by chance, suggesting no strong evidence the coin is biased. - Low p-Value (e.g., 0.01): This means getting 60 heads is very unlikely if the coin is fair. The result is unlikely due to random chance, suggesting strong evidence the coin might be biased.

In the coin-flipping experiment to get 60 heads in 100 flips, the p-value measures how likely it is to get such a result if the coin is fair. A significance level, on the other hand (usually kept at 0.05), is the threshold set to decide if a result is statistically significant. If the p-value is below this level, we consider the result significant, meaning it’s unlikely to be due to chance alone. Here are the key ideas: - Threshold for Decision: The significance level is the cutoff for deciding if a p-value indicates a significant result. - Decision Rule: If the p-value is below the significance level, the result is considered statistically significant, meaning it is unlikely to be due to chance.

A p-value is not the chance something is significant. It is more nuanced than that. As others have said, it is the likelihood seeing the observed event (or something more extreme) if the there is no relationship (a common null hypothesis). If we watched our coin emerge from the minting process at the US mint in Denver, we might forgive a small p-value and say that we still believe the coin is fair. If we found the coin in a magic shop, we might be more concerned about it being fake, even if the p-value is more modest. This "extra" information is an important caveat with p-values and a frequentist approach.

Science does have a reproducibility crisis. Their are a number of causes for this, but one might be that we often don't control for enough things when we do our experiments. A low p-value is definitely encouraging, but it is not as encouraging as we might hope. I like to do a fit my model or do my test on numerous bootstrap samples of the data to see how stable the results are. This can complete other tests.

A p-value is like a score in a game. It tells us how likely it is that we'd see the results we got if the game was fair. If the score (p-value) is low, it's like saying "Wow, that's a surprising score! Maybe the game isn't fair." If the score is high, it's like saying "That score isn't surprising for a fair game." So, a low p-value can suggest that our game (or experiment) might be biased.

Imagine you're playing a new video game and you want to test if a specific strategy helps you win more often. The p-value in this scenario helps you determine if the increase in your win rate is likely because of the strategy you're using or just due to random chance. If the p-value is very low (<0.05), it indicates that the strategy is effective in improving your win rate. On the other hand, if the p-value is higher, it suggests that any difference in your win rate could be due to luck or other factors.