This mini lecture is on p-value and nothing else. Given that p-value is generally misinterpreted, I thought a stand-alone mini lecture might be helpful to remind us what p value really is.

p-value is generally introduced as “the probability of drawing an outcome, from the null distribution, that is equal to or greater than the observed outcome.”

p-value provides strength of evidence against the null hypothesis. Providing p-value with the results is better than just stating "accept (reject) the null" at the chosen significance level.

p-value is NOT the probability that the null hypothesis is true.

Even if one prefers to interpret p-value as probability rather than a mere transformation of test statistics, it is a conditional probability. Probability of drawing an outcome from the null distribution that is equal to or greater than (at least as extreme as) the observed test statistics when the null hypothesis is true.

When p-value is large, the observed value of the statistic is consistent with random variation if null is true, then there is no evidence against null hypothesis. Hence, one cannot reject the null hypothesis.

When p-value is small, the observed value of the statistic is less likely to be due to random variation when null is true. Note that it is still possible and consistent outcome but less so. The smaller it is, the less likely it is due to random variation when null is true. That’s why smaller p-values are stronger evidence against the null hypothesis in favor of alternative hypothesis.