Assignment: Evaluating Significance of Findings

Meaningfulnessvs.statisticalsignificance.pdf

Home >Mathematics homework help >Statistics homework help >Assignment: Evaluating Significance of Findings

Meaningfulness vs. Statistical Significance

Meaningfulness vs. Statistical Significance Program Transcript

MATT JONES: Statistical significance deals with the critical value of a statistic. And in a certain philosophy, making a determination of whether the null hypothesis is rejected or you fail to reject the null hypothesis. That is statistical significance.

Meaningfulness is taking that statistic and determining it's applicability out in the real world. So all too often, as researchers, we get caught up in chasing statistical significance. And certainly while that's part of statistics, we also want to focus on the tie back to the real world.

So while we might find statistical significance in a very large sample, once we look at the effect that's present there, or the strength of the relationship, or the magnitude of the difference, they all might be extremely small. So while we might have quote unquote, "highly statistically significant results, the effect, relationship, or differences are rather small and almost meaningless in the applied real world."

It's important to understand the difference between statistical significance and meaningfulness. Because as a critical consumer of research, when you're evaluating claims, you want to match up the actual statistics with the claims being made. So if a claim is being made of how large an effect was or how important something is, you want to go back to the statistical analysis and see if that is indeed the case.

So just because something says there was a statistically significant result doesn't necessarily mean you can make claims about how profound an impact that it might have out in the real world. Because that might be an overreach of what the statistics were actually doing. Statistics will tell you only a small little piece of the pie. It's up to us as researchers to put the results into a specific context and understand that alignment between the statistics and the results with a possible misalignment.

History has given us a lot of examples of where statistics and statements don't necessarily match up. It's important to have a critical eye when it comes to statistical significance and meaningfulness to evaluate results, not only of other researchers, but also your own. And when you're making a claim about an effect or a relationship or an intervention, we're able to make that claim with some degree of authority. So just because you have statistical significance, i.e. a P value below 0.01, that doesn't necessarily mean the magnitude of the effect was strong where there's a large difference between two groups when in fact it might be very small.

Meaningfulness vs. Statistical Significance

The danger in not understanding the difference between statistical significance and meaningfulness is our potential to overreach with results. So in an era of big data with extremely large sample sizes, it's very common to find statistical significance. But when in fact if we look at the results, the differences might be minute, the effect size might be very small, the strength of the relationship might be very small. So speaking from a purely statistical standpoint, there might be an effect that's present. But if we were to change policy and spend money based upon those results, it might not be the most practical or logical way to proceed.