THE NULL HYPOTHESIS AND YOU 2
The Null Hypothesis and You – Directional Significance Tests
Researchers conduct surveys with the aim of describing the characteristics of populations.
From these surveys, key metrics such as mean, standard deviations, and others can be obtained.
Statisticians will typically perform tests on the means or differences in means obtained from
samples to determine if they’re statistically significant. Warner states “we use statistical
significance tests to try to rule out chance as an explanation for the obtained pattern of results”
(2014, p. 22). Testing for significance is a necessity in order to ascertain that the sample can
legitimately be used to describe the population, and that perceived relationships are not just as a
result of chance.
Tests of statistical significance require the individual to specify a null and an alternative
hypothesis. Per Banerjee et al., “the null hypothesis states that there is no association between the
predictor and outcome variables “(2009, p. 128). Warner describes it as “a “guess” about the
specific value of μ for a population of interest” (2013, p. 108). The null hypothesis is always
equal to a particular value, while the alternative can be less than, greater than, or not equal to
depending on whether it’s a one-sided or two-sided test; the null always aims to establish the
status quo, and the alternative challenges that.
Sometimes, researchers have an idea of what to expect the population parameter to be. In
such instances, directional significant tests can be used instead of a two-sided test which doesn’t
specify a direction and simply states that the mean is not equal to hypothesized value. A two-
sided test aims to determine whether the relationship is significant in either a positive or negative
direction. The researcher does not hypothesize which way the relationship leans, instead opting
to find out from the test. Conversely, a one-sided test focuses on one direction only, and is based
on the expectations of the researcher. Chen et al. explain how to determine which type of test to
THE NULL HYPOTHESIS AND YOU 3
conduct; they state that a one-sided test should be used “If the investigator can justify focusing
on one side (direction) based on prior knowledge” (2019, p. 1038). Researchers can hypothesize
based on past experiences or expectations, as long as they can explain the reasoning.
For example, let us assume that a group of researchers survey a representative sample of 120
people in a company that has 5,000 employees nationwide. Let us now assume that based on the
researchers’ existing knowledge from competitors in the industry, they expect the average salary
to be $55,000, however, the sample returns an average salary of $65,000. They can test the
hypotheses: Ho: μ = $55,000 & Ha: μ > $55,000 since they possess prior knowledge from others
in the industry. From this, they can compute a t-stat value which can be compared to a critical
value, or they can use the p-value approach to run the test and arrive at a decision.
As far as specifying a value of α, Shreshtha (2019) states that “α, is the probability of
rejecting the null hypothesis when it is true.” To elaborate, Emmert-Streib & Dehmer describe
the significance level (α), as a “number between zero and one,” which tells us the probability of a
false positive decision (2019, p.951). With regards to selecting a value of α, Sedwick simply
states “the cut off between a large and a small P value is conventionally set at 0.05 (5%), which
is termed the critical level of significance” (p. 2). In other words, 5% is the value most widely
used by statisticians worldwide. It is not uncommon to see tests carried out at the 10% level or
levels lower than 5%. However, it is bad practice to adjust significance levels after the fact in
order to establish significance.
In regards to the p-value, a smaller figure is always preferred because it allows the
researcher to reject the null hypothesis if it’s less than the α, thus meaning that the relationship
being analyzed is statistically significant. This is also obvious in the sense that a statistician
would not be pleased after spending hours collecting data, only to find that no significant
THE NULL HYPOTHESIS AND YOU 4
relationships exist. Sedwick states “A large P value suggests that the sample data support the null
hypothesis, whereas a small P value suggests they do not” (2014, p. 2). Banerjee et al. provide an
example explaining how a p-value of .9 symbolizes “a 9% chance of finding such an association
due to random error in the sample” (p. 130). Such a p-value would not be significant at a 5%
significance level, but would be at a 10% significance level, so the level chosen matters and must
be selected carefully.
The Null Hypothesis and You – Type 1 & Type II Errors
Per Ilakovac, a type I error “occurs when we “see” the effect when actually there is none”
(2009, p. 11). Similarly, the same author describes a type II error “when we fail to see the
difference when it is actually present.” As touched on in the previous section, α is the probability
of a type I error; in a similar way, β is the probability of a type II error. According to Ilakovac, β
is dependent on “the size of the effect we are interested in, sample size and the chosen
significance level” (p. 11). This informs us that researchers can minimize risks by controlling
significance level and increasing sample size. However, tradeoffs exist and compromise is
required.
Banerjee et al. write that (1 – β), also known as power, is “the probability of observing an
effect in the sample (if one), of a specified effect size or greater exists in the population” (p.
130). For example, if β = .15, there’s an 15% chance of missing a relationship of an effect size
between variables and an 85% chance of capturing it. This is important because in the case of a
multivariate linear regression for example, the goal of performing these tests is to include all
variables of significance in order to accurately predict the desired dependent variable. Leaving
out critical variables would inevitably lead to lackluster prediction models.
THE NULL HYPOTHESIS AND YOU 5
Jacobucci et al. conducted an experiment where they concluded that “larger-than-
expected numbers of Type II errors were committed when sample sizes were small and when
collinearity was high” (2019, p. 65). This makes theoretical sense, since as the sample size gets
larger, we get closer to the true population value, thereby eliminating errors. Also,
multicollinearity is a problem caused by “high correlations among many predictors” (Warner,
2013, p. 632). Often times, the existence of multicollinearity causes important variables to be
omitted. A researcher must eliminate variables that suffer from multicollinearity to reduce type II
errors.
Shepherd. (2019, p. 4) give examples of both types of errors by illustrating the tradeoff
between the effectiveness of a drug (avoiding type I errors), and potential side effects (avoiding
type II errors). The author is looking at it from a more complex view, but we’ll take a more
simplistic example. Let’s focus solely on testing the efficacy of the drug: A type I error in such a
scenario would be rejecting the null hypothesis when it should not be rejected, thereby claiming
that the drug is effective when it’s not. Customers would be buying the drug based on false hope.
They’ll certainly end up wasting money, face deteriorating health and possible loss of lives,
suffer from disappointment, and maybe engage in lawsuits.
A type II error would be failing to reject a false null hypothesis that the drug is not
effective. The implications here are that scientists erroneously toss aside a drug which could be
potentially life-saving, and patients end up suffering great devastation as well. Both of these
situations paint grim pictures that ideally should be avoided at all costs. An opportunity to
change many lives would be lost, and the whole scientific undertaking would turn out to be a
waste of resources and manpower. Such results would lead to an inefficient, and unfortunate
THE NULL HYPOTHESIS AND YOU 6
conclusion that would end up costing much more than necessary. Scientists could potentially
have to go back to the drawing board and work on developing alternative drugs.
The risks of a type I error can be amplified by increasing the α level and conversely can
be reduced by picking a low α level. On the other hand, the value of β, which has an inverse
relationship with power, tells us the risk of a type II error. Just as researchers can minimize the
risk of a type I error by picking lower α values, they can reduce the chances of missing out on
significant variables by picking lower values for β. However, as aforementioned, a tradeoff exists
between α and β. As one increases, the other reduces, and this could pose a challenge for
researchers trying to account for both thresholds.
“Due to the relationship between Type 1 error and power, utilizing a higher Type 1 error
rate necessarily results in increased power “(Durand, 2013, p. 2). Amplifying the chances of
rejecting a true null hypothesis would decrease the chances of failing to reject a false one,
thereby raising the power of the test. The test would end up having a higher probability of
claiming significant relationships that don’t exist; however, it would end up more likely to be
inclusive of relevant variables. From a researcher’s point of view, the colossal challenge posed is
determining where to find a good balance.
To further summarize, Emmett & Dehmer mention that researchers in a dream world
would love to have low α, β, and high power. However, they go on to state that “these three
entities are not independent from each other…reducing α leads to an increase in Type 2 error (β)
and a reduction in power” (2019, p. 954). These authors as well as Durand are in agreement that
statisticians continue to face challenges with regards to finding a compromise between the three
and establishing a sweet spot.
THE NULL HYPOTHESIS AND YOU 7
References
Banerjee, A., Chitnis, U. B., Jadhav, S. L., Bhawalkar, J. S., & Chaudhury, S. (2009). Hypothesis
testing, type I and type II errors.LIndustrial psychiatry journal,L18(2), 127-131.
Chen, G., Cox, R. W., Glen, D. R., Rajendra, J. K., Reynolds, R. C., & Taylor, P. A. (2019). A
tail of two sides: Artificially doubled false positive rates in neuroimaging due to the
sidedness choice with t‐tests.LHuman Brain Mapping,L40(3), 1037-1043.
Durand, C. P. (2013). Does raising type 1 error rate improve power to detect interactions in
linear regression models? A simulation study.LPLoS One,L8(8), e71079.
Emmert-Streib, F., & Dehmer, M. (2019). Understanding statistical hypothesis testing: The logic
of statistical inference.LMachine Learning and Knowledge Extraction,L1(3), 945-962.
Ilakovac, V. (2009). Statistical hypothesis testing and some pitfalls.LBiochemia Medica,L19(1),
10-16
Jacobucci, R., Brandmaier, A. M., & Kievit, R. A. (2019). A practical guide to variable selection
in structural equation modeling by using regularized multiple-indicators, multiple-causes
models.LAdvances in methods and practices in psychological science,L2(1), 55-76.
Sedgwick, P. (2014). Understanding statistical hypothesis testing.LBMJ,L348.
Shepherd, T. G. (2019). Storyline approach to the construction of regional climate change
information.LProceedings of the Royal Society A,L475(2225), 20190013.
Shrestha, J. (2019). P-Value: A true test of significance in agricultural research.
Warner, R. M. (2013). Applied statistics: From bivariate through Multivariate Techniques.
SAGE Publications.
Powered by TCPDF (www.tcpdf.org)
