BUSI820 ASSIGNMENT 4
Quantitative Analysis: Evidence for Reliability and Validity
School of Business, Liberty University
BUSI820: Quantitative Research Methods
Assignment 4
11/14/2024 1
BUSI820 ASSIGNMENT 4
Table of Contents
Quantitative Analysis: Evidence for Reliability and Validity ……………………………...…….4
6.1 Compute Cohen’s kappa to assess reliability using Rater 1 and Rater 2. Interpret your output
as in Output 6.1………………………………………………………………………………........4
6.2 Compute interrater reliability by running a paired t test for the essay score 1 with the essay
score 2. Interpret your output following the guide in the Interpretation of Output 6.2…………...5
Paired Samples Statistics………………………………………………………………………6
Paired Samples Correlations…………………………………………………………………...6
Paired Samples Test……………………………………………………………………………6
Effect Size……………………………………………………………………………………...6
6.3 Run an Exploratory Factor Analysis using the extraction method of principal axis factoring
as demonstrated in Problem 6.3. Run the factor analysis on the 13 items related to stress. In the
extraction window (see Fig. 6.7), request “fixed number of factors to extract” as 2. Study your
output using the Interpretation of Output 6.3............................................................................…...7
Correlation Matrix……………………………………………………………………………..8
KMO and Bartlett’s Test………………………………………………………………………8
Total Variance Explained……………………………………………………………………...9
Rotated Factor Matrix………………...………………………………………………………..9
Factor Transformation Matrix………………………………………………………………….9
6.4 Run Cronbach’s alpha for the five happiness scale items from the Chapter Six data file.
Follow the procedures outlined in Problem 6.4 and write up an explanation..…..........................11
Reliability Statistics…………………………………………………………...…….………..12
Item Statistics……………………………………………………………...………………….12
11/14/2024 2
BUSI820 ASSIGNMENT 4
Inter-Item Correlation Matrix………………………………………………………………...12
Item-Total Statistics…………………………………………………………………………..13
Reference.………………………………………………………………………………………..14
11/14/2024 3
BUSI820 ASSIGNMENT 4
Quantitative Analysis: Evidence for Reliability and Validity
6.1 Compute Cohen’s kappa to assess reliability using Rater 1 and Rater 2. Interpret your
output as in Output 6.1.
Cohen’s kappa was calculated to assess the reliability between Admissions Essay Rater 1
and Admissions Essay Rater 2 on their essay decisions (Deny or Accept). The Crosstabulation
table (Figure 1) shows that both raters agreed on 56 out of 60 cases (17 + 39 agreements), which
is approximately 93.3% agreement. Specifically, they both denied 17 cases and accepted 39
cases.
Figure 1
Cohen's Kappa for Rater 1 and Rater 2
11/14/2024 4
BUSI820 ASSIGNMENT 4
In the Symmetric Measures table (Figure 1), the kappa value is 0.846 with an asymptotic
standard error of 0.074. The kappa value indicates a strong level (≥ .70) of agreement between
the raters (Morgan et al., 2019). Additionally, the approximate significance is p < .001,
suggesting that this level of agreement is statistically significant and unlikely to have occurred by
chance. Therefore, we conclude that there is a high level of inter-rater reliability between Rater 1
and Rater 2 in their admissions decisions for the essays.
6.2 Compute interrater reliability by running a paired t test for the essay score 1 with the
essay score 2. Interpret your output following the guide in the Interpretation of Output 6.2.
The paired t-test output (Figure 2) assesses the interrater reliability between the percent
scores on essays given by Rater 1 and Rater 2.
Figure 2
Paired T-Test for Essay Score 1 and Essay Score 2
11/14/2024 5
BUSI820 ASSIGNMENT 4
Paired Samples Statistics
The mean score given by Rater 1 is 72.75 (SD = 18.779), while the mean score by Rater
2 is slightly lower at 71.08 (SD = 19.462). This indicates a small difference in average scores
between the two raters.
Paired Samples Correlations
The correlation between the scores of the two raters is 0.908, with a significance level of
p < .001. This high correlation suggests a strong positive association between the scores assigned
by Rater 1 and Rater 2, indicating substantial agreement.
Paired Samples Test
The mean difference in scores between Rater 1 and Rater 2 is 1.667 points, with a
standard deviation of 8.233 and a standard error mean of 1.063. The 95% confidence interval for
the difference ranges from -0.460 to 3.794. The t-test value is 1.568 with 59 degrees of freedom
and a two-sided p-value of 0.122. Since the p-value is greater than 0.05, we fail to reject the null
hypothesis, indicating that there is no statistically significant difference in the scores assigned by
the two raters.
Effect Size
Cohen’s d for the difference between Rater 1 and Rater 2’s scores is 0.202, which is a
small effect size. This indicates that the difference in mean scores between the two raters is
minimal.
The paired t-test shows a high correlation between Rater 1 and Rater 2’s scores,
indicating strong interrater reliability. Although there is a small difference in the mean scores,
this difference is not statistically significant, supporting the consistency between the two raters in
scoring the essays.
11/14/2024 6
BUSI820 ASSIGNMENT 4
6.3 Run an Exploratory Factor Analysis using the extraction method of principal axis
factoring as demonstrated in Problem 6.3. Run the factor analysis on the 13 items related
to stress. In the extraction window (see Fig. 6.7), request “fixed number of factors to
extract” as 2. Study your output using the Interpretation of Output 6.3.
Descriptive Statistics
The descriptive statistics table (Figure 3) for each item shows the means and standard
deviations. The average scores for items range from 2.07 to 4.50, indicating varying levels of
responses for different stress-related factors.
Figure 3
Descriptive Statistics for 13 Stress Factors
11/14/2024 7
BUSI820 ASSIGNMENT 4
Correlation Matrix
The correlation matrix (Figure 4) shows relationships among the 13 items. Significant
correlations between items (e.g., correlations above 0.5) suggest some degree of shared variance,
which is suitable for factor analysis. The determinant is 0.000, indicating multicollinearity
(Morgan et al., 2019). While this might suggest some redundancy in items, factor analysis can
still proceed as it groups correlated items into factors.
Figure 4
Correlation Matrix for 13 Stress Factors
KMO and Bartlett’s Test
The KMO Measure of Sampling Adequacy (Figure 5) is 0.898, which is very high,
indicating that the data is suitable for factor analysis (Morgan et al., 2019). Bartlett’s Test of
Sphericity (Figure 5) is significant (p < .001), which means that the correlations among items are
sufficient for factor analysis.
Figure 5
KMO and Bartlett's Test for 13 Stress Factors
11/14/2024 8
BUSI820 ASSIGNMENT 4
Total Variance Explained
The Total Variance Explained table (Figure 6) shows that two factors were extracted,
with Factor 1 explaining 45.403% of the variance and Factor 2 explaining 9.864%. Together,
they account for approximately 55.267% of the total variance, which is a satisfactory amount for
social sciences (Morgan et al., 2019).
Figure 6
Total Variance for 13 Stress Factors
Rotated Factor Matrix
The Rotated Factor Matrix (Figure 7) displays the loadings of each item on the two
factors after Varimax rotation. Factor 1 has high loadings for items such as “Unable to Control
Important Things” (0.806), “Angered from Things Out of Your Control” (0.770), and “Feeling
Nervous or Stressed” (0.711). This factor seems to represent stress related to external control.
Factor 2 has high loadings on items like “Feeling OK With My Friends” (0.743) and “Feeling
OK With Love Life” (0.574). This factor appears to represent positive social relationships. Items
with high loadings on each factor indicate the dimensions that each factor represents.
Factor Transformation Matrix
The Factor Transformation Matrix (Figure 8) shows the correlation between the two
factors after rotation, indicating that they are relatively independent.
11/14/2024 9
BUSI820 ASSIGNMENT 4
Figure 7
Rotated Factor Matrix for 13 Stress Factors
Figure 8
Factor Transformation Matrix for 13 Stress Factors
11/14/2024 10
BUSI820 ASSIGNMENT 4
Overall, the exploratory factor analysis revealed two underlying factors among the 13
stress-related items: one related to external control issues and another related to positive social
relationships. Together, these factors explain approximately 55.3% of the variance in the dataset.
The high KMO value and significant Bartlett's Test indicate that the dataset is appropriate for
factor analysis (Morgan et al., 20190. This EFA helps to understand the structure of the stress-
related items by categorizing them into distinct factors that represent different dimensions of
stress and social well-being.
6.4 Run Cronbach’s alpha for the five happiness scale items from the Chapter Six data file.
Follow the procedures outlined in Problem 6.4 and write up an explanation.
To interpret the output of Cronbach’s alpha for the five-item happiness scale (Figure 9),
we will assess the reliability of the scale and examine inter-item correlations and item-total
statistics (Morgan et al., 2019).
Figure 9
Alpha for the Five Happiness Scale
11/14/2024 11
BUSI820 ASSIGNMENT 4
Reliability Statistics
Cronbach’s Alpha is reported as 0.967, which is very high. This suggests that the five
items on the happiness scale have excellent internal consistency and are highly reliable. A
Cronbach’s alpha above 0.9 is generally considered excellent, indicating that the items are well
correlated and measure a cohesive construct (Morgan et al., 2019).
Item Statistics
The means for the items range from 4.02 to 4.32, with standard deviations between 2.127
and 2.369. This suggests a moderate spread in responses, indicating variability among
participants' perceptions of happiness.
Inter-Item Correlation Matrix
All pairs of items have positive and high correlations, with values ranging from 0.827 to
0.885. This indicates a strong relationship between items, supporting the idea that they measure
related aspects of happiness.
11/14/2024 12
BUSI820 ASSIGNMENT 4
Item-Total Statistics
Corrected Item-Total Correlation values are all high, ranging from 0.879 to 0.924. This
shows that each item is highly correlated with the total scale score, indicating that removing any
item would not improve the overall reliability of the scale. Cronbach’s Alpha if Item Deleted
values are all slightly lower than the overall alpha of 0.967, suggesting that none of the items
detracts from the scale’s reliability. Removing any item would result in a decrease in Cronbach's
alpha, indicating that all items contribute positively to the scale.
In conclusion, the happiness scale demonstrates excellent internal consistency, with a
Cronbach’s alpha of 0.967. The high inter-item correlations and item-total correlations indicate
that the items are measuring a cohesive construct (Morgan et al., 2019). None of the items would
improve reliability if removed, suggesting that each item is an integral part of the happiness
scale. This output supports the reliability and cohesiveness of the scale as a valid measure of
happiness perceptions among participants.
11/14/2024 13
BUSI820 ASSIGNMENT 4
Reference
Morgan, G., Barrett, K., Leech, N., & Gloeckner, G. (2019). IBM SPSS for introductory
statistics: Use and interpretation (6th ed.). Routledge.
11/14/2024 14
