RELIABLITY & VALIDITY 2
Benchmark Exploring Reliability and Validity Assignment
Types of Reliability and Validity
When selecting an assessment tool, counselors must comprehend its reliability and
validity. Sheperis et al. (2020) describe reliability as the extent to which test scores are
consistent, stable, and dependable across different items, forms, or repeated administrations of
the test). The reliability of an assessment tool is determined by the consistency of the results it
produces. On the other hand, validity refers to the degree to which the conclusions and decisions
based on assessment results are accurate, meaningful, and appropriate for their intended purpose
(Sheperis et al., 2020).
The Values and Motives Questionnaire (VMQ) technical manual discusses two types of
reliability and two types of validity. The first type of reliability is the test-retest reliability which
measures the stability of the test over time by administering the same test to the same group at
different times and comparing the scores. The second type of reliability is internal consistency
reliability, which specifically measures using Cronbach’s Alpha, and assesses the consistency of
results across items within the test. A high Cronbach’s Alpha value indicates that the items are
measuring the same underlying construct (Bonett and Wright, 2015).
On page 10 of the VMQ manual, the authors specifically mention using internal
consistency reliability, assessed using Cronbach’s Alpha. This data can be found in the section
titled “Reliability Of The VMI.”. Again, internal consistency is an indication of the consistency
of results across items within a test (Sheperis et al., 2020). In this case, the internal consistency
reliabilities (Cronbach’s Alpha) were computed for the total standardization sample, combining
both male and female respondents. The coefficients for the VMI Value scales are provided in
Table 2, with most scales exceeding the acceptable level of 0.7, indicating good internal
RELIABLITY & VALIDITY 3
consistency. However, some scales like Moral and Independence fall slightly below this
threshold, and the Achievement scale has a higher standard error of measurement, suggesting it
should be treated with caution.
Conversely, the type of validity specifically being used in the VMQ based on the
provided information is construct validity. This information can be located in the section titled
“validity: Construct Validity.” Construct validity involves demonstrating that the results from the
test are consistent with the results from other tests that measure similar factors and are different
from scores on tests that measure different constructs. Validation studies of a test investigate the
soundness of a proposed interpretation of that test, focusing on whether the test truly measures
the intended construct (Sheperis et al., 2020).
Reliability: Cronbach Alpha Coefficients
The VMQ manual provides Cronbach’s alpha coefficients for various scales, which can
be categorized into low, acceptable, and high reliability based on the “What Makes a Good Test”
handout. According to the handout, the ratio for scores of Cronbach’s coefficients are the
following: high reliability (alpha coefficients of 0.80 and above), acceptable reliability (alpha
coefficients between 0.70 and 0.79), and low-reliability category (below 0.70). The reliability of
the VMI scales was assessed using internal consistency reliabilities, specifically Cronbach’s
Alpha. Among the 13 scales evaluated, eight demonstrated acceptable internal consistency with
alpha values of 0.70 or higher. These scales include Affiliative (0.74), Altruistic (0.74),
Affection (0.79), Finance (0.83), Security (0.79), Aesthetics (0.83), Traditional (0.70), and
Ethical (0.70). However, five scales exhibited problematic internal consistency with alpha values
below 0.70. These problematic scales are Achievement (0.53), Moral (0.68), Independence
(0.66), Social Desirability (0.64), and Infrequency (0.52). In summary, 61.54% of the scales have
RELIABLITY & VALIDITY 4
acceptable internal consistency, while 38.46% have problematic internal consistency. Out of the
8 scales, 4 scales (50%) have high internal consistency, 3 scales (37.5%) have acceptable internal
consistency, and 1 scale (12.5%) has low internal consistency. This indicates that the majority of
the scales have acceptable to high reliability, with only one scale falling into the low-reliability
category.
Sample Size and Nature of the Population.
Validity
The Values & Motives Inventory technical manual was evaluated using several other
tests, including the 16PF, MAPP, and OPP. Each of these tests has its own distinct population
characteristics. The VMQ’s population consists of working adults from different occupational
backgrounds. This diverse pool of individuals encompasses a wide range of values and motives
relevant to the workplace. In contrast, the 16PF (Sixteen Personality Factor Questionnaire) often
includes a broader demographic, encompassing both working adults and students, which may
introduce variability in personality traits and values. The MAPP (Motivational Appraisal of
Personal Potential) and OPP (Occupational Personality Profile) also target working adults but
may have different occupational distributions and cultural contexts.
Due to these differences, the populations used in the VMQ are not comparable to other
tests as the VMQ’s focus on working adults from diverse occupational backgrounds. Thus, the
VMQ’s population may not compare properly with the differently focused samples of the 16PF,
MAPP, and OPP. This difference or discrepancy could increase concerns regarding the validity
of the VMQ’s results compared to the results of the other tests. Specifically, this discrepancy in
sample characteristics could affect the generalizability of the findings.
RELIABLITY & VALIDITY 5
In summary, while the VMQ’s validation process includes comparisons with established
tests, the differences in population characteristics do raise potential issues with the validity and
generalizability of the results. Ensuring that the populations are more closely matched would
strengthen the validity of the VMQ’s findings.
VMQ Norming Population
The VMI evaluates a numerious occupationally relevant values, but its representativeness
depends on the sample used in the validation studies. According to authors Sheperis et al. (2020),
if a sample includes a diverse range of occupations, industries, and demographic backgrounds, it
is more likely to be representative. The VMQ does provide ample infomartion regard the manual
should include detailed demographic information about the sample population, such as age,
gender, ethnicity, and occupation. However, I don’t believe the VMQ results are extensive
enough to be generalized to other populations. The norming sample for the VMQ included 159
students from both Psychology and MBA programs.The ability to generalize the results to other
populations depends on the diversity and size of the sample; a larger and more diverse sample is
more likely to yield generalizable results.
My Opinion of the VMQ
In conclusion, the VMQ produces results that have sound validity scores. This ensures
that the items in the questionaire holstically cover the constructs of values and motives.
Construct validity is evidenced by significant correlations with related measures. Moreover,
criterion-related validity is also supported, with the VMQ differentianting between different
groups based on their values and motive. However, in terms of norming for the VMQ, there are
some concerns due to the results produced. While there was a sufficent sample size, the sample
RELIABLITY & VALIDITY 6
group was homogenious; primarily consists of psychology and MBA students. Therefore, the
lack of diversity in the sample population may limit the generalizability of the findings to
broader populations. Overall, the authors of the VMQ have established its reliability and validity
to a reasonable degree. However, expanding the sample to include more diverse populations
would enhance the validity and applicability of the instrument’s psychometric properties.
RELIABLITY & VALIDITY 7
References
Bonett, D. G., & Wright, T. A. (2015). Cronbach’s alpha reliability: Interval estimation,
hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36(1),
3–15. https://www.jstor.org/stable/26610966
Psytech International. (n.d.). Values & Motives Inventory: Technical manual. Psytech
International. https://psytech.com/content/TechnicalManuals/EN/Values%20&
%20Motives%20Inventory.pdf
Sheperis, C., Drummond, R. J., & Jones, K. D. (2020). Assessment procedures for counselors
and helping professionals. Pearson.
Powered by TCPDF (www.tcpdf.org)
