2
NetConversions Influences Kelley Blue Book: Case Study Analysis
Develop the Management-Research Question Hierarchy Through Investigative
Questions for This Project
An organized method for narrowing down a wide-ranging management problem into manageable
research questions is the management-research question hierarchy. Here, Kelley Blue Book (KBB)
needed to boost their online performance to increase ad income and improve the user experience as they
shifted from a print-based publication strategy to a digital platform. KBB enlisted the help of
NetConversions to enhance the usability of their website.
Management Question:
How can Kelley Blue Book improve its website so that users are more interested in it and the company
makes more money from ads and partnerships?
Research Questions:
What parts of the current design of the website make it easier or harder for visitors to connect
with it?
How do people move around the site, and what problems do they run into?
What changes will make it more likely that people will come back and interact with ads?
Investigative Questions:
Where do users click most frequently on each page?
Are users scrolling far enough to reach key information?
Do users expect the KBB logo to function as a home page link?
Which parts of the website experience high exit rates?
Measurement Questions:
How many people click through on different parts of the design?
3
NetConversions Influences Kelley Blue Book: Case Study Analysis
How does user behavior change when interactive features are introduced?
What is the increase in ad impressions and partner link clicks post-design changes?
Using this hierarchy, NetConversions guided KBB through a research process that revealed actionable
insights based on real user behavior.
Describe and Evaluate the Research Design of This Project
NetConversions used a strong mixed-methods research approach with two main parts: site analysis and
Hill Climbing™. This design used both observation and experimentation to gather a lot of information
that could be used to make decisions.
During the site analysis process, tools such as path analysis, click density maps, and scrolling analysis
were used to keep track of how visitors used the site. This first step gave us both qualitative and
quantitative information about how people used different parts of the site. NetConversions found that
when people clicked on the KBB logo, they often thought they were going back to the home page, which
it wasn't supposed to do.
The early steps in the Research Process Model (Schindler, 2021) are similar to this phase. This is
especially true for gathering data and making a first evaluation. This method works well because it is
realistic — users were watched in their normal setting, with no outside interference, which supports
ecological validity (Tarafdar & Zhang, 2005). In the Hill Climbing™ phase, NetConversions introduced
a form of randomized control testing. Using script overlays, design elements were altered for a randomly
selected subset of users (about 1 in every 1,000 visitors). These changes were compared to the original
design using behavior metrics like click-through rates and time on page.
This experimental step aligns with the latter stages of the Research Process Model—specifically, testing
hypotheses and evaluating outcomes. It allowed KBB to validate proposed changes before site-wide
implementation, significantly reducing redesign costs and risks. According to Palmer (2002), such
controlled digital experiments offer a strong advantage in iterative design.
4
NetConversions Influences Kelley Blue Book: Case Study Analysis
While comprehensive, the research design could have been enhanced by incorporating qualitative
methods such as user interviews or feedback surveys to better understand user intent and satisfaction.
Describe and Evaluate the Sampling Design for This Project
The sampling design employed by NetConversions differed across phases. In the site analysis phase,
data was collected through a convenience sampling method, leveraging real-time site visitors over a
period of 48 hours to 7 days. Given KBB’s high site traffic, this approach yielded a large and diverse set
of behavioral data, which is appropriate for usability research (Agarwal & Venkatesh, 2002).
In the Hill Climbing™ phase, random sampling was used. A script randomly selected users to be
exposed to experimental design changes, creating a de facto A/B test. This method improved internal
validity and ensured that observed effects could be attributed to the design change rather than external
factors (Schenkman & Jönsson, 2000).
Although statistically sound, the sampling method could have been refined by stratifying the audience—
distinguishing between first-time users and returning visitors, for example. This would have enabled
more targeted insights and potentially improved design personalization.
Overall, the sampling design was highly effective for the project’s objectives. It balanced efficiency with
experimental rigor and allowed KBB to gather actionable insights without compromising user
experience or inflating costs.
Conclusion and Outcomes
The data-driven recommendations given to Kelley Blue Book by NetConversions, who conducted
extensive usability research and iterative design testing, greatly enhanced the site's performance. Page
views and ad impressions increased by 62% after certain design tweaks, such as making images pop up
instead of static and adding dropdown boxes. In addition to improving the user experience, these
enhancements increased income by a significant amount, surpassing the cost of the research program.
5
NetConversions Influences Kelley Blue Book: Case Study Analysis
This case study shows how valuable it is to combine experimental data with observational data. In order
to optimize performance, improve user experience, and promote monetization through evidence-based
design, other digital enterprises can replicate NetConversions' two-phase methodology.
6
NetConversions Influences Kelley Blue Book: Case Study Analysis
References
Agarwal, R., & Venkatesh, V. (2002). Assessing a firm’s Web presence: A heuristic evaluation
procedure for the measurement of usability. Information Systems Research, 13(2), 168–186.
https://doi.org/10.1287/isre.13.2.168.84
Palmer, J. W. (2002). Web site usability, design, and performance metrics. Information Systems
Research, 13(2), 151–167. https://doi.org/10.1287/isre.13.2.151.88
Schenkman, B. N., & Jönsson, F. U. (2000). Aesthetics and preferences of web pages. Behaviour &
Information Technology, 19(5), 367–377. https://doi.org/10.1080/014492900750000063
Tarafdar, M., & Zhang, J. (2005). Analyzing the influence of web site design parameters on web site
usability. Information Resources Management Journal, 18(4), 62–80.
https://doi.org/10.4018/irmj.2005100104
Powered by TCPDF (www.tcpdf.org)
