Empirical Methods in Software Engineering Course Assignment

Exercise # 1 – EMSE 848 DCE

To illustrate the steps involved in deciding which method or methods to use, we present two guiding

examples. Two fictional software engineering researchers, Joe and Jane, will explore how the various

research methods can be applied to their work:

‒ Jane is a new PhD student interested in the effectiveness of a novel fisheye-view file

navigator. Her research is motivated by the fact that navigation is a primary activity of

software developers requiring a lot of scrolling and many clicks to find files. ‘Fisheye-

views’ use a distortion technique that, if applied correctly, display information in a

compact format that could potentially reduce the amount of scrolling required. Jane’s

intuition is that the fisheye-view file navigator is more efficient for file navigation, but

critics argue that the more compact information is difficult to read and that developers

will not adopt it over the traditional file navigator. Her research goal, therefore, is to find

evidence that supports or refutes her intuition that fisheye-view file navigators are more

efficient than traditional file navigators for navigation.

‒ Joe is a researcher in an industrial lab. His current interests are in understanding how

developers in industry use (or not) UML diagrams during software design. This is

because, as a student, his professors’ recommended UML diagrams to be used during

software design, but his recent exposure to industrial practices indicates that UML is

rarely used. His research goal is to explore how widely UML diagrams are used in

industry, and more specifically how these diagrams are used as collaborative shared

artifacts during design.

We explore how Jane and Joe develop research strategies for their projects. We begin with an analysis

of the type of research question(s) they are asking, and the issue of what constitutes valid answers to

them. To address the latter question, we tour the main philosophical stances that underpin empirical

research. We then describe the classes of research method, and explore how Jane and Joe might use

each method as part of their research strategies. We end the chapter with a look at the practical

considerations that affect their choices.

In the early stages of a research program, we usually need to ask exploratory questions, as we

attempt to understand the phenomena, and identify useful distinctions that clarify our

understanding.

Exercise # 1 – EMSE 848 DCE

Unless they are building on existing work that already offers clear definitions, both Jane and Joe need

to formulate exploratory questions, such as:

• Existence questions of the form, “Does X exist?”

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

• Description and Classification questions such as, “What is X like?”, “What are its properties?”,

“How can it be categorized?”, “How can we measure it?”, “What is its purpose?”, “What are its

components?”, “How do the components relate to one another?”, and “What are all the types of X?”

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

• Descriptive-Comparative questions of the form, “How does X differ from Y?” investigate

similarities and differences between two or more phenomena.

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

Exercise # 1 – EMSE 848 DCE

Once we have a clearer understanding of the phenomena, we may need to ask base-rate questions

about the normal patterns of occurrence of the phenomena. If we fail to ask base-rate questions, then

we have no basis for saying whether a particular situation is normal or unusual. Example base-rate

questions include:

• Frequency and distribution questions such as,

“How often does X occur?” and, “What is an average amount of X?”

Often, these questions can be answered in terms of a standard distribution of a characteristic within a

well-defined population.

RQ Joe – PROVIDED IN THE SLIDES

RQ JANE – FORMULATE THE RQ.

• Descriptive-Process questions of the form,

“How does X normally work?”, “What is the process by which X happens?”, “In what sequence do

the events of X occur?”, “What are the steps X goes through as it evolves?”, “How does X achieve its

purpose?”.

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

Often, we are interested in the relationship between two different phenomena, and specifically

whether occurrence of one is related to occurrence of the other. Hence we need to formulate some:

• Relationship questions such as,

“Are X and Y related?” and, “Do occurrences of X correlate with the occurrences of Y?”

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

Exercise # 1 – EMSE 848 DCE

Once we have established that a relationship exists between two phenomena, it is natural to try to

explain why the relationship holds by attempting to identify a cause and effect. It is a common mistake

to confuse correlation with causality.

If high values of X correlate with high values of Y, it may be because X causes Y, or because Y causes

X. But it is also possible that X and Y share some common cause and neither causes the other. Or

perhaps they co-evolve in complex ways so that there is no clear cause-and-effect.

Causality questions include:

• Causality questions of the form, “Does X cause Y?” and “Does X prevent Y?” Plus the more

general forms: “What causes Y?”, “What are all the factors that cause Y?”, “What effect does X have

on Y?” In software engineering we often ask whether using a particular tool or technique causes an

improvement in quality, speed, and so on.

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

• Causality-Comparative questions investigate relationships between different causes: “Does X

cause more Y than does Z?” or, “Is X better at preventing Y than is Z?”

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

• Causality-Comparative Interaction questions investigate how context affects a cause-effect

relationship: “Does X or Z cause more Y under one condition but not others?”

RQ Jane – PROVIDED IN THE SLIDES

RQ Joe – FORMULATE THE RQ.

Exercise # 1 – EMSE 848 DCE

The classes of research question above are all knowledge questions focused on the way the world is.

Empirical research in software engineering addresses these types of questions.

In contrast, most non-empirical research in software engineering focuses on a very different type of

question concerned with designing better ways to do software engineering:

• Design questions of the form,

“What’s an effective way to achieve X?” or,

“What strategies help to achieve X?”

RQ Joe – PROVIDED IN THE SLIDES

RQ JANE – FORMULATE THE RQ.

These types of question are necessary when the goal is to design better procedures and tools for

carrying out some activity or to design suitable social or regulatory policies.