INTRODUCTION Software is pervasive and plays a crucial
Software is pervasive and plays a crucial role in society. We have become heavily
dependent on software, because it is used to control electrical grids, manage medical
records and assist air traffic controllers. Therefore, the software must function correctly,
even when it is changing to meet new requirements. Software maintenance is a fundamental
part of the development process. The focus of software maintenance is typically on making
routine functional changes. Each individual change may not have a large impact on the
software architecture, but the cumulative effect of many changes can begin to disrupt the
overall system architecture and make future changes more difficult to implement.
Software architectures are often overlooked when discussing the evaluation,
implementation and tool support of maintenance activities and techniques. Software
architectures however, are extremely important to the maintenance process, because
changes that affect the architectural structure of the system can be difficult to implement.
While it is important to design an architecture to be flexible so that it easily absorbs
changes, it is equally important to maintain that flexibility throughout the maintenance
phase. Therefore, maintainers must understand how changes affect the overall
architecture.
Effectively maintaining active software systems is an essential activity for software
engineers. As software systems continually evolve to keep up with technological
advancements and meet the needs of increasingly demanding and sophisticated users,
software maintenance tools and methods must also advance to remain effective. The
preceding statement addresses two very important concepts when referring to software
development: software maintenance and software evolution. Software maintenance and
software evolution have often been used interchangeably. However, these two terms
describe different stages of the system after development begins. Software maintenance
refers to:
The process of modifying a software system or component after delivery to
correct faults, improve performance or other attributes, or adapt to a changed
environment [6]. Software evolution is defined as:
The dynamic behavior of programming systems as they are maintained and
enhanced over their lifetimes [28].
Software evolution is also loosely defined as change over time. This change
however, should be consistent with the designer’s intentions even when unplanned
phenomena occur that affect the system [77]. Software engineers need tools and techniques
that not only assist in the maintenance process, but also help to characterize system
evolution so that the system can evolve in a controlled manner and deviations are properly
handled.
This research proposes two methods to address the problems associated with
increasing architectural complexity caused by software change. The first method is to
characterize software changes to assess their architectural impact prior to implementation.
The second method is to analyze historical data to identify a set of architecture change
metrics and associated models that help determine the difficulty of making a each particular
change.
The two methods provide a qualitative and quantitative perspective for
understanding change impact. The quantitative historical data analysis provides concrete
objective data to establish statistical relationships between change characteristics and risk.
However, the quantitative data alone is not sufficient. In order to provide a holistic approach
for understanding change, this research also focuses on change characterization.
Characterizing changes supplements the quantitative data with qualitative developer
experience and understanding of the system and change requests. The two-fold approach
proposed in this research relies on developer experience and real change data to provide a
framework to assist developers in making system changes without reducing the quality of
the architecture.
The next section presents important definitions related to this research. Section 1.2
describes the problem statement that this research addresses. Section 1.3 presents the
research plan.
1.1 Important Definitions
This section presents definitions of terms the reader needs to understand prior to
reading the remainder of the document. These terms are central themes used throughout
this research and are the basis for the hypotheses discussed in Section 1.3.
Software architectures are a fundamental aspect of a system’s long-term
maintainability and fulfillment of its quality attributes. Software architectures describe the
high level components of a system and details how the components will interact with each
other and the environment. Software architectures are used to facilitate communication
between stakeholders and are vital to a system’s long term maintenance.
A late change is a change that takes place after at least one cycle of software
development occurs and a working version or prototype of the system exists. When
discussing maintenance of software architectures, late changes can pose problems if the
architecture is not fully understood. These changes affect systems at the architectural level
and can be and costly to implement due to the size of the change and the related
risks.
Change characterization is the process of analyzing a change request and
characterizing how the request will impact the system. Change characterization is one way
to mitigate the risks involved when making late changes because the process of
characterizing the change prior to implementation requires that the developer consider all
aspects of the change and its impact on the architecture. As changes are characterized and
implemented, that information can be stored in a repository. Then, when new changes with
similar characteristics arise, similar approaches can be used to implement those changes.
Empirical software engineering is an important approach to conducting software
engineering research and is a vital tool for software engineers whose goal is improving the
state of the art. Empirical software engineering involves collecting and analyzing data in
order to assess relationships between software development methods, tools, or processes
and the outcomes they provide. Using empirical software engineering, researchers can
characterize, evaluate, and identify technologies that improve software quality throughout
the software lifecycle. By conducting empirical studies and analyzing the outcomes,
researchers can explain the phenomenon under study and increase the understanding of that
domain. Several empirical studies were conducted as part of this research.
Refactoring is making changes to a software system to improve its internal structure
without modifying the external functional behavior. Refactoring is performed to improve
the maintainability of the source code. The goal of refactoring is to improve quality
attributes such as readability, understandability, and maintainability and to help developers
that have to make changes to the system.
Open source software (OSS) is software in which the source code is made available
under a copyright license in the public domain for use and modification. OSS is the target
of the historical data analysis portion of this research because of its increase in popularity
and the availability of historic version of the source code.
1.2 Problem Statement
Software change is inevitable. All software systems must evolve to meet the ever
expanding needs of its users. These statements address one of two sureties in the field of
software engineering: 1) all developed software will contain bugs, and 2) all developed
software will undergo changes. Therefore, it is vital that software systems be maintained in
such a way to alleviate the complications involved when making changes and reduce their
potential to infect the system with new bugs. As changes are implemented, the increase in
the complexity of the system architecture results in an increase in the number of bugs
introduced. This phenomenon is the result a complex system being potentially less
understandable for developers leading to a decrease in quality. Due to the number and
frequency of changes that mature systems must undergo, software maintenance has been
regarded as the most expensive phase of the software lifecycle.
Late-lifecycle changes are of particular interest when trying to minimize the
increase in system complexity. Due to the time pressure caused by these crucial changes,
developers often cannot fully evaluate their impact on the system architecture. As a result,
the architecture degrades, leading to lower system quality and making future changes more
difficult [65, 107].
When dealing with late-lifecycle changes, it is important to focus on the software
architecture, which defines the structure and interactions of the system. When a change
affects the system architecture, the original architectural model must be updated to ensure
that the system remains flexible and continues to function as originally designed. When a
change to the system structure causes the interactions to become increasingly complex, the
architecture is likely to degenerate and become unmaintainable. Architectural degeneration,
also known as ‘code decay,’ leads to a mismatch between the actual functions of the system
and its original design. This situation results in confusion for developers which leads to
either a major reengineering effort or an early retirement of the system [87].
1.3 Research Plan
This section describes the goals, questions, and hypotheses of this research. The highlevel
goal of this research is stated as follows:
To provide software developers and maintainers with a framework for
software evolution analysis and design of late software architecture
changes; this framework includes capture of change characteristics and
detailed measurement data that can affect future change implementation
decisions.
This goal contains two major research thrusts: change characterization and
historical data analysis. Both approaches are necessary to develop a framework for
architecture change impact assessment and analysis. The following steps were performed
to accomplish this goal:
1. Conduct a systematic literature review of software change characteristics.
2. Develop a characterization scheme to describe software changes prior to
implementation.
3. Assess the viability and usefulness of the characterization scheme.
4. Identify metrics that provide insight into system complexity.
5. Characterize architecture changes from existing systems and compare their
effect to the metrics identified in Step 4.
6. Analyze and explain the results.
This research systematically analyzes change histories of various types of software
systems. Changes that have an impact on the software architecture are further analyzed to
determine the effect that these changes have on a set of metrics. The metrics that correlate
to change density and system complexity are examined to determine how the various
characteristics of architecture changes impact the maintenance phase. The following
detailed questions motivate the research plan:
Q1. Does characterizing change requests prior to implementation
improve the change process?
Q2. Do open-source software systems conform to Lehman’s laws of
software evolution (increasing complexity and continuing growth)?
Q3. Does refactoring cause an observable affect a system’s complexity?
Q4. Do software systems exhibit specific patterns of software evolution?
Q5. Can software metrics be used to assess system complexity and
change difficulty?
Q6. Do certain types of software changes negatively effect system
quality, maintainability, and flexibility?
The hypotheses driving this research are:.
H1. Using a change characterization scheme to analyze change requests
prior to implementation will enable software developers to create a
more acccurate estimate of the impact of the change than without
the characterization scheme.
H2. Extracting and analyzing software change metrics from historical
data gives developers measurable data on system quality which can
be used to make refactoring decisions.
H3. Refactoring has a measurable positive effect on the system
architecture.
H4. Code decay is evident in ongoing software development and can be
determined by metrics extraction during various periods of
development.
CHAPTER II
RELATED RESEARCH
This section presents literature on important concepts related to this research.
Sections 2.1 to 2.5 present background research on the important software engineering
topics introduced in Section 1.1 that is foundational to the current research. Section 2.6
presents the initial version of the change characterization scheme created during my
Master’s Degree work on which this dissertation builds.
2.1 Software Change and the Laws of Software Evolution
Change is inevitable in software systems because of the many factors that influence
the need for changes. These factors include a change in the user’s needs, a change in the
operating environment, a change due to a problem in the software, or even a change to
prevent the need for future changes. Regardless of the reason for the change, software
engineers are developing increasingly sophisticated ways to handle changes.
Software change is a well studied topic. A pioneer of the study of software changes,
Manny Lehman, created the Laws of Software Evolution [107]. These laws describe
recurring issues related to the evolution of E-type software systems. An E-Type system
solves problems in the real world and must continually evolve to maintain user satisfaction
[109]. While there are eight laws, only Laws I, II, VI, and VII are relevant to this research.
Each law will be addressed by examining methods for controlling the negative
effects of these laws. Law I – Continuing Change states that software undergoes
neverending maintenance and development that is driven by the mismatch between its
current capability and the environmental requirements [107]. This mismatch could result
from the changes in the technology that contains the protocols and standards of the
communication technology. It could also result from the changes in hardware or the need
for more efficient utilization of hardware resources. Understanding the reasons for the
change will aid developers in handling changes in a systematic fashion.
As systems change, they tend to become more complex, if not properly handled.
This situation leads to Law II – Increasing Complexity. This law simply states that changes
resulting from system adaptation lead to an increase in the interactions and dependencies
among system elements. These interactions may be unstructured and increase the system
entropy. If entropy is not properly handled, the system will become too complex to
adequately maintain. Law II is one of the primary reasons why the maintenance phase is
typically the most expensive phase of software development. In order to reduce and better
manage system complexity, developers need improved ways of understanding changes and
how to incorporate change into system architectures.
The number of modules tends to increase linearly with each system release. Law VI
– Continuing Growth focuses on user needs by stating that the functionality of software
systems must continually increase to maintain user satisfaction over the lifetime of a system
[106]. While this law is similar to Law I, it focuses on changes driven by the user base and
their tendency to become increasingly sophisticated and demand a more robust set of
features. These changes also result from features that users deem unsatisfactory [106].
Each of the previous three laws feed into Law VII – Declining Quality [106]. As
changes are made, system complexity increases. The introduction of new features to a
system causes it to grow. These factors serve to reduce the perceived quality of a system.
When the quality of the system is reduced, it becomes more expensive to maintain because
of an increase in the number of problems encountered by users. To address these problems,
changes must be made to the system. These changes are likely to further increase the
complexity and size of the system which will, in turn, further reduce the quality [106]. This
cycle results in a continuous downward spiral of quality.
The Laws of Software Evolution have been studied frequently by software
engineering researchers [59, 79, 85, 145]. In understanding these laws and the necessity of
software change, researchers have developed methods of handling changes, e.g., using
change classification schemes, performing impact analysis, and developing effort
prediction models [76, 80, 110, 137, 140]. These methods are continuing to improve. As
more research is done in the area of understanding changes, more can be done to help
practitioners them implement changes. Practitioners then will not have to suffer from the
uncontrollable increase in complexity or decline in quality.
2.2 Code Decay
The increase in complexity caused by software change is referred to by different
names. Eick, et al., named the problem code decay. They examined a 15-year old system
and found that it became much harder to change as modifications were continually made.
One cause of this decay was violation of the original architectural design of the system
[65]. Another indicator of code decay is a history of frequent class changes. Change prone
of classes have been previously studied in software engineering literature [97, 115]. These
results confirm the indications of frequent changes with code decay. Large, complex classes
were deemed change prone with changes to these classes often being more difficult than
changes to non-change prone classes.
Another name for this problems associated with code decay is architectural
degeneration. This problem occurs when the difficulty of changing the system becomes
disproportionately large relative to system size. In these systems, the architecture must be
refactored to reduce the effects of degeneration [118]. The term software aging has been
used to identify the increased complexity and degraded structure of systems. This degraded
structure increases the number of bugs introduced during incremental changes [141].
Finally, Brooks stated that “all repairs tend to destroy the structure, to increase the entropy
and disorder of the system…more and more time is spent on fixing flaws introduced by
earlier fixes” [44]. Each of these concepts describe problems associated with making late
changes to software systems. In each case, the changes tended to degrade the quality of the
system over time.
2.3 Change Classification
Lehman’s Laws and code decay reflect problems association with software change.
One way to mitigate the risks of code degeneration due to changes is to gain insight into
the impact of proposed changes prior to their implementation. After doing so, a developer
can better assess how the software will be affected by the change and plan the
implementation approach to reduce the problems that may result from the change. Change
classification schemes have been used by developers to qualitatively assess the impact and
risk associated with making certain types of changes. Several benefits of change
classification have been identified in the literature, such as identifying risks associated with
change implementation and determining change acceptability. Software change
classification schemes also allow developers to group changes based on different criteria,
e.g. the cause of the change, the type of change, the location where the change must take
place, and the potential impact of the change. By grouping changes, engineers can develop
a common approach to deal with similar changes, resulting in less overall effort compared
with addressing each change individually [137].
Lientz and Swanson’s work identified the frequency of the different types of
maintenance activities performed by a large sample of software development organizations
[113, 162]. Based on their work and work by Sommerville [156], 3 major types of changes
were identified: perfective, corrective, and adaptive changes. Perfective changes result
from new or changed requirements. These changes improve the system to better meet user
needs. Corrective changes occur in response to defects. Adaptive changes occur when
moving to a new environment or platform or to accommodate new standards or platforms.
Another type of change that often affects system architecture is a preventative change
identified by Mohagheghi [133]. Preventative changes ease future maintenance by
restructuring or reengineering the system when a potential problem is
identified.
Kung, et al. studied the impact of code changes on the class inheritance structure of
a software system. They identified the importance of understanding class dependencies
when making code changes and the role a decision support system plays in managing
change [99]. Nedstam, et al. identified changes as either architectural: affecting the
structure of the system, functional; affecting only user-observable attributes, or somewhere
in between; affecting both user-observable attributes and structure [134]. Each of the
concepts listed have been analyzed to determine their inclusion in a change characterization
method designed to assist developers in evaluating a change request prior to
implementation.
2.4 Historical Data Analysis
Analyzing the code base from past system revisions (changes) can provide insight
into current trends of a software system. In this work, data was extracted from repositories
of open source software to assess how a system has evolved. This work will build upon the
many existing studies that extract software change data from historical data sets [12, 20,
29, 61]. Most of these studies focused on identifying metrics that could be used to help the
software developer understand the types of changes that have been made to their respective
systems. Another use of historical data is investigating the relationship between metrics and
software quality [13, 50, 52, 65, 87, 111, 141]. By evaluating different metrics, dependent
variables (e.g., change density and effort), and different types of systems, one can determine
whether there was a significant correlation between metrics and some quality attribute. The
current research analyzes software change and quality metrics from historical data to
determine which metrics are effective at indicating whether a software system has degraded
over time. The change metrics include actual implementation data for previous system
changes.
Analyzing data stored in version management databases is essential to
understanding and effectively managing the change process [64]. Predictive change models
have been developed based on an analysis of change metrics collected from a system [132].
Mockus was instrumental in developing the SoftChange tool that was used to extract
development metrics from a software system [127]. He used various historical data analysis
methods to analyze the frequency of the types of changes in a large software system. He
observed a strong correlation between the difficulty of the change and the type of change
[131].
German studied fine-grained software changes of large open-source software
systems. He analyzed CVS logs and categorized various types of changes made to the open-
source systems. He also visually identified module couplings and the relationship between
authors and modification records in each system [75]. Xing, et al., identified structural
changes between designs of subsequent versions of a system using an algorithm called
UMLDiff [173]. They used CVS data to determine the similarities and differences of design
changes to the subject system.
Mining software change repositories has produced valuable change data that can be
used to model future changes and to help developers better understand the changes to be
made to a system. This previous work provides a framework for the types of analysis that
will be performed on the historical datasets targeted by this research.
2.5 Open Source Software Systems
With open source software (OSS) systems being the primary focus of the historical
data analysis, some commentary on the importance of OSS to the software engineering
community is warranted. OSS development is characterized by decentralized development
personnel. Individual contributors from around the globe can make additions and
modifications to the source code. OSS has grown in popularity and in number of active
projects in recent years. There are several factors involved in the growth of OSS: increased
reliability and software quality, lower costs of development and ownership, performance
enhancements, security, and the ability to scale systems as needed [169]. This increase in
popularity has allowed OSS systems to become more “mainstream” for both individual
users and in industry.
OSS exhibits faster growth than closed source (proprietary) software (CSS). This
growth is indicated by the increase in SLOC over a period of time as compared with CSS
[129]. This increase can be attributed to the internet enabling large number of developers
and contributors to work on successful OSS projects. With very active user communities
constantly submitting change request and active development communities responding to
those requests, a mature OSS system typically grows faster than a mature CSS system. OSS
also exhibit rapid development cycles. These releases quickly provide a large number of
new functions [78].This rapid increase in functionality and program size in mature OSS
developments provides a rich set of available changes to analyze as part of this research..
The OSS development process uses nightly builds to incorporate new
functionality and bug fixes. At various intervals, depending on the project, these nightly
builds are incorporated into beta releases. These beta releases are tested and eventually
characterized as stable releases when a certain level of quality is obtained. OSS also focuses
on modularized development. This emphasis in modularity is so that parallel development
can occur. Typically, in OSS there are no formal requirements or design documents. Many
OSS projects start as hobbies of individual developers. As the project grows in popularity,
additional developers begin to contribute to the project. These OSS development
characteristics incorporate features from extreme programming and agile development
practices. The developers continue to add new features and modify bug fixes until
modularity decreases and complexity increases. The system is typically refactored often to
maintain modularity and keep the structure of the system simple for the diverse group of
developers. One goal of this research is to determine how these refactorings affect the
quality of the system and the ease of making changes.
2.6 Masters Research
This dissertation research builds on my Master’s Degree research. During this time,
an initial change characterization scheme was developed and evaluated it with an empirical
study. This initial characterization scheme is described below. The study design is described
in detail in Section 3.2.2. This study’s design is included in Section 3.2.2 with two other
studies that were conducted to build upon this Master’s study. The results of this study are
presented in Section 6.1.
The initial focus of my Master’s work was creating an architecture change
characterization scheme designed to allow a developer to characterize a change request to
indicate the effect of that change request on the system. This initial characterization scheme
consisted of a collection of change and defect attributes. The scheme was modeled as a
decision tree for a developer to traverse while choosing values for each
attribute.
There were two types of attributes, those that were orthogonal (i.e., only a single
choice can be made) and those that were not. The high level characterization (shown in
Figure 1) of the change is an orthogonal choice. The developer determines the lifecycle
phase in which the change was requested (i.e., requirements, architecture, design,
implementation, or testing). Then he selects whether the change request was motivated by
a desire to enhance the system or as a result of a defect.
Figure 1
High-Level Change Categories
The remaining attributes require the developer to identify the causes of the defect
and/or the impact of the enhancement. For each change request that was motivated by a
defect, the developer first identifies the cause of the defect using the defect characterization
(Figure 2), then selects the attributes from the enhancement characterization (Figure 3) that
identify the impact on the system. For each defect found in the system, a change must occur
to fix the defect. The impact of the change is then characterized using the enhancement
attributes. If the change request was not a defect, the defect characterization is not used.
The Defect attributes are shown in Figure 2 and the Enhancement attributes are show in
Figure 3. In the figures, the attributes identified in the rectangle are orthogonal, and the
attributes identified by a circle are not.
Figure 2
View of Defect Attributes
The defect attributes focus on repairing an error, fault, or failure in the system and
can therefore also affect the architecture of the system. The first attribute selected is the
class of the defect. The only high-level change type is corrective, because by definition all
changes due to defects are corrective [133]. The class attribute is included in the initial
version of the characterization scheme for completeness.
The next attribute, found, identifies the stage in the development process in which
the defect was discovered. This value is either inspection, testing, or user-reported.
Following the same idea, the next attribute is the origin, or part of the documentation in
which the defect originated. The defect can originate in the requirements, design, or code.
These attributes are included because of previous research identifying a correlation
between the time and location of defect discovery and the effort required to repair that
defect. Changes later in the lifecycle tend to be more difficult and have a larger impact
because more artifacts must be changed, requiring input from multiple stakeholders [73].
The final attribute for a defect is issues. This attribute is useful for prescribing a
general means of handling the changes needed to fix the defect. Each value for the issues
attribute was identified in the software engineering literature as causing defects and are as
follows: design, data accessibility, environment, problem definition, domain knowledge,
technology, interface: system/user, and data transmission [73, 87, 121].
If the change request is strictly an enhancement, the first attribute selected is the
class of the change. The class attribute is orthogonal and has a value of either perfective,
adaptive, or preventative. The next attribute selection is the type of modification. This
determines whether information is added to, deleted from or modified in the artifacts of the
system. The type attribute allows a developer to prescribe a solution for the change by
identifying what must be done to the artifacts.
After determining the type, the next step is to describe the properties of change as
either static, dynamic, or a combination of the two. A value of static indicates that the
change affects the static structure of the system, i.e., class, package, and object diagrams,
while a dynamic value indicates an effect on the dynamic properties, i.e., collaboration,
state-chart, and activity diagrams.
The next attribute is the impacts attribute. This attribute takes advantage of the
developer’s knowledge of the system to associate the change with the area(s) of the system
that will be affected. There were various software impact areas identified in the literature
that could be affected by a software change including: processes, hardware, data, system,
protocols/standards, programs/sub-systems, file systems, interfaces, documentation, and
source code [27, 41, 119, 131, 149].
Figure 3
View of Enhancement Attributes
The last attribute to be determined is architectural vs. functional characteristic of
the change. For this attribute, a scale was used ranging from purely architectural changes
that affect the structure of a system, but not how it functions, to purely functional changes
that affect the user-observable attributes or functions of a system [134]. The value of this
attribute may be anywhere in the range between those two extremes, indicating how much
it will affect the architecture.
For each non-orthogonal Defect and Enhancement attribute, the developer chooses
a value from Table 1. The combination of these values then provides the developer with an
overall idea of the impact of that change. The characterization produced by the decision
tree is based on the developer’s experience with the system. This initial architecture change
characterization scheme was designed to help determine the difficulty of implementing the
change and to provide some support for reasoning about change impact.
Table 1
Change Impact Scale
Value
Description
0
1
2
3
No impact
Small impact
Significant impact
Major focus of change
CHAPTER III
RESEARCH APPROACH AND METHODOLOGY
This chapter describes the processes involved in conducting a systematic review to
develop the change characterization scheme (Section 3.1), the design of three empirical
studies used to validate the scheme (Section 3.2), and an overview of the historical data
analysis (Section 3.3). Section 3.1 is an excerpt from a detailed technical report describing
the creation of the characterization scheme [172].
3.1 Systematic Review of Software Architecture Changes
A systematic review is a means of identifying, evaluating and interpreting the
available research related to a research question, topic area, or phenomenon. The main
purpose for conducting a systematic review is to gather evidence on which to base
conclusions. They are commonly used to support or contradict claims made by researchers,
identify gaps in existing research, provide motivation for new research, and supply a
context for the new research [36, 83, 90, 94, 124]. A systematic review consists of planning,
conducting and reporting the review [93]. Within those three phases are the following steps:
1. Identification of the need for a systematic review
2. Formulation of a focused review question
3. A comprehensive, exhaustive search for primary studies
4. Quality assessment of included studies
5. Identification of the data needed to answer the research question
6. Data extraction
7. Summary and synthesis of study results (meta-analysis)
8. Interpretation of the results to determine their applicability
9. Report-writing
The systematic review protocol template prescribed by Biolchini was used to
perform the review [31]. Prior to performing the review, a protocol was developed that
outlined the review goals, the research questions, and the procedures to be followed. The
remainder of this section describes the steps performed to complete the review.
3.1.1 Research Questions
The review goal was to identify software change characteristics that affect
architecture. Several research questions establish the focus for the review. The high-level
question was:
Can a broad set of characteristics that encompass changes to software
architectures be identified using the current software engineering body of
knowledge and be used to create a comprehensive change assessment
framework?
This research question was then refined to 5 more specific questions. These
questions, along with the motivation for each one, are shown in Table 2.
Table 2
Research Questions
Research Question
Motivation
What are the attributes of existing
software change classification
taxonomies?
This question provides a starting point
for creating a framework for change
assessment. The answers to this
question present the basis on which to
define, build, and refine the attributes
of the scheme.
How are software architecture
elements and relationships used
when determining the effects of a
software change?
One requirement of developing the
framework is to understand the role of
architecture in a developer’s assessment
of change impact. The answer is
important in understanding
architectural characteristics that affect
change implementation difficulty.
How is the architecture affected by
functional and non-functional
changes to the system requirements?
Software architectures are important in
exhibiting the non-functional
requirements of a system. The impact of
an architectural change due to a
functional requirement may be less
important. The goal here was to
differentiate, if possible, the
architectural effects of changes to
functional/non-functional requirements.
How is the impact of architecture
changes qualitatively assessed?
Developers often have differing views
on the best way a change should be
implemented to a system. The internal
processes that developer’s use when
assessing a change is an important
abstraction to understand in developing
a change assessment scheme.
What types of architecture changes
can be made to common
architectural views?
Understanding architecture changes is
important if your goal is to provide
developers with a list of alternatives for
making decisions about changing the
architecture. Having this list of possible
changes can lessen the cognitive load
on the developer to a set of choices
given the context of the request.
3.1.2 Sources Selection and Search
The primary studies used in this review were obtained from searching databases of
peer-reviewed software engineering research. The database searches resulted in a large
number of candidate papers. The inclusion/exclusion criteria shown in Table 3 were used
to narrow the search to relevant papers.
Table 3
Inclusion/Exclusion Criteria
Inclusion Criterion
Exclusion Criterion
Papers that address change
classification at any level of
abstraction (i.e., class, package,
subsystem, architecture, etc.) Papers
that identify procedures and
techniques for change impact
analysis Papers that discuss the
effects of changing software
architectures
Empirical studies of software changes
Case studies of software change
frameworks and change assessment
methodologies
Experience reports detailing software
changes
Papers that are based only on expert
opinion
Short papers, introductions to
special issues, tutorials, and
minitracks
Studies presented in languages other
than English
Studies whose findings are unclear
and ambiguous
Papers that describe changes to
aspect-oriented software,
selfadaptive software systems,
embedded systems, and dynamic
software architectures
This inclusion/exclusion criteria was applied by:
1) Reading the title to eliminate any irrelevant papers
2) Reading the abstract and keywords to eliminate additional papers whose title
may have fit, but abstract did not relate to any of the research question
3) Reading the remaining papers and including only those that addressed the
research questions.
The research questions were reduced to a series of search strings that were executed
in the selected databases. These searches returned thousands of papers that were filtered
down to 2752 based on the specific keywords, then to 523 after reading the titles, and then
to 220 upon reading the abstract. These 220 papers were read and 130 were chosen based
on the exclusion/inclusion criteria. Of the 130 primary studies, 36 were published in
scholarly journals and 94 in conference proceedings. In addition to the primary studies, 8
books were referenced to provide additional background data on software architectures and
software evolution [19, 34, 56, 70, 107, 113, 154, 156]. Three technical reports were also
found that met the inclusion/exclusion criteria [72, 128, 157] and three standards
documents [7-9]. Prior to conducting the systematic search, I was aware of a number of
papers that were relevant. As an indicator of the completeness of the review, all those papers
were also found during the systematic review [14, 25, 27, 37, 50, 55, 66, 87, 88, 100, 101,
105-107, 119, 121, 131, 133, 134, 146, 149, 159, 166].
Table 4 lists the paper distribution by source.
Table 4
Paper Distribution
Source
Count
%
International Conference on Software Maintenance
21
16.15%
IEEE Transactions on Software Engineering
10
7.69%
International Conference on Software Engineering
9
6.92%
Table 4 (continued)
European Conference on Software Maintenance and
Reengineering
7
5.38%
IEEE Symposium on Software Metrics
6
4.62%
International Symposium on Principles of Software Evolution
6
4.62%
Journal of Software Maintenance and Evolution: Research and
Practice
6
4.62%
Working IEEE/IFIP Conference on Software Architecture
4
3.08%
IEEE Software
3
2.31%
Information and Software Technology
3
2.31%
International Conference on Computer Systems and Applications
3
2.31%
International Workshop on Mining Software Repositories
3
2.31%
Journal of Systems and Software
3
2.31%
Annual NASA Goddard/IEEE Software Engineering Workshop
2
1.54%
Asia Pacific Conference on Software Engineering
2
1.54%
Communications of the ACM
2
1.54%
Empirical Software Engineering
2
1.54%
IEEE International Workshop on Software Evolvability
2
1.54%
International Conference on Automated Software Engineering
2
1.54%
International Conference on Quality Software
2
1.54%
International Conference on Software Engineering and
Knowledge Engineering
2
1.54%
International Software Architecture Workshop
2
1.54%
International Symposium on Empirical Software Engineering
2
1.54%
Working Conference on Reverse Engineering
2
1.54%
ACM SIGPLAN Conference on Object-Oriented Programming,
Systems, Languages, and Applications
1
0.77%
Annual Hawaii International Conference on System Sciences
1
0.77%
Australian Software Engineering Conference
1
0.77%
Bell Labs Technical Journal
1
0.77%
Conference of the Centre for Advanced Studies on Collaborative
Research
1
0.77%
Cutter IT Journal
1
0.77%
EUROMICRO Conference on Software Engineering and
Advanced Applications
1
0.77%
IEEE International Requirements Engineering Conference
1
0.77%
IEEE International Symposium on Requirements Engineering
1
0.77%
IEEE International Symposium on Visual Languages
1
0.77%
Table 4 (continued)
IEEE Region 10 International Conference
1
0.77%
IEEE Symposium and Workshop on Engineering of
ComputerBased Systems
1
0.77%
International Conference on Information Systems
1
0.77%
International Software Process Workshop
1
0.77%
International Conference on Applying the Software Processes
1
0.77%
International Conference on Software Engineering, Artificial
Intelligence, Networking, and Parallel/Distributed Computing
1
0.77%
International Journal of Software Engineering and Knowledge
Engineering
1
0.77%
International Process Support of Software Product Lines
Software Process Workshop
1
0.77%
International Workshop on Program Comprehension
1
0.77%
Proceedings of the IEEE
1
0.77%
Science of Computer Programming
1
0.77%
Software – Practice and Experience
1
0.77%
Software Process Improvement and Practice
1
0.77%
Workshop on Unanticipated Software Evolution
1
0.77%
Total
130
100%
3.1.3 Data Extraction and Synthesis
A data extraction form was used to extract relevant data from each paper. The form
includes the superset of all data items examined for each study. Every paper did not provide
information for each data item, but if the information was included, it was recorded in the
form. The data extraction form is show in Table 5.
Table 5
Data Extraction Form
Data Item
Description
Focus of article
State the main objective of the article
Attributes of change classification
scheme and definitions
List any categories of changes identified by
the article and their definitions
Description of changes
Describe the different types of changes that
were made
Impact analysis techniques
Describe how the techniques are used to
perform impact analysis
Impact quantification
Describe how the impact was measured
Software architecture impact
Describe how the system architecture was
impacted by any change
Changes to software architecture
Describe what exactly changed in the
system architecture
Quantitative Results
Record quantitative empirical results
Qualitative Results
Record qualitative empirical results
Related References
Record additional references that pertain to
the research questions
Using the data extraction form, I reviewed every paper and extracted the data. Then,
my advisor, Jeffrey Carver, independently reviewed and extracted data from a sample of
the papers. We then analyzed our extracted data for consistency. We found that we had
consistently extracted information from the sample of papers. This process is consistent
with the process followed in previous systematic reviews [83, 90, 93, 124]. The data
extracted from all papers was then synthesized to answer each research question. As a result
of this systematic review, the Software Architecture Change Characterization Scheme
(SACCS) was developed, which is described in Section 4.3. The following section
describes three empirical studies used to assess the characterization scheme throughout its
development.
3.2 Empirical Studies of SACCS
To examine the viability of SACCS, three controlled experiments were conducted
at Mississippi State University in a split-level (i.e., graduate and senior undergraduates)
Software Architecture and Design Paradigms class. The first study was performed as part
of my Master’s Research, but it is included here for congruity with the other two. The three
studies had the goal of evaluating the usefulness of SACCS and suggesting improvements.
The three studies built on each other and testing slightly different hypotheses. The studies
were executed once per year for three years. The common
objectives the studies are:
Objective 1: Determine if SACCS provides a more accurate assessment of
architecture impact prior to making a change than an ad hoc method
Objective 2: Determine if SACCS provides utility to the developers
Objective 3: Identify ways SACCS can be improved for use in a real
development environment
The remainder of this section discusses the objectives, design, and data collection
for each study. Section 3.2.1 presents the experimental design that was common to each
study and the remaining subsections highlight specific goals, questions, and hypotheses
derived for each study.
3.2.1 Experimental Design
Each study was conducted in a classroom setting during one or more homework
assignments toward the end of their respective semesters. There were between 14 and 25
subjects in each study.
The subjects used artifacts from the Tactical Separation Assisted Flight
Environment (TSAFE) system, a tool designed to aid air-traffic controllers in detecting and
resolving short-term conflicts between aircraft [63]. A gold-standard implementation of
TSAFE contains about 80 Java classes and 20K lines of source code. Prior to the beginning
of each study, the subjects used the original TSAFE requirements to create their own
architecture document for the TSAFE system as a semester-long project. The process of
creating their own TSAFE architectures helped familiarize them with the system and gave
them experience with the architectural views taught throughout the semester.
The focus of each of the three studies was assessing architectural impacts, because
architecture changes tend to have an adverse effect on system quality when implemented
without taking the necessary precautions to prevent degradation [117]. SACCS was
designed to help developers understand the impact of a change prior to implementation. At
some point in each study, we gave the subjects change requests and required that they
characterize the requests using SACCS prior to making the changes. The students used a
“gold standard” version of the TSAFE architecture to make their changes rather than the
one they had created earlier in the semester. Each change, unless otherwise noted, was made
to the original TSAFE gold standard architecture. Table 6 lists the change requests used in
the studies. Prior to each study, I characterized and studied the implementation of each
change request. The characterizations were based on the actual implementation of the
changes.
Table 6
Study Change Requests
# - Name
Description & Impact
1 – Conformance
Monitor (Study 1, 2,
3)
Calculate whether flights are on set courses and visually
alert ATC if not. Add module, determine interface, and
change GUI classes.
2 – Feed Display
(Study 1)
Add connections to data feed to display raw flight
coordinates to ATC. Transfer data from low-level classes that
handle raw flight data to GUI modules.
3 – Loss of
Separation Detector
(Study 2 and 3)
Visually alert ATC when 2 flights are within certain distance
from each other. Add module, determine interface, and
change GUI classes.
4 – Dynamic
Mapping
(Study 3)
Include a new textual output window for presenting the
results (e.g. flight data); updates are in the same interval as
in the graphical output
5 – Command Line
Interpreter (Study 3)
Insert into the menu the option to change maps while
running TSAFE by adding a new tab to the “Parameters”
dialog called “Dynamic Maps.”
Qualitative and quantitative data was collected to assess the fulfillment of study
objectives and answer questions posed in each study. The qualitative data was obtained
from questionnaires, surveys, and experience reports submitted by the subjects at the end
of each study. The questionnaires also solicited information on how the characterization
scheme could be improved. Quantitative data was provided by the subject’s use of
electronic forms, which recorded their implementation details for each change. This data
included the number of modules and components changed and which architecture views
would be affected by the change. The subjects were required to make changes to the
software architecture documents and record changes in architectural structure using
Microsoft Visio®. They also used an architecture change detail form that required them to
list which modules and connections change in the architecture and provide rationale
concerning their decisions. Finally, the modified architectures were collected from each
subject in order to analyze the exact changes made to the system architecture. Sections
3.2.2 – 3.2.4 provide an overview of each of the 3 studies.
3.2.2 Study 1: Preliminary Examination
The initial change characterization scheme developed during my master’s research
(described in Section 2.6) was used in a preliminary study to assess its viability.
This study was designed to provide evidence with regard to the usefulness of the scheme.
The overall goal of this study was to gain insight into the use of the first version of the
architecture change characterization scheme and assess its viability. The goal for this study
presented in GQM format is as follows [25]:
Analyze the software change characterization scheme in order to
characterize it with respect to usability, effort prediction and architecture
impact estimation from the point of view of the researcher in the context of
a classroom study
The purpose for running the experiment was to determine if the initial change
characterization scheme created was of any practical use to a developer making a software
change. Based on these requirements, the research hypotheses were as follows:
H0: Developers will consistently characterize changes in the architecture
change characterization scheme
H1: Changes of different classes will require different amounts of effort to
implement.
H0 was chosen to test whether the subjects would first, use the scheme as it was
designed and second, reflect an adequate understanding of its attributes in characterizing a
change request. H1 was selected to test whether changes with different characterizations
required different amounts of implementation effort. This requirement is fundamental for
creating a scheme to assess the impact of various changes. If different classes of change do
not require different amount of effort, then the characterization scheme is not useful in that
regard.
3.2.3 Study 2: Further Assessment and Consensus Development
The main goal of this study was to gain insight into the feasibility and usefulness of
the architecture change characterization scheme. Stated formally in GQM format, the goal
was:
Analyze the architecture change characterization scheme in order to
understand it with respect to usability, viability, and architecture impact
estimation from the point of view of the researcher in the context of a
classroom study
The questions addressed include:
1. How well did the change characterization by the subjects match the change
characterization by the researchers?
2. Is the characterization scheme easy to use?
3. Do changes that exhibit different characterizations require different amounts of
effort to implement?
4. Does the scheme support effort estimation?
5. Does the scheme add value to the change process?
6. Does the scheme facilitate communication amongst developers?
Answers to these questions provided further insight into the usefulness of the
characterization scheme and identified its strengths and weaknesses.
3.2.4 Study 3: Quality Changes
This goal of the third study was to determine if SACCS allows the developers to
produce a higher quality change versus an ad hoc (or some other) method. We also wanted
to determine how well SACCS could be used to estimate relative effort when compared to
other changes and to determine if experienced SACCS users perform better than
inexperienced users. The GQM goal for this study was as follows:
Characterize the Software Architecture Change Characterization Scheme
(SACCS) with respect to quality software architecture changes, effort
identification, and change impact assessment from the point of view of the
researcher in the context of a classroom study.
In addition to the standard study objectives listed in Section 3.2, additional
hypotheses for the third study are as follows:
H2: A higher quality architecture will be produced when making changes
using SACCS than with an ad hoc method.
H3: Experienced SACCS users provide a more detailed assessment of
architecture impact and produce a higher quality architecture than
nonexperienced users.
If the systematic approach to change understanding that SACCS facilitates
translates to a higher quality change than without using it, this finding would add to the
benefits of SACCS.
3.3 Historical Data Analysis Overview
Code Decay (i.e., the situation where changes are harder than they should be) is
evident in ongoing software development. Developers need a way to identify the causes
and effects of code decay in order to prevent future decay and to measure the effect of
improvements to a decaying system. These product measurements (i.e., metrics) that
identify code decay have not been fully examined. This work identifies and tests metrics
that are indicators of code decay to provide a baseline set of relevant code decay metrics
that can be used when making decisions about the quality of software maintenance
activities.
3.3.1 HDA Goal, Questions, and Hypotheses
The goal of the historical data analysis research is to identify a set of metrics that
provide evidence of system decay. This goal is accomplished by analyzing change impact
data and effort data recorded during implementation of change requests. One result of code
decay is an increase in the SPAN of change over time. SPAN is defined as the number of
source files that are modified to implement a change request. An increase in SPAN indicates
that the system is decaying and increasing in complexity. It also reflects a decoupling of
system functions, because multiple files must be changed to fix a single issue [65].
If code decay exists, the next question that arises is: Have there been any changes
designed to address that decay? These changes are called refactorings. A second goal is to
understand the effect of these ‘refactoring’ changes.
Code decay is realized as system changes become increasingly more difficult to
implement [65]. For open source software systems, effort is typically measured in the
amount of work done to close an issue. Checkout time (i.e., the amount of time a file was
checked out) does not accurately show the amount of effort spent working on the file. For
open source software, change effort is determined by using the number of LOC, files,
packages, or revisions required to implement the change [80, 96, 130]. An increase in SPAN
should correlate to an increase in effort. Each file that must be changed adds to the amount
of work performed for each change. Systems that have not decayed should exhibit a higher
quality (determined by relative metric values) than systems that have decayed [43, 47, 54,
158]. The following hypotheses were created specifically to address hypothesis H4
described in the overall research approach in Section 1.3.
H4.1
Code decay is evident in ongoing software development. Systems
will show a linear/sub-linear increase in the number of files touched
over time.
H4.2
There will be an increase over time in implementation effort
required for ongoing changes for systems that have decayed. The
number of revisions, LOC, and packages modified will increase over
time.
H4.3
There is a correlation between an increase in the SPAN of changes
and a set of software metrics that can be used as code decay
indicators.
Each of the above hypotheses address the issue of code decay. Another hypothesis,
H3, from Section 1.3 has been addressed by analyzing specific instances of refactoring
changes for the subject systems. For each refactoring instance, several revisions were
downloaded prior to and after the refactoring to assess the impact the refactoring had on
the product metrics. This refactoring impact is measured for the system as a whole or for
the affected subsystems and modules. Hypothesis H2 in Section 1.3 is addressed by the
results of the overall outcome of the historical data analysis
activities.
3.3.2 HDA Extraction Overview
Open source software (OSS) systems were the targets for the historical data
analysis. OSS uses the Subversion version control system to provide access to systems
revisions. These revisions reflect the state of the system at a particular point in time. When
changes are made to a revision, even if the change is to an individual file, a new revision is
created. The new revision represents all the files of the previous revision and any file(s)
that were modified during the most recent file check-in.
Revisions were downloaded for each month for several active Apache OSS systems.
The systems were chosen based on several key characteristics: written primarily in Java,
has at least two years of data for analysis, uses the JIRA issue tracker for bugs and change
requests, and uses Subversion for version control. The JIRA issue tracker and Subversion
version control system were chosen because of the availability of system documentation,
their popularity, and because they provided a standard format for data that could be
consistently accessed for each OSS system through a standard set of
analysis scripts.
For each revision downloaded, Understand Analyst [5], a static analysis tool used
to determine system quality with software metrics, was used to extract a robust set of
product metrics (described in Section 5.3) that did not require the system to be compiled
during metrics generation. The JIRA issue tracker was used to determine the amount of
effort required for each issue. JIRA allows system stakeholders to report bugs and request
changes to the system. These bug and change requests are then assigned to a developer. The
developer making the change provides commentary on what was done to make the change
and the system tracks the revisions that are required. These revisions record the number of
LOC, classes, and packages modified. Additional details on the extraction process, systems,
and metrics analyzed can be found in Sections 5.1 to 5.3.
CHAPTER IV
SOFTWARE ARCHITECTURE CHANGE CHARACTERIZATION SCHEME
This chapter details the work performed to create the Software Architecture Change
Characterization Scheme (SACCS). Section 4.1 describes the motivation for this work.
Section 4.2 presents the finding of the systematic review described in Section 3.1 and
Section 4.3 unveils SACCS.
4.1 Motivation
This systematic review to identify and classify architecture changes was conducted
to assist in performing software maintenance. A needs assessment indicated several key
areas that must be addressed to improve the software change process.
Change Understanding and Architecture Analysis: Prior to making a change, it is
important for a software developer to understand how it will impact the architecture. A
change analysis tool should allow the developer to analyze a change prior to
implementation to understand the change, the architecture, and how the change fits with
the architecture [18, 32, 68, 139].
Build Historical Baseline of Software Change Data: The ability to compare a
change request to a system’s history provides insight into the change impact, difficulty, and
required effort. Recording information about the change type, its impact on the architecture
and the effort required provides insight into future changes [108].
Group Changes Based on Impact/Difficulty: Change requests can be grouped based
on their characteristics. Similar changes should exhibit a similar impact on the system.
Heuristics can be developed to handle certain types of changes [138].
Facilitate Discussion Amongst Developers: Methods that facilitate discussion
among the development team are useful in achieving consensus on the implementation
approach. A characterization scheme should facilitate consensus building by providing a
list of potential violations of the planned architectural structure [86].
Facilitate Change Difficulty/Complexity Estimation: The characterization scheme
should allow a developer to determine change complexity as a function of type and size.
Characterizing the context of the change request (i.e., influencing factors external to the
request) should also help facilitate difficulty estimation because certain types of changes
may be more difficult in certain domains than others [101].
The output of this review is the creation of the SACCS, which describes the effects
that changes can have on architecture. The attributes of the scheme were extracted from
change taxonomies and associated change characteristics identified during the review.
4.2 Review Findings
This section details the findings of the review. The results of the review are
presented to answer each Research Question listed in Table 2 (Section 3.1.1). This
information provides the basis for the creation of the SACCS described in Section 4.3. All
of the characteristics discussed as a result of the research questions in Sections 4.2.1 to
4.2.5 were considered for inclusion in the SACCS framework. The characteristics that were
not included were rejected because of the exclusion criteria described in Section
4.2.6 and are listed in Section 4.3.4.
4.2.1 Research Question 1
What are the attributes in existing software change classification taxonomies?
There were many studies that reported on change classification taxonomies. Several
studies focused on classifying source code changes. Other studies identified organizational
management and external factors as influencers of change implementation. There were also
studies that examined the features of the change requests to determine their effects on the
system. And finally, there were change taxonomies that focused on how system designs
would be affected by a change. Based on an examination of the general characteristics of
change classification taxonomies, the major categories of change taxonomies are listed in
the following Sections 4.2.1.1 to 4.2.1.3.
4.2.1.1 Prescriptive Change Types
Prescriptive changes ‘prescribe’ a course of action that is typical for addressing that
change type. Change taxonomies of this category generally originated from the work of
Lientz and Swanson [113, 162]. Much of their work was incorporated in the text on
software engineering by Sommerville [156]. They identified four types of maintenance
activities: perfective, corrective, preventative, and adaptive. There have been numerous
studies on these four types (or a subset of the four) that attempt to measure frequency and
potential impact [11, 24, 37, 65, 88, 91, 128, 132, 133, 144, 153].
Other researchers have expanded on the four major change types. Chapin, et al.,
identified evaluative, consultive, training, updative, reformative, performance, groomative,
reductive, in addition to the general change types which he called enhancive, adaptive, and
corrective [49]. Lin added retrenchment, retrieving, prettyprinting, and documentation to
the adaptive and corrective change types [114]. Aoyama, et al., analyzed changes to design
patterns and created design evolution patterns: intensive evolution (e.g., requirements
change, bug fix, design improvement), extensive evolution (e.g., new requirements,
accommodating new operating environment) and evolution operations (e.g., module
replacement, connection change) [15].
4.2.1.2 Source Code Changes
Some research has focused specifically on source code changes. Kim, et al.,
described a taxonomy of signature changes, that is, small changes to function names,
parameters, or orderings in source code [92]. Ren, et al., developed a taxonomy that
includes adding, deleting, and modifying fields, methods, and classes in source code [149].
Others have looked at atomic changes and their effect on code structures such as scope
changes, inheritance deviation, signature changes, modifier, attribute, class declaration,
interface and variable changes [51, 53, 69, 100]. Van Rysselberghe and Demeyer observed
frequently applied changes and classified their causes as introduction of duplicated code,
repositioning a code fragment, and temporarily adding a code fragment [167].
4.2.1.3 Organizational Influence
External factors have been identified in change taxonomies as a way of determining
how the development organization influences change management and implementation.
For example, developer experience is a characteristic of the organization responsible for
implementing a change request [62, 116]. The Prism Model of changes classified changes
based on their effect on environmental infrastructures (change and dependency structures).
The dependency structures outlined in the Prism Model defined representation of factors,
which included people, policies, laws, processes, and resources that affect change
implementation from the organizational standpoint. His change structure facilitated the
classification, recording, and analysis of change data [119, 120].
Changes have also been characterized based on their origin, cause, process
elements, phase, kind of change, and modifier (developer responsible) [135]. Others have
found organizational profiles that are important in understanding change including project
manager, participants, contractual-constraints, project size, external suppliers, customers,
operating-platform, and implementation-languages [58]. Finally, Lam identifies ten change
management issues that are essential to an effective change management process including
the importance of stakeholder consensus in making changes and assessing risk [102].
In summary, each of the papers discussed in this section provides different
approaches to characterizing software changes. The various attributes are a means of
classifying the features of a change request and interpreting its effect on the system.
There are many attributes that must be included in a comprehensive change framework.
All of the above attributes were considered for SACCS, but they were not all included.
4.2.2 Research Question 2
How are software architecture elements and relationships used when determining
the effects of a software change?
Software architectures are described in terms of components and connectors,
modules and relationships, and the topology that manages the architectural entities.
Software architectures exhibit certain properties such as coupling and cohesion. Coupling
can be used to analyze the complexity of the architecture. Changes to architectures can
positively or negatively affect coupling. Therefore, the change can affect system complexity
or understandability [39, 40, 147]. Architectural degeneration is a phenomenon that occurs
as changes increase coupling, thereby increasing complexity [87]. Architectural evaluations
have been used to determine the relationship of changes to: coupling between modules and
coupling between module classes [117, 165, 166]. In these instances, coupling affected
complexity and understandability. A distinction can also be made between design space
changes (logical structure, interface, and package) and implementation space changes
(build components and header files) [175]. Finally, modules can be characterized by the
frequency of change. Core architecture modules infrequently change while non-core or new
modules are changed more frequently [82].
Another area of focus is characterizing evolution. The “Phasic Analysis” technique
identified six evolution profiles describing how the architecture changes over time: intense
evolution, rapidly developing, restructuring, slowly developing, steadystate, and pending.
[173]. Another framework, created to facilitate component replacement in long-lived
architecture, helps to determine whether a component should be replaced or adapted to new
technology. The framework determines the quality properties defined by the architecture
that will be affected when making a change such as performance issues, stability,
scalability, and compatibility [142]. This research shows how different types of architecture
changes can impact the maintenance process. Barais, et al., developed a framework that
transformed architecture patterns using transformation rules and exhibited transformational
behaviors: superposition, conditional superposition, substitution, and conditional
substitution [22]. Others have pointed out the importance of assessing the impact of
dynamic architecture properties in addition to static architecture properties [67].
All of the topics discussed in this section used architecture characteristics to assess
and make determinations about maintenance. Several of the papers identified express the
importance of the logical decomposition at the architectural level and others address
runtime characteristics. These two facets of software architecture are important when
determining how a system may evolve. The logical and runtime characteristics described
above must be considered for inclusion in SACCS. They describe different aspects of a
system but each aspect provides relevant information for a developer that needs to make a
change.
4.2.3 Research Question 3
How is the architecture affected by functional and non-functional changes to the
system requirements?
It has been shown that for many systems the majority of changes are requested in
order to improve quality and enhance functionality [133]. Software architectures have been
used to document how quality attributes and non-functional requirements are fulfilled [46,
160, 161]. When a modification affects a non-functional attribute (e.g., increasing system
performance) often, the architecture also is affected [30]. The effect of functional changes
is not as obvious. Research Question 3 was asked to determine the effect that changing a
system’s architecture has on system quality and functional enhancements.
A software change can be a strictly functional change (affecting only user
observable attributes), a strictly architectural change (affecting only the architecture,
unnoticeable to the user), or an architectural/functional change (a mix of the two) [134].
The three major types of evolution, considering source code features, that have the greatest
architectural effect are: interface evolution, implementation evolution, and structural
evolution [163]. These categories correspond to the strictly functional (interface evolution),
architectural/functional (implementation evolution), and architectural (structural evolution)
changes. Although strictly functional changes do not impact the architecture, the
architecture does determine the location of the change. Therefore, architectural assessment
is important for all three types of changes.
Functional changes may not result in a structural change to the architecture, but they
do affect the portion of the architecture that is responsible for providing the specified
feature. Therefore, most architecture modules can be characterized based on the features
they provide. There are six functional software areas: data handling (data formats, record
segments, databases or files, and establishment of parameters), control flow (references to
changes in logic and program structure), initialization (source code modifications
establishing constraints or initial data values), user interface (modifications of
humancomputer interfaces), computation (modifications for equations and functions), and
module interface (changes in communication links between modules and/or submodules)
[23, 151]. Another way to look at functional changes is to determine the impact of the
change on a module. The important impact factors include: configuration file changes, data
changes, functionality changes to the source code, and architecture changes which were
additions and/or deletions to architecture modules and connections [123].
Strictly architectural changes are those that have an impact only on the system
architecture. These changes include refactoring and restructuring the architecture to
enhance quality attributes [70, 125]. They refer to internal changes that do not modify the
external behavior of the system [70, 89]. A preventative change is also a change that is
strictly architectural in terms of its impact. These changes involve modifying an
architecture component to ‘prevent’ problems in the future. These changes improve on non-
functional quality attributes such as understandability, modifiability, and complexity [133].
In summary, software architectures have a substantial role in changes that address
non-functional requirements (i.e., quality attributes). They play a lesser role in addressing
purely functional changes. There exists a spectrum of changes ranging from purely
architectural changes, to architectural/functional changes, and purely functional changes.
Purely architectural changes consist of refactorings and changes to system structure.
Purely functional changes affect some user-observed attribute. Both ends of this change
spectrum should be considered in SACCS. Changes ranging from those that affect the
architecture to small functional changes should be represented in SACCS. Developers
should be provided a way to distinguish between the two ends of this spectrum when
assessing how a change request will impact a system.
4.2.4 Research Question 4
How is the impact of architecture changes qualitatively assessed?
Change impact analysis is important because it can also assist in determining the
amount of effort required to implement the change [137]. Thus, impact analysis can be
approached in two ways. The most common way is to directly determine which code should
be modified. The other way is to determine the consequences of the modification(s) [18].
Change impact can be direct, modules are affected because a specific relationship to a
module that will change, or indirect, a module is affected because of dependencies on the
module that changes [33].
In terms of software architectures, determining the impact of a change can be
challenging. Architecture modules can contain other modules, packages, and classes.
Assessing change impact when architectures are affected involves highly complex
structures and external factors not present when assessing impact from a strictly source
code level. A developer must determine the underlying mechanisms of change and answer
questions such as where (location of change), when (temporal properties), what
(system properties), and how (change support) at the system and organization level [45].
Developers must also assess the abstraction level where the change takes place (i.e.,
module, subsystem, design unit, architecture, systems of systems, etc.) [126].
Architecture impact analysis can be performed by scaling up low-level, source
analyses [163]. Program traces and module dependency techniques are performed to
determine which modules must be changed along with the target module. Impact analysis
can also be performed dynamically by focusing on the module that executes after the most
recently changed module [17]. Architectural impact analysis can also be performed
statistically by determining the probabilities that a module change is required given that
another module has changed [10].
In assessing architecture impact, subjective ratings based on developer experience
are often used. For example, change impact can be cosmetic (trivial), local or global
(significant impact) [122]. The influence of a requirement change can be weak, average, or
strong [138]. Finally, the impact of a change to various classes can be characterized as low,
medium, high or no impact [137].
Lassing, et al., created a subjective impact scale in their study of architecture
flexibility. The scale included four levels:
1. No impact
2. Affects one component
3. Affects several components
4. Affects the software architecture
They also pointed out the significance of analyzing changes to the micro
architecture (internal components) and changes to the macro architecture (external
components) [103, 104].
In summary, the results for Question 4 show that change impact analysis at the
architecture/design level is subjective and involves determining which system modules will
be affected when making a change. The subjectiveness of the approach allows developers
to assess the level of impact using ordinal rating systems that require extensive developer
experience to provide accurate results. SACCS will take advantage of developer
experience by using the various types of rating scales identified above to create a
comprehensive method to subjectively rate the impact of a change request.
4.2.5 Research Question 5
What types of architecture changes can be made to common architectural views?
The architecture views used to describe software provide the architect with a means
of explaining the architecture to stakeholders. Each view presents different aspects of the
system that fulfill functional and non-functional requirements. At a very high-level,
architectures are described in terms of their logical (static) structure and their runtime
(dynamic) structure [81]. The logical views include; dependency relationships, layers,
inheritance structure, module decomposition, and source structure abstractions. The
runtime views include: control flow processing, repository access, concurrent processes,
component interaction, distributed components, and component deployment abstractions
[21, 56, 154].
There are many ways that architecture views can be modified. Logical changes
affect system structure and consist of changes to systems, subsystems, modules, packages,
classes, and relationships. Class hierarchy changes consist of modifications to inheritance
views. Class signature changes describe alterations to system interfaces [60]. Change can
be made to UML diagrams where each diagram type will signify the nature of changes
made to it: class diagrams (i.e., add/delete attributes, change attribute, add/delete method,
change method, add/delete relationship, change relationship, add/delete class, change
class), sequence diagrams and state charts [38]. A more general description includes
changes to entities (i.e., classes, modules, etc.), relations and attributes [173, 174]. Other
types of architecture changes include: kidnapping, splitting, and relocating. Kidnapping is
moving an entire module from one subsystem to another. Splitting involves dividing the
functions of a module to two distinct modules. Relocating involves moving functionality
from one module to another [164].
Runtime changes are identified based on changes to processing entities, the
connections between the processes, and connections between remote components. Changes
to components can have causal dependencies (i.e., behavior in one component causes a
behavior in another component) and ordering dependencies (i.e., where a specific ordering
relation has to be maintained between two or more component behaviors). These changes
included adding/deleting components, adding new components that refine existing
components, and adding/deleting connections and component bindings [35]. It is important
to understand the evolution of architecture components and the communications between
them [84]. There are taxonomies that also describe component changes. These changes
included adding, deleting, modifying, or substituting components, connectors, ports, and
services [16, 143, 150, 152].
In summary, there are many ways to reflect a change to the architecture in the
architecture diagrams and in the source structure. The literature describes many ways to
change high-level architecture components and their interactions. Each element in an
architecture diagram has the potential to be affected by a software change. SACCS should
allow developers to determine how the architecture should change prior to making the
change in the source code. In order to do this, SACCS must contain representative
architecture views and list the ways that each view can be modified given the types of
modifications described above.
4.2.6 SACCS Exclusion Criteria
In the previous section, a large number of potential attributes for SACCS was
identified. While each attribute has some value, including them all in SACCS was not
feasible due to the large number of attributes that were available. Therefore, exclusion
criteria were created to systematically decide whether each attribute should be included.
The 5 steps described in Table 7 were used as a method of filtering out attributes that should
be excluded from the SACCS.
Table 7
SACCS Exclusion Criteria
Steps
Title
Definition
1
Subset of another attribute
The attribute is a more specific instance of
an attribute that is include in SACCS
2
Not directly relevant to
focus on software
architecture
The attribute does not directly impact the
process of changing the architecture
3
Not relevant to scope and
goals of study
The attribute focused on characteristics that
are outside the scope of this study
4
No extensive literature
backing
The attribute did not have support from
more than three sources
5
Proposed Outcome of Study
The attribute is an output of the
characterization rather than an input.
4.3 Presentation of SACCS
After reducing the overall set of identified attributes using the exclusion criteria
defined in Table 7, the attributes were organized into a characterization scheme. The goal
of this scheme was to assist developers in making decisions about how to address a change
request.
4.3.1 Characterization Scheme Overview
SACCS was designed to capture the effects of changes to architecture and provide
a structured approach for impact analysis. To use SACCS, a developer characterizes a
change request beginning with high-level characteristics then progressing to more detailed
characteristics. The high-level characteristics describe the change’s motivation, type, size,
impact on static, impact on dynamic properties and effect on requirements
(functional and non-functional). The detailed change characteristics identify specific
changes that must be made to the major architectural views. The two-level hierarchy for
SACCS addresses both the context of the change request and its impact on the software
system.
SACCS is organized as a set of characteristics that group together two or more
related attributes. All of the characteristics and attributes originated in the literature. The
remainder of this section discusses the change characteristics and associated attributes in
detail. For each characteristic identified, the attributes are shown in rectangles in the figure
on the left and listed in the table on the right along with the references from which they
were drawn. Any attribute that is a subset of an included attribute is listed along with the
figure. Finally, for completeness, the attributes that were not included in SACCS are listed
in Table 21 along with an explanation of their exclusion.
4.3.2 General Characteristics
These high-level characteristics are used to describe how a change affects the
system and development environment. In the following subsections, each characteristic is
described in detail. In the description, the characteristic is listed in italics and the attributes
are in bold. In the figure that follows each characteristic, the shape with the bold outline is
the general characteristic. The shapes outlined with a dashed line are the values (attributes)
that can be selected for each characteristic. Attributes that are shaded use the Overall Impact
Scale (Table 8). This scale ranges from a rating of ‘0’ (no impact), meaning the change will
not have an effect on that attribute to ‘4’ (major focus of change), meaning that the change
will drastically affect that attribute [58, 104, 122, 137,
138]. Attributes that are not shaded are orthogonal and allow only a single attribute
selection for the characteristic. Figure 4 provides an example of a characteristic, an
orthogonal attribute (not shaded), and a shaded attribute that uses the Overall Impact Scale.
Figure 4
General Characterization Shapes Key
Table 8
Overall Impact Scale
Rating
Name
Description
0
No impact
The property will not be affected by the
change request
1
Cosmetic impact
The property will be minimally effected with
only a surface level impact
2
Minor impact
The property must be considered when
planning the implementation of the change
request
3
Substantial impact
This property will require considerable
attention during the planning,
implementation, and validation of the change
request
4
Major focus of
change
This property is one of the primary reasons
for the change request in will require an
extensive amount of resources to complete
4.3.2.1 Motivation
The first characteristic is the motivation for the change (Table 9). The change can
be motivated by either the need for an enhancement (i.e., to improve the system) or in
response to a defect (i.e., resulting from an error, fault, or failure) [156]. The relative
frequency of defects vs. enhancements will, over time, provide insight into system
maintenance. An increase in the relative number of defects over time may suggest that the
system quality is declining because more defects are introduced during the maintenance
process [26].
Table 9
Motivation Characteristic
Enhancement [95, 156]
Defect [95, 156]
4.3.2.2 Source
The source characteristic classifies the origin of the change request as one of four
types (Table 10). First, the resource constraint attribute indicates that the source is a change
in available resources or development environment (e.g., reduction in memory available,
reduction in available communications protocols, or reduction in budget). Second, a change
in a law or government regulation that affects the software’s domain is classified as
law/government regulation. Third, the source could be a change in organizational policy.
Finally, the source could be a stakeholder request to address changing needs.
Table 10
Source Characteristic
Resource Constraint [119, 120]
Law/Government Regulation [119,
120]
Policy [119, 120]
StakEholder Request [119, 120, 135]
4.3.2.3 Criticality/Importance
The criticality/importance characteristic contains five attributes that dictate the
consequence of making the change (Table 11). Risk indicates that a change that has a
greater than normal risk of failure or poses external risk to the organization. Time indicates
that the change must be implemented within a shorter than normal timeframe. The cost
attribute indicates that the change has greater than normal budget/resource constraints. The
safety attribute indicates that the change has implications on the safety of its users. The
requested attribute indicates that the change was requested by a stakeholder, but is not of
critical importance to the organization or software. If the requested attribute is selected,
then none of the other attributes will be selected. This feature is provided because not all
change requests are of critical importance to the system. The developer can use more than
one of the other attributes to indicate the criticality of the change request when the requested
attribute is not selected.
Table 11
Criticality/Importance Characteristic
Risk [102]
Time [136]
Cost [58, 136]
Safety [58]
Requested [119, 120, 135]
4.3.2.4 Developer Experience
The developer experience characteristic (Table 12), provides a way to assess how
well the developer(s) implementing the change request understand the system architecture
[62, 116, 119, 120, 136]. The experience of the software development personnel is an
important factor in change difficulty and in effort prediction. A minimal rating indicates
that the developer has little experience with the architecture and the components related to
the change. A localized rating indicates that the developer is experienced with the subset
of the architecture related to the change, but not with the entire architecture. Finally, an
extensive rating indicates that the developer is deeply familiar with the entire architecture
[62].
Table 12
Developer Experience Characteristic
Minimal [62, 116, 119, 120, 136]
Localized [62, 116, 119, 120, 136]
Extensive [62, 116, 119, 120, 136]
4.3.2.5 Category
The category characteristic (Table 13) classifies the type of change as perfective,
corrective, adaptive or preventative. Recording the change category is important for several
reasons. The frequency of change types can provide developers with insight about the
evolution of the system. For example, Swanson indicates that frequent adaptive changes
may be a reflection on poor system portability. Conversely, frequent perfective changes
may indicate a more mature system where maintainability is improving [162]. These
conclusions may not be true for all systems in all organizations. But, as a history of changes
develops, organization-specific insights will arise.
Table 13
Category Characteristic
Corrective [7, 9, 11, 24, 37, 49, 65, 88,
91, 113, 114, 128, 132, 133, 144, 153,
156, 162] – Subset attributes:
intensive evolution
Perfective [7, 9, 11, 24, 37, 49, 65, 88,
91, 113, 128, 131-133, 144, 153, 156,
162] – Subset attributes: performative,
groomative, reductive, enhansive,
anticipative, evolutive, design
evolution
Preventative [9, 11, 24, 37, 65, 88,
128, 131-133, 144, 156]
Adaptive [7, 11, 24, 37, 49, 65, 88, 91,
113, 114, 128, 131-133, 144, 153, 156,
162] – Subset attributes: extensive
evolution
4.3.2.6 Granular Effect
The granular effect describes the extent to which the change affects the architecture
(Table 14). Functional/module changes affect user-observable attributes and functions of
the system. These changes are within a single module. Subsystem changes have both a
functional and an architectural impact because they affect user functions across modules.
Purely architectural changes affect only the architecture and not userobservable functions
[134]. Architectural changes, which often take place to satisfy a quality attribute or non-
functional requirement, are often referred to as refactoring or restructuring [34]. System of
system changes are large-scale changes that affect interactions and architectural
components between disparate systems.
Table 14
Granular Effect Characteristic
Functional/Module [123, 134, 163]
Subsystem [126] – Subset
attributes: micro-architecture
changes
Architectural [70, 89, 125, 168] –
Subset attributes: restructuring,
refactoring, architecturally
significant change, structural
changes, macro-architecture
changes
System of Systems [126]
4.3.2.7 Properties
The properties characteristic determines the impact of the change on the logical and
runtime structures (Table 15). A static change affects logical properties, such as module
decomposition, module dependency, the inheritance structure and other static properties. A
dynamic change affects data propagation, the behavior of distributed components, the
execution of concurrent processes, and other runtime behaviors.
Table 15
Properties Characteristic
Static [67, 81, 82, 175]
Dynamic [67, 81, 82]
4.3.2.8 Features
The features characteristic (Table 16) determines how the change request will affect
the functional requirements of the system. The characteristic identifies impacts on the
following areas of the system [23, 27, 37, 119, 132, 149, 151]:
devices: hardware devices used by the system
data access: receipt of data from external systems/repositories
data transfer: flow of data from system to external systems
system interface: software interfaces with external systems
user interface: human-computer interaction interfaces
communication: protocols used to interface other systems/data
computation: algorithm functions and modification of data
input/output: format of information processed by system
It is expected that the design of a system will determine which of these aspects are
affected by a change [50]. After analyzing the change request to determine which of the
above system features will be impacted by the change, the architecture modules that handle
those system features must be identified during change analysis. This analysis helps
developers and testers focus their effort in the right place.
Table 16
Features Characteristic
Devices [23, 37, 149, 151]
Data Access [123]
Data Transfer [23, 151]
System Interface [23, 151]
User Interface [23, 151]
Communication [23, 151]
Computation [23, 151]
Input/Output [23, 151]
4.3.2.9 Quality Attributes
The quality attributes (Table 17) are areas that are impacted when the change
addresses a software quality attribute. The list includes the six quality attributes from ISO
Standard 9126 (usability, reliability, functionality, portability, maintainability, and
efficiency) plus additional attributes identified by Krutchen (availability and scalability) [8,
98]. Evaluation of the architecture is critical when addressing a change that focuses on a
non-functional (quality) attribute because the architecture may determine whether the goal
can be met. [30].
Table 17
Quality Attributes
Usability [8]
Reliability [8]
Functionality [8]
Portability [8]
Availaibility [98]
Maintainability [8]
Scalability [98]
Efficiency [8] – Subset attributes:
performance change
4.3.2.10 Logical
The logical characteristic (Table 18) includes the general architectural features that
can be used to describe the static framework of most object-oriented software. These
characteristics include: dependency relationships, layers, inheritance structure, module
decomposition, and source structure. Once a logical area is identified as important (based
on its rating on the Overall Impact Scale), the change should be characterized in more detail
using the related Specific Characterization framework, which is described in
Section 4.3.3.
Table 18
Logical Characteristic
Dependency Relationship [21, 56,
154]
Layers [21, 56, 154]
Module Decomposition [21, 56,
154] – Subset attributes: coupling
between modules
Source Structure [21, 56, 154] –
Subset attributes: header file
changes
Inheritance Structure [21, 56, 154] –
Subset attributes: inheritance
deviation
4.3.2.11 Runtime
The runtime characteristic (Table 19) lists the dynamic architecture attributes
common to most object-oriented architectures. These characteristics include: control flow
processing, repository access, concurrent processes, component interaction, distributed
components, and component deployment. Similar to the logical characteristic, once a
runtime area is identified as being important, the change should be characterized in more
detail using the related characteristic from the Specific Characterization framework.
Table 19
Runtime Characteristic
Control Flow Processing [21, 56,
154]
Concurrent Processes [21, 56, 154]
Distributed Components[21, 56,
154]
Repository Access [21, 56, 154]
Component Interaction [21, 56, 154]
Component Deployment [21, 56,
154]
4.3.2.12 Complete General Characterization Scheme
The general characteristics described in the previous subsections, together make up
the overall characterization scheme shown in Figure 5. In this figure, the arrows dictate the
that should be followed when characterizing a change. The specific characterization, which
allows the developer to provide more detail about the effect of the change on the logical
and runtimes structures, is described in the next section.
Figure 5
General Characteristics
4.3.3 Specific Characterization
The purpose of these characteristics is to allow the developer to analyze the
architecture in more detail to determine how to implement the change. The Specific Impact
Scale found in Table 20 describes the magnitude of the changes that can be made to the
various architectural structures.
Table 20
Specific Impact Scale [103, 104]
Rating
Name
0
No impact
1
Small impact – single module/component
2
Small impact – multiple modules/components
3
Significant impact – single module/component
4
Significant impact – multiple modules/components
The logical and runtime characteristics focus on static and dynamic relationships
among architectural elements. The goal of the specific characterization scheme is to
indicate, in a comprehensive manner, which portions of an object-oriented architecture are
affected when implementing a change [16, 35, 38, 84, 143, 150, 173, 174].
The logical and runtime characteristics selected during the General Characteristics
analysis are further elaborated for the Specific Characteristics. For Figure 6 and Figure 7
below, the developer selects values from the Specific Impact Scale that correspond to the
level of change necessary for each item represented by an oval in the figures. For example,
if a change request requires changes to a component within a layer and the addition of a
new layer, the developer would select a value from the Specific Impact Scale for the ‘(Add,
Remove, Modify) Layer’ and ‘(Add, Remove, Modify) Layer Module’ list of change
actions for the ‘Layers’ view.
4.3.3.1 Logical Views
The logical characteristics describe the types of changes that can be made to
elements of any view that exhibits those characteristics. Figure 6 provides a visual overview
of these characteristics along with the types of changes that can be made. These changes
include adding, modifying, and removing elements and/or the connections between them.
Figure 6
Logical Views
The Dependency Relationships view describes the system modules and the relations
between them. The Layers view abstracts how the system is divided into hierarchical layers.
The Inheritance Structure view depicts the relationship between the modules in terms of
their parent-child-sibling relationships. The Module Decomposition view is the basic view
of the system at varying levels of abstraction. The Source Structure view provides the
representation of the physical location of the source code within the folder.
4.3.3.2 Runtime Views
The runtime characteristics describe changes that can be made to portions of the
architecture that describe the dynamic aspects of the software. These views contain
executable components and connections between those components. The types of changes
that can be made to different parts of the architecture are shown in Figure 7.
Figure 7
Runtime Views
The Control Flow Processing view shows how system processes interact through a
pipe-and-filter representation of the architecture. The Repository Access view shows the
system in terms of its database and accessor relationship. The Concurrent Processes view
shows the way processes interact as system threads while the Component Interaction view
shows processes interaction through the sharing of information through a publish-and-
subscribe architecture view. The Distributed Components view shows how remote
processes interact and the Component Deployment view shows the components and their
location on system hardware.
4.3.4 Excluded Attributes
For the sake of completeness, Table 21 lists the characteristics that were identified
in the literature but not included in the SACCS. The attributes are organized relative to the
particular steps described in Table 7 that resulted in their exclusion.
Table 21
Excluded Characteristics/Attributes
Steps
Title
Attributes
2
Not directly relevant to
focus on software
architecture
documentation, function name change,
parameter change, orderings change, [fields,
methods, classes] modification, variable
scope change, modifier change, attribute
change, variable change, semantic change
3
Not relevant to scope and
goals of study
prototype change, coupling between classes,
steady-state development, pending changes,
replacement changes, slowly developing
changes, substitution change, configuration
file changes, temporal change properties,
design unit change, evolutionary couplings,
[kidnapping, splitting, relocating] modules,
causal dependency change, ordering
dependency change
4
No extensive literature
backing
evaluative, consultive, training, updating
reformative, retrenchment, retrieving,
prettyprinting, user support change,
duplicated code change, reposition a code
fragment, temporarily adding code
fragment, superposition change, direct
impact changes and indirect impact changes
5
Proposed Outcome of Study
coupling understandability, coupling
complexity, change process improvement,
feedback loop, determining change
consequences, statistical change impact
CHAPTER V
HISTORICAL DATA ANALYSIS
This section presents the details of the historical data analysis. Section 5.1 provides
an overview of the systems analyzed. Section 5.2 explains the extraction process. Section
5.3 lists the metrics collected and explains how they are used. Finally, Section 5.4 describes
the analysis process used to test the hypotheses.
5.1 Systems Analyzed
The goal of this research is to establish a framework for the analysis of historical
change data to determine to what extent a software system’s code has decayed. A secondary
purpose is to determine any measurable effects that code decay has on continuing
maintenance tasks. The initial targets for this research are open-source software systems.
The change and complexity metrics used in this research were collected by extracting data
from real-world software repositories. The Apache Software Foundation projects were the
targets for analysis. These projects are listed in Table 22.
Table 22
Overview of Apache Systems Analyzed
Name (Apache)
SLOC
# Classes
First Revision
Cayenne
171,695
2339
Jan. 21 ,2007
Derby
103,792
1367
Jul. 31, 2005
Jackrabbit
167,750
770
Dec. 3, 2006
Velocity
65,128
311
Sept. 30, 2000
Each of the projects listed in Table 22 have very active development teams and user
communities. Cayenne is an open source persistence framework that provides object
relational mapping and remoting services. Cayenne seamlessly binds one or more database
schemas directly to Java objects [1]. Apache Derby is an open source relational database
implemented entirely in Java. It has a small footprint and allows users to embed the
database with any Java application [2]. Apache Jackrabbit is fully conforming
implementation of the Content Repository for Java Technology API (JCR). It supports
structured and unstructured content, full text search, versioning, transactions, observation,
and more [3]. Apache Velocity is an open-source templating engine. It permits users to use
a simple yet powerful template language to reference objects defined in Java code and can
be easily embedded into Java applications [4].
The Apache projects use the JIRA® issue tracker and Subversion (SVN) version
control system to manage changes. The Apache projects were chosen because of the
richness of the data provided by the development groups. They record more detail about
each change compared with other open-source projects that were examined. The data in
JIRA and the SVN repository was easily extracted through use of customized scripts. The
core source folders in the SVN trunk for each project were downloaded and used for this
analysis.
5.2 HDA Extraction Process
After identifying the target systems, the complete SVN log for each system was
extracted to a text file. This file contains information about each revision stored in the
Apache SVN repository. For each revision, the file contains the date of the revision, the
developer that made the change, the developer’s comments about the change, and a list of
each file that was edited.
The revision log was then used to identify and checkout monthly revisions for each
system. The first revision of each month was checked out and stored locally. Each system
contained a minimum of two years worth of revision data (or at least 24 downloaded
revisions). The command line version of the Understand parser was used to analyze and
generate a full set of product metrics (described in the next section) for each revision. The
metrics were generated at the package (i.e. architectural) and class levels. Package-level
metrics are averages calculated based on the values for each class in the package. The
metrics generated for each revision were extracted to a comma-separatedvalues file then
copied to a statistical analysis tool (SPSS) for further analysis. At this point in the process,
the data set contained a set of product metrics for all the classes and packages in each
revision.
After generating the quality metrics for each system, the change metrics were
extracted from the JIRA repository. The JIRA repository lists the issues reported by users
of the various applications. Users report bugs and request changes. Developers are assigned
bugs to fix and changes to implement. After a developer corrects the issue and leaves
comments for the community, the issue is marked as closed. Data on every closed issue in
the target system was extracted. This data included:
the issue type (i.e., bug, new feature, and improvement)
issue priority (i.e., critical, major, minor, trivial)
a summary of the issue, a description of the work performed
the developer responsible
open and close dates
and versions affected.
The listing of JIRA issues was used with another script and the revision log to record
the number of revisions for each issue, the number of LOC changed for each revision, the
number of files and architecture modules (packages) changed for each revision. This data
was then aggregated at the issue level to calculate the number of revisions, LOC changed,
files changed, and packages changed. After collecting the quality metrics from the revisions
and the change metrics from the JIRA issues, the data was then analyzed to identify any
relationships between the quality metrics and the change data to assess the research
hypotheses found in Section 3.3.1.
The final step in the analysis was to search the JIRA issue data to identify
refactoring changes and other changes that had a substantial impact on the quality metrics
collected. This step ties SACCS to the historical data analysis by characterizing the
architecture changes found in the repository to determine how different change
characteristics impacted system quality.
5.3 Metrics Data
This section describes the metrics that were collected during the extraction process.
All quality metrics were extracted using a static analysis tool from Scientific Toolworks
(SciTools). SciTools develops a set of sophisticated static analysis tools that help
programmers measure, decipher, reengineer and maintain their source code. It easily
handles very large (over 1 million SLOC) code bases. The tools help programmers
understand, document and maintain their software. The flagship SciTools project,
Understand, is a comprehensive source code analysis and management tool that allows
developers to easily extract metrics from source code and generate graphical views of the
architecture [5]. The Understand application has both a graphical and a command line
interface. A script was created to automate the extraction of metrics for the revisions
downloaded.
The metrics that were extracted included metrics related to change implementation,
metrics that identify coupling relationships, metrics about the system architecture, and size
metrics. Table 23 lists the product metrics along with a brief description. Table 24 lists the
process metrics. In addition to these metrics listed, counts of the number of files, classes,
methods, and variables were collected for each package. The complexity and lines of code
measures were also averaged for each class and package. The next section describes the
process for analyzing the metrics data.
Table 23
Product Metrics
Metric Name
Description
Cyclomatic complexity
Number of linearly independent paths through a program
Essential Complexity
Number of edges in a program
Weighted methods per
class (WMC)
Sum of cyclomatic complexity of all nested functions or
methods
Response for a class
(RFC)
Number of different methods that can be executed when
an object receives a message
Percent lack of cohesion
for a class (LCOM)
100% minus the average cohesion for class data
members. Calculates the percentage of class methods that
uses a given class instance
Depth of inheritance
tree (DIT)
Number of levels the inheritance tree contains
Number of children
(NOC)
Number of immediate descendants of the class
Coupling between object
classes (CBO)
Number of other classes coupled to the class. Coupling
means using a type, data, or member from that class.
Fan-in
Number of input references from other classes
Number of paths
(NPATH)
Number of unique paths through a body of code
Number of Instance
[Methods, Variables]
Number of Instance [Methods, Variables]
SLOC
Number of non-commented source lines of code
LOC
Total number of lines of code
Number of Executable
Statements
Number of SLOC that contain executable statements
(e.g., expression, compound, if, while statements etc.)
Table 24
Process Metrics
Process Metrics
Description
Revisions per issue
Number of revisions for each JIRA issue
Files per revision/issue
(SPAN)
Number of files changed for each revision and JIRA issue
LOC per revision/issue
Number of LOC changed for each revision and JIRA
issue
Packages per
revision/issue
Number of packages changed for each revision and JIRA
issue
Change density
Number of changes per unit of time
5.4 Data Analysis
In order to properly analyze the data and generate testable results for the hypotheses,
the metrics listed in the previous section will be referred to as independent and dependent
variables. Independent variables are variables selected to determine the relationship that
they have with some observed phenomenon. The observed phenomenon (code decay), in
this case, are the metrics related to changes. They include the process metrics SPAN,
LOCs/files/packages changed, and change density. The independent variables consist of
the set of product metrics listed in Table 23. The independent variables were chosen
because of they were used in previous studies and have been found to provide an indication
of system quality [43, 47, 54, 158]. The following subsections provide an overview of the
statistical analysis techniques used to process the data.
5.4.1 Descriptive Statistics
For each system, the mean, median, standard deviation, and inter-quartile ranges for
independent variables were examined over the life of the system. These statistics outline
the distribution and variance of the independent variables. The range and distribution of the
measures obtained here will determine the applicability of the subsequent analysis
techniques. This data will also be used in comparison to other empirical studies where a
similar set of metrics were analyzed.
5.4.2 Principal Component Analysis
Principle component analysis (PCA) is another preliminary technique used to
determine the dimensions of the independent variables. PCA will identify independent
variables that are strongly correlated. This technique will help to explain relationships
between the variables prior to running the regression analyses.
PCA produces linear combinations of the independent variables (factors). The PCA
process is conducted by first determining the linear combination of all standardized
variables that explain a maximum amount of variance in the data set. The remaining
components are linear combinations of all standardized variables, where each new
component is orthogonal to all previously calculated components, which captures a
maximum variance under these conditions. These factors can be used in addition to
individual variables but may be more efficient than individual variables in representing the
outcome of the remaining statistical tests.
The primary focus of this analysis is to determine which metrics are redundant. The
model of multiple linear regression (Section 5.4.4) assumes that explanatory variables are
independent. Therefore, only one variable strongly correlated variable of set of variables
should be included in the final model.
5.4.3 Linear Regression
Linear regression examines the relationship between a single independent variable
and a dependent variable. Linear regression will be performed for each independent
variable against the dependent variable. If an independent variable possesses a significant
statistical relationship with a dependent variable, linear regression will capture this
relationship. The analysis also serves to eliminate measures from the subsequent
multilinear analysis if the relationships between the independent and dependent variables
are not significant. The strength of each linear regression analysis is determined by the R2
coefficient of determination which is discussed further in Section 6.2.
5.4.4 Multiple Linear Regression
Multilinear regression is used to model the dependent variable based on all
independent variables obtained during prior analyses. The independent variables that were
deemed highly correlated from the PCA analysis will be analyzed apart from other variables
that they are correlated with (i.e., one variable from each component). Such an approach
will ease the interpretation of the confidence intervals and the p-values of the coefficients.
Overfitting a model increases the standard error of the model’s prediction, making the
model more dependent on the data set it is based on and thus less
generalizable [42].
This model will determine how well we can predict the dependent variables for
successive releases of software systems. This analysis will help in answering the question
of when to refactor a system. The model that is obtained from the analysis will then be put
through a goodness of fit test to determine the practicality of using the model.
The first multilinear regression analysis that was performed was assessing how
metrics can predict change prone files. Using the change density of each class and
architectural package, we can assess the frequency of file changes. Frequent changes are
another symptom of code decay that multilinear regression can predict given a system’s
revision history. If class A is changed more often than class B, then class A is more change
prone than class B. Change proneness has also been used in other studies as an indicator of
effort [97, 112].
Given a set of metrics and the dependent variable SPAN of changes, a developer
can use this regression model to determine how difficult that change should be (i.e.,
estimated number of files touched or LOC modified) given the current set of metrics used
to feed the model. If the change is implemented and is much harder than it should be, the
developer would be prompted to refactor the system. Another potential outcome of this
model is that the developer can plug in hypothetical metric values for the model and
determine when the resulting value of SPAN is too large to be manageable. The system
could then be refactored as a preventative measure.
CHAPTER VI
COMBINED RESULTS
This chapter presents the results of the three empirical studies and the historical data
analysis. Section 6.1 describes the results of the empirical studies. Section 6.2 lists the
results of the historical data analysis, and Section 6.2.3 explains the results obtained by
characterizing the historical changes.
6.1 Empirical Studies Results
The following subsections describe the results of each empirical study. The results
from the Master’s Thesis research are contained in Study 1 and presented here for
congruity. The experimental tasks are presented first along with the results from each study.
Section 6.1.7 summarizes the results from all three studies. The design for the studies can
be referenced in Section 3.2.
6.1.1 Study 1 - Experimental Tasks
This study took place during the final homework assignment of the semester. The
subjects were given feedback on their own TSAFE architecture documents that were
created during a previous assignment and were then given the “gold standard” TSAFE
architecture. The subjects were given 2 successive change requests of different types to
make to the architecture and implement in the source code. The order in which the students
received the change request was randomized.
The change requests were designed to be complex enough to require architectural
modifications while being simple enough to implement in the allotted time. After receiving
the change request, the subjects first characterized the changes using the characterization
scheme. Next, they made the necessary modifications to the architecture. Finally, they
implemented the change in the source code. During these activities, the subjects were asked
to keep track of how many modules they changed and the amount of time they worked.
After completing and submitting the first change request, the subjects were told to
return to the original golden version of the architecture and code (i.e., not including Change
1) and were given their second change request. The subjects were given 3 weeks total to
complete both changes.
6.1.2 Study 1 - Results
The results from study one were used to evaluate hypothesis H0 and H1. Along with
each hypothesis, we provide a series of observations drawn from the associated data
analysis. As this study was part of my Master’s work, detailed results from the study can
be found in my Master’s Thesis [170].
6.1.2.1 H0 Results
H0 stated that the students would characterize the changes consistently. Based on
the data collected, we provide three observations related to this hypothesis.
Observation 1: The subjects characterized the changes similar to the way the
researchers characterized them - This observation is based on the analysis of data
submitted by the 15 subjects who submitted characterizations for Change #1 and Change
#2. The characterizations were made prior to change implementation. If the subjects
correctly understood the attributes of the scheme and how the changes would affect those
attributes, then the subjects’ characterizations should have been similar to the researcher’s
characterizations.
We measured the student characterizations in terms of consistency and accuracy.
Figure 8 and Figure 9 show how closely the median of the student’s characterizations
matches the researchers for Change #1 and Change #2 respectively. In addition, Figure 10
and Figure 11 show the consistency within the students in their characterization of the
changes. The data in these charts show: 1) because the students’ characterizations were
similar to the researcher’s, the students understood the characterization scheme, 2) the
students had a fair understanding of how to use the scheme to assess the level of impact
each change would have on the TSAFE architecture and source code based on the
agreement of their characterizations (Cronbach’s Alpha .842 for Change #1 and .879 for
Change #2), and 3) while the median values of the students’ characterization were similar
to the researcher’s characterization, they were not totally consistent as seen by the
distribution of the values.
Figure 8
Characterization Accuracy (Change #1)
Figure 9
Characterization Accuracy (Change #2)
Figure 10
Characterization Consistency (Change #1)
Figure 11
Characterization Consistency (Change #2)
Observation 2: Survey results show that the change scheme was viewed as being
useful - At the end of the assignment, we asked the subjects to describe any inconsistencies
they found in the scheme and to tell us how it could be improved. We also asked the subjects
to rate their level of agreement with several statements pertaining to the usefulness of the
scheme. The statements read as follows:
The attributes are logical and easily understood (Logical)
The scheme would be beneficial to a developer making a change (Beneficial)
The scheme has practical application in industry (Practical)
The scheme is easy to use (Easy to Use)
After classifying both changes, I had an idea of which would be the most difficult
to implement (Difficulty)
The rating scale ranged from 1 - totally disagree to 5 - totally agree. Figure 12 shows
the results. These results provide some confidence in the usefulness and practicality of the
characterization scheme. For the first three statements, there was only 1 student who
showed any level of disagreement. The fourth and the fifth statement had more
disagreement than the first three statements. The next observation provides some rationale
as to why the “easy to use” ratings were low.
Figure 12
Student Survey Results
Observation 3: The characterization scheme contains some overlapping attributes
and ambiguous definitions One of the survey questions required the subjects to identify any
ambiguity or inconsistencies in the definitions provided for the attributes of the
characterization scheme. This question was meant to elicit information that could be used
to improve the characterization scheme. The answers that were given by the subjects along
with a count (in parentheses) of the number of subjects that gave the response are shown
below:
0
0
0
0
1
0
1
0
4
6
7
5
6
4
2
9
11
9
7
7
2
1
3
3
2
0
2
4
6
8
10
12
Logical
Beneficial
Practical
Easy to
Use
Difficulty
Totally Disagree
Somewhat Disagree
Neither Agree or Disagree
Somewhat Agree
Totally Agree
Need a more clear definition of “System” for the Enhancement changes – could be
system interface (5)
Need a more granular scale for the “Functional vs. Architectural” attribute (5)
Definitions of “Adaptive” and “Perfective” are too similar (4)
Program/Subsystem attribute is not needed (2)
“Documentation” and “Source Code” are the same attributes (2)
“Corrective” class not needed for Defects category (2)
“Interface” definition should be split into separate definitions for system interfaces
and GUI interface (1)
Should define the scale used for impact measurement of each attribute by providing
examples of changes to similar systems and the results the chosen impact value had
on the system (1)
Need a clearer definition of “Static” and “Dynamic” properties (1)
Need a rating for number of modules that will be affected by the change (1)
“Requirements” should be added to the Issues section of the Defect category (1)
The confusion with some of the attributes, illustrated by the above comments,
provides some insight into why the characterization scheme was not viewed as being as
“easy to use” as we hoped. The comments also illustrate why many subjects believed it
would be difficult to predict which change would require more effort simply based on its
characterization. These comments reflect improvements that were made to the scheme in
subsequent revisions.
6.1.2.2 H1 Results
Observation 4: Change #2 (Feed Display) required more effort than Change #1
(Conformance Monitor) - Although most subjects were not able to completely implement
both changes, a majority (12/18) stated that they believed Change #2 required more effort
and was more difficult to implement than Change #1, which was the result expected. This
qualitative response is explained in part by the difference in characterization between the
two changes. Change #2 was classified as having a larger effect on the source code, and
required a larger amount of code to be added to the system.
In addition to the qualitative responses, for each change that was completed, a tool
was used to compute the actual number of modules changed and the number of LOC
changed. The data came from 3 students who completed both changes. Due to the small
number of data points, the statistical results for both the parametric t-test and the
nonparametric Mann-Whitney U test are provided. The results of this analysis showed that
that the subjects changed more modules when implementing Change #2 (7) versus Change
#1 (3). This result was statistically significant (t4 = -16.00, p = .004 [t-test]; Z = 2.121, p =
.034 [Mann-Whitney]). Furthermore, Change #2 also required more LOC to implement
than Change #1 (37 vs. 16). This result was again significant (t4 = -3.854, p = .018 [t-test];
Z = -1.964, p = .04 [Mann-Whitney]).
The statistical results coupled with the qualitative responses from the subjects
provide some evidence that if used correctly, the characterization scheme provides help in
comparing the difficulty of changes.
6.1.3 Study 2 - Experimental Tasks
Twenty-five subjects (22 seniors and 3 graduate students) participated in this study
which occurred during the final two homework assignments. To make the assignments
tractable, and due to the lessons learned from the first study, the subjects were only required
to change the architectural diagrams and not the actual source code.
Table 25 lists the training and experimental tasks.
Table 25
Training and Experimental Tasks
Task
Description
Time
T1
Software change overview
1-hr
T2
T2.1
Architecture change exercise
Review “gold standard” arch.
1-hr
A1
A1.1
A1.2
A1.3
A1.4
Individual arch. change
Record change detail
Group arch. change
Record change detail
Submit experience report
1-wk
T3
T3.1
Change characterization training
Characterization exercises
1-hr
A2
A2.1
A2.2
A2.3
A2.4
A2.5
A2.6
Individual arch. change
Characterize changes
Record change detail
Group arch. change
Characterize changes
Record change detail
Submit experience report
2-wk
A3
Post-study survey
1-hr
A = Activity | T = Training
After the trainings, the subjects were given a single TSAFE change request to
complete for the first homework assignment. For this task, the subjects were required to
analyze the architecture, change the architecture diagrams, record the details of the change,
and provide justification and rationale. After each subject completed their individual
changes, they were randomly assigned a partner. Each pair repeated a similar process that
was done individually. The subjects turned in their updated group architecture diagrams
and detail forms along with a report describing their interaction and comparing the
architecture created by the pair to the ones created individually.
In the second homework assignment, the subjects were given two TSAFE change
requests. The students were asked to return to the original “gold standard” version of the
architecture before making the each additional change. This assignment was given after the
third training session, allowing the subjects to use SACCS. The subjects performed the
same steps as in the first assignment plus the additional step of characterizing the change
requests with the change characterization scheme. The steps followed by the subjects
included: characterization of the change request, modification of the architecture diagrams,
and documentation of the change. The subjects were then assigned a different partner to
perform the changes as a group. Again, each pair had to come to a consensus on the change
characterization, implementation detail, and provide a description of their experiences.
They were asked to describe how they used the characterization scheme and any differences
between their individual characterization and changes and the group ones.
6.1.4 Study 2 - Results
This section is organized around the six research questions posed in Section 3.2.3.
For each question, a brief statement of the results is provided. More detailed results have
been published elsewhere [171].
1. How well did the change characterization by the subjects match the change
characterization by the researchers?
The answer to this question comes from the characterization data submitted by the
subjects for Changes 2 and 3. If the subjects correctly understood the attributes of the
characterization scheme and how the change requests would affect those attributes, then
their characterization should be similar to the “gold standard” characterization as defined
by me. Any discrepancies in the result are likely caused by a partial or complete
misunderstanding of the attributes of the characterization scheme and/or the TSAFE
architecture.
In order to determine the closeness of the characterizations, the mean of values of
the subject’s characterizations were computed. This value was compared to the gold
standard for each change. The subjects characterizations were viewed to be close to the
gold standard if the absolute value of the difference between the two values was less than
or equal to 1. Of the 28 general characteristics, 22 attributes met the standard for Change 1
and Change 2.
Figure 13 and Figure 14 show a comparison of mean subject values to the gold
standard for Change 2 and Change 3 respectively. Based on these results, a majority of
subjects seemed to understand how to use the scheme.
Figure 13
Characterization Comparison - Change 2
Figure 14
Characterization Comparison - Change 3
2. Is the characterization scheme easy to use?
In the survey completed after the two assignments, the subjects were asked to
indicate their level of agreement with the following statements about the usefulness of the
characterization scheme:
1. The attributes are logical and easily understood
2. The scheme is beneficial for a developer making a change
3. I understood the effect of the changes to the system architecture better using the
scheme than without it
4. The scheme was detailed and covered all aspects of the architectural
implementation
5. The change scheme helped me to understand the impact of the change request
For each question, a 5-point Likert rating scale was used, ranging from 1 - totally
disagree to 5 - totally agree. The results were analyzed using a one-sample t-test with a test
value of 3 representing the neutral response. The results in Table 26 show that each attribute
was significantly greater than 3. These generally positive results provide support for the
idea that characterization scheme is both useful and practical.
Table 26
Statistical Survey Results
Statement
Mean
T-Value
P-Value
1.
3.68
3.302
.003
2.
4.12
9.333
.000
3.
3.56
2.133
.045
4.
4.32
7.333
.000
5.
3.96
6.080
.000
3. Do changes that exhibit different characteristics require different amounts of
effort to implement?
Change 2 was a perfective enhancement that was a functional change and did not
require the addition of any modules. This change required the modification of a small
number of LOC in a relatively large number of modules.
Change 3 was also a perfective enhancement that was both a functional and
architectural change because it required the addition of a significant architecture module.
Table 27 shows the change implementation detail for Change 2 and Change 3. Changes 2
and 3 had different characterizations. This difference resulted in a differing amount of
implementation effort in terms of LOC and modules modified or added.
Table 27
Change Detail
Detail
Change 2
Feed Display
Change 3
LOS Detector
Modules Modified
7
7
Modules Added
0
1
LOC Mod./Add.
37
190
4. Does the scheme support effort estimation?
For each change the subjects were required to estimate the number of module and
component changes that would be required. First, the subjects were asked which change
would require the most effort to implement. A majority of 16 subjects identified Change
3 as the most difficult, 6 subjects chose Change 2, and the remaining 3 chose Change 1.
Changes 1 and 3 were very similar changes in terms of their impact on the architecture and
actual implementation detail (both were implemented by adding one module and modifying
7). We hypothesize that the majority of the subjects chose Change 3 as the most difficult
because of the rigor of the change characterization process in forcing them to consider
which aspects of the architecture would be affected.
Changes 1 and 3 in actuality required changing the same number of modules. The
mean number of module changes estimated for Change 1 was 1.84 and for Change 3 was
2.88. This difference was statistically significant (t4 = -2.153, p=.036 [t-test]; Z = -2.399,
p=.016 [Mann-Whitney]). This result suggests that the subjects were able to identify
additional architectural changes when using the characterization scheme (Change 3) that
were not apparent without the change scheme (Change 1).
5. Does the scheme add value to the change process?
In the post-study survey the subjects provided their opinions of the scheme, how
the scheme could be improved, and any problems that they encountered while using the
scheme. These questions were used to elicit information to help with improvement of the
characterization scheme and to better understand its strengths. In the list below the number
of subjects who gave each response is in parentheses. The subjects said that the
characterization scheme:
Aids in determining what changes should be made to each architecture view and the
impact the change will have on the view (7);
Helped ensure thoroughness of change detail (6);
Would be a good communication tool for project managers, software architects,
maintainers, and developers (6);
Is good for large changes but not practical for small changes (5);
Has too many attributes (5);
Requires more training in its use than was provided (3);
Is complete with the right level of detail (2);
6. Does the scheme help facilitate communication amongst developers?
For both assignments, the subjects first worked individually then with a partner.
These two steps were used to capture the interaction between the subjects to determine if
the characterization scheme facilitated the discussion of the impact of a change request.
The reports were analyzed and coded to extract information about the use of the
characterization scheme during the group meeting. Each group did not specifically
comment on their use of the characterization scheme in the group meeting, but any
statements about to the use of the scheme were extracted from the experience report. Some
of the comments (paraphrased) made in the reports include:
Four groups reported that they recorded the characterization of the changes after discussing
their individual change rationale. Next, they determined how their individual changes
compared to the changes made jointly using the scheme. Finally, they recorded the
change detail reflected by the scheme and updated the architecture diagrams to reflect
this new combined architecture. They used the scheme to determine the change detail.
One group used the characterization scheme as a checklist while recording the architecture
changes on the change detail form. The group stated that the scheme helped their
decision process by focusing their discussion on which changes listed for each view
were needed.
Three groups did not use the scheme to make the actual decisions. They simply used it at
the end of the process to record the characterization of the changes after their change
decisions were made.
Two groups used the characterization scheme after their analysis but prior to modifying
the architecture diagrams. By using the characterization scheme at this point, they were
able to determine what changes they would have to make to each architecture view.
Based on the observations above, SACCS provides a basis for discussion amongst
multiple developers considering a single change request. Strict process conformance will
ensure that this discussion takes place with SACCS as the platform for communication.
6.1.5 Study 3 - Experimental Tasks
The class was divided into two random groups, one ad hoc (control) group and one
SACCS (experimental) group. The SACCS group was trained to use SACCS, and the ad
hoc group did not receive SACCS training.
Both the control group and the SACCS group were divided into two subgroups. The
members of each subgroup were given 1 of 2 change requests. The subjects were asked to
make changes to the gold standard TSAFE architecture in response to a change request.
The changes were made by each individual group member. The SACCS group used SACCS
to analyze and implement the change while the control group did not. The high-level tasks
include:
1. Change the architecture document and diagrams
2. Record change detail (e.g., number of modules changed)
Within each major group (SACCS and control), members of the subgroups switched their
modified architectures with another individual that completed the other change request.
Each individual inspected a classmate’s architecture document using a checklist-based
inspection technique. They recorded the number of defects found to assess the quality of
changes made. Then, the ad hoc group received training on how to use SACCS. Both
groups used SACCS to modify the original TSAFE architecture in order to implement two
additional change requests.
The quality of the changes was determined in two ways. First, each modified
architecture document was analyzed to determine whether the subjects were able to present
the correct approach to making the change and to correctly identify the modules and
connections that needed to be modified. The change detail forms provided this information.
Second, the students used a reading technique to inspect another student’s change within
the same group (SACCS or control). The students however, did not inspect the architecture
for the same change request that they had made themselves. This approach was used to
eliminate any potential bias that might result from a partiality to their own designs.
Each subject was asked to write a report about their experiences, and they were
given specific survey questions regarding the entire task. Figure 15 provides an overview
of the experimental tasks in Study 3.
Figure 15
Study 3 Overview
6.1.6 Study 3 Results
The hypotheses for study 3 dealt with the ability of SACCS to assist the developer
in producing quality changes compared with an ad hoc method. Another goal associated
with these hypotheses was determining if the novice developers that have experience with
SACCS are better able to assess change impact than those without. The quantitative data
collected from the subjects is associated with the two hypotheses for Study 3 and presented
in the following subsections. As with each study, suggestions for improvement and a quality
assessment of SACCS is presented in a later subsection.
6.1.6.1 Quantitative Results
The first two changes given for Assignment 1, Command Line Interface (CLI) and
Dynamic Maps (DM), were functional changes and did not have a large effect on the
architecture. The main focus for the subjects making the changes was to determine which
modules were affected and what connections between modules were required to implement
the change requests. The goal was to determine whether the members of the SACCS group
were better able to correctly identify the modules and views that would be impacted and
make the changes to the architecture diagrams. The inspections were used to determine the
number of defects contained in each architecture document. The mean of the change detail
results for the first set of changes are shown in Table 28.
Table 28
Change Set 1 (DM/CLI) Results
DM
Modules
Added
DM
Modules
Changed
DM
Defects
CLI
Modules
Added
CLI
Modules
Changed
CLI Defects
Control
Group
2
3
6.33
5.33
2
8.66
SACCS
Group
1.75
3
5
3
2.25
4.5
The mean of the change decisions that were made to the architecture based on the
change requests were very similar. The average number of defects recorded for the CLI and
DM changes were higher for the control group that did not receive SACCS training.
This number of defects recorded for the control group for the CLI change was significantly
higher than the number of defects found for the SACCS group (t4 = -2.240, p=.075 [t-test]).
This t-test compared the mean of the defects found for the CLI control group and the CLI
SACCS group.
The results are similar for the Assignment 2, where the subjects received a second
set of changes (LOS and CM). The Loss of Separation Detector (LOS) and the
Conformance Monitor (CM) change requests had a larger architectural impact than the
previous two. The control group received training on how to use SACCS to implement the
change. We wanted to see if there was any improvement to the original SACCS and if the
quality of their architecture changes was improved versus the control group. Table 29
contains the mean of the results from change set 2.
Table 29
Change Set 2 (LOS/CM) Results
LOS
Modules
Added
LOS
Modules
Changed
LOS
Defects
CM Modules
Added
CM
Modules
Changed
CM
Defects
Control
Group
2.33
6
3.66
3
4.33
3.33
SACCS
Group
2.25
4.5
4
1.75
3.25
4.75
There were no significant differences in the values between the two groups after
each group received SACCS training. The control group did however average less defects
for both of the changes after receiving SACCS training.
These results show that by using SACCS, the subjects were able to create a higher
quality change evidenced by the comparison of defects found for the CLI and DM change
request comparison. This number of defects found for the control was significantly higher
than the SACCS group for the CLI change. Experienced SACCS users however, did not
show a significant improved in the quality of changes made compared with nonexperienced
SACCS users. One potential reason for this result is that the members of the control group,
which received SACCS training for Assignment 2, may have made higher quality changes
compared with their changes during the assignment. This conclusion is could not be testing
statistically in this study and should be tested in a future study.
6.1.6.2 Qualitative Results
The version of SACCS used in this study was the most comprehensive because it
was updated based on feedback from prior studies and from the results of the systematic
literature review. The subjects answered several questions concerning the utility of SACCS
and their overall opinions of using it. Several statements that the subjects made are listed
below. We paraphrased a statement that represents the idea of the group of statements that
follows. All related comments are provided below along with any others that were useful
to improve SACCS and gain insight into its use.
Supports the thought process:
The scheme helped to classify changes, but also, at times, made me think
which category changes fell into.
The characterization scheme made me think some about the scope of
changes before I started looking through the document.
It provides a path to follow when making changes, and makes the process
more intuitive.
Helps make you think more intensely on the effects of the change.
I don’t feel that it helped or hindered, but it did make sure that I did not
forget any aspect of the document.
Supports thorough analysis of the architecture
It helped me catch some modules that would need changing that I might
have missed otherwise.
I think it was helpful and takes some of the guesswork out of finding all of
the area of a system that a given change would affect.
The scheme aided in identifying which aspects of the architecture would be
affected by the change.
It helped me to decide how much change I would put into each view
presented in the architecture document.
It makes you look at everything that the change might affect.
It creates extra unnecessary work
It hinders by creating extra work, but helps you to understand the amount
of change needed when deciding whether to implement the change.
I would decrease the number of items or categories, in other words, shorten
the procedure somehow.
Some of the fields seemed unnecessary or redundant, so it could probably
be reduced in size without sacrificing utility.
It’s useless
I would not totally go by the characterization scheme as I found it confusing.
It was at times more troubles than what it was worth. It was easier to forget
the table and just make any change you deemed necessary.
If I know how large a change is, how does that help me, it doesn’t change what
has to be done.
You won’t know what exactly what you have to do until after you make the
change, and what sense does it make to characterize it then.
Serves as a good checklist
The characterization scheme helped me with the change request by
providing an iterative list of possible change locations. I though it served
as a good reminder of things that needed to be checked for.
I feel it worked well as a checklist, but I’m not sure if marking changes with
minor to complex is helpful.
Insightful
I thought the change characterization scheme was helpful in documenting
changes to the system. It was simple, yet comprehensive enough to detail
changes to most architectures.
Honestly, I could have done without it. However, it did get me thinking about
what I was going to change and every once in a while I would refer myself
back to the scheme to see how much change I had initial planned on for
each view.
While the scheme aids in documenting the change in an organized manner,
it did not cause the change to be any easier to design.
The qualitative results provide the subjective impressions from subjects who used
SACCS to make a software change. These results offer genuine insight into the benefits
and drawbacks of using SACCS. SACCS requires its users to thoroughly think about the
change before implementation. It compels developers to consider all aspects of the system
that may be affected as a result of the change request. According to some users, SACCS
adds additional overhead to the process. With its many attributes, it also adds to the overall
complexity of change implementation. The benefits that SACCS provides to developers
does however, appear to outweigh these costs.
6.1.7 Conclusions
The initial goal for these studies was to use them to transform the original
architecture change characterization scheme to a useful tool that developers and
development organizations could use throughout the maintenance and evolution cycles of
their software systems. The purpose for creating the scheme was to provide a tool that could
be used to record the characteristics of a software change. These characteristics can be used
as an input to a decision support system that incorporates the characterization, impact
analysis, and risk assessment to aid developers in making go/no-go decisions for changes.
These studies examined several hypotheses and answered several questions. The following
statements summarize the lessons learned:
SACCS provides insight into the difficulty of a change request
SACCS helps to facilitate discussion amongst developers
SACCS is a useful tool for change implementation
6.1.8 Threats to Validity
There are several threats with each of the studies and the statements made above.
These threats include:
Using Students to Perform Analysis: Students are frequently used in empirical
studies to provide some evidence of the usefulness of software engineering products and
processes [48]. Students were able to provide data about the use of the scheme and answer
some important questions about it. The threats associated with using students in this study
include their potential bias in answering survey questions for fear of criticizing. They also
may not have the appropriate experience to evaluate the scheme’s usefulness in a
professional setting.
Small Sample Size: Study 2 had the largest set of subjects for the studies with 25.
Study 1 had 18 subjects and study 3 has 14 subjects. We took this issue into account when
running statistical tests by using both the parametric and nonparametric tests when
appropriate, which showed similar results.
Actual Effort not Collected: We did not collect actual effort data in man-hours in
any of the studies. All measures of effort were based on the work done to the architecture
or the source code.
6.2 Historical Data Analysis Results
The analysis was conducted on four OSS systems of varying size. Section 6.2.1
presents the results of the metrics analysis used to predict change density at the class and
package levels. Section 6.2.2 describes the results of analyzing SPAN during monthly
development cycles for one of the four systems. Section 6.2.3 presents the results of the
refactoring and architecture change analysis using SACCS.
6.2.1 Change Density Analysis
The goal of this analysis is to determine which metrics predict the change proneness
of classes and architecture modules (packages). Prior to conducting any of the analyses,
descriptive statistics were calculated to summarize the data and check assumptions.
Multilinear regression was the method of choice for this analysis because of the availability
of multiple independent variables (metrics). The analysis began with one of the largest
systems in terms of classes so that the metric model chosen for this initial system could
easily scale to smaller systems with fewer data points. The goal is to generalize this model
to other systems.
6.2.1.1 Class-Level Change Density Analysis
First, a principal component analysis (PCA) identified the relationships and reduced
redundancy among the metrics. PCA was conducted using an orthogonal rotation (varimax)
so that the information explained by one component is independent of the other
components. Thirteen product metrics were used in this analysis (see Table 23). The metrics
were computed separately for each class.
Table 30 shows the results of the PCA on the class metrics of Apache Jackrabbit.
Three components account for 72% of the total variance among the measures. Factor
loadings less than |.40| were omitted from the table to improve clarity. The three
components represent size and complexity (PC1), average values of size (PC2), and a set
of remaining metrics not strongly correlated to size such as LCOM (PC3). PC1 accounts
for 43% of the variance and contains 11 of the 17 metrics. This strong correlation between
such a large set of the metrics could cause multi-colllinearity problems in the regression
analysis if not accounted for.
Table 30
Change Density - Principal Compnent Analysis
Variable Principal Component
PC1
PC2
PC3
WMC
.954
RFC
.678
-.415
AvgCyclomatic
.952
LCOM
.488
NOC
AvgLineCode
.889
#InstanceMeth
.950
#InstanceVar
.599
.424
CBO
.712
AvgEssential
.873
FAN_IN
.593
CountStmtExe
.913
SLOC
.932
MaxCyclomatic
.594
.586
DIT
-.867
SumCyclomatic
.936
SumEssential
.934
% Variance
43%
18%
11%
Next, the PCA components were used on exploratory variables to conduct a
multilinear regression analysis with the dependent variable change density (number of
changes) for each class. This step was performed to determine the relationship of
components with change density. The combination of factors significantly influence change
density (F(3,508) = 303.76 p < .001). The adjusted R squared value is .63. There is also a
significant correlation (p < .001) between PC1 and change density (.763) indicating that
class size is a strong factor in the number of times that it will be changed.
Table 31 provided a summary of this analysis. Each component significantly contributed to
the model. Tolerance and VIF values of 1.0 indicate that each component represents
different dimensions of information and are not redundant. The coefficients for each
variable are also included in this table.
Table 31
Principal Compnents Regression Analysis (Number of Changes)
Model
Unstandardized
Coefficients
Standardized
Coefficients
t
Sig.
Collinearity
Statistics
B
Std. Error
Beta
Tolerance
VIF
(Constant)
3.871
.129
29.965
.000
1.000
PC1
3.717
.129
.763
28.745
.000
1.000
PC2
.747
.129
.153
5.773
.000
1.000
1.000
1.000
PC3
.929
.129
.191
7.187
.000
1.000
In order to determine the actual model of metrics and their relationship to change
density, another multilinear regression analysis was conducted using the individual metrics.
The PCA step showed that the metrics related to size and complexity were correlated and
would have resulted in redundant measures if they were analyzed separately. Multilinear
regression can produce misleading or inaccurate results if there are high inter-correlations
among some of the predictor variables. Using the entire set of metrics in a multilinear
regression model would not be sufficient (This approach was tested, F(17,494) =79.75. p <
.001 with adj. R2 = .724 but resulted in only 8 of the 17 measures meeting the tolerance
standard [1-R2] to prevent multi-collinearity).
Stepwise multilinear regression was used to generate a set of meaningful metrics
that significantly contribute to a model of change density prediction. Stepwise regression
is a technique that starts with a linear regression of the variable most correlated with the
dependent variable (in this case, it was SLOC). Additional metrics are then added to the
model. Variables that significantly add to the model are kept and others are discarded. While
stepwise multilinear regression is a good automated technique for determining significant
variables, it does have some drawbacks if used in isolation. The variables selected using
stepwise regression may not be the best object-oriented metrics for the model as some other
independent variables may be substituted with the ones chosen without a significant loss of
fit. Other variables belonging to the same component of PCA that are highly correlated to
each other may also fit in the stepwise model. Therefore, stepwise multilinear regression
should be used with caution. With these considerations, highly correlated variables were
replaced and explored within the best stepwise model to determine if they provide a better
logical representation of the dimensions of the model.
The stepwise technique, the results of the PCA, and the PCA multilinear regression
model were used to determine a set of metrics that both contribute to a prediction model
and indicate a high level of variance. This model contains metrics from each of the three
PCA components, each metric significantly contributing to the model.
The model contains the following metrics: LOC, CBO, MaxComplexity, WMC,
#InstanceVariables, and LCOM. This combination of metrics significantly predicted
change density (F(6,505) = 214.21 p < .001), with all six metrics significantly contributing
to the prediction. The adjusted R squared value is .715 indicating that 71.5% of the variance
in change density is explained by the model. Table 32 shows the coefficients, significance,
beta and tolerance of each variable used in the model.
Table 32
Change Density Model – Apache Jackrabbit
Model
Unstandardized
Coefficients
Standardized
Coefficients
t
Sig.
Collinearity
Statistics
B
Std. Error
Beta
Tolerance
VIF
(Constant)
.927
.005
.209
4.438
.000
SLOC
.001
.212
3.373
.001
.141
7.086
CBO
.166
-.015
.097
.359
.071
.024
.247
6.941
.000
.441
2.268
LCOM
.004
-.100
-3.474
.001
.672
1.488
MaxCyclomatic
.029
.119
3.353
.001
.446
2.242
#InstanceVar
.037
.311
9.698
.000
.544
1.838
WMC
.022
.181
3.165
.002
.170
5.876
The model represented by Table 32 was then tested on the three remaining systems.
For each class in each system, the model metrics (SLOC, CBO, LCOM, MaxCyclomatic,
#InstanceVar, and WMC) were inputs to a multilinear regression analysis with the number
of changes as the dependent variable. The results of this analysis are show in Table 33. This
table shows the number of classes, the R, R2 and adjusted R2 values, the standard error of
the estimate, and the degrees of freedom of the residual and regression (in parentheses)
with the F-value. All of the models were statistically significant with a p-value of < 0.001.
Table 33
Regression Model for Class Changes
System
N
R
R2
Adj. R2
Std. Error
F-Value
Jackrabbit
512
.847
.718
.715
2.603
(6, 505) = 214.21
Cayenne
1600
.592
.352
.345
3.223
(6, 1593) = 143.45
Velocity
255
.658
.433
.419
12.18
(4. 250) = 36.43
Derby
894
.823
.677
.675
4.64
(6, 887) =310.32
The metrics were able to compute the variance in the number of changes greater
than 40% of the time in 3 of the 4 systems. This is considered a large effect and shows how
well future outcomes can be predicted by this model [57]. Table 34 shows the mean relative
error (MRE) of the estimates for each of the four systems. MRE shows the percentage of
relative error that can be expected when running the model on actual data. MRE should be
less than 1 to be considered useful. The standard deviation is also presented in the table and
shows a large dispersion in the relative error prediction. The median of the MRE is also
shown to allow the reader to see that there are a few outlier predictions that increase the
mean, but the median value is still relatively low for this type of analysis.
Table 34
Relative Error Analysis
Jackrabbit
Cayenne
Velocity
Derby
Median MRE
0.448
0.513
0.425
0.201
Mean MRE
0.711
0.841
1.43
0.476
Std. Deviation
0.829
1.02
2.48
1.09
If a developer could predict modules that are change prone, he can then make a
decision to refactor the module given this data. A change-prone module can be hazardous
because frequent changes are an indicator of code decay. Frequent changes can also
introduce faults to a system.
6.2.1.2 Package-Level Change Density Analysis
The same analysis conducted for change density at the class level was conducted
for change density at the package level. For each package, a set of product metrics and a
value for the number of changes made to that package were calculated. The analysis at the
package level did not produce a definitive metric model as the class metrics did.
Cayenne was analyzed first because it consists of the largest number of packages (128).
The goal was to determine a set of metrics that significantly predict the variation in the
number of changes for each package.
Because of the larger degree of redundancy between package-level variability, PCA
for Cayenne generated two components. PCA was then performed for each system to see
if the measures captured any additional dimensions for the other systems. Derby was the
only system whose metric values produced three components. It appears that this third
component was generated for Derby because of the relatively small number of classes per
package for the Derby system. The mean of classes per package in Derby is
13.45 while Jackrabbit, Cayenne, and Velocity contain an average of 45.35, 36.73, and
27.97 respectively. This difference was reflected in the PCA generation of the third
component for Derby. The num_classes metric for Derby had a very high loading for the
third component and a weak correlation to the other size metrics which were contained in
the first component. This observation indicates that the packages with the largest number
of classes were the ones that were most often changed. But, for Derby, this observation was
not true.
The attempts to find a single model to predict the variance in the number of changes
at the packaged level proved to be unsuccessful. There were not any strong common
correlations between the metrics and the number of changes in any of the datasets. This
observation is explained by the potential for packages to have a variety of classes in them.
Some classes could be large and complex, which have a tendency to be change prone,
versus others that may be much smaller. Given the previous analysis, it would prove more
beneficial to focus on change proneness solely at the class level as packages are not focused
enough for this analysis to be useful.
6.2.2 SPAN Analysis
The goal of this analysis was to determine which metrics affect the SPAN of changes
for the overall system. The change in LOC and files was calculated monthly for Cayenne.
In addition to the total amount of change calculated during the period, the
SPAN ratios of LOC changed per issue and file changes per issue were also computed. At
the end of each period, the system revision reflecting the total amount of change from the
previous month was downloaded and metrics were extracted for the entire system.
Cayenne was the only system used in this analysis because of its unique ‘Resolved Date’
JIRA field that specifies the date that each issue was completed. The other systems use an
‘Updated Date’ field that is overwritten whenever changes are made to the issue, regardless
of whether the change affected the source code. This situation led to misleading results for
the other systems. Cayenne also uses the ‘Updated Date’ field for any issue updates that are
made, but the ‘Resolved Date’ remains the same.
Metrics were calculated at the system level (i.e., for every class and package in the
system, a single value was tallied and/or averaged) per month for two years. These metrics
were then analyzed to determine if any correlations exist between the metric values and the
amount of effort required to implement the changes. Table 35 shows a sample of the data
collected.
Table 35
SPAN Metrics Collected
Month
1
5
10
15
20
NumFiles_Changed
16
3
66
57
24
NumLOC_Changed
1032
32
2603
1233
629
Files Per Issue
5.33
3.00
9.43
6.33
2.67
LOC Per Issue
344.00
32.00
371.86
137.00
69.89
Revisions Per Issue
1.00
1.00
2.86
1.33
1.11
AvgCyclomatic
1.72
1.7
1.68
1.68
1.69
Avg_SLOC
75.82
73.26
72.69
73.65
73.89
#Classes
2589
2734
2917
3034
3151
#Files
1850
1981
2122
2197
2276
#Functions
14054
14873
15859
16452
17100
#lInstanceMeth
13165
13862
14815
15396
16004
#InstanceVar
2950
3017
3124
3246
3400
WMC
13649
14351
15301
15890
16526
RFC
37754
41446
46702
49171
50897
SLOC
140258
145135
154247
161819
168174
Count_StmtExe
49723
51164
54246
56514
58976
Cyclomatic
24253
25253
26683
27727
28918
Essential
17759
18650
19742
20479
21323
No statistically significant correlation was found between the product metrics and
measured effort in terms of LOC and file changes. There were however, some interesting
observations of the trend lines for the SPAN measures over time. Figure 16 shows SPAN
of files per issues for two years of Cayenne development. In this figure, there are several
large spikes of activity that serve to reduce the SPAN measure after the change is
completed. These large spikes include three major architectural changes. The three changes
were removed during months 11, 14, and 19. These changes are discussed in more detail in
the next section. The resulting data is shown in Figure 17. The bars highlight the period
where the major changes were removed. These change significantly increased the ratio of
files per revision during those months.
Figure 16
Cayenne Files Per Issue
0
20
40
60
80
100
120
140
160
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Months
Files Per Issue
Figure 17
Cayenne Files Per Issue - 3 Architecture Changes Removed
6.2.3 Refactoring Analysis and SACCS Characterization
This analysis was used to determine when refactorings performed by the developers
significantly impacted the metrics for the parts of the system that was refactored. In order
to conduct this analysis, the JIRA issue tracker was searched for instances of refactoring.
The refactoring issues were identified when the keyword ‘refactor’ (and any other
variations of the word) was present in the description of the issue. Restructurings and other
architectural changes were also identified by large file, package, and LOC changes
0
2
4
6
8
10
12
14
16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Months
Files/Issue
contained in the change databases for each system. The revision that precedes the
refactoring and the revision immediately following the refactoring were downloaded and
analyzed. The size of the refactoring was measured in terms of LOC and files modified.
The change in metric values was analyzed using the two-tailed independent samples T-test
to compare the means of the values before and after refactoring to determine if there was a
significant change.
Each of the refactoring changes was also characterized using the SACCS General
Characteristics. Additional changes that were not defined as refactoring, but that did have
an impact on the system architecture were also characterized. For these changes, the same
tests were performed as the refactoring changes. The issues were characterized to determine
how various types of changes impact each system. A minimum of three
refactoring/architecture changes were analyzed for each of the four systems. For each
change a table was created to provide information about the change. The title from the issue
tracker is included with the number of files, LOC, and packages modified are listed with
characteristics from SACCS. Several examples of this output are provided in the tables
below. The remaining tables are shown in Appendix A.
The first example shown in Table 36 is from Jackrabbit. This issue resulted in over
22 thousand lines of code and 176 files modifications. This huge change consisted of a
refactoring of a several major Jackrabbit components, the Persistence Managers. This
change involved moving the persistence manager components to another module to
improve system maintainability and developer understanding. This change did not impact
system functionality and therefore is characterized as a preventative architecture change.
This preventative change was requested and implemented by one of the lead developers
who has been involved in system development for over four years. This change had a large
impact on static system properties since most of the work was performed on the source
structure. The dynamic properties are impacted because this change affected how the
runtime components interact. The only feature that was impacted by the change was data
access. The change was designed to improve the maintainability of the system.
Table 36
Jackrabbit Issue #595 Summary and Characterizataion
Project-Issue #
Modified: LOC Files Packages
Jackrabbit-595
22036 176 14
JIRA Title
Refactoring of the Persistence Managers
SACCS
Characteristics
Motivation Source Criticality Dev.
Experience
Enhance Stakeholder Requested Extensive
Request
Granular
Effect
Category
Properties
Subsystem
Preventative
Static (4)
Dynamic (2)
Logical
Dependency (1) – Layers (0) – Mod. Decomposition (2) – Source
Structure (4) – Inheritance (2)
Runtime
Control Flow (0) – Repos. Access (2) – Conc. Processes (1) –
Comp. Interaction (1) – Dist. Components (0) – Comp.
Deployment (0)
Features
Data Access (2)
Quality
Attributes
Maintainability (4)
Overall Impact Scale Values in Parentheses
The large system modification represented by Table 36 caused a minimal impact on
the system metrics. There were no statistically significant changes to the overall metric
values resulting from this change. Even though this change represents a huge refactoring,
the change simply represents a module moving from one folder to another. While the class
complexity of the affected classes was reduced, this reduction was minimal.
The next two change summaries are for Cayenne. Table 37 presents the
characterization of an extremely large change. This change is one of the outlier architecture
changes during month 11 that was removed from the analysis described in the previous
section. This change consisted of modifying the Cayenne source files to use the updated
Java Development Kit (JDK). The developers updated the system to JDK 1.5 and removed
all references to the previous JDK from the development trunk. This change was a large
change, but was not the change responsible for the improvement in the SPAN observed in
Figure 17. This change did not result in a significant change in the metric values at the class
or package level.
Table 37
Cayenne Issue #904 Summary and Characterizataion
Project-Issue #
Modified: LOC Files Packages
Jackrabbit-904
133157 901 44
JIRA Title
Switch cayenne to Java 5: merge cayenne-jdk1.4-unpublished to
cayenne-jdk1.5-unpublished
SACCS
Characteristics
Motivation Source Criticality Dev. Experience
Enhance Resource Requested Extensive
Constraint
Granular
Effect
Category
Properties
Architectural
Adaptive
Static (4) Dynamic (0)
Logical
Dependency (3) – Layers (0) – Mod. Decomposition (0) – Source
Structure (4) – Inheritance (0)
Runtime
Control Flow (0) – Repos. Access (0) – Conc. Processes (0) –
Comp. Interaction (0) – Dist. Components (0) – Comp.
Deployment (0)
Features
N/A
Quality
Attributes
Maintainability (4) – Functionality (1) – Scalability (1) – Reliability
(1)
Overall Impact Scale Values in Parentheses
There were several refactorings and module restructurings that took place during
months 11 and 12. One of these refactorings is recorded in Table 38. This refactoring was
conducted to make certain types of Cayenne system modules “easier to extend.”
Table 38
Cayenne Issue #926 Summary and Characterizataion
Project-Issue #
Modified: LOC Files Packages
Jackrabbit-926
4651 82 6
JIRA Title
Refactoring class generator classes
SACCS
Characteristics
Motivation Source Criticality Dev. Experience
Enhance Stakeholder Requested Extensive
Request
Granular
Effect
Category
Properties
Architectural
Preventative
Static (3)
Dynamic (1)
Logical
Dependency (3) – Layers (0) – Mod. Decomposition (2) – Source
Structure (3) – Inheritance (3)
Runtime
Control Flow (1) – Repos. Access (0) – Conc. Processes (0) –
Comp. Interaction (0) – Dist. Components (0) – Comp.
Deployment (0)
Features
Input/Output (1)
Quality
Attributes
Maintainability (4) – Scalability (3)
Overall Impact Scale Values in Parentheses
The characterization of this change reflects a relatively large modification to the
static source and inheritance structures. Control flow is marginally impacted because
several of the classes modified determine how system processes are implemented. This
refactoring change along with the previous two did not significantly impact the computed
metrics.
Each system analyzed contained issues that required modifying the system
structure. The goal of this analysis was to determine if these individual structural changes
significantly impacted the metric values reflected by the source code from the preceding
revision and the revision immediate following the refactoring. There were no occasions
where the refactorings made a statistically significant impact. The refactorings did however
result in observed changes to metrics at the class level. In line with the previous analysis,
the refactorings tended to create cohesive classes and well structured source code. This
cohesiveness reflects the decrease in the observed SPAN from the previous analysis.
6.2.4 Historical Data Analysis Conclusions
The historical data analysis confirmed the results of previous studies of change
prone classes. It also provided insight into the perceived versus actual benefits of
refactoring. Large, complex classes tend to be change prone [97, 115]. This information is
beneficial to a developer that is considering refactoring a system to improve its quality.
This refactoring can divide a single large class into multiple smaller classes that provide
specific functionality. Reducing the amount a change for a single class also reduces the
potential to introduce faults when changing the class. Frequent changes could also be the
result of a fault prone class because changes are made to correct faults in this class. By
reducing the class’s size through refactoring, a developer can assess which specific
functions of the class are contributing to the fault/change proneness of the class.
For the SPAN analysis, there were no statistically significant correlations between
the observed monthly metric values and SPAN. The monthly SPAN trend lines for Cayenne
did show increases in SPAN over periods of time. These increases were followed by a
decrease in SPAN. This decrease in SPAN was not from a specific refactoring, but appeared
to result from multiple refactorings that took place at various periods throughout the
lifecycle. There were over 28 separate instances of refactoring or restructuring issues
identified for Cayenne. The analysis of the individual refactorings showed that refactoring
did not have a significant impact on any metric. It seems that multiple refactorings over
time serves to reduce SPAN. This conclusion has not been verified and is discussed as
future work in Section 7.2.
The purpose of performing a refactoring is to reduce system complexity and
improve understandability and maintainability [71]. The refactorings performed on each
system may have all had this goal, but the sample of large refactorings that were analyzed
did not statistically confirm any positive results. The refactoring could have reduced the
subjective complexity and understandability of the system to a developer making a change.
One refactoring in particular required a large number of files to be changed (82) and
resulted in several classes being split into multiple classes. This refactoring was performed
to make it easier to extend the system. The perception of “easier” cannot be statistically
determined with metrics in this case, but a survey of the developers may confirm the result.
There were several cases (3) analyzed for each system where a refactoring or restructuring
was performed that fulfilled the goals of the developers but did not impact the metric
values.
SACCS was used to characterize several historic refactorings and large architecture
changes. This step was conducted to determine the types of changes that impact the
maintenance process and determine the usefulness of SACCS to do so. The SACCS
characterizations for each system can be found in Appendix A. For each system, changes
that were made to adapt the system to a new environment resulted in a relatively larger
impact to the source code than other types of changes. Preventative changes included
refactorings that did not modify the functionality, but were made to improve quality
attributes. These types of insights are beneficial for developers maintaining software
systems. New change requests that have similar characteristics as previous changes, can be
compared to those previous changes to determine the potential of the new changes to impact
the system.
CHAPTER VII
CONCLUSIONS AND FUTURE WORK
This chapter presents the conclusions and future work.
7.1 Conclusions
The conclusions of this research are drawn from the hypotheses generated in
Section 1.3. The hypotheses are repeated below.
H1. Using a change characterization scheme to analyze change requests
prior to implementation will enable software developers to create a more
exact estimate of the impact of the change than without the characterization
scheme.
SACCS was created to assist developers making software changes. It was designed
to provide a framework for change analysis and feedback as well as to facilitate developer
discussions about a change request. Three empirical studies were conducted using SACCS.
These studies provided evidence that using SACCS was beneficial for developers making
changes. SACCS compelled novice developers to consider additional aspects of a change
request that may not have been considered without it. This thorough analysis resulted in
change implementation designs that contained more detail and were matched more closely
to what was actually required than if SACCS was not used.
SACCS provides a structured approach to architecture change impact analysis.
H2. Extracting and analyzing software change metrics from historical data
gives developers measurable data on system quality which can be used to
make refactoring decisions.
Refactoring has been shown to improve system understanding and changeability
[74]. The ability to predict when a module should be refactored to prevent potential
problems before they arise is a needed skill. Part of this research involved extracting metrics
from historical data in order to predict attributes that determine system quality. A metric
model of change proneness was developed and tested on four open source software
systems. This model predicted greater than 40% of the variability in the dependent variable
change density in 3 of the 4 system. This is considered a large effect [57]. Being able to
predict which files are change prone provides a means to determine which files should be
refactored.
H3. Refactoring has a measurable positive effect on the system architecture.
SPAN of changes is an indicator of code decay. SPAN was computed monthly for
one system and shown via trend lines over time. As SPAN increased, the system was
restructured which appeared to affect the trending of SPAN toward a downward
direction. I was not able to identify any statistically significant relationships between the
metrics, refactoring, and SPAN. The issue tracker for Cayenne contained over 28 separate
instances of named refactorings and system restructurings that were identified from the
JIRA data. The Cayenne developers refactor the system frequently. This feature of Cayenne
development could indicate why no sustained increase in SPAN was observed. This
observation is consistent with refactoring having a positive effect on system architecture,
but the results could not be statistically determined from the given data.
H4. Code decay is evident in ongoing software development and can be
determined by metrics extraction during various periods of development.
OSS development is characterized as an agile development environment. In this
environment, formal architecture and design documents are not typically used when
creating new systems. These documents are often derived from the source after a working
prototype exists. With no formal design step, OSS source code is often developed in an ad
hoc manner. This ad hoc development from multiple remote developers could lead to
potentially complex systems as developers make changes without understanding the overall
structure. This problem is alleviated by the agile design philosophy of frequent refactorings
[148, 155]. In analyzing OSS systems that use this agile design methodology, clear
indications of lasting code decay were not present. While there were indications of classes
that change frequently and an uptrend in relative metric values, there were no obvious
indications of code decay in the systems analyzed.
7.1.1 Contributions
The contributions of this research are as follows:
Developed a novel framework for change analysis using SACCS that assists
novice developers making changes, facilitates discussion between
developers making changes, forces a thorough analysis of the architecture
prior to making a change, and provides insight into how various types of
changes will impact a software system.
Validated the usefulness of SACCS through three empirical studies. A detail
lab manual was created that included the resources needed to duplicate the
studies and has been made available for others to use.
Showed that individual refactorings did not significantly impact software
quality metrics.
Developed a model for determining change prone files that resulted in a large
(>40%) amount of variability being predicted in 3 of four systems analyzed.
7.1.2 Publications
The following articles have been produced for publication from this research. The
title of the article and the venue is listed with the date of submission. The papers that have
been published are referenced.
Refereed Journal Articles
The Journal of Information and Software Technology – “A Systematic
Review of Software Architecture Changes” – (Submitted December 2008)
Empirical Software Engineering: An International Journal – “The
Empirical Analysis of the Software Architecture Change Characterization
Scheme” – (To be Submitted April 2009)
Journal of Software Maintenance and Evolution: Research and Practice –
“Using Change Characterization and Historic Data to Assess the
Architectural Evolution of a Software System” - (To be Submitted May
2009)
Refereed Conference Papers
B. Williams and J. Carver, "Characterizing Software Architecture
Changes: An Initial Study," in Proceedings of the First International
Conference on Empirical Software Engineering and Measurement, 2007,
Madrid, Spain, pp. 410-419.
Technical Reports
B. Williams and J. Carver, " Characterizing Changes to Assess Architectural
Impact.", Technical Report MSU-070115, Department of Computer Science
and Engineering, Mississippi State University, 2007, Starkville, MS.
B. Williams and J. Carver, "Characterizing Software Architecture Changes:
A Systematic Review", Technical Report MSU-081216, Department of
Computer Science and Engineering, Mississippi State University, 2008,
Starkville, MS.
7.2 Future Work
The purpose of this doctoral research is to provide the preliminary framework for a
model of application-dependent change difficulty prediction for a change decision support
system. The decision support system will take a change request as input and output
predictions about what is expected. The ideal implementation of the decision support
system is described below.
1. Stakeholder issues change request
2. Developer formalizes request using template
3. Developer isolates and characterizes change using SACCS
4. Proposed changes discussed between project group
5. Planned implementation recorded
6. System compares current change to set of previous similar changes
7. System provides difficulty prediction and analysis
8. Actual change detail and SACCS characteristics recorded for future use
The type of questions that the output of the decision support system will provide
include: What amount of relative effort is required to implement the change? After the
change is isolated to a module, what other historically coupled modules are likely to be
affected? Will the change result in a more complex architecture? Will the change violate
any architecture constraints? Will the changes be made to historically effort-intensive
modules? Should a part of the system with a high handling rate be refactored? Answers to
questions such as these will assist a developer to make quality decisions concerning the
software project from relevant, application-dependent data.
The results of this research provide a piece of the framework for a data driven
change support model that analyzes relevant change data to provide a developer with the
tools needed to maintain today’s complex software systems. This unique approach that
blends quantitative data analysis and qualitative characterizations that benefit from
developer experience is key to this larger system. Additional historical data analysis must
be conducted to fully determine the impact of refactoring on quality metrics. Additional
systems should be analyzed from both open source and proprietary software development
projects. This analysis will help determine the true quantitative impact of refactoring.
