Application 2 – Annotated Bibliography

Information and Software Technology 53 (2011) 276–290

Contents lists available at ScienceDirect

Information and Software Technology

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / i n f s o f

Agile methods rapidly replacing traditional methods at Nokia: A survey of opinions on agile transformation

Maarit Laanti a,⇑, Outi Salo a, Pekka Abrahamsson b a Nokia Corporation, P.O. Box 407, 00045 Nokia Group, Finland b University of Helsinki, P.O. Box 68, 00014 University of Helsinki, Finland

a r t i c l e i n f o

Article history: Received 25 July 2009 Received in revised form 21 November 2010 Accepted 28 November 2010 Available online 3 December 2010

Keywords: Software engineering Agile software development Software processes Agile deployment

0950-5849/$ - see front matter � 2010 Elsevier B.V. A doi:10.1016/j.infsof.2010.11.010

⇑ Corresponding author. Tel.: +358 71808000; fax: E-mail addresses: [email protected] (M. Laa

Salo), [email protected] (P. Abrahams

a b s t r a c t

Context: Many organizations have started to deploy agile methods, but so far there exist only a few stud- ies on organization-wide transformations. Are agile methods here to stay? Some claim that agile software development methods are in the mainstream adoption phase in the software industry, while others hope that those are a passing fad. The assumption here is that if agile would not provide real improvement, adopters would be eager at first but turn pessimistic after putting it into practice. Objective: Despite the growing amount of anecdotal evidence on the success of agile methods across a wide range of different real-life development settings, scientific studies remain scarce. Even less is known about the perception of the impacts of agile transformation when it is deployed in a very large software development environment, and whether agile methods are here to stay. This study aims to fill that gap by providing evidence from a large-scale agile transformation within Nokia. While we have yet to confirm these findings with solid quantitative data, we believe that the perception of the impacts already pin- points the direction of the impacts of large-scale agile transformation. Method: The data were collected using a questionnaire. The population of the study contains more than 1000 respondents in seven different countries in Europe, North America, and Asia. Results: The results reveal that most respondents agree on all accounts with the generally claimed ben- efits of agile methods. These benefits include higher satisfaction, a feeling of effectiveness, increased quality and transparency, increased autonomy and happiness, and earlier detection of defects. Finally, 60% of respondents would not like to return to the old way of working. Conclusion: While the perception of the impact of agile methods is predominantly positive, several chal- lenge areas were discovered. However, based on this study, agile methods are here to stay.

� 2010 Elsevier B.V. All rights reserved.

1. Introduction

Since the emergence of agile software engineering (SE) methods in the mid 1990s, the software industry has grown increasingly interested in them. Nowadays, a wide range of organizations claim to be agile or report interest in the adoption of agile methods. According to a Forrester study where IT decision-makers repre- senting each a different company were interviewed during Sep- tember 2006, around 17% of enterprises already used agile methods, while more than half of the remaining companies were interested in adopting them [1]. This implies that the benefits of agile methods have been recognized in the IT software industry. In addition, the existing numerous publications on agile methods on both scientific and non-scientific forums mainly seem to report positive impacts of agile adoption. However, adoption of agile

ll rights reserved.

+358 718068810. nti), [email protected] (O. son).

methods has proven to be a challenging task [2]. It has been claimed that agile adopters are often unaware of what agile adop- tion really means, and how broad a change is actually required [1].

While the benefits of agile methods have been widely acknowl- edged, the strength of empirical evidence is still claimed to be fairly low [1,3,4]. One reason for this is that companies rarely have comparable data to explain the impact of agile methods before and after adoption [1]. From a scientific viewpoint, the vast majority of the agile-related research currently addresses qualitative case studies of individual development projects and lacks broader scope. In addition, a need has been identified for ‘‘more and better empirical studies of agile software development within a common research agenda’’ ([3] p. 833). Currently, it is still rare to come across scientific studies presenting quantitative evidence on agile methods to provide more generalizable results (e.g., [1,5,6]). In addition, one challenge identified among the current agile software engineering research is that the reported studies tend to focus on the few most familiar agile methods, namely Extreme Program- ming (XP) [7] and Scrum [8]. In fact, recent studies reveal an almost

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 277

exclusive focus on XP among agile-related SE studies (76%) while proposing an urgent need to broaden the focus of studies to include other agile methods [3].

Thus, the need to expand the current quantitative scientifically valid research regarding various aspects of agile methods is evi- dent. This study aims to provide empirical evidence from a survey conducted in 2008 for a large product development organization (Nokia Corporation) during the course of its agile transformation. The main agile model used can be defined as scaled-up Scrum [8] as defined in [9]. In total, the survey data include more than 1000 responses regarding different aspects of agile development from the viewpoint of various organizational stakeholders. The goal of this paper is to provide empirical evidence as to whether a linkage can be found between a person’s attitude toward agility and the following two factors: (1) experience (time) in applying agile methods in practice and (2) previous experience (time) in applying traditional (non-agile) methods. In addition, this study provides empirical evidence regarding the areas perceived as being the main challenges and benefits of agile methods. To provide broader perspective, this was done by comparing both the most receptive and most resistant segments of the population.

Organizations that are still contemplating agile transformation will find the presented results useful from at least two perspec- tives. First, we study how practitioner’s experience influences their opinions towards agile adoption. Secondly, we identify the most challenging areas that are likely to be faced when attempt- ing to transform a large organization toward higher degrees of agility.

From the industrial viewpoint, these results indicate that once the respondents have experience on agile methods, these are per- ceived to provide added value. This claim is supported by the find- ing that those with lengthy experience using both agile and traditional methods have positive attitudes towards agility. The ac- tual experience in using agile methods seems to play a more important role than the earlier prejudices. In the study, three espe- cially interesting areas of agile development were identified: (1) deployment of agile methods, (2) requirements management and planning in a flexible and iterative manner and (3) visibility and transparency. While deployment itself was perceived as the most challenging task, the areas of requirements management and transparency were perceived as among the most beneficial, yet also among the most challenging, areas of agile development throughout the survey population.

This paper is structured as follows: Section 2 contains back- ground information. Section 3 discusses research aims, context, and data collection and analysis methods. Section 4 contains re- sults with representative statistics, research questions, qualitative analysis, and limitations. Section 5 covers discussion of the main results, and lessons learned.

2. Related literature

The fundamentals of agile methods, i.e., the values and princi- ples behind them (http://agilemanifesto.org/), define the underly- ing aspiration of agility. The values stated in the Agile Manifesto highlight the values of communication and collaboration, respon- siveness, and focusing on the implementation of working software. In agile principles, several more detailed ambitions for agile meth- ods have been set. Based on this information, a software organiza- tion could be interested, for example, in: increased return on investment (ROI) for their efforts in the deployment of agile meth- ods; increased ability to deliver software early and continuously; better responsiveness to changes in product content; closer collab- oration and communication within the organization and with the customer; higher motivation and increased self-organization of

development teams; a sustainable and steady working pace; con- tinuously increasing work productivity; and higher quality of end-products.

Schwaber et al. [1], among many others, argue that agile meth- ods yield several benefits. These include reduced time-to-market, increased quality, reduced waste, better predictability, and better morale. Among other suggested benefits are, for instance, an in- creased ability to respond to dynamic market changes [10]. It should be noted that different agile methods aim to tackle different aspects of software development. For example, XP [7] focuses on the implementation of software, Test Driven Development [11] fo- cuses on unit testing, and Scrum [8] focuses on the management of product planning and implementation. A large amount of predom- inantly positive anecdotal evidence indicates benefits associated with agile methods, including quality of the end-product, produc- tivity, and increased transparency and collaboration. Most (76%) of the publications seem to have focused solely on XP while commu- nicating an urgent need to broaden the focus of studies to include other agile methods [3]. Meanwhile, sound empirical evidence on the actual benefits and challenges of agile methods remains rare [3].

Many studies report that the adoption of individual agile prac- tices, or certain fundamentals of agile software development, has been undertaken to complement the organization’s existing pro- cesses [12]. Often, the adoption of existing agile methods may re- quire their radical modification to fit the context at hand [13]. Thus, it is no surprise that several case studies report challenges in the adoption of agile software development methods [2]. It has been reported that agile adopters are often not aware of what agile adoption really means, and how broad a change is actually re- quired [1]. It has been stated that just having shorter iterations resulting in better quality may not be sufficient for large compa- nies producing complex software; a more holistic view on agility may be needed [14].

Various success and failure factors have been proposed to be significant in the agile adoption phase ([5,15]). For example, it has been suggested [15] that for various reasons, agile methods seem to polarize people and stakeholder groups into opponents and supporters having very different standpoints regarding the usefulness of agile methods for the organization. To overcome this, a fundamental change of philosophy and the development of new behaviors are claimed to be required across the organization [15]. A recent survey study proposes that the appreciation of agile meth- ods seems to increase once they have been adopted and applied in practice (5), which indicates that while there is likely to be resis- tance among the agile adopters in the organization, time and expe- rience in applying the methods will have a positive effect on this resistance.

It has been claimed that organizations that have failed in plan- ning agile adoption in systematic manner will generate sporadic results across the organization [15]. Kettunen and Laanti have proposed a framework [14] for understanding the multidimen- sional nature of agility that could be used as a basis for software process improvement (SPI). Not all problems in software projects can be solved by software process methodologies, but during agile transformation an organization can be seen as a force field con- sisting of forces for and against agility [14]. Kettunen and Laanti have described this force field as Goals (what we try to achieve with agility), Means (what we need to put in place in order to be agile) and Enablers (what needs to be in place before agility can happen). They also present the outline for an agility evalua- tion grid containing several hundred of items collected from liter- ature and categorized into Goals, Means, and Enablers. Due to the lack of other suitable frameworks, this ‘‘Agility evaluation grid outline’’ [14] was adopted as the basis for the qualitative analysis in this study.

278 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

3. Research approach

In this section, the research aims, context, data collection, and analysis methods are presented.

3.1. Research aims

The original reason why the organization started to adopt agile methods is scoped out from this study: i.e. the initial set-up is that the organization in some way or other following agile methods and we want to reflect the reality via the practitioners’ opinions, atti- tudes, and experience on agile methods. It is of major importance to study and understand the underlying factors affecting the satis- faction of stakeholders towards agility in organizations adopting agile methods, because the level of success in software process improvement can be characterized in terms of personnel satisfac- tion and whether the new process is actually used [16]. Currently, agile methods are subject of much discussion yet limited empirical evidence exists to support their usefulness [3]. Thus, it is important to evaluate how sound agile methods really prove to be. The assumption here is that if agile methods provide no real improve- ment, adopters would initially be eager but would then turn pessi- mistic after experimenting with agile methods in practice. Alternatively, if following agile methods would prove to be very difficult in practice, the followers might first be quite optimistic and then become realistic, seeing it as something laborious but necessary. The second assumption is that past experience with tra- ditional methods might affect attitudes and opinions negatively. Also, we wanted to know if the old saying, ‘‘You can not teach an old dog a new trick,’’ applies to agile adoption.

In this study, we first studied how strongly a person’s back- ground – i.e., the length of the person’s experience with agile and traditional methods – affects the perceptions of and satisfaction with, agile methods. From a broader perspective, this provides us a way to find out whether agility is making a permanent entrance or if those methods will fade away. Thus the first research aim of this study is to determine respondents’ opinions and attitudes to- ward agility and how they change as the person gains more and more experience in using agile methods. This leads us to the fol- lowing two research questions that guide the quantitative analysis of the study:

(R1) How does the length of practitioners’ experience using agile methods affect their opinions toward agile methods? (R2) How does the length of practitioners’ experience in using traditional methods affect their opinions toward agile methods?

The research method is one-way ANOVA (analysis of variance) [17]. In the analysis of the opinion statements, we use the original ordinal data with range 0.00–120.00 from the survey tool. As one-

Fig. 1. Respondents’ background in non-agile

way ANOVA assumes Homogeneity of Variance, Levene’s test was used for homogeneity [17]. A post-hoc analysis is done with Tukey HSD [17].

The second research aim of this study is to take a closer look at qualitative findings to reveal respondents’ perceptions of new, emerging challenges and benefits of agile development. In order to provide the view on this topic, respondents are first categorized as having negative, neutral, or positive attitude towards the adop- tion of agile methods. Then, the goal is to detect any significant dif- ferences in underlying perceptions between the two extreme groups, i.e., the negative and positive groups of respondents.

3.2. Research context

The sample of the study contains more than 1000 respondents from seven different countries in Europe, North America, and Asia. 90% of the respondents represented Research and Development (R&D), while the rest represented mainly Marketing, Design, or Process Development organizations. The total response rate was 33% from the population under the study. Fig. 1 shows the respondents’ background in agile and non-agile development as percentages.

The deployment method and main agile model applied among the survey population was scaled-up Scrum, as defined in more de- tail in [9]. Traditional methods in this case refer to waterfall-type of product lines, as defined in [18]. Ninety percentage of respon- dents were able to identify their agile role. Distribution of different agile roles can be seen in Fig. 2.

3.3. Data collection and analysis

This research was conducted as a questionnaire-based survey. Personal interviewing, telephone interviewing, and mail data col- lection have earlier been proposed as suitable data collection methods for such questionnaire-based surveys [19,20]. However, in this case a computer assisted web-based survey was conducted, because it is faster and easier to fill than a paper survey and also the handling of the responses is of lesser work. We also had proper tools to do a web-based survey at hand. Nardi [21] proposes using web-based surveys to eliminate human errors caused by manual data handling, and to enable a higher response rate while also increasing speed and decreasing cost. A low response rate – rarely over 30% – has been identified as one of the major problems of di- rect mail data collection [19]. The questionnaire was conducted in March 2008 and it was open for two weeks in April 2008.

The questionnaire consisted of multiple sections: (1) back- ground (e.g. organizational unit), (2) agile role and experience, (3) support, site distribution, and collaboration in agile develop- ment, (4) current use and usefulness of agile methods, (5) effects of agile methods, and (6) opinions. A questionnaire form was

and agile development as percentages.

Table 1 Use of sliding scale values in data analysis.

Respondent’s selection on sliding scale (1–7)

Corresponding integer #a

Corresponding selection range values generated by the systemb

1.0–1.49 1 0.00–9.99 1.5–2.49 2 10.00–29.99 2.5–3.49 3 30.00–49.99 3.5–4.49 4 50.00–69.99 4.5–5.49 5 70.00–89.99 5.5–6.49 6 90.00–109.99 6.5–7.0 7 110.00–120.00

a Used in graphs. b Used in ANOVA calculations.

Fig. 2. Respondents’ agile role as percentages (of those 90% who reported they had an agile role).

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 279

automatically generated based on the respondents’ answers to the following questions: (1) respondent’s organizational unit, (2) respondent’s role in agile development, and (3) respondent’s expe- rience in applying agile methods. Thus, depending on how a respondent replied to the previous questions the system generated different sections to be answered. The following scenarios form the case sensitive questionnaire:

– ALL respondents: Sections 1 and 6. – Respondents from R&D, marketing, design, or process organiza-

tions: Section 2. – Respondents with identifiable agile role and experience in agile

development: Section 3. – Agile team members with experience: Section 4. – Program and product owners: Section 5.

This study focuses on Sections 1, 2 and 6. Among other questions, the background section focused on

respondents’ experience both in non-agile (i.e., traditional SE) methods and agile methods. Collecting data about respondents’ experience with traditional methods provided the ability to evalu- ate the linkage between earlier experience and attitude and opin- ions regarding agile methods. The following scales were used to collect the data regarding experience with traditional software engineering methods:

1. <1 year. 2. 1–2 years. 3. 3–5 years. 4. 5–10 years. 5. 10 years.

Gathering information about respondents’ experience in apply- ing agile methods, on the other hand, provided the ability to eval- uate whether respondents’ opinions and attitude toward agile methods could be considered to be the result of actual experiences or rather to be mere assumptions. The agile experience level of the respondent was determined using the following scale:

1. Not at all. 2. 1–3 months. 3. 4–6 months. 4. 7–11 months. 5. 1–2 years. 6. 2 years.

In the opinions section, nine statements were presented (Table 2; also Appendix A). A seven-step sliding scale (ranging from 1 = totally disagree to 7 = totally agree, the middle position being ‘‘neutral’’) was used to collect respondents’ opinions, see Appendix

A. Table 1 aims at clarifying how the two decimal precise selections of responses on sliding scale were converted into integer (auto- matic conversion by survey tool used) and range values (system conversion). These values have then been used to illustrate the re- sults of the analysis throughout the paper. The more precise value range (0.00–120.00) have been used for ANOVA calculations and illustrations while the integer range (1–7) has been used for other graphs.

In addition, both open questions and multi-choice questions were included in the questionnaire, providing possibilities for both qualitative and quantitative data analysis. In the multi-choice questions the satisfaction or opinion of the respondent, or the experienced use and usefulness of specific agile methods and whether one would like to go back to the old methods were asked.

Kitchenham [22] points out one limitation of surveys being their ability to only confirm association instead of causality. In addition, Miles and Huberman [23] have proposed linking qualita- tive and quantitative data to provide richer detail and deeper understanding for the research. Taking these issues into consider- ation, the analysis includes both quantitative and qualitative data, the latter collected through free-form text entry fields in the survey.

In the qualitative analysis first all data was checked against the categories, and these were cross-checked and the unclear cases were discussed i.e. whether the comments were interpreted in the right way and the true meaning of the comment was captured or diluted in the process. Few of these cases led to changing the category. Finally, we checked that the different interpretation of the open comments would not have an impact on the final results (i.e. if a change in the interpretation of a comment that had multi- ple interpretations would have changed the result or the ordering of the categories). When we concluded that different interpreta- tions could not affect the order of the categories or the conclusions, we accepted the results and that the process was correct and prob- ability of any errors small enough.

The large number of responses allowed not only descriptive data analysis (i.e., frequencies and cross tabulation) but also more

Table 2 Opinions statements and descriptive statistics.

Statements N Mean Variance Standard deviation

1. Agile development increases the effectiveness of development 751 4.97 1.652 1.285 2. Agile development increases the quality of the product 751 4.70 1.854 1.362 3. Agile development increases the transparency of development 751 5.13 1.917 1.385 4. Agile development increases collaboration 751 5.04 1.769 1.330 5. Agile development makes work more fun 751 4.61 2.259 1.503 6. Agile development makes work more organized/planned 751 4.50 2.341 1.530 7. Agile development increases the autonomy of development teams 751 4.86 1.901 1.379 8. Agile development enables the earlier detection of bugs/errors/defects 751 4.77 2.005 1.416 9. Agile development makes work less hectica 751 3.64 2.318 1.522

a The original question in the survey was phrased as ‘‘agile development makes work more hectic’’. In the analysis, the question and scales are reversed so that the descriptives are better comparable with other questions.

280 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

sophisticated quantitative data analysis. Clustering used in this study [24] provides an analysis method that seeks to build up a number of clusters. It is a widely used method in biology, medicine, and market research (especially segmenting). Cluster techniques are used to create taxonomies, identify patterns of associations, and distinguish sub-groups in samples. Clustering analysis empha- sizes diversity rather than central tendency. In clustering analysis the distance between data points is measured, and the data points closer to each other are marked as belonging to the same cluster. Cluster analysis is especially suitable for studying the differences in a sample. Even though general linkages exist, cluster analysis can be used to demonstrate patterns that do not conform to the majority in the sample [24].

In this study, cluster analysis was used to discover small differ- ences between the groups, primarily consisting of two different attitude shifts. First there is analysis of the (negative) impact of the length of experience with traditional methods on opinions to- wards agility and, secondly, there is analysis of the (positive) im- pact of actual experience with agility on the attitude and opinions detected. Then we drill behind the differences by analyz- ing different respondent groups using the attitude question ‘‘would you go back’’. Finally, a cluster analysis of answers to the same atti- tude question, ‘‘would you go back’’ was used to determine the opinions of those people who belong to both of these groups, i.e., who have long experience with both non-agile and agile methods.

4. Results of the study

In this section, we present research data and analysis to evalu- ate findings in general (Section 4.1), answer the research-questions set for the study (Section 4.2), examine qualitative data for finding underlying factors (Section 4.3), analyze quantitative data to see if the agile methods will prevail (Section 4.4) and list limitations of the study (Section 4.5).

The focus of this study is investigation of the impact of agile and non-agile experience to opinions and toward agile methods. We also analyzed other possible factors that could have impact on respondents’ opinions (such as respondent’s agile role and location) but decided to scope out all additional explaining factors from this study in order to keep it into reasonable length.

4.1. Representative statistics

The perceptions of the respondents regarding agile develop- ment were captured in the ‘‘opinions’’ section of the survey. Firstly, nine specific statements (Table 2) were presented to evaluate the respondents’ view on the impact of agile methods in the organiza- tion. The statements were formulated based on the underlying fun- damentals and claimed benefits of agile methods (discussed in Section 2). Secondly, an attitude question was presented to evalu-

ate the respondents’ overall attitude toward the use of agility in the organization. In the following sub-sections the analysis of the respondents’ opinions (4.1.1) is presented.

It should be noted that even though the ‘‘opinions’’ section of the questionnaire was available for the entire survey population, not all respondents replied to all of the questions.

4.1.1. Opinions on the impact of agile development In the Opinions section, nine statements were presented (Table

2). A sliding scale from 1 to 7 was used to collect the responses: 1 = totally disagree, 4 = neutral and 7 = totally agree. The data dis- tribution to each of the questions was symmetric and normal.

In Table 2, descriptive statistics regarding the nine statements are presented. As can be seen, responses to only one of the state- ments were below neutral. In other words, on the average the respondents thought that agile makes work more hectic. All other statements received positive (i.e., above neutral) responses on the average. The most positive effects of agile development, within the limits of the above statements, were the ‘‘increase of transparency’’ and ‘‘increase of collaboration.’’ The rest of the statements also gained results above neutral (4), i.e., closer to the value of ‘‘some- what agree’’ (5).

4.1.2. Attitude towards the use and deployment of agile methods In the Opinions section, the following attitude question was

asked:

1. Would you go back to the old way of working?

This attitude question had the following options: 1 = Yes, 2 = I see no difference, 3 = No.

Fig. 3 shows the overall distribution of responses to the ques- tion and how the responses were distributed between respondents without agile experience (Binary agile experience = 0) and with agile experience (Binary agile experience = 1). The overall attitude towards agile methods could be considered as positive, as 60% wanted to stay in an agile mode of working, while only 9% wanted to go back to traditional working methods. However, as many as 31% of respondents did not see any difference or did not have a clear opinion on this.

In the analysis the respondents were divided in three groups based on the attitude question as follows: Yes = negative group, I see no difference = neutral group, No = positive group. The large portion of neutral answers (31%) is partially explained (by 12%, see Fig. 3) by the large number of people not yet having experience with agile methods (21% of all respondents).

4.2. Answering the research questions

The research-questions set for the study were:

Fig. 3. Overall distribution of responses to the question, ‘‘Would you go back to the old way of working?’’, categorized by agile experience.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 281

(R1) How does the length of practitioners’ experience using agile methods affect their opinions toward agile methods? (R2) How does the length of practitioners’ experience in using traditional methods affect their opinions toward agile methods?

In this section, we present the analysis of the research data against the above questions. Note that the values 1–7 have been used only in the graphs and the calculations have been done with the using the more accurate data values from 0.00 to 120.00 as de- fined in Section 3.3 and Table 1.

4.2.1. Analysis of variance between agile experience and opinions on agile development

To shed light on the first research question, the nine opinions statements were further analyzed to find whether the length of practical experience with agile methods has a positive impact on respondents’ opinions.

ANOVA assumes the homogeneity of variances i.e. the variance of data in groups should be the same. In order to check this assumption we used Levene’s test which tests the null hypothesis that population variances for groups are equal. The null hypothesis can be rejected if the p-value is less than a certain threshold; in this study we use a threshold of 0.05 (See Table 3).

For certain statements, ‘‘effectiveness of development’’, ‘‘quality of product’’, ‘‘more fun’’, ‘‘more hectic’’, ‘‘more organized/planned’’ and ‘‘autonomy of development teams’’ the p-value is less than 0.05, hence there is a significant difference in population variances and we cannot use ANOVA in these cases. For other statements,

Table 4 Means of opinions per group/agile experience.

Means of opinions of groups Not at all 1–3 months

Transparency 57.0 66.1 Collaboration 19.4 19.2

Table 3 The homogeneity of variances of opinions/agile experience.

Levene statistic p-Value

Effectiveness of development 2.792 0.017 Quality of Product 4.137 0.001 Transparency 0.425 0.832 Collaboration 0.684 0.636 More fun 4.196 0.001 More hectic 4.882 0.000 More organized/planned 3.126 0.008 Autonomy of development teams 3.256 0.007 Earlier detection of errors 0.939 0.455

the p-value is greater than 0.05, hence we cannot reject the null hypothesis. For these statements, we can say that the differences in variances might have occurred by chance. ANOVA is used for these statements only.

The ANOVA results (Appendix B) indicate a significant differ- ence between the experience with agility and the opinions ‘‘Agile increases the transparency of development’’ (F(5, 698) = 10.79, p < 0.05) and ‘‘Agile development increases collaboration’’ (F(5, 694) = 4.425, p < 0.05). The respective means of opinions for these groups are presented in Table 4. As seen in Table 4, the means of opinions tend to become more positive as agile expertise grows.

The Tukey HSD was used for post hoc analysis. The opinions transparency and collaboration were divided into two groups by Tukey HSD. See Appendix B.

Fig. 4 presents a plot on agile experience means on the ‘‘Agile in- creases the transparency of development’’ opinion (scale 0.00– 120.00).

As a result, a statistically significant positive shift was found in two of the nine opinion statements, namely collaboration and transparency.

4.2.2. Analysis of variance between non-agile experience and opinions on agile development

Nine opinions statements were studied to reveal whether the length of experience with non-agile methods (i.e., with traditional software development) seems to have a negative impact on opin- ions towards agile methods. Levene’s test for homogeneity reveals that (Table 5) variances for opinions for effectiveness, collabora- tion, more organized/planned work, autonomy, and earlier detec- tion of bugs are likely to have occurred by chance. The analysis of variance (Appendix C) indicates a significant difference between the groups and the opinions on ‘‘Agile development increases the effectiveness of development’’ (F(4, 683) = 6.791, p < 0.05), ‘‘Agile development increases collaboration’’ (F(4, 671) = 7.811, p < 0.05), ‘‘Agile makes work more organized/planned’’ (F(4, 670) = 7.707, p < 0.05) and ‘‘Agile development enables the earlier detection of er- rors’’ (F = 4, 670) = 7.510, p < 0.05).

Fig. 5 presents a plot on experience with traditional methods and opinions means of ‘‘Agile development increases transparency’’, (scale 0.00–120.00).

4–6 months 7–11 months 1–2 years >2 years

67.8 68.8 73.6 72.9 21.3 19.7 17.8 21.3

Table 5 Homogeneity of variances of opinions variables/non-agile experience.

Effectiveness of development

Transparency Collaboration More organized/ planned

Autonomy of development teams

Earlier detection of errors

Levene statistic 0.894 3.631 1.099 0.972 1.124 2.144 p-Value 0.467 0.006 0.356 0.422 0.344 0.074

Fig. 5. Means plot of transparency attitude per experience on traditional methods (x represents non-agile development background, y represents respondents’ opinion means with 95% Confidence Interval with scale 0.00–120.00).

Fig. 4. Effect of agile experience on the opinion, ‘‘Agile development increases transparency’’ (x represents agile background, y represents respondents’ opinion means with 95% Confidence Interval, with scale 0.00–120.00).

Table 6 Means of opinions grouped by non-agile experience.

Means of opinions

<1 year 1–2 years 3–5 years 5–10 years >10 years

Effectiveness 70.4 69.0 64.1 60.6 61.0 Collaboration 17.1 18.2 19.7 18.9 21.9 More organized/

planned 66.3 62.7 59.1 53.3 54.4

Earlier detection of errors

69.0 67.2 57.9 58.4 59.0

282 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

Table 6 represents the means of opinions within each group. We can see the trend towards a more negative attitude as non-agile experience increases, except for collaboration, where the trend is the opposite.

The post hoc analysis with Tukey HSD (Appendix C) revealed that opinions for effectiveness, collaboration, more organized/ planned, and earlier detection of errors could be divided into two groups based on non-agile experience. People with more than 3– 5 years of experience in traditional development seemed to think more negatively about agile.

Fig. 6. Means of all opinions questions classified by agile experience (0 = no agile experience, 1 = agile experience) (x represents opinions; y represents opinion means with 95% confidence interval with scale 0.00–120.00. Figure on left: respondents having no agile background only. Figure on right: respondents having agile background only.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 283

Based on the analysis above, it can be concluded that long expe- rience with non-agile development negatively affects certain opin- ions of agile development (i.e., increased effectiveness, work being more organized/planned, and earlier detection of errors), and pos- itively on the opinion of increased collaboration. Fig. 6 shows plots of means for all opinions. It can be seen that in general when agile experience grows, practitioners’’ opinions sharpen and get more positive towards agility.

4.3. Perceived benefits and challenges of agile methods: qualitative analysis

As the parametric research and clustering analysis do not pro- vide a comprehensive explanation of opinions towards agility, we decided to also analyze the collected qualitative data. The goal was to find the perceived benefits and challenges reported by the respondents and to see whether significant differences could be detected among respondents with positive and negative attitudes towards agile methods.

Two open questions were selected for further analysis: (1) ‘‘What do you consider the main benefits of agile methods in your organization?’’ and (2) ‘‘What do you consider the main challenges of agile methods in your organization?’’ Firstly, respondents were grouped based on their answer to the attitude question, ‘‘Would you like to go back to the old way of working’’. This question was selected for grouping as it was considered to distinguish the positive group (‘‘No’’ responses) and the negative group (‘‘Yes’’ re- sponses) from the survey population. Neutral responses were not included in the analysis. 16% of the negative group did not report any challenges and 26% did not list any benefits of agile adoption.

20% of the positive group did not answer either of these free-form open questions.

No suitable frameworks were found for classification of the qualitative data of the study. Thus we based the classification to our own earlier work and formed the initial groups from the Key Areas presented in Table 7 on ‘‘agility evaluation grid outline’’ [14]. Then we went through the qualitative responses in several increments to find a set of classes that would fit in grouping them all. The following increments were gone through: (1) classification of negative responses of the negative group, (2) classification of positive responses of negative group, (3) classification of positive findings of positive group, and (4) classification of negative find- ings of positive group. During each increment, new findings were added to the already identified classes, and if no suitable classes existed, new classes were created or existing ones were modified. Table 7 presents the distribution of findings in each key area for the positive and negative groups of respondents.

Somewhat surprisingly, the single largest key area among the negative group was that no benefits or challenges were identified at all (key area 12) or that benefits were unidentified; as for exam- ple, is the case with the following comment: ‘‘There is not any, if you count out momentary increase of work effort which eventually increases workers stress level!!!!!!!!!!!!!!!!!!’’ this comment is an extreme example – most respondents just wrote that there are no improvements. When leaving this particular key area out of the scope, the following Top 3 challenges (Table 8) and benefits (Table 9) could be identified among the two groups.

In the following, the key areas common for both of the control groups – i.e., deployment of agile methods, requirements engineer- ing, and visibility and transparency — are discussed in more detail.

Table 7 Agile benefits and challenges perceived by positive and negative respondents.

Key area Negative group Positive group

+Benefits (#61) (%)

�Challenges (#42) (%)

+Benefits (#347) (%)

� Challenges (#349) (%)

(1) Requirements management/planning/changing requirements/ flexibility

13 17 16 9

(2) (Product) integration – 5 – 1 (3) Resourcing/effort management – 14 – – (4) Visibility/transparency 20 10 16 2 (5) Deployment/tailoring/process improvement 2 24 8 21 (6) Self-organization/cross-functional teams/teamwork 8 10 11 3 (7) Competence – 2 – 7 (8) Management 3 7 2 6 (9) Quality/testing/tools/continuous integration 8 7 7 7

(10) Team size/distribution – 2 – – (11) Other 9 2 5 4 (12) No benefits or challenges or I do not know 22 – 3 – (13) Frequent delivery/speed/responsiveness to change 11 – 6 2 (14) Productivity/focus/efficiency/predictability – – 13 1 (15) People/communication/motivation/collaboration 5 – 12 3 (16) Dependencies/co-operation/work distribution/subcontracting – – – 9 (17) Error fixing/maintenance/ad hoc work – – – 5

284 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

4.3.1. Deployment of agile methods Clearly, the largest hindrance found by both the negative group

and positive group of respondents was the deployment of agile methods itself (key area 5). Respondents in the positive group saw problems with, for example, rolling the deployment out to all teams simultaneously as is also the case in this typical case: ‘‘Rolling it out to people who are not familiar with it and ensuring that it is followed correctly.’’ The proper usage (adaptation) and deploy- ment of agile methods on a large scale was also much commented upon. One of the negative group’s comments concerned, for in- stance, defining the right balance and level of agility: ‘‘We must re- sist the temptation to go too far with the agile idea at the cost of predictable and consistent deliverables to our product programs’’ and ‘‘If one team does agile, then all teams must do (and respect) agile in order for it to be effective.’’ These comments are also typical to the negative group concerned with deployment, i.e. that we would go too extreme with agility.

4.3.2. Requirements management / iterative planning The second largest group of challenges was the same in both of

the compared groups: namely, managing and planning require-

Table 8 Top three perceived challenges of agile development.

Top three perceived challenges

Positive group Negative group

1. Deployment of agile methods 1. Deployment of agile methods 2. Requirements management/iterative

planning 2. Requirements management/

iterative planning 3. Dependencies, co-operation, work

distribution, subcontracting 3. Resourcing/effort management

Table 9 Top three perceived benefits of agile development. (Note that visibility/transparency and requirements management/iterative planning got equal number of votes.).

Top three perceived benefits

Positive group Negative group

1. Visibility and transparency 1. Visibility and transparency 2. Requirements management/

iterative planning 2. Requirements management/

iterative planning 3. Productivity/focus/efficiency/

predictability 3. Frequent delivery/speed/

responsiveness to change

ments in a more flexible and iterative manner (key area 1). The negative group interpreted agility as missing plans and planning as is also the case in this typical comment: ‘‘Software is now created as fast as possible without any planning or design. Major decisions are made off the cuff in the hallway, based on whatever the project man- ager happens to think at the moment.’’ The positive group saw prod- uct management, frequent changes, and the new way of managing requirements in prioritized backlogs as a challenge: ‘‘Not clear how to get new tasks prioritized on backlog(s). Planning is always hard, be- cause of surprises.’’ The comment is typical to the group, represent- ing a true concern on how we get the deployment done correctly.

However, planning was considered beneficial by both groups too. The negative group found beneficial the flexibility to take in new requirements any time. The attitude behind most comments is visible also in this example: ‘‘The only idea around agile develop- ment that I see as positive is the idea of flexibility. Beyond that, I think the agile concepts do not mix well with the need to provide predictable and consistent delivery dates and content information to device.’’ The positive group saw agile planning and priorities as increasing pre- dictability – and typically as simply better way to do planning, as is case also in this comment: ‘‘Product plans are more realistic and thus is easier to plan other things related to products’’.

4.3.3. Visibility and transparency Visibility and transparency (key area 4) was found as another

top benefit of agile development by both groups of respondents. Most respondents from both groups gave no specific explanations as to why visibility and transparency was perceived as a benefit. Two respondents from the negative group, however, named daily scrum meetings as the particular reason for visibility/transpar- ency: ‘‘Visibility into work and reduced slack in developers time due to daily scrums’’. From the positive group, a few respondents gave fast error handling and cross-team communication as explanations as in this comment: ‘‘Visibility inside the team: everybody in the team knows the priorities and why we do what we do’’. Some respondents from the positive group identified increased visibility especially on progress and problems.

Visibility was listed as one of the major problems in agility by the negative group. This comment is clearly reflecting the typical concern that we will lose visibility with agility: ‘‘To get the big pic- ture what is happening at program level’’. Here, visibility and trans- parency were defined as information sharing, and (traditional) program/project tracking.

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 285

4.4. Cluster analysis of attitude question ‘‘Would you like to go back to the old way of working?’’

We wanted to dive deeper to the question whether agile meth- ods are here to stay or not. Assuming opinions and attitudes would really give an answer, we selected the attitude question ‘‘would you like to go back’’ as a basis of further analysis. Obviously, the same person may have long experience with both non-agile and agile development. These may be typical opinion leaders, so it might be valuable to know what especially this group thinks. For this analysis, clustering analysis was used (Table 10).

In this study, the K-means cluster analysis was used due to the large number of data points. The K-means clustering analysis uses simple Euclidean distance analysis to compute the clusters and re- quires the user to specify the number of clusters. Hierarchical anal- ysis with the first 100 respondents gave 2 cluster centers only, leaving all those who would like to go back to traditional methods as outliers. That simply did not fit our purpose of finding out how people with both extensive agile and extensive non-agile experi- ence think. Thus, several sets of numbers for clustering were tried out. As stated by Rapkin and Luke [24], group differences become evident only after a sufficient number of clusters is identified. Rap- king and Luke also suggest the generation of two or more clustering solutions to determine whether cluster structures in a sample gen- eralize across specific measures of interest, and they claim that the proper number of clusters can be determined empirically. As we were interested to find out who would be those who would like most to go back to the old methods, several cluster numbers were tried out until it was noticed that eight cluster centers were needed to find a cluster with a positive answer to this question. Also, large numbers of clusters (like 16 and 32) were tried out, but they pro- vided no additional information compared to the eight-cluster case.

Groups 5, 6 and 7 belong to the group of people we are inter- ested in, having lengthy experience with both agile and non-agile development. Groups 5 and 6 answered that they would like to

Table 10 Quick cluster results for experiences and agile favoring.

Cluster

1 2 3

Final cluster centers How much experience do

you have using agile in practice?

Not at all 1–2 years 7–11 months

How much experience do you have using non-agile development?

1–2 years 1–2 years 1–2 years

Would you go back to the old way of working?

I see no difference No I see no difference

Number of cases per cluster 62 84 20

Fig. 7. Analysis result presented as a bubble graph. Note! The bubbles represent found clu cluster.

stay agile, while group 7 would like to go back to the traditional way of working. When comparing the sizes of these groups, we found that the size of group 7 was roughly 1/3 the size of group 5 – i.e., more people having lengthy experience with both agile and non-agile development methods favor agile than have a nega- tive attitude toward it. See Fig. 7 for a graphical representation of cluster analysis and this result. Could it simply be that people with an extensive development background do not easily ‘‘buy’’ any new development method, but once they have experienced agile development, the majority of them will have a positive attitude?

Fig. 8 illustrates the number of people with extensive experi- ence in both agile and non-agile methods if they would like to stay in the agile mode of working.

Because the sub-group of those who would like to go back is quite small, small variations in the background may not be visible in the cluster analysis. Thus, a cross tabulation was done of all the answers of those who would like to go back to old methods, in order to verify the results from the clustering analysis. The cross tabulation is presented in Table 11.

In Table 11 it can be seen that the backgrounds of the respon- dents are quite evenly distributed. Thus there need to be other explaining factors why these people would like to go back to old methods than the respondent’s background in agile and traditional methods.

To conclude, clustering analysis revealed that majority of the respondents with lengthy experience in both agile and non-agile (traditional) development have a positive attitude towards agility. It is highly plausible that original negative attitudes towards the new working mode are replaced by actual positive experience, once the subjects have been exposed to agile development meth- ods. It could be stated that, based on the positivity of the study group and the positive shift in attitudes based on their experience and willingness to stick to an agile way of working, the future usage of agile methods looks promising. From a practical view- point, this is an important finding, as it is known that persons

4 5 6 7 8

Not at all 7–11 months 1–2 years 7–11 months 4–6 months

5–10 years 5–10 years >10 years 5–10 years <1 year

I see no difference No No Yes No

137 143 84 59 70

ster centers, and their size is relative to number of respondent’s belonging that that

Table 11 Cross tabulation of experience (agile/non-agile) when respondent would not like to use agile methods.

How much experience do you have with non-agile development?

<1 year 1–2 years 3–5 years 5–10 years >10 years Total

Experience from agile in practice � experience from non-agile development (Cross tabulationa) How much experience do you have with agile in practice? Not at all 1 0 1 7 3 12

1–3 months 2 3 0 1 1 7 4–6 months 2 1 2 6 2 13 7–11 months 1 2 9 6 1 19 1–2 years 2 1 1 1 1 6 >2 years 0 0 1 1 0 2 Total 8 7 14 22 8 59

a Would you go back to the old way of working? = Yes.

Fig. 8. Cluster Responses to ‘‘Would you like to go back,’’ paneled by non-agile expertise, and separated by agile experience. This graph is showing only answers from respondents having long expertise in both in agile and non-agile development.

286 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

having lengthy experience are more likely to have recognized and unrecognized leading positions.

4.5. Limitations

This study is scoped only into respondent’s opinions and their agile and non-agile background. The reported benefits and prob- lems with agile methods are scoped out from this study. Also we do not know how much the success in agile deployment at the pro- gram and team level actually impacted the results – this would have required a comparative study over a lengthy time period.

The survey was done in one organization only, so it might be biased by the organizational culture and the specific interpretation of agility; so there remains a question for external validity [25], i.e. how largely these results can be generalized. It could well be e.g. that the resistance to agile methods has some relation to the speed of the transition, and the deployment method itself. This can only be verified by making similar studies in other organizations.

External reliability deals with the issue of whether or not inde- pendent researchers can replicate studies in the same or similar settings [26]. There is a question how much the results were af- fected by the response rate – i.e. would it be possible that only those respondents who are more satisfied with agile methods an- swered the survey. Some smaller studies show similar results [27,28], so this is likely out of the question.

The impact of applicability of agile model to respondents’ role is also scoped out from the study, even though agile impact to some roles have definitely been bigger than to some other roles.

The classification decisions made in qualitative analysis were double-checked for validation and in cases of possible multiple interpretations discussed thoroughly. This adds to the reliability of the study [25] – the results should not be biased by the interpre- tation of the individual researcher. We also double-checked that ranks of the classes were correct and not affected the final inter- pretation made – i.e. if the classes were chosen differently in the questionable cases the order would still have remained the same (in few cases the interpretation of the comment impacted the class or classes chosen).

Construct validity [25] is a concern of obtaining the right mea- sures for the concept being studied. We did a separate pilot survey with a team of roughly one hundred respondents before the survey was sent to a wider distribution.

5. Conclusions

The motivation of this study was to provide empirical evidence of the perceived usefulness of agile methods in practice, especially from the viewpoint of agile transformation in a large-scale indus- trial setting. Firstly, the focus set was to study whether agile meth- ods will provide any real improvement from the viewpoint of adopters. Thus, in this study it is firstly studied how a person’s background – i.e., the length of a person’s experience with agile and traditional methods – correlates with satisfaction and percep- tions of agile methods. From a broader perspective, this provides us with a way to find out whether agility will make a permanent en- trance or if it shall just fade away. As a result, it can be claimed that

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 287

the longer experience with agile methods in practice positively af- fects opinions regarding their usefulness. This finding is in line with earlier reported results [29].

As an answer to the first research question ‘‘How does the length of practitioners’ experience using agile methods affect their opinions’’ we can conclude, that although the opinions were all very positive towards the agile methods we could see that in general the opin- ions changed more positive as the experience grew and that this change was large enough regarding transparency and collaboration to be statistically significant. Likewise we could answer to the sec- ond research question ‘‘How does the length of practitioners’ experi- ence in using traditional methods in affect their opinions’’ that if you had more than 3 years experience on traditional methods it af- fected somewhat negatively attitude towards agile methods, and that this difference was significant with opinions on effectiveness, increased collaboration, work being more organized/planned and agile enabling the earlier detection of errors.

Although a linkage was found between longer experience with traditional methods and a negative attitude towards agile meth- ods, further investigation of ‘‘would you like to go back’’ attitude with clustering analysis revealed that actual experience with agile methods overrides any possible prejudices. Overall, actual experi- ence with agile methods seems to play a more important role than does earlier experience with traditional methods. From the indus- trial viewpoint, these results indicate that once the respondents have experience on agile methods, these are perceived to provide added value.

The second research goal of this study was to contribute to the scientific and industrial body of knowledge on the perceived chal- lenges and benefits of agile development. As a result, three espe- cially challenging areas of agile development were identified: (1) deployment of agile methods, (2) requirements management and planning in a flexible and iterative manner, and (3) visibility and transparency. Thus, our qualitative results confirm earlier proposi- tions [15] that agile deployment and adaptation of agile methods to fit the organization are seen as a challenge. In this study, this was the biggest challenge reported by several survey respondents. Interestingly, although the deployment of agile methods was con- sidered the largest challenge, the perceived usefulness of agile methods still grew with agile experience among the survey respondents. The areas of requirements management and visibility and transparency were perceived as among the most beneficial, yet also among the most challenging, areas of agile development among the survey respondents. Thus, it could be claimed that the above mentioned three areas in particular should be carefully con- templated by any organization planning, or in transition towards, agile methods.

As in some other reported cases [14], it could be identified that respondents fall into groups for and against agility. In this study 60% of respondents could be considered advocates for agile meth- ods (would like to stay in an agile mode of working) and 9% as opponents (would like to go back to a traditional way of working). The rest of the respondents (31%) were neutral. Among the small opponent group, a negative shift was found with non-agile experi- ence in practice. This, again, indicates that agile methods prove to be useful in practice, once carefully adopted and adapted.

Agile transformation is sometimes described as a development culture change and the organizational culture has been mentioned as one explanation to the encountered difficulties with this trans- formation [30]. An organizational culture however is never fixed. In this study we can see how the viewpoints of agile advocates and opponents are different, even when the conclusion is the same; e.g. that the agile deployment is a challenge. Some opinions like effectiveness and quality might have been widely discussed in the organization and this could be the reason why there was no significant difference between the opinions of more and less expe-

rienced respondents. Other opinions such as transparency and col- laboration (were the significant difference was found) may be less discussed and thus less understood – or simply not yet practiced widely and properly enough. This study differs from the preceding studies (such as Dybå and Dingsøyr [31] or Petersen and Wohlin [32]) because of the large number of respondents going through an agile deployment transition at the time when the survey was conducted.

This study supports the findings of Dybå and Dingsøyr [31] about the benefits of (customer) collaboration and increased quality. Some benefits listed by Dybå and Dingsøyr [31] like work processes for handling effects, learning in pair programming, thinking ahead for management, focusing on current work for engineers and estimation, and increased standardization of the job were not questioned in our survey. Dybå and Dingsøyr [31] list the lack of attention to design and architectural issues as limitations of agile methods. These did not emerge as agile transformation challenges in this qualitative study. There may be various reasons to this: firstly, it could be that mostly architects are concerned about these subjects and the per- centage of architects among the respondents is small. Or it may be that the model deployed takes architectural concerns better into account. Or it may be that the other concerns at this state of deployment were simply bigger.

Petersen and Wohlin [32] mention better knowledge transfer due to better communication and frequent feedback as well as early feed- back via discussions with customers as the main benefits of agile methods. These benefits have been categorized under visibility and transparency in our qualitative study. Petersen and Wohlin [32] also mention lacking focus to architectural design but then further list agile development not scaling well and communication between teams as the biggest challenges in incremental and agile development. The concern about scaling agility came up in the positive group of the qualitative part of this study as well: we categorized this under the deployment of agile methods. Com- munication between teams was seen as a third largest problem by the positive group but also in a wider context including organi- zational challenges, work distribution, subcontracting, and co- operation in general.

The survey was done in one company, where agile deployment has been executed in a certain way, so the results may not be gen- erally applicable elsewhere. It would be interesting to evaluate how much the organizational culture and the deployment ap- proach, for instance, affect the results. In addition, it could be ana- lyzed whether other factors such as location, role, or personality of respondents affect their opinions. For instance, one typical assumption is that introverted people who like working alone op- pose agile methods, which emphasize co-operation and teamwork. However, this cannot be verified in our study, as post-interviewing is not an option due to the anonymity of the study. It would be also interesting to study whether the Technology Adoption Lifecycle [33] affects people’s ability to adopt agile practices – i.e., can the respondents wanting to go back to traditional methods be charac- terized as ‘‘Skeptics’’?

At the time of the survey, a large proportion of the population had no experience with agile development. It would be interesting to repeat the survey after some time to see how their opinions have developed as their experience with and understanding of agile methods have increased. It is also unclear how much the po- sitive attitude towards agility has affected people’s decision to start using agile methods – i.e., what came first, the positive atti- tude or the experience? The positive attitude detected among peo- ple who have used the agile method longest can partially be explained by the fact that people who think positively about agile are the first ones to start using it. However, the survey shows con- stant increasing satisfaction towards agile methods over the time that people use those methods. Would this be the case, if only peo-

288 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

ple having a positive attitude in the first place would start to use these methods? Typically, individuals cannot make the decision to use agile methods themselves; the whole team needs to start using these methods simultaneously. This means that even though people are forced to follow company policy in using agile methods, they still develop a more positive attitude towards agile methods after a certain amount of exposure to them.

This survey revealed some new aspects of agility and experi- ence and it also raises some questions for future research. It would be extremely interesting and useful, e.g. to study what level of rela- tion exists with programs or teams that start and succeed in the agile deployment i.e. does the actual level success of agile deploy- ment (e.g. measured as number of agile practices followed or the personal satisfaction how in that team works or the results they get) have any relation to how the team’s or project member’s atti- tudes develop. This type of study would require a close follow-up of team or project attitudes over a longer period of time, and pref- erably from the beginning of the agile deployment.

This survey changed minds across Nokia how agile methods are seen. Before the survey we spoke about agile adoption – after the survey we started to speak with more active tone about agile deployment – the idea being that once people get first-hand expe- rience with agile methods they start to understand these better. The fact that agile methods received such a positive feedback was a surprise to many. For anyone imposing loud criticism to- wards agile methods his or her actual experience with agile meth- ods is then discussed.

1. Agile development increases the effectiveness of development (Li 2. Agile development increases the quality of the product (Line) 3. Agile development increases the transparency of development (Li 4. Agile development increases collaboration (Line) 5. Agile development makes work more fun (Line) 6. Agile development makes work more hectic (Line) 7. Agile development makes work more organized/planned (Line) 8. Agile development increases the autonomy of development team 9. Agile development enables the earlier detection of bugs/errors/de 10. How important you consider agile and iterative development in

(Multi Choice Question) Choices:

1. Very important 2. Important 3. Neutral 4. Not very important 5. Not important at all

11. How satisfied you are with agile and iterative development with work? (Multi Choice Question) Choices:

1. Very satisfied 2. Satisfied 3. Neither satisfied nor dissatisfied 4. Dissatisfied 5. Very dissatisfied

12. Would you go back to the old way of working? (Multi Choice Qu Choices:

1. Yes 2. I see no difference 3. No

13. What do you consider as the main benefits of agile methods in y organization? (Free Answer)

14. What do you consider as the main challenges of agile methods i organization? (Free Answer)

15. If there was one thing you could change, what would it be? (Fre

It would be interesting to conduct another study later to see if the perceived challenges and perceived benefits will change over time. The organization may be able to overcome some of the deployment challenges and the understanding of the benefits may depend on how many agile practices were actually deployed. This would be an interesting hypothesis to test in the future. Also the relation between agile methods and the organizational culture is widely still not understood or studied properly.

Acknowledgements

Maarit Laanti would like to thank Dr. Judy Sheard from the Monash University Australia, and Professor Lauri Malmi for the Quantitative Methods course at Helsinki University of Technology. Without that course this paper would not exist. We would also like to thank Danielle Arvanitis for proof reading the manuscript and Abhishek Tripathi, from University of Helsinki for his valuable comments related to the statistical methods used.

Appendix A. Opinions questions in the survey

A.1. Opinions on agile development

Please, respond based on your current experiences on agile development.

ne)

Answering Areas:

Line ne)

s (Line) fects (Line) the future?

in your own

estion)

our

n your

e Answer)

Appendix B. ANOVA and Tukey HSD results for agile experience and opinions

ANOVA

Sum of squares df Mean square F p-Value

Agile development increases the transparency of development (X) Between groups 21288.962 5 4257.792 10.792 .000 Within groups 275383.476 698 394.532 Total 296672.438 703

Agile development increases collaboration (X) Between groups 8580.965 5 1716.193 4.425 .001 Within groups 269181.828 694 387.870 Total 277762.793 699

Agile development enables the earlier detection of bugs/errors/defects (X) Between groups 5069.697 5 1013.939 2.303 .043 Within groups 304178.937 691 440.201 Total 309248.634 696

Agile development increases collaboration (X) Agile development increases the transparency (X) Tukey HSDa Tukey HSDb

Length of agile experience N Subset for alpha = 0.05

Length of agile experience N Subset for alpha = 0.05

1 2 1 2

Not at all 153 59.0576 Not at all 153 56.9689 1–3 months 90 64.2907 64.2907 1–3 months 91 66.1159 7–11 months 196 65.1451 65.1451 4–6 months 137 67.8189 4–6 months 135 65.6467 65.6467 7–11 months 197 68.8189 1–2 years 29 66.4721 66.4721 >2 years 29 72.9183 >2 years 97 70.6624 1–2 years 97 73.5965 Sig. .164 .317 Sig. 1.000 .162

Means for groups in homogeneous subsets are displayed. a Uses Harmonic Mean Sample Size = 80.055. b Uses Harmonic Mean Sample Size = 80.329.

Appendix C. Tukey HSD for opinions grouped by non-agile expertise, homogeneous subsets

ANOVA

Sum of squares df Mean square F p-Value

Agile development increases the effectiveness of development (X) Between groups 9925.163 4 2481.291 6.791 .000 Within groups 249536.583 683 365.354 Total 259461.747 687

Agile development increases collaboration (X) Between groups 11701.943 4 2925.486 7.811 .000 Within groups 251306.902 671 374.526 Total 263008.846 675

Agile development makes work more organized/planned (X) Between groups 15012.716 4 3753.179 7.707 .000 Within groups 326262.434 670 486.959 Total 341275.150 674

Agile development increases the autonomy of development teams (X) Between groups 2687.577 4 671.894 1.577 .179 Within groups 283247.177 665 425.936 Total 285934.754 669

Agile development enables the earlier detection of bugs/errors/defects (X) Between groups 12736.691 4 3184.173 7.510 .000 Within groups 284077.272 670 423.996 Total 296813.963 674

M. Laanti et al. / Information and Software Technology 53 (2011) 276–290 289

Appendix C (continued)

Agile development increases the effectiveness of development (X)

Agile development increases collaboration (X)

Tukey HSD Tukey HSD

Non-agile experience N Subset for alpha = 0.05 Non-agile experience N Subset for alpha = 0.05

1 2 1 2

5–10 years 230 60.6142 >10 years 127 59.9668 >10 years 129 60.9547 5–10 years 225 61.7078 3–5 years 143 64.1315 64.1315 3–5 years 141 65.9684 65.9684 1–2 years 93 68.9910 1–2 years 92 68.9129 <1 year 93 70.3806 <1 year 91 72.1008 Sig. .599 .078 Sig. .113 .100

Means for groups in homogeneous subsets are displayed.

Agile development makes work more organized/planned (X) Agile development enables the earlier detection of bugs/errors/defects (X)

Tukey HSD Tukey HSD

Non-agile experience N Subset for alpha = 0.05 Non-agile experience N Subset for alpha = 0.05

1 2 1 2

5–10 years 222 53.2573 >10 years 143 57.9318 >10 years 126 54.4899 5–10 years 221 58.4302 3–5 years 143 59.1130 59.1130 3–5 years 127 59.0463 1–2 years 92 62.6925 1–2 years 92 67.2321 <1 year 92 66.2714 <1 year 92 68.9928 Sig. .235 .086 Sig. .993 .963

Means for groups in homogeneous subsets are displayed.

290 M. Laanti et al. / Information and Software Technology 53 (2011) 276–290

References

[1] K. Schwaber, G. Laganza, D. D’Silva, The Truth about Agile Processes: Frank Answers to Frequently Asked Questions, Forrester Report, 2007.

[2] H. Svensson, M. Höst, Introducing an agile process in a software maintenance and evolution organization, in: Proceedings of 9th European Conference of Maintenance and Reengineering, 2005.

[3] T. Dybå, T. Dingsøyr, Empirical studies of agile software development: a systematic review, Information and Software Technology 50 (2008) 833–859.

[4] T. Grant, P. Burris, Z. Reiss-Davis, Inquiry Insights: Agile Development, Forrester Report, 2008.

[5] T. Chow, D. Cao, A survey study of critical success factors in agile software projects, Journal of Systems and Software 81 (2008) 961–971.

[6] A. Sillitti, M. Ceschi, B. Russo, G. Succi, Managing uncertainty in requirements: a survey in documentation-driven and agile companies, in: Proceedings of 11th IEEE International Symposium on Software Metrics, 2005.

[7] K. Beck, Extreme Programming Explained: Embrace Change, Addison Wesley Longman, Inc., 2000.

[8] K. Schwaber, M. Beedle, Agile Software Development with Scrum, Prentice- Hall, Inc., 2002.

[9] M. Laanti, Implementing program model with agile principles in a large software development organization, in: Proceedings of 32nd Annual IEEE International Conference on Computer Software and Applications (Compsac ‘08), 2008.

[10] M. Lycett, R. Macredie, C. Patel, R. Paul, Migrating agile methods to standardized development practice, Computer 36 (2003) 79–85.

[11] K. Beck, Test-Driven Development by Example, Addison-Wesley, 2003. [12] P. Manhart, K. Schneider, Breaking the ice for agile development of embedded

software: an industry experience report, in: Proceedings of the 26th International Conference on Software Engineering (ICSE 2004), 2004.

[13] J. Grenning, Launching XP at a process-intensive company, IEEE Software 22 (1) (2001) 3–9.

[14] P. Kettunen, M. Laanti, Combining agile software projects and large-scale organizational agility, Software Process: Improvement and Practice 13 (2008) 183–193.

[15] D. Norton, Five reasons organizations fail to adopt agile methods, Gartner 9 (2008).

[16] P. Abrahamsson, Measuring the success of software process improvement: the dimensions, in: Proceedings of European Software Process Improvement Conference (EuroSPI 2000), 2000.

[17] A. Field, Discovering Statistics using SPSS, third ed., Sage Publications Ltd., 2009. ISBN: 978-1-84787-906-6.

[18] F. van der Linden, J. Bosch, E. Kamsties, K. Känsälä, H. Obbink, Software Product Family Evaluation, Lecture Notes in Computer Science, Springer Link, 2004.

[19] P. Alreck, R. Settle, The Survey Research Handbook: Guidelines and Strategies for Conducting a Survey, IRWIN Professional Publishing, 1995.

[20] A. Oppenheim, Questionnaire Design, Interviewing and Attitude Measurement, Continuum International Publishing Group, 1992.

[21] P. Nardi, Doing Survey Research: a Guide to Quantitative Methods, Pearson Education, Inc., 2003.

[22] B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El-Emam, J. Rosenberg, Preliminary Guidelines for Empirical Research in Software Engineering, National Research, 2001.

[23] M. Miles, A. Huberman, Qualitative Data Analysis; An Expanded Sourcebook, second ed., SAGE Publications, 1994.

[24] B. Rapkin, D. Luke, Cluster analysis in community research: epistemology and practise, American Journal of Community Psychology 21 (1993) 247–277.

[25] K. Petersen, C. Wohlin, D. Baca, The waterfall model in large-scale development, 10th International Conference on Product-Focused Software Process Improvement, 2009. ISBN: 978-3-642-02151-0.

[26] W. Wiersma, S. Jurs, Research Methods in Education, eighth ed., Pearson Education Inc., 2005.

[27] B. Tessem, F. Mauer, Job Satisfaction and Motivation in a Large Agile Team. Lecture Notes in Computer Science, 2007, vol. 4536/2007, 54-61, doi:10.1007/ 978-3-5.40-73101-6_8.

[28] G. Melnik, F. Mauer, Comparative analysis of job satisfaction in agile and non- agile software development teams, Lecture Notes in Computer Science, 2006, vol. 4044/2006, 32-42, doi:10.1007/11774129_4.

[29] O. Salo, P. Abrahamsson, Agile methods in European embedded software development organizations: a survey on the actual use and usefulness of Extreme Programming and Scrum, IET Software 2 (2008) 58–64.

[30] T. Dingsøyr, T. Dybå, N.B. Moe, Agile Software Development: Current Research and Future Directions, Springer, 2010.

[31] T. Dingsøyr, T. Dybå, Empirical studies of agile softw are development: a systematic review, Information and Software Technology (2008).

[32] K. Petersen, C. Wohlin, A comparison of issues and advantages in agile and incremental development between state of art and an industrial case, Journal of Systems and Software (2009).

[33] J.M. Bohlen, G.M. Beal, The Diffusion Process, Agriculture Extension Service, Iowa State College, 1957.