Background: Agile methods have increasingly attracted interest in the Software Industry. SCRUM is currently one of the most studied agile method, because of both its novelty and the assumption that SCRUM is able to improve project productivity. Objective/Method: This article describes a systematic literature review, which aims to find scientific evidence of the correlation between the use of SCRUM and productivity in Software Projects. Results/Conclusion: Among 274 primary studies, this research selected 28 papers presenting strong evidence on the research questions. According to these results, the relation between SCRUM and productivity may be positive.

SCRUM, Productivity, Systematic Literature Review, Software Project, Agile Methods.

1. INTRODUCTION

Agile methods have increasingly attracted interest in the Software Industry. A survey performed in the United Stated and Europe has shown that 14% of software companies are already using agile methods, and 49% of the companies which know something about agile methods are interested in adopting them. [13]. This increasing interest may be due to a belief that Agile Methods are powerful development alternatives, supposedly able to avoid project problems such as low productivity, schedule delays, high costs, lack of people motivation, and others, which still are significant problems in the Software Engineering field [13].

SCRUM is currently one of the most studied agile method, because of both its novelty and the assumption that SCRUM is able to improve project productivity. SCRUM is an agile project management method, which uses an empirical process based on flexibility, adaptability and productivity. SCRUM sets high-frequency management activities in order to track real-time problems during a project development process. SCRUM is usually adopted in organizations which have not achieved success by using traditional development or management methods, and it is also usually combined to other development methodologies, such as XP, according to the organization needs.

This article describes a systematic literature review, which aims to find scientific evidence of the correlation between the use of SCRUM and productivity in

Software Projects. Even though there are studies such as [13] seeking for evidence on the benefits, limitations and implications of using agile methods in software projects, those studies are mainly about Extreme Programming.

This research follows guidelines defined by [16, 17] and looks to the literature to answer the following research questions:

(RQ1) Is the use of SCRUM related to productivity improvement in software development projects?

(RQ2) What are the other performance outcomes, which might be confounded with productivity, related to the use of SCRUM in software development projects?

This research found strong evidence about the relation between SCRUM and productivity. Moreover, we found other outcomes such as product quality, client satisfaction, cost reduction and team motivation, which are not necessarily related to productivity but represent general success characteristics in software development projects.

2. SYSTEMATIC LITERATURE REVIEW

This research was conducted in three main steps: planning, execution and data analysis [15]. In the first step, the research protocol was designed, including the definition of the queries terms, choice of scientific

Cardozo A. Neto Barza França Da Silva

resources databases, design of the document selection criteria, and quality assessment procedures. In the second step, the queries were executed, the material was read and selected according to its relevance, and then they were assessed according to their quality. The last step was the data analysis, where the results were organized according to each research question, in order to answer them. The three research steps are better described in the following sessions.

2.1 Planning

The search terms used in this SLR were developed using the same steps as in [16] and [17], and are summarized in Table 1.

Table 1: Research Terms

Population (Software projects)

Intervention

AND (Methodology scrum OR scrum methodology OR agile scrum OR scrum process OR scrum agile process OR agile scrum methodology OR scrum projects)

Outcome AND (productivity OR performance OR efficiency)

Context

AND (software development OR development of software OR software projects development OR development of software projects)

Then, the search was performed only in those search engines and digital libraries available on the Internet and which have partnership with the Universidade Federal de Pernambuco (Federal University of Pernambuco): ACM Portal, IEEE, Scopus, EI Compendex and Science Direct.

After executing the search query, the resulting documents were independently selected by three different researchers, according to the procedure described in Table 2. The selection criteria are also specified in Table 3.

Table 2: Document Selection Procedure

Step 1.1

The three researchers (R1, R2, R3) independently read all titles and abstracts. Then, they selected only the relevant articles and discarded the others.

Step 1.2

The three researchers (R1, R2, R3) independently read the introduction and conclusion of the chosen articles. Then, they selected only the relevant articles and discarded the others.

Step 1.3

The articles selected so far were compiled, in order to remove duplications. Each article was completely read by two researchers (R1+R2, R1+R3, R2+R3).

Step 1.4

The researchers filled a Quality Assessment form for each article, and elaborated a summary containing their key-points such as objectives and findings.

In order to assess the quality of the selected articles, a Likert-scale form was designed covering five characteristics: research validity, threats to validity,

relevance, applicability, and consistency of the presented evidence. Each researcher has assessed all selected articles, so the article grade is the sum of the two grades, assigned in the Step 1.4. After grading, the articles were classified in four types: Very Good (between 31-40), Good (between 21-30), Regular (between 11-20) and Poor (between 0-10).

Table 3: Document Selection Criteria

The articles must

be available among the selected resources;

be written in English.

have some study about SCRUM.

present some result on the relation between the use of SCRUM and software projects productivity.

clearly describe its methodology. ... have sufficient data for the analysis. ... have been completed and concluded. ... be unique (have no duplication).

It is important to report that the researchers also had ad-hoc help from specialists in agile methodologies during the document selection steps, which has added some quality and reliability to this selection process.

2.2 Execution

The first search retrieved 274 studies from the chosen scientific databases. After performing the document selection procedure, 28 relevant articles were selected. Table 4 shows the distribution of these articles among the resources.

Table 4: Distribution of studies among resources

Resource

Search Results

Relevant Studies

ACM Portal 202 21 IEEE Xplorer 19 3 Compendex 13 3 Science Direct 37 1 Scopus 3 0 Total 274 28

This SLR did not restrict the period of publications, although all selected studies were carried out between 2000 and 2009, as shown in Figure 1, which therefore portraits the relevance this theme has recently acquired.

Figure 1: Temporal Distribution of selected studies

Table 5 shows the final classification of the selected articles, according to their respective grades. It is

SCRUM and Productivity in Software Projects: A Systematic Literature Review

important to notice in Table 5 there are no Poor articles, which means that the found evidence have a reasonable level of reliability. This fact also confirms the efficacy of the document selection procedure.

Table 5: Classification according to grades

Class. Articles Total

Very Good [3], [8], [12], [13], [18], [21], [24], [28], [29],

Good [1], [2], [5], [6], [9], [10], [11], [19], [20], [22], [23], [26], [27], [30], [32], [33]

Regular [4], [7], [25]

3 Poor - 0

2.3. Data Analysis

Among the selected studies, this research found relevant evidence to satisfactorily answer its two research questions, as described below.

(RQ1) Is the use of SCRUM related to productivity improvement in software development projects?

The productivity improvement was discussed in 14 out of the 28 selected articles, from which 5 are Very Good, 7 are Good and 2 are Regular. It indicates that the answer of this question may be positive. Even though each study has its particularities, all of them have evidenced some productivity benefit in software engineering projects using SCRUM methodology. It is important to notice that most SCRUM implementations are usually combined with another software development methodology or process model. Therefore, further research may identify what others methods, which combined to SCRUM, have generated evidence of productivity improvement in software development projects.

(RQ2) What are the other performance outcomes, which might be confounded with productivity, related to the use of SCRUM in software development projects?

Among the selected articles, the productivity was the most frequently cited outcome of using SCRUM methodology. However, some relevant evidence refer to outcomes also related to project performance which may be confounded to productivity, such as customer satisfaction, product and process quality, team motivation, and cost reduction. They are properly summarized in Table 6. This study did not find sufficient evidence to consider other project aspects, such as communication, reliability, cohesion, and business value, as a significant outcome, even though they seem to be incidental benefits, of using SCRUM [4, 10, 13, 19, 20, 21, 23].

Table 6: Outcomes of using SCRUM methodology

Outcome Class. Articles Total

Productivity

V. Good [3] [12] [18] [28] [29]

14 Good

[2] [5] [6] [11] [26] [27] [30]

Regular [7] [25]

Customer Satisfaction

V. Good [8] [18] [29]

5 Good [26] [33]

Regular -

Quality V. Good [8] [29]

6 Good [1] [22] [27] [33]

Regular -

Team Motivation

V. Good [24]

5 Good [1] [9] [30] [33]

Regular -

Cost Reduction

V. Good [12]

3 Good [27] [32]

Regular -

3. FINAL CONSIDERATIONS

This study performed a systematic literature review to answer two main questions: Is the use of SCRUM related to productivity improvement in software development projects? What are the other performance outcomes, which might be confounded with productivity, related to the use of SCRUM in software development projects? This research most important contribution is the organization of 28 relevant studies, which generated evidence that SCRUM is related to productivity in software projects. Moreover, other project performance factors supposedly related to the use of SCRUM were mapped.

This research data is still being analyzed in order to address other complementary questions which could not be covered in this short paper, such as identify the context of organizations, project, teams and people in which SCRUM have been related to productivity and also what are the other methodologies and models which have been combined to SCRUM and have contributed to productivity.

It is important to notice, however, that most of the evidence on this theme is recent. A possible explanation for having only a few relevant studies in 2009 is that relevant publications and proceedings have not yet been indexd in the search engines. Hence, even though the selected studies are considered good evidence, further research should replicate this study, covering a broader sample of studies, which would enable wider analyses and evidence comparison. Accessing only five sources can be considered as a threat to validity of his research, because many other sources seem to be relevant but, for access restrictions, their papers could not be retrieved and analyzed. Future research must address this problem too.

Cardozo A. Neto Barza França Da Silva

4. REFERENCES

[1] Fisher K. G., Bankston A., 2009 From Cradle to Sprint: Creating a Full-Lifecycle Request Pipeline at Nationwide Insurance . 2009 Agile Conference. IEEE Computer Society.

[2] Bondi A. B., Ros J. P., 2009 Experience with Training a Remotely Located Performance Test Team in a Quasi-Agile Global Environment . 2009 Fourth IEEE International Conference on Global Software Engineering. IEEE Computer Society.

[3] Sutherland J., Viktorov A., Blount J., Puntikov N., 2007 Distributed Scrum: Agile Project Management with

Outsourced Development Teams . Proceedings of the 40th Hawaii International Conference on System Sciences

2007. IEEE Computer Society.

[4] Nerur S., Mahapatra R., K., Mangalaraj G.,2005 Challenges of Migrating to Agile Methodologies .

Communications of the ACM May 2005/Vol. 48, No. 5

[5] Ramesh B., Cao L., Mohan K., Xu P., 2006 CAN DISTRIBUTED SOFTWARE DEVELOPMENT BE AGILE? Communications of the ACM October 2006/Vol. 49, No. 10

[6] Nottonson K., DeLong K., 2008 Baby Steps: Agile Transformation at BabyCenter.com . IEEE Computer Society, 59-62.

[7] Cordeiro L., Mar C., Valentin E., Cruz F., Patrick D., Barreto R., Lucena V., 2008 An Agile Development Methodology Applied to Embedded Control Software under Stringent Hardware Constraints . ACM SIGSOFT Software Engineering Notes, January 2008 Volume 33

[8] Rottier P. A., Rodrigues V., 2008 Agile Development in a Medical Device Company . Agile 2008 Conference. IEEE Computer Society.

[10] Abrahamsson P., Warsta J., Siponen M. T., Ronkainen J., 2003, New Directions on Agile Methods: A Comparative Analysis , 25th International Conference on Software Engineering (ICSE'03), Portland, Oregon. IEEE Computer Society.

[11] Sutherland J., Jakobsen C. R., Johnson K. Scrum and CMMI Level 5: The Magic Potion for Code Warriors . 2007 Conference, Washington D.C. ACM Digital Library.

[12] Sutherland J., Schoonheim G., Rustenburg E., Rijk M., 2008 Fully Distributed Scrum: The Secret Sauce for Hyperproductive Offshored Development Teams , Agile 2008 Conference, 339-344.IEEE Computer Society.

[13] Dyba T., Dingsoyr T. 2008 Empirical studies of agile software development: A systematic Review Information and Software Technology 50 (2008) 833 859

[14] No:453/2006 Protocol for a Systematic Literature Review of Motivation in Software Engineering Technical Report, UH

University of Hertfordshire

[15] RT

ES 679/2005 Systematic Review in Software Engineering Technical Report, Programa de Engenharia de Sistemas e Computação

PESC, COPPE/UFRJ

[16] TR/SE-0401ISSN:1353-7776/2004 Procedures for Performing Systematic Reviews , Technical Report, Keele University.

[17] No. EBSE 2007-001 Guidelines for Performing Systematic Literature Review in Software Engineering , EBSE Technical Report, 2.3, Keele University.

[18] Barton B., Campbell E., 2007 Implementing a Professional Services Organization Using Type C Scrum . Proceedings of the 40th Hawaii International Conference on System Sciences

2007. IEEE Computer Society.

[19] Uy E., Rosendahl R., 2008 Migrating From SharePoint to a Better Scrum Tool . Agile 2008 Conference. IEEE Computer Society.

[21] Rising L., Janoff N. S., 2000 The Scrum Software

Development Process for Small Teams AG Communication Systems. IEEE Software.

[22] Najafi M., Toyoshiba L., 2008 Two Case Studies of User Experience Design and Agile Develo . Agile 2008 Conference. IEEE Computer Society.

[23] Ktata O., Lévesque G., 2009 Agile development: Issues and avenues requiring a substantial enhancement of the business perspective in large projects . C3S2E-09 2009, May 19-21, Montreal [QC, CANADA]. B C. DESAI, Copyright ©2009 ACM.

[25] Sutherland J., Altman I., 2009 Take No Prisoners: How a Venture Capital Group Does Scrum . 2009 Agile Conference, Chicago, USA. IEEE Computer Society.

[26] Bates C. D., Yates S., 2008 Scrum down: a software engineer and a sociologist explore the implementation of an agile method . International Conference on Software Engineering, Leipzig, Germany. ACM.

[27] Cohen D., Lindvall M., Costa P., 2004 An Introduction to Agile Methods . ADVANCES IN COMPUTERS, VOL. 62.

[28] Cheng T. H., Jansen S., Remmers M., 2009 Controlling and monitoring agile software development in three dutch product software companies . International Conference on Software Engineering. IEEE Computer Society.

[29] Ferreira C., Cohen J., 2008 Agile systems development and stakeholder satisfaction: a South African empirical study . ACM International Conference Proceeding Series; Vol. 338, Wilderness, South Africa. ACM.

[30] Law A., Charron R., 2005 Effects of agile practices on social factors . ACM SIGSOFT Software Engineering Notes Volume 30, Issue 4 (July 2005).

[31] Maurer F., Melnik G., 2007 Agile Methods: Crossing the Chasm . 29th International Conference on Software Engineering (ICSE'07 Companion). IEEE Computer Society.

[32] Sutherland J., Schoonheim G., Kumar N., Pandey V., Vishal S., 2009 Fully Distributed Scrum: Linear Scalability of Production between San Francisco and India . 2009 Agile Conference, Chicago, USA. IEEE Computer Society.

[33] Long K., Starr D., 2008 Agile Supports Improved Culture and Quality for Healthwise . Agile 2008. IEEE Computer Society.