Course: Software Requirements Definition and Analysis - Discussion Board

Requirements Elicitation Techniques Analyzing the Gap between Technology Availability and Technology Use

ANN M. HICKEY, ALAN M. DAVIS, and DENALI KAISER

279

INTRODUCTION

The current state of building information systems is terrible. At least $185 billion is wasted on development projects that fail, often because the software does not satisfy users’ needs (Standish Group, 1995). Although there are many possible reasons for these failures, problems related to

ABSTRACT Many software development projects fail because the resulting software does not satisfy user needs. The pro- cess of determining user needs is generally termed requirements elicitation. Although there are many possible reasons for software failures, if analysts practiced more effective requirements elicita- tion, fewer projects would fail. Although hundreds of requirements elicitation techniques have been developed by researchers to aid analysts in effectively determining user needs, few have ever been used by practitioners. This paper reports on research to study the nature of the gap between requirements elicitation technique avail- ability and use, identifies the major factors that impact the transfer of elicitation techniques to practice, and explores how to improve the transfer of elicitation techniques from research to practice.

Comparative Technology Transfer and Society, volume 1, number 3 (December 2003):279–304 © 2003 by Colorado Institute for Technology Transfer and Implementation

understanding users’ needs are consistently identified as among the most important (Standish Group, 1999). Requirements elicitation techniques are the means by which systems analysts determine the problems, oppor- tunities, and needs of the customers, so that systems development person- nel can construct systems that actually resolve those problems, leverage those opportunities, and/or address customers’ needs. In response to a less-than-acceptable rate of failure of systems, hundreds of elicitation tech- niques have been created by researchers. But the majority of these tech- niques are rarely, if ever, used by practitioners. Solutions appear to be available, yet we continuously fail to make use of them. Given this gap be- tween elicitation technique technology availability and use, barriers to the successful transfer of requirements elicitation techniques from researchers to practitioners appear to exist.

Problems with the transfer of new requirements elicitation techniques to practice impact more than the software development industry. First, information technology, and more specifically the information systems software that supports an organization’s operations, is critical to the suc- cess of the vast majority of organizations today. Therefore, any problem that impacts the successful development of those systems impacts all organizations. Second, information technology (e.g., knowledge manage- ment systems) is viewed as a key enabler of technology transfer and infor- mation dissemination. Therefore, the successful development of informa- tion systems potentially impacts the successful transfer of many other technologies. Third, continuous process improvement, and the transfer of new processes and knowledge required to support that improvement, is a necessity in today’s rapidly changing and competitive environment. There- fore, problems with the transfer of processes such as new requirements elicitation techniques in an industry that has traditionally been viewed as an early adopter of new technology may provide important, early insights into the transfer of new processes and knowledge in other, less pro-inno- vation industries.

The research reported in this paper seeks to investigate the nature of the gap between requirements elicitation technique availability and use; identify the major factors that impact the transfer of new processes and knowledge, such as requirements elicitation techniques, to practice; and explore how the transfer of requirements elicitation techniques and other new process technologies may be improved. This paper describes our overall research plan, provides details of the exploratory research we have conducted using surveys and in-depth interviews with requirements ex- perts, and summarizes our results.

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

280

BACKGROUND

Requirements elicitation is the process of learning and understanding the problems, opportunities, and desires of customers. It is part of the set of activities usually termed requirements management, whose goal is to understand the customers’ requirements for, and document the desired external behavior of, a new or revised system. Although much of the re- quirements literature describes elicitation as an early activity of the re- quirements phase, it is actually conducted throughout the requirements phase in conjunction with the other requirements activities. But even more importantly, because customers’ problems constantly change, the entire set of requirements activities is performed continuously throughout the sys- tem development process. Therefore, requirements elicitation should be considered an ongoing process throughout the life of a system.

Requirements elicitation is performed by analysts (also known as sys- tems analysts, requirements engineers, and requirements analysts) using elic- itation techniques. In 1996, Capers Jones estimated that there would be approximately 12 million software developers worldwide by 1998. Assum- ing 5% growth per year, and assuming that 1 out of every 15 developers is an analyst, there were approximately 1 million practicing analysts in 2003; so it is clear that there are many potential users of requirements elicitation techniques. To better understand elicitation techniques, and why they are so important to product success, Table 1 provides a brief overview (ex- tracted from Davis, Hickey, & Zweig [2003]) of several major classes of elicitation techniques including interviewing, brainstorming, collaborative workshops, prototyping, questionnaires, observation, and modeling.

When faced with a new requirements elicitation situation, analysts se- lect from the following types of elicitation techniques using a variety of ap- proaches:

1. They select an elicitation technique because it is the only one that they know.

2. They select their favorite elicitation technique.

3. They select an elicitation technique because they understand intuitively that the technique will be effective in the current circumstance.

Using either of the first two approaches yields the potential for disas- trous consequences: alienated customers, incorrect requirements, and sys- tems that fail to meet the customers’ needs. Both researchers and practi- tioners seem to recognize that poor requirements elicitation, and by exten-

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

281

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

282

Table 1

REQUIREMENTS ELICITATION TECHNIQUES

General Classes of Elicitation Techniques History of Techniques in Survey

Interviewing is the process of analysts asking Interviewing has been performed since the questions and prompting one or more stake- dawn of computers in the 1950s. Goal-Ori- holders to verbalize their thoughts, opinions, ented Elicitation during interviews or work- concerns, and needs. Gause and Weinberg shops was first referenced in 1993 (1989) provide a wealth of ideas on how to (Dardenne, van Lamsweerde, & Fickas, interview. 1993).

Brainstorming is the process of gathering mul- Brainstorming was invented by Osborne in tiple stakeholders in a room, posing an issue 1939 (Osborne, 1963), and has been used or question, encouraging the stakeholders to throughout the history of software express their ideas aloud, and having those development. ideas recorded.

Workshops involve gathering multiple stake- The earliest documented cases of using facili- holders together in structured, facilitated tated workshops for elicitation date back to workshops to define the requirements for a the Joint Application Development (JAD) ap- system. Facilitators lead stakeholders through proach developed by IBM in the late 1970s a series of preplanned activities designed to (Wood & Silver, 1995). Role Playing in work- produce the requirements deliverables needed. shops was first referenced as a means to elicit Gottesdiener (2002) provides an outstanding requirements in 2000 (Leffingwell & Widrig, compendium of ideas on how to use work- 2000). shops for requirements elicitation.

Prototyping is the process of creating a partial Prototyping has been around for as long as implementation of a system, demonstrating it humans have manufactured products. Its use to stakeholders, and perhaps allowing them to gather requirements feedback from poten- to play with it. Davis (1995) provides an ex- tial users and customers seems to go back as cellent overall summary of prototyping tech- far as 1977 (Bally, Brittan, & Wagner, 1977). niques and effects.

Questionnaires are collections of closed- and Questionnaires are one of the oldest tech- open-ended questions that are distributed to niques for gathering data from large popula- many stakeholders. Responses are analyzed to tions, and have been used since the dawn of understand general trends and stakeholders’ computers in the 1950s. opinions.

Observation is an ethnographic technique Anthropologists have been using ethno- whereby the analyst observes the users and graphic observation of people since at least customers performing their regular activities. the late 1880s (Stocking, 1996). Its use to A good survey of techniques involving obser- elicit requirements goes back to the early vation can be found in Goguen & Linde 1980s (Parkin, 1980). (1993).

Modeling involves the creation of graphical or Modeling of systems has been performed textual representations of the problem or its since the dawn of computers with the earliest solutions to increase communication and pro- record of flow charts and decision trees dating vide fresh insights into the problem or solu- back to the 1950s (Couger, 1973); data flow tion. A wide range of modeling approaches diagrams to 1977 (Ross, 1977); scenarios to

continued

sion, poor use of requirements elicitation techniques or selection of inap- propriate techniques, is almost guaranteed to result in a poor product (Hickey & Davis, 2002). One would think that the plethora of available elicitation techniques would make it easy for analysts to find one with a high probability of working effectively, yet few are used in practice.

From a researcher’s perspective, the intended transfer of elicitation technique technology from research to practice is as follows: Researchers create new technology, which, along with existing technology, may be adopted by consultants and practitioners. The technology could move directly from researchers to practicing analysts, or from researchers to con- sultants (who in theory should be early adopters [Moore, 1991]), and then transferred to practicing analysts via the consultative activity. Unfortun- ately, given the gap between the number of elicitation techniques available and those that are actually used in practice, the transfer of this technolo- gy does not appear to be taking place as anticipated. The purpose of this paper is to explore the nature of the gap between the availability and use of requirements elicitation technique technology.

TECHNOLOGY TRANSFER THEORIES

Many theories have been posed by the information systems community concerning why elicitation techniques are created but not used in practice (Hickey & Davis, 2002; Jones, 1995; Zmud, 1983). The following theories seem the most plausible (these are not necessarily mutually exclusive).

More time is needed for transfer to occur (Redwine & Riddle, 1985). Stated another way, perhaps the techniques are too immature for transi-

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

283

Table 1

REQUIREMENTS ELICITATION TECHNIQUES (continued)

General Classes of Elicitation Techniques History of Techniques in Survey

exists, including: object diagrams, data flow the 1970s (Alford & Burns, 1975; Davis, diagrams (DFD), the Unified Modeling 1982); Quality Function Deployment (QFD) Language (UML), Z, finite state machines appearing in 1986 (Haag, Raja, & Schkade, (FSM), Petri nets, the System Description 1996); statecharts to their invention in 1987 Language (SDL), statecharts, flow charts, (Harel, 1987); use cases to 1993 (Jacobson, use cases, scenarios, decision tables and Christerson, Jonsson, & Overgaard, 1992) trees, and so on. See Davis (1993a), Kowal and the Unified Modeling Language (UML) (1992), and Wieringa (1996) for descrip- to 1995 (Rational Software Corporation, tions of most of these notations. 1997).

tioning (Heslop, McGregor, & Griffith, 2001). As shown in Figure 1, any new elicitation technique must undergo a series of maturity levels before it is ripe for widespread use by the community of practitioners. Specifically,

• Practicing analysts do not know of the existence of techniques. Technology transfer does not occur because path A2 in Figure 1 is not traversed due to the analysts and consultants not knowing about the availability of techniques. This could be caused by either researchers insufficiently marketing their new technology or practi- tioners insufficiently paying attention.

• Practicing analysts know of techniques’ existence, but do not know how to apply them. Thus, technology transfer does not occur be- cause path A3 in Figure 1 is not traversed. This could be caused by researchers or educators insufficiently teaching practitioners about the new technology or consultants and analysts failing to take the appropriate courses.

• Practicing analysts know of techniques’ existence, and know how to apply them, but do not know when to apply them. In this case, path A4 in Figure 1 is not traversed, again negatively impacting technology transfer of elicitation techniques. Preliminary ideas concerning when to apply specific techniques can be found in Davis (1993a), Hudlicka (1996), Kotonya & Sommerville (1998), Lauesen (2002), Leffingwell & Widrig (2000), Macaulay (1996), Maiden & Rugg (1996), and Wiegers (1999), but more comprehen- sive, situation-specific guidance is not available (Hickey & Davis, 2003).

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

284

Figure 1 Elicitation technique maturity levels.

In conclusion, technology transfer becomes effective once enough time passes, allowing the technique to move at a natural pace through paths A2, A3, and A4.

Many other theories relate to the readiness of the technology and the organizations/individuals for the transfer and the communication required to facilitate that transfer. For example, the available techniques may not be practical or sufficiently demonstrated to be so (Davis & Hickey, 2002). Or, practicing analysts may be perfectly content with what they are doing (in- cluding, for example, using ad hoc techniques), so do not care about alter- native techniques and have no motivation to absorb any techniques that are “ready” for adoption. Or, analysts may not be doing elicitation at all!

More generally, Rogers (1995) identifies four main elements in the dif- fusion of technological innovations: time, the innovation, the social sys- tem, and communication channels. By definition, technological innovations may include hardware, software (including computer software as well as other information or process knowledge), or both. However, Rogers (1995) observes that most research has been conducted on hardware or hardware/software combinations with virtually none on the diffusion of software-only (e.g., information or process) innovations. Regardless, the proposed elicitation technique transfer theories seem to fall within these categories. Therefore, the current research will use Rogers’ framework to determine which conditions or sets of conditions are the most prevalent factors impacting the transfer of requirements elicitation techniques from research to practice.

RESEARCH APPROACH

Given the multitude of theories regarding technology transfer in gen- eral, but lack of specific research on the transfer of requirements elicita- tion techniques, we have chosen a multi-method exploratory research ap- proach. Specifically, we have conducted qualitative in-depth interviews (Seidman, 1998) with selected requirements experts, and quantitative sur- veys (Fowler, 1993) of practicing analysts to explore the research goals posed in the introduction. Specifically, we will

1. Determine whether or not a gap really exists between the avail- ability and use of new requirements elicitation techniques and their transfer from research to practice;

2. Identify the major factors that impact the transfer of elicitation techniques to practice;

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

285

3. Explore how the transfer of elicitation techniques may be improved.

In-depth interviews with 11 of the world’s experts at elicitation were conducted to explore all three research questions. Key to this step was the purposeful selection of the interviewees (Seidman, 1998). Our current cri- teria include (1) practicing with both a breadth and depth of elicitation experience, (2) self-aware, (3) considered by many to be experts, (4) are looking for ways to improve, and (5) demonstrated a desire to share their knowledge with others via publication. The experts interviewed were Grady Booch, Larry Constantine, Tom DeMarco, Don Gause, Ellen Gottes- diener, Tim Lister, Lucy Lockwood, Don Reifer, Suzanne Robertson, Karl Wiegers, and Ed Yourdon. Combined, these experts have over 285 years experience analyzing requirements on more than 700 projects around the world.

A survey of practicing analysts was conducted to investigate the first two research questions. Based on an extensive literature search to identify existing requirements elicitation techniques, we prepared our initial sur- vey instrument (Appendix A). This survey captures the state of the respondents’ knowledge, use, and assessment of selected elicitation tech- niques. Because of the vast numbers of techniques identified, we included only 17 techniques in our initial survey. We selected this subset out of the more extensive list based on the following criteria: (1) have made the ini- tial technology transfer to at least some innovators (Moore, 1991), (2) are not proprietary, (3) have been successfully used in practice, (4) provide category breadth (i.e., coverage of all the categories described in the Back- ground section of this paper), (5) provide age breadth (i.e., coverage of techniques introduced from 40 years ago to recently), (6) provide applica- tion applicability breadth (i.e., coverage of techniques applicable to a wide range of applications such as engineering, real-time, management infor- mation systems, and so on), and (7) provide generality breadth (i.e., in- clude techniques that are generally applicable as well as ones that are de- signed for specific purposes). We have collected responses to this instru- ment from 46 individuals; their educational background, number of years’ experience in requirements engineering, and number of requirements pro- jects worked on are depicted in Figures 2, 3, and 4, respectively. Although the respondents represent a convenience sample of attendees at two inter- national conferences, and practicing analysts in several large software development organizations, the breadth of their education spanning tech- nical, business, and other fields (see Figure 2) and experience ranging from novice to expert (shown in Figures 3 and 4) indicate a reasonable

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

286

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

287

Figure 2 Survey respondents: educational background.

Figure 3 Survey respondents: years of requirements experience.

Figure 4 Survey respondents: number of requirements projects.

representation of the target population. We also interviewed eight of the practicing analysts to explore their responses in more detail.

We have organized our research results into three sections correspon- ding to our research questions: (1) interview and survey results about the existence of a gap between requirements elicitation technique availability and use, (2) interview and survey results about the factors impacting transfer of elicitation techniques to practice, and (3) interview results about ideas for improving that transfer. Whenever counts are provided (e.g., 5 of 11) in the interview results, the reader should assume the other experts did not mention that issue, not that they necessarily disagreed with the others.

RESULTS: EXISTENCE OF A GAP BETWEEN AVAILABILITY AND USE

The interviews with the 11 experts strongly support our hypothesis concerning the existence of a gap between the availability and use of new requirements elicitation techniques. In fact, there was no dissention. Gen- erally, the state of practice (per experts) is that analysts either interview stakeholders or conduct group discussions. A few are considerably better, but most have their heads in the sand. Some use modeling.

The survey provides two key indicators of the existence of a gap im- pacting the use of new requirements elicitation techniques in practice. First, the survey asks respondents if they know of the existence of selected elicitation techniques, which is a necessary precursor to use (see Figure 1). Results were somewhat equivocal on this indicator. On average, res- pondents knew of the existence of 14 of the 17 techniques included in the survey. However, although the majority of techniques were familiar to vir- tually all respondents, one-third (6) of the techniques were familiar to fewer than 80% of the respondents, with 2 techniques familiar to 50% or less. Although these results show a generally positive transfer of knowledge of the selected techniques’ existence, they do show the possible existence of knowledge gaps for some of the techniques. These gaps were somewhat unexpected given the experience level of the survey respondents (see Figure 3) and the nature of the techniques included in the survey.

The second indicator asks respondents if they have ever used the selected techniques. Results on this indicator provided stronger support for the existence of a gap between technology availability and use. On average, respondents had used approximately half of the techniques in- cluded in the survey. Whereas some techniques such as interviewing and

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

288

brainstorming were widely used (93% and 89%, respectively), about half of the techniques have been used by between 40% and 66% of the respon- dents, and 4 have been used by fewer than one third of the respondents. These data are consistent with the experts’ perception that interviews and brainstorming are widely used in practice and that many, even well- known, techniques are less used. Therefore, we believe these results gen- erally support our hypothesis that a gap does exist between the availabili- ty and use of new requirements elicitation techniques.

RESULTS: FACTORS IMPACTING TRANSFER TO PRACTICE

The theories concerning the factors that impact the transfer of require- ments elicitation techniques to practice were listed in an earlier section. This section presents our results concerning these factors, grouped by Rogers’ (1995) four main elements in the diffusion of innovations.

time

To assess the impact of time on the adoption of requirements elicita- tion techniques, we need to correlate the degree of awareness of the tech- niques to the age of the techniques. Table 1 summarizes what we learned concerning the documented dates of availability of each of the techniques included in the survey instrument. The survey results show the degree of knowledge and use that each of these techniques exhibit. When we plot the introduction date against the degree of knowledge by the respondents, we arrive at Figure 5. When we plot the introduction date against the degree of use by the respondents, we arrive at Figure 6.

Although these charts do not show a clear trend, grouping the tech- niques by decade does (Table 2). In particular, these preliminary results show a classic correlation between age of the technique and both the knowledge and the use of techniques within the user community. Thus, Table 2 shows a general increase in knowledge of techniques as they get older (53%, 88%, and 97% as we move from 10 to 20 to 40 years old) as well as a general increase in their use (28%, 62%, and 72%). In both cases, we see a spike in popularity among the most recently introduced tech- niques due to the lemmingineering phenomenon (Davis, 1993b). See the classic age-related technology absorption graph with the lemmingineering spike in Figure 7.

More specifically, to assess the impact of the maturity of the tech-

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

289

niques, we explored each of the maturity levels shown in Figure 1. On average, survey respondents knew of the existence of 84% of the 17 elicita- tion techniques included in the survey (see Table 2). If we believe that our survey respondents are representative of the population at large, then there

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

290

Figure 5 Timeline of elicitation technique introduction versus knowledge.

Figure 6 Timeline of elicitation technique introduction versus use.

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

291

appears to be no problem here except with a few techniques as discussed in the previous section. If we believe that our survey respondents are early adopters (Moore, 1991), then this number seems low, indicating that knowledge of existence is a potential problem. In addition, if we believe that the 17 elicitation techniques represent some of the most commonly known techniques, then knowledge of these techniques may not be gen- eralizable to knowledge of the more than 200 techniques identified in preparation for this research, and knowledge of the existence of tech- niques may still be a potential problem. Many of the experts did agree that knowledge is part of the problem for practitioners. In fact, when describ- ing how they learn about the existence of techniques, all the experts iden- tified the large number of industry and academic publications they read to stay current—an option simply not practical for the majority of practicing analysts.

Figure 7 Absorption curve.

Table 2

TECHNIQUE KNOWLEDGE AND USE AS FUNCTION OF AGE

Average percentage Average percentage Age (years) of techniques known of techniques used

≥ 40 97 72 20–30 88 62

10–20 53 28

< 10 77 39

All 84 56

The degree of respondents’ use of various techniques is shown in Table 2; on average, the survey respondents have used half of the elicitation tech- niques included in the survey. Technique usage ranged from 93% for inter- views to less than one third for 4 of the 17 techniques. Although this does not indicate whether the techniques were used correctly or effectively, it does seem to imply at least some knowledge of how to use at least half of the techniques respondents have used. Techniques with lower usage levels (e.g., Joint Application Development [JAD], quality function deployment [QFD]) generally require more specific knowledge to use, so it may be that their low usage was caused by lack of knowledge of how to use.

We have no data from our survey results to support whether or not analysts know when to use techniques, but several of our experts felt that analysts seemed overwhelmed by the number of available elicitation tech- niques and that guidance on when to use techniques could help to reduce the information overload. Hickey and Davis (2003) report on efforts to better understand when various techniques are applicable.

the innovation

Eight of the experts highlighted concerns relating to the readiness of elicitation techniques for transfer to practice. The interview results exhib- ited quite a bit of comment on whether techniques created by researchers were useful or practical. In particular, six felt strongly that most of the techniques created by researchers are not useful or practical. For example, one interviewee stated that “90% of academic research is crap.” A second stated that only about 25% to 33% of new techniques are even seen by practitioners, and of these, only 25% to 33% ever get into practice. If this is the case, then only 6% to 11% of new elicitation techniques are ever used by practitioners. Another was even more pessimistic, estimating that only 1 in 100 will make it. Still another stated that 65% to 70% of elicita- tion research represents “useless academic results,” 5% is “stimulating,” and the rest “may have some value.” More specifically, one interviewee stated, “For each book [that I read on elicitation, I] may find 1 to 3 ideas that [could] change practice. From papers, [the most I can expect to get is] possibly a great quote/question/table.” Two others implied that too many academics do not have real-world experience, and that as a result, much of what is developed is totally disconnected from the real world.

On a similar note, two of the experts challenged whether new tech- niques are actually being developed by researchers, or whether old tech- niques are just being renamed or repackaged. Specifically, one stated that there has been nothing new in the past 30 years. The other was concerned

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

292

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

293

that similar attitudes would cause practitioners to miss new techniques. The ability to clearly define what is new or different about a technique is important to successful transfer, as Greiner and Franza (2003) found when they observed that “technology which is more understandable, de- monstrable, and unambiguous is easier to transfer.”

Experts were also concerned about the ability to make a business case for a specific elicitation technique. One expert observed that more case studies are needed where “real” practitioners report successful use of the technique. Similarly, another emphasized that organizations/analysts need to experience (or observe) positive results from a technique before they will adopt it. A third highlighted that there is not even any agreement on the “best practices” for elicitation. At the opposite end of the spectrum, one expert stated that a major problem is solution vendors who oversell their technique/tool as a “silver bullet” and preach that one size fits all, that is, that their solution works all the time. According to Heslop, Mc- Gregor, and Griffith (2001), the ability to offer “significant identifiable and quantifiable benefits” is a primary indicator of technology readiness, so we must do a better job of documenting the benefits of elicitation techniques if we hope to successfully transfer them to practice.

the social system

Seven of the experts we interviewed laid part of the blame for poor technology transfer on the practitioners and their organizations. One ex- pert emphasized that organizations do not understand the risk of not doing requirements, or, as another stated, the resources it takes to do requirements correctly. Another expert felt that organizations often do not recognize the need to improve: “We’re doing fine. Our products fail in the marketplace for reasons other than poor elicitation.” A third expert ob- served that organizations have been burned before by oversold techniques, and are now risk averse. Not surprisingly, therefore, as mentioned by four of those interviewed, few companies or analysts dedicate the resources to finding, let alone learning or implementing, new elicitation techniques. Another said that there are too many political and cultural barriers in industry to be able to effectively assimilate new elicitation techniques. Furthermore, they are mostly looking for “silver bullets that require less effort, not more effort. Developing on time is far more important to most companies than doing it right.” Another stated that corporate egos and limited budgets can interfere with acceptance of a new technique. Finally, one of those interviewed summarized his feelings by saying that industry has too much “apathy, ignorance, and inertia.”

communication channels

The problem of communicating research results to practitioners, essen- tial to successful technology transfer, became evident when we asked the experts how they learn about the existence of new elicitation techniques. Seven of the experts read a very wide range of academic and practitioner journals and books on a regular basis, with two emphasizing the need to read outside the field (e.g., other design disciplines). Four mentioned re- viewing conference or journal papers, and five stated they regularly at- tended conferences. Four of the experts felt that networking with associ- ates, students with work experience, and/or researchers at universities helped them to stay current. However, several observed that the academic literature is extremely inaccessible to practitioners. One specifically stated that academics tend to speak in a language quite different from that of practitioners, and that there is a need for translation as well as closer inter- action between academics and practitioners. The sheer volume and inac- cessibility of these information resources make them impractical for the vast majority of practitioners. Because effective communication is critical for successful technology transfer (Kremic, 2003), it is clear that commu- nication mechanisms for disseminating information about requirements elicitation techniques must be improved.

RESULTS: IDEAS FOR IMPROVING TRANSFER

The interviews provided us with numerous, consistent ideas about how to improve the transfer of requirements elicitation techniques to prac- tice. However, one expert felt that was no problem to be solved; very sim- ply, researchers are not producing useful results, so why should we try to increase the transfer of such [useless] technology to practice? This is con- sistent with Rogers’ (1995) observation that an innovation must provide an advantage before it will be used.

Many of the experts’ ideas related to the social system. Most important- ly, the software industry needs to increase the percent of organizations doing requirements. Capers Jones (1991, p. 228) discovered that more than 75% of organizations do not pay adequate attention to requirements activ- ities. To overcome this, researchers need to create a better business case for performing requirements. Much of this business case should emphasize the costs and risks of not doing requirements. This business case also needs to report the necessary costs, hard work, and dedication by all stakeholders

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

294

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

295

that are associated with doing good requirements. One expert emphasized the need to blame poor requirements more often during litigation; this type of negative press is bound to make more companies pay more attention to requirements. Two experts expressed the key role that the technology champion plays in making new techniques work in practice.

Other ideas for improvement relating to the innovation itself include the need for a clearer definition of requirements elicitation techniques. Most of the experts interviewed complained that when a new technique is unveiled, they are unable to understand what the technique is or whether it is any different from techniques they already know. If this is a problem for the experts, it must be overwhelming for the typical practitioner. Re- searchers, working with industry, also need to determine and document which techniques have value in which situations; they must identify best practices. Only after this is done, can we begin to build business cases for individual elicitation techniques. In our research, we found just one (Got- tesdiener, 2002) documentation of a template for making the business case for a specific technique; in this case, collaborative workshops. Most of those interviewed had not themselves even thought about how they select a technique. To make matters worse, almost all the experts reported their frustration with vendors who oversell their techniques as “silver bullets.” Another expert suggested that techniques tend to be more easily adopted when packaged with some type of automated tool. Two of the experts em- phasized the critical role played by business cases, case studies, or project retrospectives to help future projects understand how well or poorly cer- tain techniques worked in the past.

The experts and our own observations indicate a critical need for im- provements in the communication channels used to transfer requirements elicitation techniques from research to practice. First, we must eliminate the mismatch between where researchers publish and what practitioners read. Faculty members are rewarded for publishing in the most research- oriented journals, yet practitioners have time to read only the most practi- cal of publications. We also must improve education. Unfortunately, most academic institutions teach students prescriptive approaches, rather than the more important skill of critical thinking, which is essential for effective requirements elicitation. In addition, mechanisms must be established to encourage practitioners to become academic instructors. One expert said that educational institutions do not give their students enough experience working in teams. To support continuing education, we need to increase the number of practitioner-oriented conferences. One expert interviewed stated that enlightened organizations allow and in fact encourage their employees to attend conferences so they can improve their understanding

of the field. However, the same expert also reported that few conferences are designed for the practitioner. Many claim to be for the practitioner, but are organized by academics and attract mostly fellow academics.

Three final areas address both the social system and communication channels. Many studies (Potts, 1993) have shown that successful technol- ogy transfer can be enhanced via close collaboration between industry, consultants, and academe. The result of such collaboration is that re- searchers better understand real problems in industry and are able to develop techniques to solve them, and industry better understands how new technologies apply to their situation. However, only two experts sug- gested closer collaboration in response to our open-ended question, “How would you fix the technology transfer problem?” One expert did suggest changing the consultant role so that consultants do less of the actual work while undertaking an assignment. Instead, they should spend more time facilitating work so the organization becomes trained, leaving nuggets of new ideas. Consultants need to keep up to date and translate academic research into terms that practitioners understand. Finally, although not mentioned directly in the interviews, analysis of many of the experts’ rec- ommendations made it clear that the inconsistency between the academic reward structure and technology transfer must be resolved. Academic rewards at most research institutions are based on successful publication of research results in journals of little interest to practitioners. Moreover, once published, little or no incentive exists to see the results transferred to practice. Some possibilities for addressing this issue are profit sharing from patents and encouraging faculty to take sabbaticals in industry.

LIMITATIONS

The primary limitations of our research results relate to the survey. The survey respondents may not be representative of our desired practitioner population from two perspectives: (1) 29 of the respondents were atten- dees at requirements engineering and information systems conferences, which attract a high percentage of attendees from academia versus indus- try, and (2) the 17 practicing analysts who responded may be more expe- rienced than average practitioners. Limiting the survey to 17 techniques, while providing interesting results regarding those techniques, impacts the generalizability of our results to all elicitation techniques. We also need to fine tune the measures to gain a better understanding of the extent of the knowledge and use of those techniques.

In addition, although expert interviews provided a rich source of

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

296

information, they also have their limitations. Alternative methods may be needed to collect more detailed information on the specific factors impact- ing technology transfer so that we more adequately assess the validity of our proposed theories and better identify mechanisms for improving the transfer of elicitation techniques to practice.

FUTURE RESEARCH OPPORTUNITIES

Our research goals, survey and interview results, and the limitations of our research provide the following opportunities for future research on the transfer of requirements elicitation techniques from research to practice.

A more comprehensive survey instrument could be developed based on these results to evaluate a greater number of elicitation techniques using more exact measures of knowledge that could then be distributed to a larg- er, more representative sample of practicing analysts. This survey could also explore respondents’ assessments of the factors impacting technology transfer of requirements elicitation techniques identified in this paper.

The recommendations presented in this paper could be extended in several ways. For general factors relating to Rogers’ four elements, a liter- ature search should identify how other fields are addressing these issues. For factors unique to requirements elicitation, ideas could be identified through further discussions with experts, working groups at requirements conferences, and additional targeted research. Specifically, anticipating that one of the factors is analysts’ lack of understanding of when to apply elicitation techniques, we have already begun complementary research into how experts select elicitation techniques so that their expertise can be shared with practicing analysts (Hickey & Davis, 2002, 2003).

The graph depicted in Figure 7 seems logical to us, but represents a model of technology adoption that we have not seen in the technology transfer literature. More research could be done to validate the rate of ab- sorption, and understand the conditions under which adoption demon- strates this phenomenon rather than following more traditional S-curves.

SUMMARY AND CONCLUSIONS

This paper reports results of our research into the factors impacting the transfer requirements elicitation techniques from research to practice. Results of our interviews with 11 experts and a survey of 46 individuals support our hypothesis that a gap between the availability and use of

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

297

requirements elicitation techniques does exist, and identified several crit- ical factors that impact the transfer of elicitation techniques to practice. The interview results also provided interesting ideas about how to improve that transfer. Our goal is to share these recommendations to improve elic- itation technique transfer and the state of requirements elicitation prac- tice, and, as a result, improve future software so that it will more fully sat- isfy users’ needs. Better software will positively impact all organizations using that software for operations, and enable improved information dis- semination and transfer of other technological innovations.

This research also highlighted academia’s pro-innovation bias. Re- searchers often refer to technology transfer barriers, assuming that their new innovations should be adopted, and tend to blame industry for not accepting their research results. That is as silly as a company with a prod- uct blaming the customer for not buying it. The onus is always on the sup- plier to construct a useful product. The most successful mass marketers of products understand the needs of their intended customers before they build their product! Similarly, researchers who want to transfer their re- sults to industry need to understand the desires of industry before they commence their research and clearly demonstrate the value of their new innovations to potential customers. This research has shown that this has not been done for requirements elicitation techniques.

More generally, these results provide insight into the transfer of other process innovations to practice. First, the research demonstrated that even industries known for the early adoption of hardware and computer soft- ware innovations can still have problems with the transfer of process inno- vations. Second, many of the factors impacting the transfer of elicitation techniques also impact the transfer of other process innovations. Third, the recommendations for improving elicitation technique transfer are therefore also applicable to the transfer of other process innovations. Fi- nally, the results demonstrate that Rogers’ framework for the diffusion of technological innovations aided understanding of the transfer of require- ments elicitation techniques and therefore also applies to software-only and process innovations such as this. These results and Rogers’ (1995) analyses of the impact of the characteristics of the innovation, time, social system, and communication channels on the diffusion of technological in- novations can therefore aid the practitioner in the diffusion of many other process innovations.

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

298

ACKNOWLEDGMENTS

The authors would like to thank Daedra Studniarz, the Colorado Insti- tute for Technology Transfer and Implementation (CITTI), and El Pomar Foundation for their financial support and encouragement to pursue this research. We would also like to thank Dean Gary Klein and the entire College of Business at the University of Colorado at Colorado Springs for establishing an environment where research is rewarding, rewarded, and enjoyable.

REFERENCES

Alford, M., & Burns, I. (1975). An approach to stating real-time processing requirements. Conference on Petri Nets and Related Methods. Cambridge, MA: MIT Press.

Bally, L., Brittan, J., & Wagner, K. (1977). A prototype approach to information system design and development. Information & Management, 1(1), 21–26.

Couger, D. (1973). Evolution of business system analysis techniques. ACM Computing Surveys, 5(3), 167–98.

Dardenne, A., van Lamsweerde, A., & Fickas, S. (1993). Goal-directed require- ments acquisition. Science of Computer Programming, 20, 3–50.

Davis, A. (1982). The design of a family of applications-oriented requirements languages. IEEE Computer, 15(5), 21–28.

Davis, A. (1993a). Software Requirements: Objects, Functions and States. Upper Saddle River, NJ: Prentice Hall.

Davis, A. (1993b). Software lemmingineering. IEEE Software, 10(5), 79–81, 84. Davis, A. (1995). Software prototyping. Advances in Computers, 40 (pp. 39–63).

New York: Academic Press. Davis, A., & Hickey, A. (2002). Requirements researchers: Do we practice what

we preach? Requirements Engineering Journal, 7(2), 107–11. Davis, A., Hickey, A., & Zweig, A. (2003). Requirements management in a

project management context. In Morris, P. & Pinto, J. (Eds.). The Wiley Managing Projects Resource Book. New York: Wiley.

Fowler, F., Jr. (1993). Survey Research Methods (2nd ed.). Newbury Park, CA: Sage.

Gause, D., & Weinberg, G. (1989). Exploring Requirements: Quality Before Design. New York: Dorset House.

Goguen, J., & Linde, C. (1993). Software requirements analysis and specifica- tion in Europe: An overview. First International Symposium on Requirements Engineering (pp. 152–64). Los Alamitos, CA: IEEE Computer Society Press.

Gottesdiener, E. (2002). Requirements by Collaboration. Reading, MA: Addison- Wesley.

Hickey, Davis, and Kaiser | REQUIREMENT ELICITATION TECHNIQUES

299

Greiner, M., & Franza, R. (2003). Barriers and bridges for successful environ- mental technology transfer. Journal of Technology Transfer, 28(2), 167–77.

Haag, S., Raja, M., & Schkade, L. (1996). Quality function deployment—Usage in software development. Communications of the ACM, 39(1), 41–49.

Harel, D. (1987). Statecharts: A visual formalism for complex systems. Science of Computer Programming, 8, 231–74.

Heslop, L., McGregor, E., & Griffith, M. (2001). Development of a technology readiness assessment measure: The cloverleaf model of technology transfer. Journal of Technology Transfer, 26(4), 369–84.

Hickey, A., & Davis, A. (2002). The role of requirements elicitation techniques in achieving software quality. Requirements Engineering Workshop: Founda- tions for Software Quality (REFSQ ’02) (pp. 165–71). Essen, Germany: Essener Informatik Beiträge.

Hickey, A., & Davis, A. (2003). Elicitation technique selection: How do experts do it? 11th IEEE International Requirements Engineering Conference. Los Alamitos, CA: IEEE Computer Society Press.

Hudlicka, E. (1996). Requirements elicitation with indirect knowledge elicita- tion techniques: Comparison of three methods. 2nd International Conference on Requirements Engineering (ICRE ’96) (pp. 4–11). Los Alamitos, CA: IEEE Computer Society Press.

Jacobson, I., Christerson, M., Jonsson, P., & Overgaard, G. (1992). Object- Oriented Software Engineering: A Use Case Driven Approach. Reading, MA: Addison-Wesley.

Jones, C. (1991). Applied Software Measurement: Assuring Productivity and Quality. New York: McGraw Hill.

Jones, C. (1995). Why is technology transfer so hard? IEEE Computer, 28(6), 86–87.

Jones, C. (1996). Patterns of Software Systems Failure and Success (p. 55). Boston, MA: Thomson Computer Press.

Kotonya, G., & Sommerville, I. (1998). Requirements Engineering. New York: Wiley.

Kowal, J. (1992). Behavior Models. Upper Saddle River, NJ: Prentice Hall. Kremic, T. (2003). Technology transfer: A contextual approach. Journal of

Technology Transfer 28, 149–58. Lauesen, S. (2002). Software Requirements: Styles and Techniques. London:

Addison-Wesley. Leffingwell, D., & Widrig, D. (2000). Managing Software Requirements.

Reading, MA: Addison-Wesley. Macaulay, L. (1996). Requirements Engineering. London: Springer. Maiden, N., & Rugg, G. (1996). ACRE: Selecting methods for requirements

acquisition. Software Engineering Journal, 11(5), 183–92. Moore, G. (1991). Crossing the Chasm. New York: HarperCollins. Osborne, A. (1963). Applied Imagination. New York: Scribner. Parkin, A. (1980). Systems Analysis. Cambridge, MA: Winthrop. Potts, C. (1993). Software engineering research revisited. IEEE Software, 10(5),

19–28.

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

300

Rational Software Corporation (1997). UML Summary, January 1997. Redwine, S., & Riddle, W. (1985). Software technology maturation. Eighth IEEE

International Conference on Software Engineering (ICSE) (pp. 189–200). Los Alamitos, CA: IEEE Computer Society Press.

Rogers, E. (1995). Diffusion of Innovations (4th ed.). New York: Free Press. Ross, D. (1977). Structured analysis: A language for communicating ideas.

IEEE Transactions on Software Engineering, 3(1), 16–34. Seidman, I. (1998). Interviewing as Qualitative Research: A Guide for Researchers

in Education and the Social Sciences (2nd ed.). New York: Teachers College Press.

Standish Group (1995). The chaos report. Available: www.standishgroup.com. Standish Group (1999). Chaos: A recipe for success. Available: www.standish-

group.com. Stocking, G. (1996). After Tylor: British Social Anthropology 1888–1951.

Madison: University of Wisconsin Press. Wiegers, K. (1999). Software Requirements. Redmond, WA: Microsoft Press. Wieringa, R. (1996). Requirements Engineering. Chichester, UK: Wiley. Wood, J., & Silver, D. (1995). Joint Application Development (2nd ed.).

New York: Wiley. Zmud, R. (1983). The effectiveness of information channels in facilitating inno-

vation within software development groups. MIS Quarterly, 7(2), 43–58.

APPENDIX A SURVEY INSTRUMENT

1. Name: (optional) ___________________________________________________________

2. Field of study during education: ___________________________________________________________

3. Number of years involved in requirements: ___________________________________________________________

4. Approximate number of projects in which you played a requirements role: ______________________________________________________

5. Please complete the following table to indicate your knowledge and experience with a variety of techniques. We understand that this list of elicitation techniques is but a small fraction of those that are avail- able, and that some of the techniques listed may overlap with others.

NOTES from the FIELD

301

Thank you for your time!

Comparative Technology Transfer and Society, December 2003, vol. 1, no. 3

302

Value (check one column for each technique)

I know of I have It has It has no its existence used it value for value for I don’t

Technique (Y or N) (Y or N) elicitation elicitation know

Brainstorming

Interviewing

Joint Application Development (JAD)

Quality Function Deployment (QFD)

Questionnaires

Prototyping

Facilitated Workshop

Scenarios

Use Cases

Modeling

Unified Modeling Language (UML)

Data Flow Diagrams

Observation

Goal-Oriented Elicitation

Role Playing

Decision Trees

Statecharts

Other (be specific)

____________________

Other (be specific)

____________________

Other (be specific)

____________________