paper I want to fix it - clean it up -organize it and add

Communications of the Association for Information Systems

Volume 26 2010 Article 27

Focus Groups for Artifact Refinement and Evaluation in Design Research

Monica Chiarini Tremblay∗ Alan R. Hevner†

Donald J. Berndt‡

∗Decision Sciences and Information Systems, Florida International University, trem- blay@fiu.edu †Information Systems and Decision Sciences, University of South Florida ‡Information Systems and Decision Sciences, University of South Florida

Copyright c©2010 by the authors. Communications of the Association for Information Systems is produced by The Berkeley Electronic Press (bepress). http://aisel.aisnet.org/cais

Focus Groups for Artifact Refinement and Evaluation in Design Research

Monica Chiarini Tremblay, Alan R. Hevner, and Donald J. Berndt

Abstract

Focus groups to investigate new ideas are widely used in many research fields. The use of focus groups in design research poses interesting opportunities and challenges. Traditional focus group methods must be adapted to meet two specific goals of design research. For the refinement of an artifact design, exploratory focus groups (EFGs) study the artifact to propose improvements in the design. The cycle of build and evaluate using EFGs continues until the artifact is released for field test in the application environment. Then, the field test of the design artifact may employ confirmatory focus groups (CFGs) to establish the utility of the artifact in field use. Rigorous investigation of the artifact requires multiple CFGs to be run with opportunities for quantitative and qualitative data collection and analyses across the multiple CFGs. In this paper, we discuss the adaptation of focus groups to design research projects. We demonstrate the use of both EFGs and CFGs in a design research project in the health care field.

KEYWORDS: design research, focus groups, data quality, health care information systems

Volume 26 Article 27

Focus Groups for Artifact Refinement and Evaluation in Design Research

Monica Chiarini Tremblay

Decision Sciences and Information Systems, Florida International University

tremblay@fiu.edu

Alan R. Hevner

Information Systems and Decision Sciences, University of South Florida

Donald J. Berndt

Information Systems and Decision Sciences, University of South Florida

Focus groups to investigate new ideas are widely used in many research fields. The use of focus groups in design research poses interesting opportunities and challenges. Traditional focus group methods must be adapted to meet two specific goals of design research. For the refinement of an artifact design, exploratory focus groups (EFGs) study the artifact to propose improvements in the design. The cycle of build and evaluate using EFGs continues until the artifact is released for field test in the application environment. Then, the field test of the design artifact may employ confirmatory focus groups (CFGs) to establish the utility of the artifact in field use. Rigorous investigation of the artifact requires multiple CFGs to be run with opportunities for quantitative and qualitative data collection and analyses across the multiple CFGs. In this paper, we discuss the adaptation of focus groups to design research projects. We demonstrate the use of both EFGs and CFGs in a design research project in the health care field. Keywords: design research, focus groups, data quality, health care information systems

Volume 26, Article 27, pp. 599-618, June 2010

Focus Groups for Artifact Refinement and Evaluation in Design Research

Focus Groups for Artifact Refinement and Evaluation in Design Research

600 Volume 26 Article 27

I. INTRODUCTION

Design research in information systems (IS) builds and evaluates artifacts that address particular business needs. The design of artifacts can be described as having two phases repeated in an iterative pattern: the development of the artifact and its evaluation. This is a process which involves frequent iteration between development and evaluation, rather than a procedural approach [Kuechler and Vaishnavi, 2008]. A design researcher not only designs an artifact, but must provide evidence that this artifact solves a real problem. In fact, artifact evaluation is crucial in design research. This requires that the artifact be evaluated for utility and efficacy within the technical infrastructure of the business environment.

The design science community has struggled in the selection of methods by which to accomplish this sort of evaluation of an artifact [Cleven et al., 2009]. Due to the lack of descriptive techniques, design researchers are increasingly suggesting that nontraditional techniques be used for evaluation [Baskerville et al., 2007; Vaishnavi and Kuechler, 2004; Venable, 2006]. Several information systems artifact evaluation methods have been outlined by researchers, including: observation, analytics, experiments, testing, descriptive analysis, and, more recently, action research [Baskerville and Myers, 2004; Cole et al., 2005; Hevner et al., 2004; Jakob et al., 2004; Rikard et al., 2004].

The use of focus group methods to evaluate and refine design artifacts is relatively new to the IS field [Mantei and Teorey, 1989; Smolander et al., 2008]. Thus, our goal in this paper is to adapt traditional focus group techniques for use in design research projects. We propose two types of focus groups: exploratory focus groups (EFG), which are used for the design and refinement of an artifact; and confirmatory focus groups (CFG), which are used for the confirmatory proof of an artifact’s utility in the field. The primary challenge is the structuring of focus groups so participants can collectively use an information systems artifact in order to provide feedback. We describe one potential approach, in which participants collectively decide on an outcome both with and without the artifact, in order to compare decision making strategies. We use a recently completed research study as an example of how to conduct focus groups for artifact refinement and evaluation highlighting potential pitfalls.

II. RESEARCH FOCUS GROUPS

The Focus Group technique has long been utilized in social research to study ideas in a group setting [Morgan, 1988]. A focus group is defined as a moderated discussion among six to twelve people who discuss a topic under the direction of a moderator whose role is to promote interaction and keep the discussion on the topic of interest [Stewart et al., 2007]. The term focus in the title refers to the fact that the interview is limited to a small number of issues. The questions in a focus group are open ended but are carefully predetermined. The set of questions or ―questioning route‖ is meant to feel spontaneous, but is carefully planned. Usually, the moderator encourages the sharing of ideas and careful attention is paid to understanding the feelings, comments, and thought processes of the participants as they discuss issues [Krueger and Casey, 2000]. A typical focus group lasts about two hours and covers a predetermined range of topics. Multiple focus groups allow for understanding the range of opinions of people across several groups and provide a much more natural environment than personal interviews because people are allowed to interact, which allows them to both influence and be influenced by others [Krueger and Casey, 2000]. This is valuable to gain shared understandings but yet allows for individual differences of opinion to be voiced.

Focus groups have been effective both as a self-contained means of collecting data (as a primary research tool) or as a supplement to other methods of research (as a secondary research tool) [Krueger et al. 2000; Morgan 1988]. The focus group technique is particularly useful as an exploratory method when little is known about the phenomenon but also can be used as a confirmatory method to test hypotheses [Stewart et al., 2007].

Originally coined focused interviews, focus groups were used during World War II by social scientists to explore morale in the U.S. military for the War Department [Krueger and Casey, 2000; Merton and Kendall, 1946; Stewart et al., 2007]. Though invented by academics, the focus group technique was mostly ignored by researchers because of the difficulties in demonstrating rigor in analysis and the fear of possible contamination of the interview process. Focus groups were, however, widely embraced by market researchers in the early 1950s. In fact, the use of focus groups continues to grow in the for-profit sector, accounting for 80 percent of industry related qualitative research, and firms have been created to support all aspects of focus groups [Krueger and Casey, 2000; Stewart et al., 2007; Wellner, 2003].

In the 1980s, academics rediscovered focus groups as an alternative to other qualitative research methods, such as interviews and participant observation. Focus groups are now one of the most widely used research tools in the

Volume 26 Article 27 601

social sciences [Stewart et al., 2007]. Researchers in both basic and applied behavioral science disciplines have utilized focus groups as a source of primary data. Education, management, sociology, communications, health sciences (particularly by clinicians), organizational behavior, social psychology, political science, policy research, and marketing are some of the disciplines utilizing focus groups. The diversity of the aforementioned fields suggests that focus groups can be effectively designed, fielded, and analyzed from varying perspectives and priorities.

Information systems researchers have called for a broader variety of available empirical methods to improve relevance of research [Benbasat and Weber, 1996; Galliers, 1991]. As a result, in the IS field we have seen increased attention on the use of focus groups for research [Baker and Collier, 2005; Debreceny et al., 2003; Jarvenpaa and Lang, 2005; Manning, 1996; Mantei and Teorey, 1989; Smith et al., 1996; Torkzadeh et al., 2006; Xia and Lee, 2005]. Similarly, the software engineering community has suggested a need for a wider availability of empirical methods to improve validity and generalizability of their designs [Basili, 1996; Kontio et al., 2004]. Thus, the field of software engineering has also suggested their use as an evaluation and knowledge elicitation technique [Kontio et al., 2004; LeRouge and Niederman, 2006; Massey and Wallace, 1991; Nielsen, 1997]. In addition, across the IT industry, focus groups are widely used in human-computer interface usability studies.

1

We contend that there are several key reasons focus groups are an appropriate evaluation technique for design research projects [based on Stewart et al. (2007), p. 42]:

1. Flexibility. Focus groups allow for an open format and are flexible enough to handle a wide range of design topics and domains.

2. Direct Interaction with Respondents. The researcher is put into direct contact with domain experts and potential users of the design artifact. This allows for the researcher to clarify any questions about the design artifact as well as probing the respondents on certain key design issues.

3. Large Amounts of Rich Data. The focus group interactions produce a large amount of information in the form of qualitative and quantitative feedback. This rich data set allows deeper understandings, not only on the respondents’ reactions and use of the artifact, but also on other issues that may be present in a business environment that would impact the design.

4. Building on Other Respondent’s Comments. The group setting with its opportunities for interactions allows for the emergence of ideas or opinions that are not usually uncovered in individual interviews.

III. ADAPTING FOCUS GROUPS TO DESIGN RESEARCH

The traditional literature outlines several steps for the conduct and analysis of focus groups. Figure 1 summarizes the basic steps that would be applicable for any research-oriented use of focus groups as found in Krueger et al [2000], Bloor et al [2001], Stewart et al [2007], and Morgan [1988]. We analyze each step taking into consideration the two primary goals of design research: (1) refinement (EFG) and (2) evaluation (CFG) of a design artifact and outline some changes to the traditional focus group methodology that allow for the rigorous steps of building and evaluating an artifact.

Formulate Research Problem

In order to effectively define the content and focus groups, the research goals must be clearly identified. Design researchers seek to design an artifact, incrementally improve the design, and evaluate its utility and efficacy. These are two complementary, yet different, research goals. In Figure 2, we illustrate the positioning of the two types of focus groups—exploratory and confirmatory—in the design research process. As discussed more fully in Hevner [2007], two forms of artifact evaluation are performed in a design research project—the evaluation of the artifact to refine its design in the design science build/evaluate cycle and the field testing of the released artifact in the application environment. We discuss the similarities and differences between EFGs and CFGs as follows.

1 For example, usability.gov is a U.S. Government Web site managed by the U.S. Department of Health & Human

Services that outlines the use of focus groups in the design of web pages (see http://www.usability.gov/methods/ focusgroup.html).

602 Volume 26 Article 27

Formulate Research Problem

Identify Sample Frame

· Number of Groups

· Size of Groups

· Source of Participants

Identify Moderator

Develop and Pre-Test a Questioning Route

Recruit Participants

Conduct Focus Group

Analyze and Interpret Data

Report Results

Figure 1. Focus Group Steps

Both focus groups have participants with similar characteristics and similarly designed questioning routes. Both are used to solicit participant feedback about an artifact’s usefulness and utility. Based on feedback from participants, (1) exploratory focus groups (EFGs) achieve rapid incremental improvements in artifact design and (2) confirmatory focus groups (CFGs) demonstrate the utility of the design.

EFGs have two roles: (1) the provision of feedback to be utilized for design changes to both the artifact and to the focus group script and (2) the refinement of scripts and the identification of the constructs to be utilized in future focus groups. Feedback for improvement of the design of the artifact [Hevner et al., 2004; Hevner, 2007; Kuechler and Vaishnavi, 2008; Markus et al., 2002] is an essential component of design research. Additionally, the questioning scripts can be refined to improve the quality of feedback received in subsequent EFGs. EFGs can be used to define and consequently refine the coding scheme that will be used for the analysis and interpretation of field testing in CFGs. The number of EFGs run depends on the number of build/evaluate cycles that use focus groups for evaluation. It is important to note that other evaluation methods (e.g., analytic optimization) may be used for early design cycles while focus groups may be used for later cycles of design refinement based on feedback from human experts and users.

The CFGs are used to demonstrate the utility of the artifact design in the application field. The unit of analysis is at the focus group [Krueger and Casey, 2000] and not the individual participants. Thus, it is crucial not to introduce any changes to the interview script and the artifact when multiple CFGs are conducted. This allows for the comparison of the results across CFGs to demonstrate and corroborate proof of utility of the artifact. The number of CFGs run in the field test depends on the consistency of results across the focus groups, the level of rigor required in the design research project, and the availability of resources needed to run additional CFGs.

Volume 26 Article 27 603

Field Test

Confirmatory

Focus Groups Evaluate

Exploratory

Focus Groups

Build Design

Artifact

ENVIRONMENT DESIGN SCIENCE RESEARCH

Release

Artifact to

Field Test

Iteratively Refine

Artifact Design

Figure 2. Focus Groups in Design Research

Identify Sample Frame

Three decisions are made in this step: (1) number of each type of focus group to run, (2) the desired number of participants in each group, and (3) what type of participant to recruit.

Number of focus groups

Deciding how many focus groups to run can prove to be quite challenging. The literature states that focus groups should continue until nothing new is learned [Krueger and Casey, 2000], yet deciding ―nothing new‖ is being learned is a difficult and somewhat arbitrary task. This is especially challenging in design research. When conducting an EFG, the designers will find that there is always room for improvement of an artifact and certainly a fair amount of subjectivity in interpreting when the design of an artifact is indeed complete. There is a point where we choose to satisfice in order to move forward [Simon, 1996]. For CFG, the decision that enough evidence of utility has been collected is somewhat subjective. Additionally, there is a need to balance available people and resources, since focus groups can be expensive both in terms of time and money (most participants receive some sort of compensation) and expert participants may be difficult to find.

In our experience, at the minimum, one pilot focus group, two EFGs and least two CFGs should be run. The pilot is informal (one could use students) and is used to understand timing issues and any kinks in the questioning route. A design researcher should allow for at least two design cycles and enough contrast for field test analysis. Since the unit of analysis is the focus group, it would be difficult to make a compelling argument for the utility of the designed artifact with just one CFG. In the example we outline in the later part of this manuscript, we used two CFGs.

Number of participants

Selecting group size has several considerations. It may seem simpler (and less expensive) to run fewer, larger focus groups, since it takes fewer focus groups to hear from the same number of participants. Yet this could lower ―sample size,‖ since there are fewer groups to compare. Additionally, the dynamics of smaller versus larger groups are different; smaller groups require greater participation from each member, larger groups can lead to ―social loafing‖. Morgan [1998] suggests a lower boundary of four participants and an upper boundary of twelve participants. Depending on the approach taken to demonstrate the artifact to the group (for example, whether each individual uses the artifact, versus if a moderator demonstrates it), large focus groups (more than six) could be tricky in design research since the subject matter is more complex than traditional focus group topics.

Participant recruitment

The identification of focus group participants is not a random selection, but rather is based on characteristics of the participants in relation to the artifact that is being discussed. A diversity of participants will potentially produce more creative ideas (and perhaps more conflict depending on topic), but segregation of participants based on skills and knowledge may provide more in-depth tradeoffs in values and success measures. In fact, research shows that

604 Volume 26 Article 27

bringing together groups which are too diverse in relationship to the topic of interests could result in data of insufficient depth [Bloor et al., 2001].

For design research, the participants should be from a population familiar with the application environment for which the artifact is designed so they can adequately inform the refinement and evaluation of the artifact. Care should be taken that the participant groups are from a similar pool for both EFGs and CFGs, so that CFGs are in fact confirming a final design. Though the authors have never attempted this, an interesting approach is to use the same groups of participants twice, once to evaluate and once to confirm.

Research is mixed on whether to use pre-existing groups, though for design topics this may be advantageous since the participants have problem solved together and the focus group may approximate a realistic environment [Kitzinger, 1994]. Interaction among participants is one of the most important aspects of focus groups. For example, a group consisting of all technical experts may be very different than an expert/non-expert group [Stewart et al., 2007]. A design researcher must consider membership of the focus groups and how it aligns with the research objective early in the participant selection process. For example, if the artifact is a software requirement methodology, the group membership may consist predominately of requirements analysis experts. If the artifact is a decision aid tool, a design researcher may purposely mix different skill sets: such as systems analysis, business analysis, and context experts in order to include different aspects of the aid in the conversation.

Design researchers should strive to recruit participants that are familiar with the application environment and would be potential users of the proposed artifact. Unfortunately, in many cases such individuals are not easy to find, so plenty of time and effort should be allotted for this task. For instance, it might be possible to conduct the focus group in the evening (most participants will likely work) and offer dinner. Another good approach is to conduct the focus group at a place where the potential participants work, again enticing them with lunch or breakfast. Phone calls and e-mails should be placed at least a month before the focus groups are planned. A few days before the focus groups the participants should be reminded. Researchers should plan for a few participants to not show up, so if the goal is six people, invite eight. However, care should be taken if the no-shows upset the diversity of the focus group. For example, if both medical doctors invited are no-shows, then the group may be left with no doctors.

Identify Moderator

Due to the open-ended nature of focus groups, moderation can be complex, especially in social research. Several skills are important when moderating a focus group. Krueger et al. [2000] find the following skills to be highly important: (1) respect for participants, allowing all participants the opportunity to express their views, (2) the ability to communicate clearly, both orally and in writing, (3) the ability to listen and the self-discipline to control personal views, and (4) a friendly manner and a sense of humor, and (5) the ability to involve all participants in the conversation. For design research, the moderator not only needs to have these skills, but also a clear understanding of various aspects of the designed artifact. The moderator should be familiar with the artifact and be comfortable presenting it to focus group participants.

In some cases, the moderator may be one of the artifact designers. In this case, the moderator has to be very careful not to introduce any personal bias in the presentation of the artifact (we tend to be proud of our work), particularly when conducting an EFG. It may be possible to enlist a second observer to guard against the encroachment of personal views (at least during the initial groups). This is an excellent time to receive good suggestions for improvement of the design and the designer has to be receptive to criticism and suggestions given by the participants; being careful to justify or defend the design work only in appropriate ways that do not discourage discussion and feedback.

Develop and Pre-Test a Questioning Route

The questioning route is the agenda for the focus group. In the questioning route you are setting the direction for a group discussion [Stewart et al., 2007] and it should closely align with your research objectives. There should be no more than twelve questions for a two-hour session [Krueger and Casey, 2000; Stewart et al., 2007]. Two general principals outlined by Stewart et al. [2007, p. 61] are to order the questions from the most general to the more specific and to order the topics by the relative importance to the research agenda. Thus, the topics to be discussed are ordered by importance, and within those topics, the questions are ordered from general to specific.

For a designed artifact, this means beginning with an explanation of the motivation behind the design of this artifact, followed by a broad explanation of different scenarios on where and how the artifact could be utilized, a description of the details of the design of the artifact, training on its use, and finishing with a task where focus group participants are asked to utilize and evaluate the artifact.

Volume 26 Article 27 605

For an EFG, the ―rolling interview guide‖ [Stewart et al., 2007] is an excellent approach. With a rolling interview guide, a script is created for the first EFG but is modified for the next EFG, based on the outcome of the previous EFG. One of the advantages of this approach is that it allows for information to unfold over time as you discover more about how people would understand and use the artifact. However, it is imperative that no revisions are made to the interview guide in the CFGs, since continuous change would make comparisons across the focus groups difficult, compromising rigorous interpretation of the results [Stewart et al., 2007].

A promising evaluation approach in design research focus groups (both EFGs and CFGs) is to create a manipulation within the focus group. Participants can be asked to collectively complete a task without the artifact and then again with the artifact. The ensuing discussion should revolve around how the artifact was used and how the completion of the task was altered by its use.

Conduct the Focus Group

Focus group sessions should be fun and stimulating for the participants and moderator [Stewart et al., 2007]. The moderator usually greets the participants as they enter and may ask them to fill out demographic information and informed consent forms (e.g., IRB forms). The participants are generally seated in a U-shape arrangement to encourage collaboration [Krueger and Casey, 2000] and allow space for the moderator to demonstrate the artifact. Seating arrangements are also very important. A good approach is to get to know the participants before the questioning route begins. Greeting them when they arrive is a good first step. The most assertive and expert participant should be seated next to the moderator and the least talkative directly across from the moderator [Krueger and Casey, 2000; Stewart et al., 2007]. One potential risk is that an assertive participant could manipulate the conversation and the moderator will have to discreetly refocus the group (in fact we describe such a case in our example).

Depending on your research protocols, focus groups may be video and/or audio taped. Generally, the participants are told they are being recorded and most institutional review boards require written consent. It is also a good idea to have an observer. The observer will not participate in the focus group, rather will take careful notes, noting in particular any strong reactions, the participants’ facial expression and general tone of any exchange between participants or between the participant and the moderator [Stewart et al., 2007].

Time management is also important when conducting a focus group. A moderator should be able to recognize when all possible issues for a topic have been covered and move on to the next topic. Pilot focus groups can help anticipate and manage the timing of focus groups.

Additional guidelines for running focus groups can be found in many excellent texts, such as: Kruger and Casey [2000], Stewart et al. [2007], Bloor et al. [2001], and Morgan [1988].

Analyze and Interpret Data

The two design research goals for using focus groups are the incremental improvement of the design of the artifact and the demonstration of the utility of the design. For this reason, we have suggested the different focus group types of EFG and CFG. While the objectives of the two group types are very different, the methods of analyzing the focus group data from both EFG and CFG can be similar. The interpretation of the focus group discussions has many of the same challenges in demonstrating rigor that all qualitative research encounters share. Several techniques that are used for qualitative data analysis can be considered, carefully selecting those techniques that emphasize the reliability and replicability of the observations and results [Stewart et al., 2007].

One possible approach is template analysis. Template analysis normally starts with at least a few predefined codes which help guide analysis. The first step in template analysis is to create an initial template by exploring the focus group transcripts, academic literature, the researchers’ own experiences, anecdotal and informal evidence, and other exploratory research [King, 1998]. The contents of the discussions are also examined for the meanings and implications for the research questions. Analysts will look for common themes and variations within the transcripts that would provide rich descriptions of the participants’ reactions to design features.

In template analysis, the initial template is applied in order to analyze the text, but is revised between each EFG session. Once the final template is created after the final EFG, it is used to code the CFG sessions.

Report Results

King [1998] suggests that qualitative results can be reported by creating an account structured around the main themes identified; drawing illustrative examples from each transcript as required. A similar approach can be taken when reporting focus group results. Short quotes are used to aid in the specific points of interpretation and longer passages of quotation are used to give a flavor of the original discussions. Summary tables can be very helpful, displaying both evidence and counter-evidence of the utility of the artifact by focus group. Rich descriptions can

606 Volume 26 Article 27

further corroborate results by using quotes from the focus group participants. Miles and Huberman [1994] provide many options for reporting qualitative results.

IV. FOCUS GROUP USE IN A DESIGN RESEARCH PROJECT

To illustrate the use of focus groups in a design research project, we discuss a recently completed research project [Tremblay, 2007] in which an artifact was designed and evaluated in the health care context. The research investigated issues of data quality in the context of public policy health planning.

Research Context

In a recent field study, our research group had observed health planners in a regional health planning agency in the State of Florida utilize an Online Analytical Processing (OLAP) tool to navigate data from a data warehouse [Berndt et al., 2003; Tremblay et al., 2007].

This field study provided an exceptional opportunity to study knowledge workers in a real life context. It offered a rich understanding of the health planner’s tasks and their use of OLAP technology. After we completed the study, we noted that users were making important public policy decisions, but were unaware of data quality problems. When data are examined in the aggregate in an OLAP cube, data quality problems are masked and can result in poor decision making. We subsequently designed three methods to inform decision makers about data quality problems so that they could consider data quality information along with the context of a decision. The goal of these methods was to aid decision makers to better understand the quality of aggregated data and to help them overcome inherent decision-making biases.

Data Quality Metrics Description

Our research developed artifacts comprised of methods for calculating metrics that provide data quality information that can be presented to decision makers at decision time. The decision maker is not involved in the calculations, but considers the qualities of the data as he or she formulates a context-specific decision. We selected and designed these metrics based on problematic scenarios observed in the field study of health planning decision-making, using an online analytical processing (OLAP) environment. We surveyed the database and data quality literature [Ballou et al., 1998; Ballou and Pazer, 1985; Ballou and Pazer, 2003; Cappiello et al., 2003–2004; Chen, 1976; Chengalur-Smith et al., 1999; Parssian, 2006; Parssian et al., 2004; Pipino et al., 2002; Shankaranarayan and Cai, 2006; Shankaranarayan et al., 2003; Shim et al., 2002; Shin, 2003; Wixom and Watson, 2001] and decision biases literature [Tversky and Kahneman, 1982] and aligned them with the three key data quality dimensions suggested by Wang and Strong [1996]. The data quality metrics are summarized in Table 1. We considered several alternative evaluation techniques and selected the focus group technique as the most appropriate for the research context. The subsequent discussion describes how we appropriated the focus group steps shown in Figure 1 to this design research project.

Design Research Problems

In order to correctly design the focus group scripts and identify qualified participants, the research goals were clearly identified for each type of focus group. Both EFG and CFG shared the first goal: To solicit participant feedback about the three metrics’ utility and efficacy. Utility was defined as ―usefulness of the metric‖ and efficacy as ―having the ability to change data analytic strategies.‖ The additional research goals for the EFGs were:

· To solicit participant feedback on how to improve the usability and presentation of the metric (how to improve the metrics)

· To analyze transcribed data in order to improve the focus group script and coding schema.

For the CFGs the additional research goal was:

· To collect evidence of metric utility and efficacy.

Identify Sample Frame

A total of five focus groups with six to twelve participants were performed. The first focus group was a pilot and was conducted in a Master’s of MIS Data Warehousing course. The pilot was instrumental in identifying timing issues, refining the questioning route, evaluating the moderator’s style, and surfacing any potential logistical issues. The pilot data were not used further for analysis since the participants were students and not actual healthcare decision makers. The decision of two EFGs and two CFGs was made primarily due to constraints in: (1) the ability to find

Volume 26 Article 27 607

participants that met our requirements, (2) time, and (3) cost containment (we offered dinner to each focus group). We did find that two EFGs were sufficient, but felt that we probably should have run more CFGs (we discuss this in the closing remarks).

The requirements outlined for the participants of both EFGs and CFGs included previous experience with decision- making in the healthcare field, an advanced college degree, and some training in statistics and decision-making software systems.

Table 1: Data Quality Metrics

Data Quality Problem [Wang and Strong, 1996] Metric

Completeness. A problem is encountered when

combining or aggregating data from multiple sources that are missing codes or have codes that do not match other sources of data. This results in data that are not assigned to any of the possible cells in a data cube.

Unallocated data presentation methodologies (UDPM)

which considered the effects of null values in any of the grouping or filtering variables for counts and for averages.

Representational Consistency. When considering

aggregated data or when observing trends, decision makers rely on point estimates, such as an average, which may be biased by noisy data. This is particularly true in healthcare, where data are frequently not normally distributed.

Information volatility was defined as the rate of change in

the values of stored data. In order to judge the volatility of this data we transformed the data in order to achieve a more ―well-behaved‖ distribution. Assessment of reliability was outlined by presenting the decision maker with a metric and benchmarking information. Two types of information volatility were defined: intra-cell (trends) and inter-cell (summarized data) [Tremblay et al., 2009].

Appropriate Amount of Data. Insensitivity to

sample size by decision makers when considering/ comparing groupings

Sample size indicator was a simple method of drawing the

attention of the decision maker in order to mitigate a well known judgment bias with the use of highlighting and indicating sample size.

Identify Moderator

The moderator was the primary researcher who had some experience in moderating focus groups in both educational and industrial settings. Another researcher served as an observer, taking careful notes, providing a final summary to participants, and supporting the moderator in time keeping.

Develop a Questioning Route

The planning process included creating a carefully planned script in which all three of the designed metrics were presented to the participants (see Appendix A for a partial script). The research utilized the ―rolling interview guide‖ [Stewart et al., 2007] for the EFGs. A script was created for the first focus group. Then, based on the outcomes of the first EFG, the guide was revised for use in the second EFG. Based on the outcome of the second EFG, the script, the coding template and the metrics were revised again. No revisions were made to the questioning route during the execution of the CFGs.

―Vignettes‖ or storylines were used to create decision scenarios based on current healthcare situations (in recent news reports) and sample healthcare data. These data included data from a statewide cancer registry, which has been collecting incidence data since 1981, county data from the US Census Bureau, demographic data from commercial sources, and an internally generated time dimension. These decision scenarios were based on the decisions we observed in the field study [Berndt et al., 2003; Tremblay et al., 2007]. The strategy was to present the data with and without the metrics information in order to detect differences in the collective decision-making processes. Thus, we developed an experimental manipulation within the context of a focus group. Table 2 shows examples of the vignettes used in the focus groups.

608 Volume 26 Article 27

Table 2: Example of Vignettes

Metric Evaluated Vignette Decision

Unallocated data presentation methodologies (UDPM)

Studies have shown that smoking is responsible for most cancers of the larynx, oral cavity and pharynx, esophagus, and bladder. In addition, it is a cause of kidney, pancreatic, cervical, and stomach cancers, as well as acute myeloid leukemia.

Is there correlation between smoking and certain types of cancer?

Unallocated data presentation methodologies (UDPM)

When Hispanics are diagnosed with a certain cancer (fictitious example), they’re less likely to receive chemotherapy than non-Hispanics.

Is there disparity in care between ethnic groups?

Information volatility metric

Counties neighboring the target county are better at early detection/prevention of breast cancer based on volumes of cases.

Examine trend; is this a true claim?

Sample size indicator

Tumor size has been shown to be a good predictor of survival for certain cancers, including: breast, lung, and endocrine. Compare average tumor size in the target county to that of neighboring counties.

How does the target county compare to other counties?

Participant Recruitment

Potential participants were identified via personal contacts and phone calls to county public health departments. The potential focus group members were given a brief description of the study, and their participation was requested. They were offered dinner after the focus group session. Many of the participants had taken university courses in data warehousing and/or data mining. Several other participants had jobs that required use of data analytics (e.g., spreadsheets, business intelligence tools, statistics packages). To illustrate the qualifications of participants, Table 3 shows the demographic characteristics from one of the focus groups, which is representative of the other three groups.

Conduct Focus Groups

The focus groups were held in state-of-the-art conference rooms. The participants were seated in a U-shape arrangement to encourage collaboration [Krueger and Casey, 2000] and to allow space for the moderator to demonstrate the design artifacts and PowerPoint presentation. The moderator presented the experimental vignettes and encouraged the participants to play the role of a healthcare decision-maker. In order to analyze the data, the focus group guided the moderator in exploring the health care data. For example, participants were encouraged to ask the moderator to drill down or roll up data in order to thoroughly understand and compare data for different counties as part of their decision making process.

The participants were asked to come to consensus on a particular task without the data quality metric. A PowerPoint presentation was used to describe the vignettes and the metrics and the OLAP decision aid was projected to the screen. The moderator presented the health care decision-making problem and navigation in the OLAP decision aid was guided by participants (for example, participants could ask to drill from yearly data to monthly data, or to filter by ethnicity). The participants collectively decided first without data quality metrics. They were then asked to reconsider the decision utilizing the data quality metric. The ensuing discussion revolved around how the data quality metric was used and how the metric affected their decision making. The sessions were recorded and professionally transcribed.

After conducting each of the EFGs, significant changes were made to both the design artifacts (the data quality metrics) and to the focus group scripts and coding templates. The observer helped refine the focus group script used in the EFG. He carefully observed people’s understanding of the scenarios, their reaction to the metrics and the flow of the conversation and took detailed notes. In our case, the focus groups were not videotaped. Videotaping could be useful in studying people’s reactions post-facto. The observer’s notes were carefully analyzed and changes were made to the focus group script for the next EFG. For example, the observer noted that the moderator needed to better clarify the goal of research, in particular he needed to give a clearer description of who normally would utilize these types of tools and for what sorts of tasks. Once the first CFG began, no changes were made to the questioning route.

Volume 26 Article 27 609

The EFGs were useful in indicating changes needed in the design of the metrics. Table 4 presents several example design changes introduced after the two EFGs. These changes required the researchers to create new calculations and implement them in the OLAP environment.

Table 3: Sample Focus Group Participants G

e n

d e

r

A g

e

L a

s t

D e

g re

e

C u

rr e

n t

P o

s it

io n

C o

u rs

e i

n

S ta

ti s

ti c s

?

Y e

a rs

o f

W o

rk

E x

p e

ri e

n c e

Y e

a rs

o f

H e

a lt

h c

a re

E x

p e

ri e

n c e

S e

lf R

e p

o rt

e d

C o

m fo

rt w

it h

D a

ta A

n a

ly s

is

(7 p

o in

t s

c a

le )

M 34 Ph.D. Health Economist Y 7 6 7

M 51 Ph.D. Assistant Director of Measurement and Evaluation

Y 28 28 7

F 49 Ph.D. Researcher Y 28 28 5

F 35 Ph.D. Project Manager/

Data Analyst/

Health Science Specialist

Y 9 9 5

M 56 Ph.D. Health Services Researcher Y 25 20 6

F 31 MA/

MPH

Program Specialist Y 8 7 5

F 31 MSPH Project Manager Y 8 6 7

F 36 Ph.D. Health Economist Y NR 3 7

Table 4: Example Design Changes Made after EFGs

Metric EFG Design Change Reason

Information Volatility

EFG1 Addition of a graph to the OLAP screen for a pictorial feel of the variability in the data (along with the numeric value of volatility)

Participants found numeric value difficult to interpret.

Unallocated data presentation methodologies (UDPM)

EFG2 Introduced a case based scenario which showed: the cube without allocation (all null data grouped together), the proportional allocation and the worse cased scenario (based on context of decision)

Originally, we were just presenting the cube with null data grouped together, and proportional allocation. The participants felt that they would also like to consider the case where the missing data was not equally distributed, especially in cases where the final decision would have dire impact.

Information Volatility

EFG2 Benchmarking was added, which included a graphical presentation, a numerical presentation and a categorical presentation (medium, high, low) of benchmarking data.

Even with the graphical presentation of volatility introduced after EFG2, focus group participants felt that they would have a better understanding of the volatility of the data if it was benchmarked against a standard.

Sample Size Indicator

EFG2 Gradients were added (using color), allowing for the analyst to conduct a sensitivity analysis.

The focus groups indicated they would want to know the severity on the sample size issues.

Analyze and Interpret the Data

The analysis and interpretation of the focus groups had two goals: (1) to report on the evidence and counter- evidence of utility of the proposed metric and (2) to show evidence of the efficacy of the proposed metric. To conduct this analysis, the focus group sessions were recorded and professionally transcribed. The transcriptions were entered into QSR NVivo where they were coded by two independent coders.

Several approaches are available to analyze qualitative data, including grounded theory [Corbin and Strauss, 1990] or interpretive phenomenological analysis [Smith, 1996]. For our study, we selected template analysis as outlined in the references [King, 1998; King, 2004].Template analysis is flexible, has fewer defined procedures than its more formal alternatives, and is adaptable to our requirements.

610 Volume 26 Article 27

We began by creating an initial template with high order codes and some lower order codes that focused on aspects of the data quality metrics. According to King [2004], ―a code is a label attached to a section of text to index it as relating to a theme or issue in the data which the researcher has identified as important to his or her interpretation.‖ We worked systematically through the transcript of the first EFG and identified sections of the text which were relevant to our aim. Our aim was primarily to identify changes in data analytic strategies and to find evidence or counter-evidence of the metrics’ usefulness. In addition, we also created several other coding categories in order to explore the entire range of participant reactions.

Using QSR NVivo (see Figure 3 for an example), two independent coders marked the transcribed text with the appropriate code from the initial template. The coders met to discuss the areas of disagreement, stopping when agreement was reached on all higher ordered codes and most of the lower order codes. The coders settled on inadequacies in the initial template and made incremental changes. The transcripts were then recoded based on the reconciliation between the two coders.

We continued this process through the EFGs until our final template was created and used for the CFG (no further changes were made to the template during the CFG analysis). We provide Table 5 as an example of a partial coding template for one of our metrics, the information volatility metric.

Figure 3. Example of Screen Used in Coding

Table 5: Partial Final Coding Scheme for Information Volatility

Goal Code Definition

Efficacy— change in data analytic strategy

Information Volatility Before

· Strategies to deal with volatility prior to receiving metric.

· Interpretation before

Information Volatility After · Strategies to deal with volatility after receiving metric

· Interpretation after

Utility Design Feature Volatility Mention of the information volatility feature, design improvement suggestion

Further analysis Speculation Speculation on data quality problems

Other Factors in Decision Making

Including stakeholder issues

We analyzed utility of the metric by investigating all the passages that were coded in our template analysis as ―design feature.‖ As was previously described, the participants were asked to come to consensus on a particular task without the data quality metric and again with the data quality metric. To understand efficacy we contrasted data analytic strategies in the passages that were coded as ―before‖ and ―after‖ the presentation of each metric.

Volume 26 Article 27 611

Report Results

For each of the metrics proposed in the study, utility and efficacy of the metrics were presented. The qualitative data were summarized for both utility and efficacy and then rich descriptions were given using quotes from the focus group participants to corroborate the results. In this section we give partial results for one of the metrics, information volatility. Table 6 provides an example of the summary results for utility.

Table 6: Utility of Information Volatility Metric

Focus group Evidence of Utility Counter-Evidence of Utility

EFG1 Yes Difficulty interpreting

EFG2 Yes Difficulty interpreting

CFG1 Yes. Saw several instances where this would be useful in their daily data analysis

None

CFG2 Yes None

Additionally, we are able to provide evidence by using passages from the focus groups, for example, this quote from a participant in CFG1 that showed evidence of utility:

“I like that calculation and the idea of having a metric or measuring and giving you this kind of information.”

In this quote, a participant of EFG1 showed counter-evidence of utility:

“Yeah, the volatility one is a little—I think a little more—difficult. Well, because people don’t have a lot of background in what that means.”

Similarly, Table 7 shows how the efficacy of the same metric is evaluated. This particular example points out one of the limitations of the use of focus groups. We ran two CFGs to field test the set of design metrics. For the volatility metric, no efficacy data were collected from one of the CFGs. This group was dominated by an individual who rejected the validity of the vignette presented to evaluate the metric. The participant in CFG1 had difficulty ―buying into‖ the reality of such a scenario, though the group found this metric useful and saw the potential for its use in their daily tasks. The individual convinced the group to refuse to make a decision. Thus to study this metric, at least one more focus group would need to be run to show stronger evidence.

Table 7: Efficacy of Information Volatility Metric

Focus group Change in Data Analytic Strategies?

Comments/Observed Changes

EFG1 Yes

EFG2 Yes

CFG1 N/A Rejected task, group questioned realism of the vignettes, refused to make decision.

CFG2 Yes

In the same way, passages can be used as evidence. For example, prior to seeing the IVM metric a participant in CFG2 noted:

“No, they're not doing as well, because they have a straight across line, and there's no decrease; whereas the other two counties that you showed had a decrease.”

When the IVM metric was made available, the participant reversed his prior decision, since the counties that were being compared to Miami-Dade had high IV numbers.

V. CLOSING REMARKS

Our goal in this paper was to propose and demonstrate the use of focus group methods for design research. We showed that focus groups can be effectively applied to achieve two of the fundamental goals of design research: refinement of a proposed artifact and evaluation of its utility. We outlined how traditional focus group methods can be adapted for these purposes. For the evaluation of an artifact design, exploratory focus groups (EFGs) study the artifact to propose improvements in the design, continuing the cycle of build and evaluate until the artifact is released for field test in the application environment. Then, the field test of the design artifact may employ confirmatory focus groups (CFGs) to establish the utility of the artifact in field use.

612 Volume 26 Article 27

Rigorously designing, planning, selecting participants, conducting, analyzing and reporting the results of the focus groups have unique concerns as adapted to design science research. These concerns are discussed at length in this paper, and we outlined several potential approaches for each step in the focus group process. Additionally, the data generated by this methodology are qualitative, and we described a process to capture, code, analyze, and report these data in a rigorous manner.

Based on our experiences in applying focus groups in a healthcare research project, we were encouraged by the emergence of rich ideas and concepts from the qualitative and interactive focus group technique. Unlike traditional interviews and one-on-one prototyping, focus groups generated interactive conversations among the participants. Different forms of interactive communication, such as joking, teasing, and arguing provided valuable data. The comments and the language used by the target audience regarding the specified artifacts and the discussions that led to a final consensus helped point to design flaws and areas for improvement. Causes of disagreement recognized problem areas with the proposed artifact. The open-ended nature of focus groups allowed for the identification of new artifact ideas, several of which we are pursuing in our ongoing research.

Other rich findings emerged. For example, the intent of the example research study was the evaluation of the proposed data quality metrics, but several other ―user views" of data quality emerged that merit serious consideration and will stimulate further research. The focus group technique allowed the researchers to observe data quality in action in actual decision making. One interesting finding, for example, was that although participants were skeptical of the data in the examples (which for the most part was from an actual healthcare data warehouse), they were not skeptical about their own data (data that they had utilized in their jobs), perhaps because they have very high ownership of those data and believe their data to be of high quality, even though this was unlikely to be true. We also observed that several ―irrational‖ approaches were taken to analyze the data. These included speculating on the reasons for poor data quality without any real evidence (e.g., Hispanics don’t go to the doctor as much as other ethnic groups).

We observed some limitations in the use of focus groups in the evaluation of artifacts. One of the major difficulties is deciding how many focus groups to run. When considering the design of the artifact, the EFGs continuously produced new ideas and suggestions, making the decision to stop and move on to CFG somewhat subjective. Deciding how many CFGs to run is also difficult. In our study the two CFGs found some differing results, thereby highlighting the need for additional CFGs which we did not have the time or resources to run. The choice will most likely be driven by the costs of running focus groups and difficulty in finding additional expert participants.

A very important aspect of conducting focus groups is an effective moderator who is skilled in drawing information from the participants, encourages interaction between participants, and is nonauthoritarian and nonjudgmental [Stewart et al., 2007]. The moderator has to be careful to not bias the results during the focus groups. In our example, the moderator had control of the interface in which the data and metrics were presented to the groups, which certainly led to different results than if the focus-group participants had been able to access them directly at their own workstations. However, the goal was to focus the attention of all the participants at the same point to enable common discussion. Thus, decisions must be made on the balance of control between the moderator and the focus group in the presentation of the artifact design. We noted that the lack of a skilled moderator would be a severe limitation on the use of focus groups for design research. Even differences in skilled moderator approaches could lead to different focus group results. For example, we found that crowd control proved challenging. In one of our CFGs we had a very assertive participant whose opinions resulted in the entire group losing focus. A skilled moderator should understand techniques to refocus conversation.

Certainly there were many other avenues to explore in the use of focus groups in design science research. In every step we outline, there may be diverse approaches that are contingent on the artifact and the application domain. For example, the design of the focus group script will be very different for varying application domains. In our case, we dealt with a decision-making environment in the healthcare industry, but another approach may be needed if the context is significantly different (for example, a supply chain bid recommendation agent). In fact, focus groups are most appropriate where observational methods can be used for evaluation. Also, our application of template analysis worked well for our study, but we undoubtedly can draw from the work of other qualitative researchers for guidance on other ways to analyze the data from transcribed focus groups.

To conclude, we believe that focus groups are a highly relevant and rigorous approach for refining and evaluating design artifacts. A key insight was the adaptation of traditional focus group methods to the goals of design science research projects in the forms of Exploratory Focus Groups and Confirmatory Focus Groups. An exemplar design research project illustrated the effective adaptation and use of focus groups for design research.

Volume 26 Article 27 613

ACKNOWLEDGMENTS

The authors acknowledge research support of resources and use of facilities provided by the James A. Haley Veterans’ Hospital in Tampa, Florida. Dr. Hevner received support from the National Science Foundation during a temporary assignment during the period of this research. The Associate Editor provided many helpful suggestions that improved the content and presentation of the paper.

REFERENCES

Editor’s Note: The following reference list contains hyperlinks to World Wide Web pages. Readers who have the ability to access the Web directly from their word processor or are reading the paper on the Web, can gain direct access to these linked references. Readers are warned, however, that: 1. These links existed as of the date of publication but are not guaranteed to be working thereafter. 2. The contents of Web pages may change over time. Where version information is provided in the References, different versions may not contain the information or the conclusions referenced. 3. The author(s) of the Web pages, not AIS, is (are) responsible for the accuracy of their content. 4. The author(s) of this article, not AIS, is (are) responsible for the accuracy of the URL and version information.

American Community Survey, "Using the Data Quality Measures‖, http://www.census.gov/acs/www/UseData/ sse/ita/ita_def.htm (current November, 2006).

Baker, T. and D.A. Collier (2005) "The Economic Payout Model for Service Guarantees‖, Decision Sciences (36)2, p. 197.

Ballou, D., et al. (1998) "Modeling Information Manufacturing Systems to Determine Information Product Quality‖, Management Science (44)4, pp. 462–484.

Ballou, D.P. and H.L. Pazer (1985) "Modeling Data and Process Quality in Multi-Input, Multi-Output Information Systems", Management Science (31)2, pp. 150–162.

Ballou, D.P. and H.L. Pazer (2003) "Modeling Completeness versus Consistency Tradeoffs in Information Decision Contexts‖, IEEE Transactions on Knowledge and Data Engineering (15)1, pp. 240–243.

Basili, V.R. (1996) "The Role of Experimentation in Software Engineering: Past, Current, and Future" in Proceedings of the 18th International Conference on Software Engineering, Berlin, Germany: IEEE Computer Society.

Baskerville, R. and M.D. Myers (2004) "Special Issue on Action Research in Information Systems: Making Is Research Relevant To Practice-Foreword‖, MIS Quarterly (28)3, p. 329.

Baskerville, R., J. Pries-Heje, and J. Venable (2007) "Soft Design Science Research: Extending the Boundaries of Evaluation in Design Science Research", 2nd International Conference on Design Science Research in Information Systems and Technology (DESRIST 2007), Pasadena, CA.

Benbasat, I. and R. Weber (1996) "Research Commentary: Rethinking 'Diversity' in Information Systems Research‖, Information Systems Research (7)4, p. 389.

Berndt, D.J., A.R. Hevner, and J. Studnicki (2003) "The CATCH Data Warehouse: Support for Community Health Care Decision-Making‖, Decision Support Systems (35)3, p. 367.

Bloor, M., et al. (2001) Focus Groups in Social Research, London.

Cappiello, C., C. Francalanci, and B. Pernici (2003–2004) "Time-Related Factors of Data Quality in Multichannel Information Systems‖, Journal of Management Information Systems (20)3, pp. 71–92.

Chen, P.P.-S. (1976) "The Entity-Relationship Model—Toward a Unified View of Data," ACM Transactions on Database Systems (TODS) (1)1, pp. 9–36.

Chengalur-Smith, I.N., D.P. Ballou, and H.L. Pazer (1999) "The Impact of Data Quality Information on Decision Making: An Exploratory Analysis‖, IEEE Transactions on Knowledge and Data Engineering (11)6, pp. 853– 864.

Cleven, A., P. Gubler, and K.M. Huner (2009) "Design Alternatives for the Evaluation of Design Science Research Artifacts" in Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology, Philadelphia, PA: ACM.

Cole, R., et al. (2005) "Being Proactive: Where Action Research Meets Design Research‖, Twenty-Sixth International Conference on Information Systems, Las Vegas.

614 Volume 26 Article 27

Corbin, J. and A. Strauss (1990) "Grounded Theory Research—Procedures, Canons and Evaluative Criteria‖, Zeitschrift Fur Soziologie (19)6, pp. 418–427.

Debreceny, R., et al. (2003) "New Tools for the Determination of E-Commerce Inhibitors‖, Decision Support Systems (34)2, p. 177.

Galliers, R.D. (1991) "Choosing Appropriate Information Systems Research Approaches: A Revised Taxonomy" in Nissen, H.E., H.K. Klein, and R. Hirschheim (eds.) Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier Science Publishers.

Hevner, A., et al. (2004) "Design Science Research in Information Systems‖, Management Information Systems Quarterly (28)1, pp. 75–105.

Hevner, A.R. (2007) ―A Three Cycle View of Design Science Research‖, Scandinavian Journal of Information Systems (19)2, pp. 87–92.

Jakob, H.I., M. Lars, and N. Peter Axel (2004) "Managing Risk in Software Process Improvement: An Action Research Approach‖, MIS Quarterly (28)3, pp. 395.

Jarvenpaa, S.L. and K.R. Lang (2005) "Managing the Paradoxes of Mobile Technology‖, Information Systems Management (22)4, p. 7.

King, N. (1998) "Template Analysis", in Symon, G. and C. Cassell (eds.) Qualitative Methods and Analysis in Organizational Research, London: Sage Publications.

King, N. (2004) "Using Templates in the Thematic Analysis of Text" in Essential Guide to Qualitative Methods in Organizational Research, London, Thousand Oaks, CA: SAGE Publications, pp. xviii, 388.

Kontio, J., L. Lehtola, and J. Bragge. (2004) "Using the Focus Group Method in Software Engineering: Obtaining Practitioner and User Experiences", in Proceedings of the International Symposium on Empirical Software Engineering (ISESE), IEEE Computer Society, Redondo Beach, CA, August, pp. 271–280.

Krueger, R.A. and M.A. Casey (2000) Focus Groups: A Practical Guide for Applied Research, 3rd edition, Thousand Oaks, CA: Sage Publications.

Kuechler, B. and V. Vaishnavi (2008) "On Theory Development in Design Science Research: Anatomy of a Research Project‖, European Journal of Information Systems (17)5, p. 489.

LeRouge, C. and F. Niederman (2006) "Information Systems and Health Care Xi: Public Health Knowledge Management Architecture Design: A Case Study‖, Communications of the Association for Information Systems (18)9.

Manning, P.K. (1996) "Information Technology in the Police Context: The 'Sailor' Phone‖, Information Systems Research (7)1, p. 52.

Mantei, M.M. and T.J. Teorey (1989) "Incorporating Behavioral Techniques into The Systems Development‖, MIS Quarterly (13)3, p. 257.

Markus, M.L., A. Majchrzak, and L. Gasser (2002) "A Design Theory for Systems that Support Emergent Knowledge Processes‖, MIS Quarterly (26)3, pp. 179–212.

Massey, A.P. and W.A. Wallace (1991) "Focus Groups as a Knowledge Elicitation Technique: An Exploratory Study‖, IEEE Transactions on Knowledge and Data Engineering (3)2, pp. 193–200.

Merton, R.K. and P.L. Kendall (1946) "The Focused Interview‖, The American Journal of Sociology (51)6, pp. 541– 557.

Miles, M.B. and A M. Huberman (1994) Qualitative Data Analysis: An Expanded Sourcebook, 2nd edition, Thousand Oaks: Sage Publications.

Morgan, D.L. (1988) Focus Groups as Qualitative Research. Newbury Park, CA: Sage Publications.

Nielsen, J. (1997) "The Use and Misuse of Focus Groups‖, IEEE Software (14)1, pp. 94–95.

Parssian, A. (2006) "Managerial Decision Support with Knowledge of Accuracy and Completeness of the Relational Aggregate Functions‖, Decision Support Systems (42)3, pp. 1494–1502.

Parssian, A., S. Sarkar, and V.S. Jacob (2004) "Assessing Data Quality for Information Products:Impact for Selection, Projection, and Cartesian Product‖, Management Science (50)7, pp. 967–982.

Pipino, L.L., Y.W. Lee, and R.Y. Wang (2002) "Data Quality Assessment‖, Communications of the ACM (45)4, pp. 211–218.

Volume 26 Article 27 615

Rikard, L., H. Ola, and S. Ulrike (2004) "Design Principles for Competence Management Systems: A Synthesis of an Action Research Study‖, MIS Quarterly (28)3, p. 435.

Shankaranarayan, G. and Y. Cai (2006) "Supporting Data Quality Management in Decision-Making‖, Decision Support Systems (42)1, pp. 302–317.

Shankaranarayan, G., M. Ziad, and R.Y. Wang (2003) "Managing Data Quality in Dynamic Decision Environments: An Information Product Approach‖, Journal of Database Management (14)4.

Shim, J.P., et al. (2002) "Past, Present, and Future of Decision Support Technology‖, Decision Support Systems (33)2, p. 111.

Shin, B. (2003) "An Exploratory Investigation of System Success Factors in Data Warehousing‖, Journal of the Association for Information Systems (4), pp. 141–170.

Smith, H.J., S.J. Milberg, and S.J. Burke (1996) "Information Privacy: Measuring Individuals' Concerns about Organizational Practices‖, MIS Quarterly (20)2, p. 167.

Smith, J.A. (1996) "Beyond the Divide between Cognition and Discourse: Using Interpretative Phenomenological Analysis in Health Psychology‖, Psychology & Health (11)2, pp. 261–271.

Smolander, K., M. Rossi, and S. Purao (2008) "Software Architectures: Blueprint, Literature, Language or Decision?" European Journal of Information Systems (17)6, pp. 575–589.

Stewart, D.W., P.N. Shamdasani, and D.W. Rook (2007) Focus Groups: Theory and Practice, 2nd edition, vol. 20, Newbury Park, CA: Sage Publications.

Torkzadeh, G., J.C.-J. Chang, and G.W. Hansen (2006) "Identifying Issues in Customer Relationship Management at Merck-Medco‖, Decision Support Systems (42)2, p. 1116.

Tremblay, M.C. (2007) Uncertainty in the Information Supply Chain: Integrating Multiple Health Care Data Sources, Doctoral Dissertation, Information Systems/Decision Sciences Department, University of South Florida.

Tremblay, M.C., et al. (2007) "Doing More with More Information: Changing Healthcare Planning with OLAP Tools‖, Decision Support Systems (43)4, pp. 1305–1320.

Tremblay, M.C., D.J. Berndt, and A.R. Hevner (2009) "Measuring Information Volatility in a Health Care Information Supply Chain", in Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology. Philadelphia: ACM.

Tversky, A. and D. Kahneman (1982) "Judgment Under Uncertainty: Heuristics and Biases" in Kahneman, D., P. Slovic, and A. Tversky (eds.) Judgment under Uncertainty: Heuristics and Biases, Cambridge: Cambridge University Press.

Vaishnavi, V. and B. Kuechler (2004) "Design Research in Information Systems‖, http://www.isworld.org/ Researchdesign/drisISworld.htm.

Venable, J. (2006) "A Framework for Design Science Research Activities", Information Resource Management Association Conference, Washington, DC, pp. 24–26.

Wang, R.Y. and D.M. Strong (1996) "Beyond Accuracy: What Data Quality Means to Data Consumers‖, Journal of Management Information Systems (12)4, pp. 5–34.

Wellner, A.S. (2003) "The New Science of Focus Groups‖, American Demographics (25)2, p. 29.

Wixom, B.H. and H.J. Watson (2001) "An Empirical Envestigation of the Factors Affecting Data Warehousing Success‖, MIS Quarterly (25)1, p. 17.

Xia, W. and G. Lee (2005) "Complexity of Information Systems Development Projects: Conceptualization and Measurement Development", Journal of Management Information Systems (22)1, p. 45.

616 Volume 26 Article 27

APPENDIX A: PARTIAL FOCUS GROUP SCRIPT

This is sample script. We have removed several sections in order to give a flavor of the script but limit the length.

Tasks

Arrange furniture for focus group, set up tape recorder & test, set out pencils and questionnaire Open all examples:

· PPT presentation

· Cancer and Smoking

· New Chemo Treatment

· Volatility in Cancer Volumes Trend Greet and chat with people as they come in. Encourage them to fill out questionnaire and consent forms while they wait.

Introduction

―Thanks for filling out the questionnaires and forms, again. Please hand them all in now.‖ We will be showing you a typical OLAP interface. OLAP interfaces (which are usually embedded in Business Intelligence tools) are increasingly being used for decision support. They display the results in a tabular form, and allow flexibility to reconfigure the results depending on the task. Often there are some data quality problems that are hidden from the decision maker. In fact, most decision makers assume that the information is 100 percent accurate. You are being asked to participate because: 1. We want to understand how including information about data quality in a business intelligence tool will affect your

decision-making process. 2. To get your opinion on the way it is presented 3. To get your suggestions on how you would improve it Our goal is to eventually automate these data quality calculations so they match the information shown to you by the OLAP tool. Every time you reconfigure, these calculations will match the information on the screen. Keep in mind this is not the final tool. We are at a ―prototype‖ stage, and we seek to understand how to present this information in a useful and understandable way. The data we are going to look at is real data from the cancer registry. This data comes from several data sources (all Florida hospitals). I am going to take several cases and demonstrate three different ways to tell you about the quality of the data. After each case we will discuss for each case what decision you would make for each scenario. This should take about 1½ hours, after which we will go to dinner (have lunch). Imagine yourself in a position that helps define public policy. For example, making decisions about where in the state you may open a cancer center, or whether a certain ethnicity or race is underserved. These are the three cases we are considering:

1. The amount of data that we could not place because of missing information 2. The variability in the data; for example, if we observing a trend across a time period, how much does it fluctuate? 3. When comparing data, do we know enough about the data to make valid comparisons? The tape recorder is here to allow us to tape the discussion so that we can listen and study the conversation later— ―rigorous qualitative analysis.‖ Everything you say is strictly confidential—your real names will not be used in any report. Please try to speak one at a time so that we can all hear what is being said and so that we’ll be able to follow the conversation on the tape. Let’s begin with introductions. Please tell us your name and a brief description on your current job. I’ll start; then we can go around the room.

Vignette Script

The participants are shown several vignettes - One will highlight how information about missing data is illustrated. - One will illustrate information about variability in the data. - One will illustrate how information on sample size will be shown.

Volume 26 Article 27 617

For each vignette: 1. Ask participants to discuss how this extra information on data quality and on sample size would impact their

decision. 2. Allow conversation to flow—the goal of focus groups is to stimulate conversation from comments of other

participants.

3. Ask participants to make a final decision, allow the use of scrap paper—have them write down their choice(s) before discussing them as a group

Case 2—Hispanic Disparity with Cancer Treatment

The participants are shown the data—a part of the chart shows a percentage for which we do not have information on Hispanic/not Hispanic or whether they received treatment. Start with PowerPoint slide on Hispanic and Cancer Risks and ―Ignotus‖ cancer. We are looking at a particular cancer and I should point out that—that this is a fake cancer, okay? The premise here is that when Hispanics are diagnosed with a certain cancer, this cancer which I called Ignotus (ignotus: unknown, obscure, ignorant, ignoble), they’re less likely to receive chemotherapy than non-Hispanics.

· Explain all the types of unallocated data.

· Start with PowerPoint slide on Hispanic and Cancer Risks and ―Ignotus‖ cancer

· Show all counties together, then break down by counties (Miami Dade is a good example)

1. Is there a disparity based on this data (alone)?

· Allow discussion.

· What might be some of the approaches you might take to consider this data (allow them to talk! Even if they do not have suggestions!)?

2. If not brought up, illustrate how this is more complex since there are unallocated data amounts in several fields. 3. Discuss approaches with PowerPoint (3 slides). 4. Describe worst case scenario.

· How confident would you be using this data?

· How would you explain your answer? 5. Show distributed approach (use PowerPoint slide).

· How confident would you be using this data?

· How would you explain your answer? 6. How do you feel about the three scenario approach (show without nulls, worse, distributed)?

· Does it change your opinion?

· How about your confidence?

· How would you change it?

Wrap up —This questionnaire is filled out individually and then discussed as a group.

1. Do you think receiving data quality information would be beneficial to you? 2. Do you find this information useful? 3. Would data quality information improve the way you do your work? 4. Given data quality information, how many of you would utilize it? 5. Do you think you could figure out how to use the tool? 6. Do you understand what the data quality metrics mean? 7. Do you think data quality information may complicate your work? 8. Do you think the tool may make you waste time on mechanical operations? 9. Take too long to learn or understand?

10. Do you think the decisions you would make would be more/less effective if you had data quality information? Hand out final questionnaire—I realize these are the same questions, but I am interested in individual opinions. Thank everybody for participating—invite everyone to join for lunch/dinner.

618 Volume 26 Article 27

ABOUT THE AUTHORS

Monica Chiarini Tremblay is an Assistant Professor in the Decision Sciences and Information Systems Department in the College of Business Administration at Florida International University. Dr. Tremblay’s areas of research interest are in health informatics, particularly in data warehousing, OLAP tools, data mining, text mining, data quality, and data visualization. She is currently working as a co-investigator in two Veterans Administration, Health Services Research and Development funded studies. She received a PhD in Information Systems and Decision Sciences from the University of South Florida. Dr. Tremblay is a member of ACM, IEEE and AIS. Prior to academia, Dr. Tremblay was a systems analyst at Exxon Company, U.S.A.

Alan Hevner is an Eminent Scholar and Professor in the Information Systems and Decision Sciences Department in the College of Business at the University of South Florida. He holds the Citigroup/Hidden River Chair of Distributed Technology. Dr. Hevner's areas of research interest include information systems development, software engineering, distributed database systems, healthcare information systems and service-oriented computing. He has published more than one hundred and fifty research papers on these topics and has consulted for several Fortune 500 companies. Dr. Hevner has a Ph.D. in Computer Science from Purdue University. He has held faculty positions at the University of Maryland and the University of Minnesota. Dr. Hevner is a member of ACM, IEEE, AIS and INFORMS. From 2006 to 2009, he served as a Program Manager in the Computer and Information Science and Engineering directorate at the U.S. National Science Foundation.

Donald J. Berndt is an Associate Professor in the Information Systems and Decision Sciences Department in the College of Business at the University of South Florida. His research interests include data warehousing, knowledge discovery, data mining, and health informatics. Dr. Berndt received a PhD in Information Systems from the Stern School of Business at New York University.

Copyright © 2010 by the Association for Information Systems. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than the Association for Information Systems must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists requires prior specific permission and/or fee. Request permission to publish from: AIS Administrative Office, P.O. Box 2712 Atlanta, GA, 30301-2712, Attn: Reprints; or via e-mail from ais@aisnet.org.

Volume 26 Article 27

. ISSN: 1529-3181

EDITOR-IN-CHIEF Ilze Zigurs

University of Nebraska at Omaha

AIS SENIOR EDITORIAL BOARD Guy Fitzgerald Vice President Publications Brunel University

Ilze Zigurs Editor, CAIS

University of Nebraska at Omaha

Kalle Lyytinen Editor, JAIS

Case Western Reserve University

Edward A. Stohr Editor-at-Large Stevens Institute of Technology

Blake Ives Editor, Electronic Publications University of Houston

Paul Gray Founding Editor, CAIS Claremont Graduate University

CAIS ADVISORY BOARD Gordon Davis University of Minnesota

Ken Kraemer University of California at Irvine

M. Lynne Markus Bentley University

Richard Mason Southern Methodist University

Jay Nunamaker University of Arizona

Henk Sol University of Groningen

Ralph Sprague University of Hawaii

Hugh J. Watson University of Georgia

CAIS SENIOR EDITORS Steve Alter University of San Francisco

Jane Fedorowicz Bentley University

Jerry Luftman Stevens Institute of Technology

CAIS EDITORIAL BOARD Monica Adya Marquette University

Michel Avital University of Amsterdam

Dinesh Batra Florida International University

Indranil Bose University of Hong Kong

Thomas Case Georgia Southern University

Evan Duggan University of the West Indies

Sy Goodman Georgia Institute of Technology

Mary Granger George Washington University

Ake Gronlund University of Umea

Douglas Havelka Miami University

K.D. Joshi Washington State University

Michel Kalika University of Paris Dauphine

Karlheinz Kautz Copenhagen Business School

Julie Kendall Rutgers University

Nancy Lankton Michigan State University

Claudia Loebbecke University of Cologne

Paul Benjamin Lowry Brigham Young University

Sal March Vanderbilt University

Don McCubbrey University of Denver

Fred Niederman St. Louis University

Shan Ling Pan National University of Singapore

Katia Passerini New Jersey Institute of Technology

Jackie Rees Purdue University

Thompson Teo National University of Singapore

Chelley Vician Michigan Technological University

Padmal Vitharana Syracuse University

Rolf Wigand University of Arkansas, Little Rock

A.B.J.M. (Fons) Wijnhoven University of Twente

Vance Wilson Arizona State University

Peter Wolcott University of Nebraska at Omaha

Yajiong Xue East Carolina University

DEPARTMENTS Global Diffusion of the Internet Editors: Peter Wolcott and Sy Goodman

Information Technology and Systems Editors: Sal March and Dinesh Batra

Papers in French Editor: Michel Kalika

Information Systems and Healthcare Editor: Vance Wilson

ADMINISTRATIVE PERSONNEL James P. Tinsley AIS Executive Director

Vipin Arora CAIS Managing Editor University of Nebraska at Omaha

Sheri Hronek CAIS Publications Editor Hronek Associates, Inc.

Copyediting by S4Carlisle Publishing Services