615 forum

Watkins, R., & Altschuld, J. M. (2014). A final note about improving needs assessment research and practice. In J. W. Altschuld& R. Watkins (Eds.), Needs assessment: Trends and a view toward the future. New Directions for Evaluation, 144, 105–114.

8

A Final Note About Improving Needs Assessment Research and Practice

Ryan Watkins, James W. Altschuld

Abstract

As with any field of inquiry and application, needs assessment is an evolving topic of research and practice. The theories, models, tools, and approaches are not idle targets. In this chapter, the authors explore the emerging trends and issues of the field, suggesting elements of an exciting future and the continuing call for more research that can improve practice. They include an examination of how technologies are expanding the methods employed in assessments and a dis- cussion of how many of the challenges that have nagged at assessors for decades continue to limit our results. The future for needs assessments remains bright, but much work remains to be done in order for needs assessment to mature as a field. © Wiley Periodicals, Inc., and the American Evaluation Association.

Introduction

We would be remiss if we did not close this issue with a few observations about the alignment of research, theory, and practice in needs assessment methods. The landscape of needs assessment has been changing (as de- scribed in Chapter 1), as is likely most evident in the methods being applied today. Epidemiological-type studies, for instance, appear more frequently with the advent of “big data.” At the same time, almost paradoxically, the number of qualitative techniques being applied to probe deeply into situations has been growing as well. Technology is also creating new

NEW DIRECTIONS FOR EVALUATION, no. 144, Winter 2014 © 2014 Wiley Periodicals, Inc., and the American Evaluation Association. Published online in Wiley Online Library (wileyonlinelibrary.com) • DOI: 10.1002/ev.20106 105

106 NEEDS ASSESSMENT: TRENDS AND A VIEW TOWARD THE FUTURE

updates to assessment data collection methods, such as cyber meetings, on- line focus groups, photovoice, rapid participatory appraisal, real-time sur- veys, and other possibilities. In this chapter, we will explore some of these new opportunities and speculate as to what needs assessment could look like methodologically in the future.

Evolving Methods

In 1979, Kamis-Gould called on professionals to do a better job of mea- suring needs. More than a decade later, Witkin (1994) supported that call to action, and in 2010 Altschuld (see Altschuld & White, 2010; White & Altschuld, 2012) again called for more research and investigation into assessment methods. This continuing mandate to improve techniques for assessing needs is a reflection of insufficient research being done and the evolving options available to assessors.

In the 21st century, we have more choices than ever before. As dis- cussed in Chapter 2, the definition of needs, and in turn needs assessment, should guide the choice of methods rather than the reverse of letting the methods we prefer shape the needs we assess. In other words, don’t enter an assessment with the method picked a priori. Instead, define the needs to be assessed and then select appropriate ways for measuring them, which may include many different approaches. Flipping the initial focus from how (methods) to what (needs) is essential for understanding when to use the variety of available procedures and what results you can expect from your work.

In the field of needs assessment, there is a particularly close relation- ship between practice, theory, tools, and techniques. With practice inform- ing theory and vice versa, leaders in the field have frequently had one foot in practice and one in the academy. This has led to a dynamic set of tools for needs assessment, with many borrowed and customized from other fields. For example, needs assessments are frequently integrated in strategic planning initiatives, with tools such as Future Search (Weisbord & Janoff, 2009) and SWOT (Leigh, 2009) employed to assist in specify- ing the “desired” state in terms of results. Likewise, multidisciplinary tools such as surveys have been tailored for needs assessments to utilize dual- and triple-response scales to collect data on needs as gaps (see Altschuld, 2010; Kaufman, Guerra-Lopez, Watkins, & Leigh, 2008).

In the future, we anticipate that we will see more of this, with newer techniques (such as photovoice, crowdsourcing, or “big data” analysis) be- ing adapted for needs assessment purposes. Many of these techniques are due to new technologies making it possible for assessors to fit them into var- ied practical contexts. Technologies (from the spread of smartphone apps and broadband access to Web 2.0 tools and new qualitative data analysis software) provide opportunities to improve what we do from designing and developing to implementing and disseminating results. As examples, Skype

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

A FINAL NOTE ABOUT IMPROVING NEEDS ASSESSMENT RESEARCH AND PRACTICE 107

or Google Hangouts are free tools that assessors can use to engage with stakeholders who may not be available in person (e.g., Google Hangouts al- lows for video calls with up to nine participants). Likewise, SurveyMonkey and related online survey tools now make it faster and easier to collect data from people regardless of location, and SMS text surveys are often used in international development contexts. These illustrate how technology pro- vides more choices; next are major trends that we believe will continue to shape how needs assessments are done.

Big Data and Real-Time Data

The amount of data that is now available for needs assessment has grown rapidly in the last two decades. From detailed data on customer behavior to real-time data on market prices collected by cell phones, there is in- creasingly more information for decisions. Analyzing, interpreting, and us- ing that data are all challenges (and opportunities) for needs assessments. And it is easy to get overwhelmed by the sheer number of options. For in- stance, an HR department can likely provide your assessment with literally hundreds of interesting types of data within most any organization (per- formance appraisals of new hires vs. long-term staff, number of training courses attended by administrative vs. sales staff).

The good news is that all of this affords new perspectives in defining needs (and assets) within an organization, community, or society. Utilizing existing data, you have evidence to support trends that would have been based primarily on assumptions (or limited data) in the past. Likewise, con- ducting state-wide assessments (or assessments across multiple locations of a multinational organization) is now feasible, based upon broad represen- tation of the populations involved.

Preparing to conduct assessments from big data (and/or real-time data) does take special considerations. The capturing and analysis of such data sets often requires numerous meetings with organizational IT staff and the utilization of specialized software. Similarly, it can be challenging to de- termine (before expending resources) what data are most useful for guid- ing your needs assessment. Therefore, you may have to start by analyzing smaller sets of information. You can then use the results to simulate deci- sions, and thereby to ensure that whatever is collected in the end will really be used in decisions before making large investments.

With these considerations in mind, needs assessments can gain valu- able insights from the vast amount of what is now available on a variety of variables within institutions. Take time to explore the options with those who know the data and what they can (and cannot) tell you. Often oppor- tunities to collect real-time data can help you capture current perspectives on needs and assets. This can be especially important in assessments where stakeholders have high turnover, or when external realities (such as market

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

108 NEEDS ASSESSMENT: TRENDS AND A VIEW TOWARD THE FUTURE

prices for food or housing) shift dramatically on a daily, weekly, or monthly basis.

Geographic Information Systems (GIS)

Systems, like Google Maps, allows for inexpensive mapping of needs, assets, or participant groups. These services can offer helpful visual elements to the analysis and interpretation of data you collect in your assessment. A map illustrating the available homeless shelters in an urban area can be more effective in guiding decisions than multiple tables with text describing the locations. When combined with other forms of infographics, GIS can help in communicating the results of the assessment.

Mobile Technologies

While there are many technologies influencing how we collect and ana- lyze data, cell phone technologies (smartphone apps, SMS surveys, pho- tovoice, and videos) are pushing these trends like few others. For example, the integration of cameras into most cell phones greatly reduces the cost of photovoice data collection. Photovoice allows stakeholders to describe the needs and assessments within their institution or community in a visual manner, providing new insights that survey questions on a scale of 1 to 5 never could. Similarly, photos can be used to document performance issues. In India, as a case in point, cell phone pictures are utilized in combination with GIS technology to document issues around trash collection—with ob- servers reporting issues by taking pictures and sending them to a group site where they are mapped in real time. (For this and other examples see http://www.clearsouthasia.org/resources/mobile-based technology.pdf)

SMS (or text message) surveys also offer opportunities to collect real- time data. In Latin America, SMS surveys are used to monitor the price of food in relation to household income (see http://go.worldbank.org/ KV9LUG0G40). Services such as SurveyMonkey afford survey access via smartphone and through iPhone or Android apps; all afford your assess- ment new ways to collect data closer to the time of analysis and subsequent decision making.

Social Networking

Social media tools (Facebook, Twitter, Kickstarter, Google+, We- ShareScience, and others) are new ways to connect with your stakeholder audiences. From raising funds for your needs assessment (Kickstarter) to getting the word out of stakeholders about opportuni- ties to participate (Twitter), social media tools can be used effectively throughout many assessments.

During the needs assessment, it is often worthwhile to monitor so- cial media since it can work equally for you as against you. While positive

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

A FINAL NOTE ABOUT IMPROVING NEEDS ASSESSMENT RESEARCH AND PRACTICE 109

messages might expand opportunities to reach stakeholders, negative mes- sages from groups with alternative agendas might reduce participation in your assessment. Using free services, you can, search for mentions of key words (e.g., the name of your organization + needs assessment) on major social media sites like Facebook, Reddit, and Twitter.

It should be noted that many of these exciting technology opportuni- ties remain primarily accessible by the most affluent in the world. We must not be blind to the digital divide and should counterbalance technology- driven methods with those that are more equitable for the communities and stakeholders with which you are working. If for budget reasons your pri- mary survey tool is online, then you may have to offset it by holding focus groups in less economically developed parts of the community to ensure that its voice is included in the assessment as well.

Globalization

Needs assessments take place within the context of the world around us. Therefore, globalization continues to affect the development of assess- ment methods (Sachs, McArthur, & Schmidt-Traub, 2004). Even for a local community- or organization-specific needs assessment, it is of benefit to be aware of how globalization influences our actions. It has significantly increased cultural awareness in terms of who and how we engage with a variety of stakeholder groups. From considerations on how to word survey items to be culturally sensitive to selecting appropriate focus group facil- itators for different audiences, the changing diversity in workplaces and communities means that we must be keenly aware of our actions. After all, you would not want your assessment results to be skewed because of an un- intentional oversight that led to important stakeholders not participating. Any needs assessment must resonate with the cultures and backgrounds of those who will implement the activities that follow. This is as true in Indiana as it is in India.

With these considerations in mind, new methods have evolved to help assessors. Participatory methods (affected by parallel work in evaluation) are now common in many assessments. Similarly, new models for needs assessment (including the hybrid approach described in Chapter 7 of this issue) also strengthen the inclusion of diverse stakeholder groups into the assessment process.

Globalization has brought with it renewed interest in cross-disciplinary applications of needs assessment. From community psychology, medicine, and business to education and international development, we now find col- laborations on assessments that span traditional boundaries. This is not to say that all assessments should cross fields or disciplines; many do not for very practical reasons. But we are learning from other disciplines to improve our theories, models, and methods. This cross-fertilization often happens in very practical ways (a needs assessment done in a business is shared by

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

110 NEEDS ASSESSMENT: TRENDS AND A VIEW TOWARD THE FUTURE

email with a community psychologist without much fanfare). And these small events lead to new techniques being tried out and new theories being developed.

It is an exciting time to be working in the field of needs assessment. Technology, globalization, and new methods make the identification and prioritization of needs better and more impactful. Nevertheless, these trends can at times obscure, hide, or even amplify some of the methodological challenges that professionals assessing needs have confronted for years. As a result, it is critical that we pay closer attention to the validity of our methods as these trends continue to shape our research and practice.

Concerns

Amid the excitement of 21st century needs assessment, we must highlight a few of the concerns that should guide our future research and practice. One is at the core of needs assessment theory and practice, and is derived specifically from how we define and therefore measure needs. The common discrepancy definition of needs is practical on many levels, but its mea- surement can lead to issues that influence recommendations coming from an assessment. On the surface, needs assessment seems very simple—ask about what should be (desired) and what is (current) status and then com- pare one to the other to see if a need is there. A higher desired score than the one for the current condition is the discrepancy indicator, and the larger the discrepancy, the greater the need. Yet there are many complications. For instance, we commonly subtract two related scales (desired − current = need), each with inherent measurement error in it. Difference scores com- pound error and may lead to unreliable results with limited or specious utility.

If on a survey, for example, you measure the desired results on a scale of 1 to 6, with 1 equaling “not important” and 6 equaling “very important,” you will naturally get some measurement error since individual participants may interpret the scale differently (a 4 for you may be a 5 for me). Add to this the fact that we don’t know the rationale for why an item is rated the way it is by respondents (is it based on accurate knowledge of an area, limited knowledge, or perhaps no knowledge whatsoever?), and you could find that you have a fair amount of uncertainty within the survey.

White and Altschuld (2012) explored the meaning of the “what should be” scores given by respondents and came to the conclusion that they are difficult to interpret without more understanding of what underlies them. When we factor in that in many instances Don’t Know (DK) or Not Appli- cable (NA) choices were not available, it gets harder to interpret data.

Additionally, if you then compare what should be with what is data, where 1 equals “never” and 6 equals “always,” you add to the problem. Subtracting a “frequency” rating (for what is) from an “importance” (for desired) would not be responsible. It is quite challenging to write needs

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

A FINAL NOTE ABOUT IMPROVING NEEDS ASSESSMENT RESEARCH AND PRACTICE 111

Table 8.1. Sampling of Methodological Concerns

Concern What Should Be What Is Comments

Double or multiple scaled forms

Meaningfully worded scales permitting appropriate subtraction between scales

Frequent use of multiple scales without required attention to wording

Can lead to erroneous conclusions about needs; better wording somewhat reduces the problem

Not Applicable (NA) or Don’t Know (DK) responses

Increased usage of such options allows for clearer interpretation of data

Not employed as frequently as they should be

Can allow surveyors to get a more realistic perspective of the respondents and the basis for their replies

Analysis of multiple scaled forms

Weighted needs indices, proportionate reduction in error, means difference analysis, and other approaches

Simple item-by-item mean subtractions with inherent problems predicated on less-than- optimum wording and compounded error

Sophisticated procedures for analysis are warranted in order to properly inform decisions

Ways to portray data from multiple sources

Straightforward and utilitarian formats for information that facilitate decision making

Some discussion of this in the literature and in reports, though not taking into account when results do not agree

Not enough emphasis given on how to array, summarize, and integrate their results

Research about the assessment of needs

Robust study of methods and approaches for looking at needs (i.e., NA as a research area)

Research is done but falls short of what is required to guide practice

With research-based practice, the quality of assessments should be enhanced; practitioners should review literature as they go about assessing needs

Evaluation of the enterprise itself

Needs assessments should be evaluated formatively and in a summative fashion for outcomes

Hardly any evaluations are reported, some for joint needs assessments and asset capacity building endeavors

We must evaluate what we are doing and whether it has produced positive change and growth; we should be learning from what did and did not work

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

112 NEEDS ASSESSMENT: TRENDS AND A VIEW TOWARD THE FUTURE

Figure 8.1. Examples of Data-Collection Tools and Techniques for Each Data Type (Watkins, 2007; Watkins, West Meiers, & Visser,

2012)

assessment questions that measure desired and current results on equivalent scales. We must be aware of (and forthright about) measurement problems and error when we design, implement, report, and make recommendations.

Table 8.1 contains an overview of some of the concerns that commonly occur in needs assessment studies (there could be more entries). The pur- pose here is to sensitize you to the challenges we have as assessors and to the vast possibilities for research into the field.

As a result of these concerns, rarely should you rely on one mea- surement method (e.g., survey or focus group) alone to identify needs. Cross-validate your findings with a mix of hard (externally verifiable) and soft (not externally verifiable) methods, as well as qualitative (words) and quantitative (numbers) methods (see Figure 8.1). And when you report on

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

A FINAL NOTE ABOUT IMPROVING NEEDS ASSESSMENT RESEARCH AND PRACTICE 113

results, align them to data sources so that accurate comparisons and con- trasts can be made. These improvements to how we collect and analyze data will become all the more critical as we use new technologies (discussed above) to broaden our use of data to drive assessment findings and recom- mendations.

Summary

Obviously, many more issues could have been included, such as transla- tion of items and scales across cultures, assessor subjectivity or bias, and response rates. We encourage you to conduct “action research” in your as- sessments or design investigations that could help make collecting data about needs and methods used substantially better. It is only by those of us engaged in assessment activities working to improve our methods that substantial gains in the validity and reliability of assessment can be made in the coming years. And if possible, write up what you have done and sub- mit it for publication. People who have done so have improved the work in needs assessment and we are all grateful for such efforts.

As a field of practice (rather than a field of “pure” research), we do rec- ognize that many of the valuable improvements in methods may not come through traditional academic research and publications. We have therefore created a repository (NA Docs) on the www.NeedsAssessment.org website for sharing tools and techniques (as well as research) that can help all of us improve our methods. These will not be vetted, as peer-reviewed research articles would be, but we hope that by sharing resources in common place where practitioners and researchers can collaborate, everyone can improve in needs assessment practice over time.

References

Altschuld, J. W. (Ed.). (2010). The needs assessment kit. Thousand Oaks, CA: Sage. Altschuld, J. W., & White, J. L. (2010). The needs assessment kit—Analysis and prioriti-

zation (Book 4). Thousand Oaks, CA: Sage. Kamis, E. (1979). A witness for the defense of needs assessment. Evaluation and Program

Planning, 2, 7–12. Kaufman, R., Guerra-Lopez, I., Watkins, R., & Leigh, D. (2008). The assessment book: Ap-

plied strategic thinking and performance improvement through self-assessments. Amherst, MA: HRD Press.

Leigh, D. (2009). SWOT analysis. In R. Watkins & D. Leigh (Eds.), The handbook of im- proving performance in the workplace, Volume 2: Selecting and implementing performance interventions (pp. 115–140). San Francisco, CA: Wiley.

Sachs, J., McArthur, J., & Schmidt-Traub, G (2004). Millennium develop- ment goals needs assessments: Country case studies of Bangladesh, Cambodia, Ghana, Tanzania and Uganda. New York, NY: United Nations. Retrieved from http://www.unmillenniumproject.org/documents/mp ccspaper jan1704.pdf

Watkins, R. (2007). Performance by design: The systematic selection, design, and develop- ment of performance technologies that produce useful results. Amherst, MA: HRD Press.

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

114 NEEDS ASSESSMENT: TRENDS AND A VIEW TOWARD THE FUTURE

Watkins, R., West Meiers, M., & Visser, Y. (2012). A guide to assessing needs: Tools for col- lecting information, making decisions, and achieving development results. Washington, DC: World Bank.

Weisbord, M., & Janoff, S. (2009). Future search. In R. Watkins & D. Leigh (Eds.), The handbook of improving performance in the workplace, Volume 2: Selecting and implement- ing performance interventions (pp. 91–114). San Francisco, CA: Wiley.

White, J., & Altschuld, J. (2012). Understanding the “What Should Be Condition” in needs assessment data. Evaluation and Program Planning, 35(1), 124–132.

Witkin, B. R. (1994). Needs assessment since 1981: The state of practice. Evaluation Practice, 15(1), 17–27.

RYAN WATKINS is an associate professor at George Washington University in Washington, DC, and founder of www.NeedsAssessment.org and www.WeShareScience.com.

JAMES W. ALTSCHULD is professor emeritus in the College of Education and Human Ecology, The Ohio State University.

NEW DIRECTIONS FOR EVALUATION • DOI: 10.1002/ev

Copyright of New Directions for Evaluation is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.