U7D1-68 - **READ AND FOLLOW ALL INSTRUCTIONS AS OUTLINED BELOW.. Research Methodology Peer Review.

CHAPTER

6 Fieldwork Strategies and ObservationMethods The intersection of lines of equal length is an extremely old ideogram found in every part of the world, in prehistoric caves and engraved on rocks. In pre- Columbian America, the sign seems to have been associated with the four points of the compass. Semiotically, the vertical beam sometimes stands for the human arena whereas the transverse beam represents the physical world, so the intersection is where human culture and ecological context interact.

The cross with arms of equal length was common in the Neolithic Age. The so-called Ice Man, who died on the top of an Alpine pass at 3,000 meters about 3200 BCE, had this sign tattooed on one of his legs.

The law of the polarity of meanings of elementary graphs is well illustrated by , which is both a sign

that unites, as in logic, mathematics, and chemistry, and a sign that separates, as in numismatics and cartography; and is positive in many systems, but negative in the French hobo or gypsy system: here they give nothing. (Symbols, 2006)

The symbol also represents exploration, being open to going in any direction. Here, it is our

symbol for fieldwork.

Enter Into the World; Observe and Wonder; Experience and Reflect

And the children said unto Halcolm, “We want to understand the world. Tell us, O Sage, what must we do to know the world?”

“Have you read the works of our great thinkers?”

“Yes, Master, every one of them as we were instructed.”

“And have you practiced diligently your meditations so as to become One with the infinity of the universe?”

“We have, Master, with devotion and discipline.”

“Have you studied the experiments, the surveys, and the mathematical models of the Sciences?”

“Beyond even the examinations, Master, we have studied in the innermost chambers where the experiments and surveys are analyzed, and where the mathematical models are developed and tested.”

“Still you are not satisfied? You would know more?”

“Yes, Master. We want to understand the world.”

“Then, my children, you must go out into the world. Live among the peoples of the world as they live. Learn their language. Participate in their rituals and routines. Taste the world. Smell it. Watch and listen. Touch and be touched. Write down what you see and hear, how they think and how you feel.

“Enter into the world. Observe and wonder. Experience and reflect. To understand a world you must become part of that world while at the same time remaining separate, a part of and apart from.

“Go, then, and return to tell me what you see and hear, what you learn, and what you come to understand.”

—From Halcolm’s Methodological Chronicle

Chapter Preview

You know that you’ve decided to work within theoretical tradition. You know what, if any, applied contribution you want to make. So your theory is informing your practice, and your practice is linked to your theory. You have a design. You know how and why you are purposefully sampling. You know where and from whom you will gather data. You are ready to enter the “field” to collect data—the “work” of qualitative inquiry—thus the designation fieldwork. When you undertake qualitative inquiry, you become a fieldworker. Add it to your resume. Announce your status to friends, family, lovers, and Facebook.

This chapter provides guidance for fieldwork and observation. The next chapter covers in-depth interviewing. Every interview is also an observation. Observation gives rise to interviewing opportunities. The two kinds of data collection involve different, distinct, but complementary skills. There’s a lot to learn to become adept at either observation or interviewing, thus the length of each of these chapters.

Module 43 opens this chapter with reflections on and examples of the power of direct observation.

Module 44 presents an overview of variations in observational methods. Subsequent modules deal with specific issues of scope and method.

Module 45 Variations in Duration of Observations and Site Visits: From Rapid Reconnaissance to Longitudinal Studies Over Years

Module 46 Variations in Observational Focus and Summary of Dimensions Along Which Fieldwork Varies

Module 47 What to Observe: Sensitizing Concepts

Module 48 Integrating What to Observe With How to Observe

Module 49 Unobtrusive Observations and Indicators, and Documents and Archival Fieldwork

The chapter then moves into the nature and stages of fieldwork, beginning with observing yourself and documenting your personal and interpersonal experiences (Module 50) and the data- gathering process (Module 51). That leads to three stages:

Module 52 Stages of Fieldwork: Entry Into the Field

Module 53 Routinization of Fieldwork: The Dynamics of the Second Stage

Module 54 Bringing Fieldwork to a Close

The chapter concludes with reflections on the observer and what is observed (Module 55) and summary guidelines for conducting fieldwork (Module 56). Let’s move into the field.

MODULE

43 The Power of Direct Observation

The Power of Direct Observation

Some things must be observed directly. Bridges must be inspected. The sick must be examined. Births must be attended. Children on playgrounds must be watched. Melting ice caps must be seen and seen again and seen again. Weddings must be witnessed. Testimony at trial must be seen as well as heard. Astronomers observe the cosmos, biologists nature, and social scientists people. Our ancestors observed fire and figured out how to control it. Copernicus observed the movement of the planets and placed the sun at the center of the solar system. Newton observed falling objects and figured out gravity. Darwin observed species variation and theorized about evolution. All scientific knowledge is rooted in observation.

Timely and Important Contemporary Observations

Our knowledge of the modern world depends on trained, skilled, and dedicated observers paying attention to what’s going on, systematically documenting what they see, and reporting what they learn for our common benefit—and scrutiny by other observers. Consider these diverse findings from observational studies and fieldwork.

Source of Escherichia coli outbreak found: German authorities concluded that contaminated sprouts from an organic farm in the country’s north were the most likely cause of one of the world’s worst outbreaks of E. coli. To reach their conclusion, health officials said they relied on an epidemiological study of the pattern of infection among patients, tracking the outbreak along the food chain from hospital beds to restaurants and back to the farm, southeast of Hamburg, at Bienenbüttel. The breakthrough came with the discovery of infected sprouts in a garbage can at the home of two infected patients in Cologne. At least 30 people in Germany died, “unsettling the nation” and throwing European agriculture into “disarray.” “As the outbreak spread, German authorities blamed cucumbers, tomatoes or lettuce imported from Spain and urged people, particularly in northern Germany . . . to avoid the products.” By observing the pattern of the infection as it spread, observers narrowed down the cause epidemiologically to sprouts. To do so, investigators examined 112 people, 19 of whom had been infected with E. coli during a group visit to a single restaurant, examined recipes for the food they had eaten, interviewed chefs, and examined photographs diners had taken of one another with their choices of food on the table (Cowell, 2011, p. A5).

Human testicles subtly but effectively regulate sperm temperature: Observations of temperature variation and control explain why one testicle is usually slightly lower than the other, why the skin of the scrotum sometimes becomes wrinkled, why the testicles retract during sexual arousal, and why blows to the testicles are so excruciatingly painful. Until recent systematic observations, these subtle temperature-regulating features had gone largely unnoticed by doctors and researchers, much less by ordinary people. The cremasteric muscle retracts the testicles to draw them up closer to the body in cold and lowers them when it gets warm. This up-and-down action

occurs on a moment-to-moment basis so that the male body continually optimizes the gonadal climate for creating sperm, sperm storage, and, ultimately, ejaculation (Bering, 2012).

People with power tune out the less powerful: A growing body of recent research shows that people with the most social power pay scant attention to those with little such power. This tuning out has been observed, for instance, with strangers in a mere five-minute get-acquainted session, where the more powerful person shows fewer signals of paying attention, like nodding or laughing. Higher-status people are also more likely to express disregard, through facial expressions, and are more likely to take over the conversation and interrupt or look past the other speaker (Goleman, 2013, p. SR12).

Understanding toddler tantrums: Pediatric researchers dressed toddlers in a special outfit with a high-quality microphone inside. They recorded more than 100 naturally occurring tantrums and analyzed the acoustic patterns. They found that tantrums have two distinct phases. First comes anger, which involves yelling, screaming, and physical aggression, such as throwing things and/or writhing on the floor. This is followed by sadness, including tears, whimpering, and the desire for comfort. The researchers advise parents to end a tantrum by getting the child past the peak of anger as quickly as possible to sadness because sad children seek comfort. The quickest way past the anger is to do nothing (Green, Whitney, & Potegal, 2011).

Climate change in the Great North Woods: University of Minnesota forest researchers have established 100 observation sites in the Boundary Waters Canoe Area, a 1,090,000-acres (4,400 square kilometers) wilderness area within the Superior National Forest in northeastern Minnesota of the United States along the Canadian border. One major finding is the spread of earthworms, invasive creatures that “are re-engineering the forest floor as they go. . . . Each year, the worms can eat a season’s worth of basswood leaves, depriving the forest floor of ‘duff,’ the carpet-like layer of decaying matter that is a critical component of northern American forests. In a healthy forest, the duff keeps tree roots cool, germinates tree seeds and mushrooms, and provides a home for ovenbirds, salamanders and other small creatures.” But below a worm-infested basswood tree, the earth is bare, a circle of hard-packed dirt 30 feet in diameter. Hardwood trees that might fare better as the climate warms—red maple and basswood—can’t take root in the packed dirt. Instead, the worms create ideal conditions for invasive, brushlike buckthorn and garlic mustard. University of Minnesota forester Lee Frelich calls it “global worming,” because they are amplifying the effects of climate change (Marcotty, 2013, pp. A10–A11).

Folk Wisdom About Human Observation

In the fields of observation, chance favors the prepared mind. —Louis Pasteur (1822–1895)

French chemist who discovered the principles of vaccination

Every student who takes an introductory psychology or sociology course learns that human perception is highly selective. When looking at the same scene or object, different people will see different things. What people “see” is highly dependent on their interests, biases, and backgrounds. Our culture shapes what we see; our early-childhood socialization forms how we look at the world; and our value systems tell us how to interpret what passes before our eyes. How, then, can one trust observational data?

In their classic book Evaluating Information: A Guide for Users of Social Science Research, Katzer, Cook, and Crouch (1998) titled their chapter on observation “Seeing Is Not Believing.” They open with an oft-repeated story meant to demonstrate the problem with observational data.

Once at a scientific meeting, a man suddenly rushed into the midst of one of the sessions. Another man with a revolver was chasing him. They scuffled in plain view of the assembled researchers, a shot was fired, and they rushed out. About twenty seconds had elapsed. The chairperson of the session immediately asked all present to write down an account of what they had seen. The observers did not know that the ruckus had been planned, rehearsed, and photographed. Of the forty reports turned in, only one was less than 20-percent mistaken about the principal facts, and most were more than 40-percent mistaken. The event surely drew the undivided attention of the observers, was in full view at close range, and lasted only twenty seconds. But the observers could not observe all that happened. Some readers chuckled because the observers were researchers, but similar experiments have been reported numerous times. They are alike for all kinds of people. (pp. 21–22)

Using this story to cast doubt on all varieties of observational research manifests two fundamental fallacies: (1) these researchers were not trained as social science observers and (2) they were not prepared to make observations at that particular moment. Scientific inquiry using observational methods requires disciplined training, systematic preparation, and readiness.

The fact that a person is equipped with functioning senses does not make that person a skilled observer. The fact that ordinary persons experiencing any particular incident will report different things does not mean that trained and prepared observers cannot report with accuracy, authenticity, and reliability that same incident. Exhibit 6.1 lists the training and attributes necessary for skilled observation.

Training observers can be particularly challenging because so many people think that they are “natural” observers and therefore have little to learn. Training to become a skilled observer is a no less rigorous process than the training necessary to become a skilled survey researcher or statistician. People don’t “naturally” know how to write good survey items or analyze statistics—and people do not “naturally” know how to do systematic research observations. All forms of scientific inquiry require training and practice.

EXHIBIT 6.1 Becoming a Skilled Observer

Training to become a skilled observer includes the following:

1. Learning to pay attention: Seeing what there is to see, and hearing what there is to hear

2. Writing descriptively writing descriptively

3. Acquiring expertise and discipline in recording field notes

4. Knowing how to separate detail from trivia in order to achieve the former without being overwhelmed by the latter

5. Using systematic methods to validate and triangulate observations

6. Reporting the strengths and limitations of one’s own perspective, which requires both self- knowledge and self-disclosure

Careful preparation for entering into fieldwork is as important as disciplined training. Though I have considerable experience doing observational fieldwork, had I been present at the scientific meeting where the shooting scene occurred, my recorded observations might not have been significantly more accurate than those of my less trained colleagues because I would not have been prepared to observe what occurred and, lacking that preparation, would have been seeing things through my ordinary eyes rather than my scientific observer’s eyes.

Preparation has mental, physical, intellectual, and psychological dimensions. Pasteur said, “In the fields of observation, chance favors the prepared mind.” Part of preparing the mind is learning how to concentrate during the observation. Observation, for me, involves enormous energy and concentration. I have to “turn on” that concentration—turn on my scientific eyes and ears, my observational senses. A scientific observer cannot be expected to engage in systematic observation on the spur of the moment any more than a world class boxer can be expected to defend his title spontaneously on a street corner or an Olympic runner can be asked to dash off at record speed because someone suddenly thinks it would be nice to test the runner’s time. Athletes, artists, musicians, dancers, engineers, and scientists require training and mental preparation to do their best. Experiments and simulations that document the inaccuracy of spontaneous observations made by untrained and unprepared observers are no more indicative of the potential quality of observational methods than an amateur community talent show is indicative of what professional performers can do.

Two points are critical, then, in this introductory section. First, the folk wisdom about observation being nothing more than selective perception is true in the ordinary course of participating in day-to-day events. Second, the skilled observer is able to improve the accuracy, authenticity, and reliability of observations through intensive training and systematic preparation. The remainder of this chapter is devoted to helping evaluators and researchers move their observations from the level of ordinary looking to the rigor of systematic seeing.

The Value of Direct Observations

I’m often asked by students,

Isn’t interviewing just as good as observation. Do you really have to go see a program or spend time in a community to understand it? Can’t you find out all you need to know by talking to people without going out and seeing things firsthand.

I reply by relating my experience of evaluating a leadership development program with two colleagues. As part of a formative evaluation aimed at helping staff and funders clarify and improve the program’s design before undertaking a comprehensive follow-up study for a summative evaluation, we went through the program as participant-observers. After completing the six-day leadership retreat, we met to compare experiences. Our very first conclusion was that we would never have understood the program without personally experiencing it. It bore little resemblance to our expectations, what people had told us, or the official program description. Had we designed the follow-up study without having participated in the program, we would have completely missed the mark and asked inappropriate questions. To absorb the program’s language, understand nuances of meaning, appreciate variations in participants’ experiences, capture the importance of what happened outside formal activities (during breaks, over meals, in late-night gatherings and parties), and feel the intensity of the retreat environment—nothing could have substituted for direct experience with the program. Indeed, what we observed and experienced was that participants were

changed as much or more by what happened outside the formal program structure and activities as anything that happened through the planned curriculum and exercises.

Indeed, a major purpose of observation is to see firsthand what is going on rather than simply assume we know. We go into a setting, observe, and describe what we observe.

What does all that description do for us? Perhaps not the only thing, but a very important one, is that it helps us get around conventional thinking. A major obstacle to proper description and analysis of social phenomena is that we think we know most of the answers already. We take a lot for granted, because we are, after all, competent adult members of our society and know what any competent adult knows. We have, as we say, “common sense.” We know, for example, that schools educate children and hospitals cure the sick. “Everyone” knows that. We don’t question what everyone knows; it would be silly. But, since what everyone knows is the object of our study, we must question it or at least suspend judgment about it, go look for ourselves to find out what schools and hospitals do, rather than accepting conventional answers. (Becker, 1998, p. 83)

The Characteristics and Strengths of High-Quality Observations

The first-order purpose of observational data is to describe in depth and detail the setting that was observed, the activities that took place in that setting, the people who participated in those activities, and the meanings of what was observed from the perspectives of those observed. The descriptions should be factual, accurate, and thorough without being cluttered by irrelevant minutiae and trivia. You can judge the quality of observational reports by the extent to which the observation takes you into the situation described and deepens your understanding. In this way, evaluation users, for example, can come to understand program activities and impacts through detailed descriptive information about what has occurred in a program and how the people in the program have reacted to what has occurred.

Naturalistic observations take place in the field. For ethnographers, the field is a cultural setting. For organizational researchers, the field will be an organization. For evaluators, the field is the program being studied. Many phrases are used for talking about field-based observations, including participant observation, fieldwork, qualitative observation, direct observation, or field research. “All these terms refer to the circumstance of being in or around an ongoing social setting for the purpose of making a qualitative analysis of that setting” (Lofland, 1971, p. 93).

In addition to the value of direct, personal contact with and observations of a setting through fieldwork, the inquirer is better able to understand and capture the context within which people interact—for understanding context is essential to a holistic perspective. In a site visit to an employment program, it became clear that the location of the program in a run-down, unsafe area was affecting how participants who came from elsewhere in the city felt about the program and contributed to the high dropout rate.

SIDEBAR

WHAT IS INVOLVED IN PARTICIPANT OBSERVATION: AN EXEMPLAR

The Study of the Boys in White: Student Culture in Medical School

Organizational sociologists Howard Becker, Blanche Geer, Everett] Hughes, and Anselm Strauss set the standard for organizational participant observation in their classic and revered study of student experience at medical school. Here’s an overview of their methodology that provides insight into the commitment and rigor involved in participant observation.

They participated in the daily lives of the students. They did this by attending medical school with the students, “following them from class to laboratory to hospital ward.” In studying the clinical years, they “attached” themselves to a group and “followed the members through the entire day’s activities.” They “went with students to lectures and to the laboratories in which they studied the basic sciences, watched their activities, and engaged in casual conversation with them.” They “followed students to their fraternity houses and sat with them while they discussed their school experiences.” They accompanied students in the hospital, watching them examine patients. They sat in on discussion groups and oral exams, had meals with the students, and took night calls with them. In short, they “went with the students wherever they went in the course of the day.”

Two aspects of our participant observation are important: it was continued and it was total. When we observed a particular group of students, we observed them day after day, more or less continuously, for periods ranging from a week to two months; and we observed the total day’s activities of such a group (so far as possible) rather than simply some segment of them. (Becker, Geer, Hughes, & Strauss, 1961, p. 27)

They did not record every event they observed, commenting that “encyclopedic recording is neither possible nor particularly useful.” They “made it a rule” to dictate a complete account of each day’s observations as soon as possible. The recorded conversations they overheard were a part of “verbatim.” Thus, they not only engaged in informal, conversational interviewing but also conducted formal interviews with a purposeful sample of students and faculty.

Their field notes and interviews yielded some 5,000 single-spaced typed pages. They indexed the field notes and labeled each entry with topical codes. Their conclusions were emergent: “What we ended with in the way of a theoretical position is somewhat different from what we started with” (Becker et al., 1961, pp. 25–32).

Moreover, firsthand experience with a setting and the people in the setting allows an inquirer to be open and discovery oriented and inductive because, by being on-site, the observer has less need to rely on prior conceptualizations of the setting, whether those prior conceptualizations are from written documents or verbal reports. In a visit to a rural African school that was supposed to serve both boys and girls, the first thing that stood out was that there were no girls. Routine monitoring reports submitted to headquarters showed roughly equal numbers of boys and girls. So where were the girls? They were on a waiting list. The school lacked space to serve everyone, so they enrolled everyone, boys and girls, and reported enrollment data to headquarters and the government, but only boys were allowed to actually attend.

Another strength of observational fieldwork is that the inquirer has the opportunity to see things that may routinely escape awareness among the people in the setting. For someone to provide information in an interview, he or she must be aware enough to report the desired information. Because all social systems involve routines, participants in those routines may take them so much

for granted that they cease to be aware of important nuances that are apparent only to an observer who has not become fully immersed in those routines. In an evaluation of early-childhood programs, I was struck immediately by the bare walls in one center. All other centers had walls covered with children’s art, posters of animals, and scenes of children playing in places around the world. I asked the program director about the bare walls. She looked quizzically, like she didn’t understand my question, and then exclaimed, “Oh no, oh no. The walls are bare.” She grabbed a child’s drawing from her desk and taped it on the wall while she explained, “We formed a committee to decorate the walls. They got into a personality conflict and couldn’t agree. Nothing happened. I guess we got busy and just got used to the bare walls. This is so embarrassing.” (Note: Observing and interviewing in the field can affect what is going on, a matter we’ll discuss at some length later in this chapter.)

The participant-observer can also discover things no one else has ever really paid attention to. One of the highlights of the leadership training program we experienced was the final evening banquet at which the staff were “roasted.” For three nights after the training ended, the participants worked to put together a program of jokes, songs, and skits for the banquet. Staff were never around for these preparations, which lasted late into the night, but they had come to count on this culminating event. Month after month for two years each completely new training group had organized a final banquet event to both honor and make fun of staff. Staff assumed that either prior participants passed on this tradition or it was a natural result of the bonding among participants. We learned that neither explanation was true. What actually occurred was that, unknown to program staff, the dining hostess for the hotel where participants stayed initiated the roast. After the second evening’s meal, when the staff routinely departed for a meeting, the hostess would tell the participants what was expected. She even brought out a photo album of past banquets and offered to supply joke books, costumes, music, or whatever. This 60-year-old woman had begun playing what amounted to a major staff role for one of the most important processes in the program—and the staff didn’t know about it. We learned about it by being there.

Yet another value of direct observation is the chance to learn things that people would be unwilling to talk about in an interview. Interviewees may be unwilling to provide information on sensitive topics, especially to strangers. During participant observation of the wilderness leadership program, observing the emergence of romantic encounters raised issues that we would not have been likely to ask about if just doing interviews or surveys for the inquiry.

Ongoing fieldwork offers an opportunity to move beyond the selective perceptions of others. Interviews present the understandings of the people being interviewed. Those understandings constitute important, indeed critical, information. However, it is necessary for the inquirer to keep in mind that interviewees are always reporting perceptions—selective perceptions. A program director told me that one of the strengths of her program was its welcoming environment. And, indeed, whenever she was around, the reception staff were unfailingly friendly and attentive. When she wasn’t around, they were rude, inattentive, and condescending toward participants.

At the same time, an important point to keep in mind is that field observers will also have selective perceptions. By reflecting on the basis for their perceptions and making their own perceptions part of the data—a matter of training, discipline, and self-awareness—observers can arrive at a more comprehensive view of the setting being studied and move beyond their preconceptions.

Finally, getting close to the people in a setting through firsthand experience permits the inquirer to draw on personal knowledge during the interpretation stage of analysis. Reflection and introspection are important parts of field research. The impressions and feelings of the observer

become part of the data to be used in attempting to understand a setting and the people who inhabit it. The observer takes in information and forms impressions that go beyond what can be fully recorded in even the most detailed field notes.

SIDEBAR

OBSERVATION SOCIETY

The recent sudden increase of ethnography can be explained with the hypothesis that we are entering an “observation society,” a society in which observing has become a fundamental activity, and watching and scrutinizing are becoming important cognitive modes. . . .

The clues that we are living in an “observation society” are many. Wherever we go, there is always a television camera ready to film our actions (unbeknownst to us): camera phones and the current fashion for making video recordings of even the most personal and intimate situations and posting them on the Internet; or logging on to webcams pointed at city streets, monuments, landscapes, plants, bird nests, coffee pots, etc., to observe movements, developments, and changes. Then, there is the trend of webcams worn by people so that they can lead us virtually through their everyday lives. These are not minor eccentricities but websites visited by millions of people around the world.

Observing and being observed are two important features of contemporary Western societies. Consequently, there is an increasing demand in various sectors of society—from marketing to security, television to the fashion industry!—for observation and ethnography. All of which suggests that ours is becoming an observation society.

—Giampietro Gobo (2011, p. 25)

Professor of Methodology and Social Research and Evaluation

University of Milan

Because he sees and hears the people he studies in many situations of the kind that normally occur for them, rather than just in an isolated and formal interview, he builds an ever-growing fund of impressions, many of them at the subliminal level, which give him an extensive base for the interpretation and analytic use of any particular datum. This wealth of information and impression sensitizes him to subtleties which might pass unnoticed in an interview and forces him to raise continually new and different questions, which he brings to and tries to answer in succeeding observations. (Becker & Geer, 1970, p. 133)

In the course of fieldwork, the inquirer develops an empathetic understanding: coming to understand a setting and its people not just intellectually but emotionally, what it feels like to be there, in that place, doing those things, with those people. In all of my participant observation experiences, I could look back through my notes and reflect on my experiences and identify when I had begun to feel the experience and understand the feelings of the people I was interacting with. Participant observation is a holistic encounter. All senses come into play. That is the great strength of being present, truly present, in the field.

Exhibit 6.2 summarizes the 10 strengths of direct observational fieldwork just presented.

Going Into the Real World

In a moment, we’ll offer guidance on how to do fieldwork, but to inform that transition and reinforce the importance of direct observation in the real world, let me offer a perspective from the world of children’s stories. Some of the most delightful, entertaining, and suspenseful fables concern tales of kings who discard their royal robes to take on the apparel of peasants so that they can move freely among their people to really understand what is happening in their kingdom. Our modern-day kings and political figures are more likely to take television crews with them when they make excursions among the people. They are unlikely to go out secretly disguised, moving through the streets anonymously, unless they’re up to mischief. It is left, then, to applied researchers and evaluators to play out the fable, to take on the appropriate appearance and mannerisms that will permit easy movement among the people, sometimes secretly, sometimes openly, but always with the purpose of better understanding what the world is really like. They are then able to report those understandings to our modern-day version of kings so that policy wisdom can be enhanced and programmatic decisions enlightened. At least that’s the fantasy. Turning that fantasy into reality involves a number of important decisions about what kind of fieldwork to do. We turn now to those decisions.

EXHIBIT 6.2 Ten Strengths of High-Quality Observations

1. Rich description: Detailed, rich descriptions take readers into the setting observed, providing a vicarious experience and deepened understanding.

2. Contextual sensitivity: Being in a setting allows observation of the context and environment that is likely to affect what happens in the setting.

3. Being open to what emerges: Direct observation supports being open, discovery oriented, and inductive, taking in whatever is there in addition to and beyond any predetermined observational protocol.

4. Seeing the unseen: In the field, the inquirer has the opportunity to see things that may routinely escape awareness among the people in the setting. (The fish doesn’t know it’s swimming in water.)

5. Testing old assumptions and generating new insights: The participant-observer can discover things no one else has ever really paid attention to.

6. Opening up new areas of inquiry: Observations generate questions that can be pursued in interviews to help understand and interpret what has been observed.

7. Delving into sensitive issues: Fieldwork provides an opportunity to learn things that people may be unwilling to bring up in an interview.

8. Getting beyond selective perceptions of others: Direct observation provides an opportunity to move beyond the selective perceptions of others and see for oneself. Interview data alone do not allow the comparison between what is said to a fieldworker and what is directly observed.

9. Getting beyond one’s own selective perceptions: By reflecting on the basis for their perceptions and making their own perceptions part of the data—a matter of training, discipline, and self-awareness—observers can arrive at a more comprehensive view of the setting being studied and move beyond their own preconceptions.

10. Experiencing empathy: Observers come to understand a setting and its people not just intellectually but emotionally, what it feels like to be there, in that place, doing those things, with those people.

©2002 Michael Quinn Patton and Michael Cochran

MODULE

44 Variations in Observational Methods

Observational inquiry explores the world in many ways for varied purposes. Observation for evaluation or action research is different from basic social science research because the purposes, questions, and timelines are typically different. While fieldwork methods originated in anthropology and qualitative sociology, using those methods for evaluation often requires adaptation. The sections that follow will discuss both the similarities and differences between evaluation field methods and research field methods and the options that exist for both.

Variations in Observer Involvement: Participant, Onlooker, or Both?

The first and most fundamental distinction that differentiates observational strategies concerns the extent to which the observer will be a participant in the setting being studied. This involves more than a simple choice between participation and nonparticipation. The extent of participation is a continuum that varies from complete immersion in the setting as full participant to complete separation from the setting as spectator, with a great deal of variation along the continuum between these two end points.

Nor is it simply a matter of deciding at the beginning how much the observer will participate. The extent of participation can change over time. In some cases, the observer may begin the study as an onlooker and gradually become a participant as fieldwork progresses. The opposite can also occur. An evaluator might begin as a complete participant to experience what it is like to be initially immersed in the program and then gradually withdraw participation over the period of the study until finally taking the role of occasional observer from an onlooker stance.

Full participant observation constitutes an omnibus field strategy in that it “simultaneously combines document analysis, interviewing of respondents and informants, direct participation and observation, and introspection” (Denzin, 1978b, p. 183). If, on the other hand, an evaluator observes a program as an onlooker, the processes of observation can be separated from interviewing. In participant observation, however, no such separation exists. Typically, anthropological fieldworkers combine in their field notes data from personal, eye witness observation with information gained from informal, natural interviews and informants’ descriptions (Pelto & Pelto, 1978, p. 5). Thus, the participant-observer employs multiple and overlapping data collection strategies: being fully engaged in experiencing the setting (participation) while at the same time observing and talking with others about whatever is happening.

In leadership programs I have evaluated through participant observation, I have been a full participant in all exercises and program activities, using the field of evaluation as my leadership arena (since all participants had to have an arena of leadership as their focus). As did the other participants, I developed close relationships with some people as the week progressed, sharing meals and conversing late into the night. I sometimes took detailed notes during activities if the activity permitted (e.g., group discussion), while at other times, I waited until later to record notes (e.g., after meals). If a situation suddenly became emotional, for example, during a small group encounter, I would cease to take notes so as to be fully present as well as to keep my note taking

from becoming a distraction. Unlike other participants, I sat in on staff meetings and knew how staff viewed what was going on. Much of the time, I was fully immersed in the program experience as a participant, but I was also always aware of my additional role as evaluation observer.

The extent to which it is possible for an observer to become a participant in a setting will depend partly on the nature of the setting. In human service and education programs that serve children, an observer cannot participate as a child but may be able to participate as a volunteer, parent, or staff member in such a way as to develop the perspective of an insider in one of those adult roles. Gender can create barriers to participant observation. Males can’t be participants in female-only settings (e.g., a battered women’s shelter). Females doing fieldwork in nonliterate cultures may not be permitted access to male-only councils and ceremonies. Programs that serve special populations may also involve natural limitations on the extent to which the observer can become a full participant. For example, a researcher who is not chemically dependent will not be able to become a full participant, physically and psychologically, in a chemical dependency program, even though it may be possible to participate in the program as a client. Such participation in a treatment program can lead to important insights and understanding about what it is like to be in the program; however, the observer must avoid the delusion that participation has been complete. This point is illustrated by an exchange between an inmate and an observer who was doing participant observation in a prison.

Inmate: What are you in here for, man?

Observer: I’m here for a while to find out what it’s like to be in prison.

Inmate: What do you mean—“find out what it’s like”?

Observer: I’m here so that I can experience prison from the inside instead of just studying what it’s like from out there.

Inmate: You got to be jerkin’ me off, man. “Experience from the inside” . . . ? Shit, man, you can go home when you decide you’ve had enough can’t you?

Observer: Yeah.

Inmate: Then you ain’t never gonna know what it’s like from the inside.

Social, cultural, political, and interpersonal factors can limit the nature and degree of participation in participant observation. For example, if people in a setting all know each other intimately, they may object to an outsider trying to become part of their close circle. Where marked social class differences exist between a sociologist and people in a neighborhood, access will be more difficult; likewise, when, as is often the case, an evaluator is well educated and middle class while welfare program clients are economically disadvantaged and poorly educated, the participants in the program may object to any ruse of “full” participant observation. Program staff will sometimes object to the additional burden of including an evaluator in a program where resources are limited and an additional participant would unbalance staff–client ratios. Thus, in evaluation, the extent to which full participation is possible and desirable will depend on the precise nature of the program, the political context, and the nature of the evaluation questions being asked. Adult training programs, for example, may permit fairly easy access for full participation by evaluators. Offender treatment programs are much less likely to be open to participant observation as an evaluation method. Evaluators must therefore be flexible, sensitive, and adaptive in negotiating the precise degree of participation that is appropriate in any particular observational study, especially where reporting timelines are constrained such that entry into the setting must be

accomplished relatively quickly. Social scientists who can take a long time to become integrated into the setting under study have more options for fuller participant observation.

As these examples illustrate, full and complete participation in a setting, what is sometimes called “going native,” is fairly rare, especially in program evaluation. The degree of participation and nature of observations vary along a wide continuum of possibilities. The ideal is to design and negotiate that degree of participation that will yield the most meaningful findings given the characteristics of the setting, the people in the setting, and the sociopolitical context of the program. In summary, the purpose, scope, length, and setting of the study will dictate the range and types of participant observation that are possible.

One final caution: The observer’s plans and intentions regarding the degree of involvement to be experienced may not be the way things actually turn out. Lang and Lang (1960) reported that two scientific participant-observers who were studying audience behavior at a Billy Graham evangelical crusade made their “decision for Christ” and left their observer posts to walk down the aisle and join Reverend Graham’s campaign. Such are the occupational hazards (or benefits, depending on your perspective) of real-world fieldwork.

Insider and Outsider Perspectives: Emic Versus Etic Approaches

People who are insiders to a setting being studied often have a view of the setting and any findings about it quite different from that of the outside researchers who are conducting the study.

—Bartunek and Louis (1996, p. 1) Insider/Outsider Team Research

Ethnosemanticist Kenneth Pike (1954) coined the terms emic and etic to distinguish classification systems reported by anthropologists based on (a) the language and categories used by the people in the culture studied, an emic approach, in contrast to (b) categories created by anthropologists based on their analysis of important cultural distinctions, an etic approach. Leading anthropologists like Franz Boas and Edward Sapir argued that the only meaningful distinctions were those made by people within a culture, that is, the emic perspective. However, as anthropologists turned to more comparative studies, engaging in cross-cultural analyses, distinctions that cut across cultures were based on the anthropologist’s analytical perspective, that is, an etic perspective. The etic approach involved “standing far enough away from or outside of a particular culture to see its separate events, primarily in relation to their similarities and their differences, as compared to events in other cultures” (Pike, 1954, p. 10). For some years, a debate raged in anthropology about the relative merits of emic versus etic perspectives (Headland, Pike, & Harris, 1990; Pelto & Pelto, 1978, pp. 55–60), but as often happens over time, both approaches came to be understood as valuable, though each contributes something different. Nevertheless, tension between these perspectives remains:

Today, despite or perhaps because of the new recognition of cultural diversity, the tension between universalistic and relativistic values remains an unresolved conundrum for the Western ethnographer. In practice, it becomes this question: By which values are observations to be guided? The choices seem to

be either the values of the ethnographer or the values of the observed—that is, in modern parlance, either the etic or the emic. . . . Herein lies a deeper and more fundamental problem: How it is possible to understand the other when the other’s values are not one’s own? This problem arises to plague ethnography at a time when Western Christian values are no longer a surety of truth and, hence, no longer the benchmark from which self-confidently valid observations can be made. (Vidich & Lyman, 2000, p. 41)

Methodologically, the challenge is to do justice to both perspectives during and after fieldwork and to be clear with one’s self and one’s audience how this tension is managed.

A participant-observer shares as intimately as possible in the life and activities of the setting under study to develop an insider’s view of what is happening, the emic perspective. This means that the participant-observer not only sees what is happening but also feels what it is like to be a part of the setting or program. Anthropologist Hortense Powdermaker (1966) has described the basic assumption undergirding participant observation as follows:

To understand a society, the anthropologist has traditionally immersed himself in it, learning, as far as possible, to think, see, feel and sometimes act as a member of its culture and at the same time as a trained anthropologist from another culture. (p. 9)

SIDEBAR

CAPTURING THE INSIDERS’ PERSPECTIVE IN THEIR OWN WORDS

When social scientists study something—a community, an organization, an ethnic group—they are never the first people to have arrived on the scene, never newcomers to an unpeopled landscape who can name its features as they like. Every topic they write about makes up part of the experience of many other kinds of people, all of whom have their own ways of talking about it, their own distinctive words for the objects and events and people involved in that area of social life. Those words are never neutral, objective signifiers. Rather, they express the perspective and situation of the people who use them. The natives are already there, were there all along, and everything in that terrain has a name, more likely many names.

If we choose to name what we study with words the people involved already use, we acquire, with the words, the attitudes and perspectives the words imply. Since many kinds of people are involved in any social activity, choosing words from any of their vocabularies commits us to one or another of the perspectives in use by one or another of the groups already on the scene. Those perspectives invariably take much for granted, making assumptions about what social scientists might better treat as problematic. . . .

What we call the things we study has consequences [italics added].

—Howard S. Beck (2007, p. 224)

Telling About Society

Experiencing the setting or program as an insider accentuates the participant part of participant observation. At the same time, the inquirer remains aware of being an outsider. The challenge is to combine participation and observation so as to become capable of understanding the setting as an insider while describing it to and for outsiders.

Obtaining something of the understanding of an insider is, for most researchers, only a first step. They expect, in time, to become capable of thinking and acting within the perspective of two quite different groups, the one in which they were reared and—to some degree—the one they are studying. They will also, at times, be able to assume a mental position peripheral to both, a position from which they will be able to perceive and, hopefully, describe those relationships, systems, and patterns of which an inextricably involved insider is not likely to be consciously aware. For what the social scientist realizes is that while the outsider simply does not know the meanings or the patterns, the insider is so immersed that he may be oblivious to the fact that patterns exist. . . . What field workers eventually produce out of the tension developed by this ability to shift their point of view depends upon their sophistication, ability, and training. Their task, in any case, is to realize what they have experienced and learned and to communicate this in terms that will illumine. (Wax, 1971, p. 3)

Who Conducts the Inquiry? Solo and Team Versus Participatory and Collaborative Approaches

The ultimate “insider” perspective comes from involving the insiders as coresearchers through collaborative or participatory research. Collaborative forms of fieldwork, participatory action research, indigenous peoples’ ethical research principles, feminist research, and empowerment evaluation have become sufficiently important and widespread to make degree of collaboration a dimension of design choice in qualitative fieldwork. Participatory action research has a long and distinguished history (Kemmis & McTagery, 2000; Lincoln, Lynham, & Guba, 2011; Whyte, 1989). Collaborative principles of feminist inquiry include participatory processes that support equity and mutuality (Olesen, 2000; Podems, 2010, 2013). Guidelines for ethical research in indigenous studies call for collaboration between researchers and the indigenous people involved in a setting under observation: “Indigenous people have the right to full participation appropriate to their skills and experiences in research projects and processes” (Australian Institute of Aboriginal and Torres Strait Islander Studies, 2011, p. 10). In evaluation, participatory and collaborative approaches are widely used (Cousins & Chouinard, 2012; Cousins & Earl, 1995; Cousins & Whitmore, 2007; King & Stevahn, 2013). Empowerment evaluation, often using qualitative methods (Fetterman, 2000; Fetterman & Wandersman, 2005), involves the use of evaluation concepts and techniques to foster self-determination and help people help themselves by learning to study and report on their own issues and concerns.

What these approaches have in common is a style of inquiry in which the researcher or evaluator becomes a facilitator, collaborator, coach, and teacher in support of those engaging in the inquiry. While the findings from such participatory processes may be useful, a supplementary agenda is often to increase participants’ sense of being in control of, deliberative about, and reflective on their own lives and situations. Chapter 4 discussed these approaches as examples of how qualitative inquiry can be applied in support of organizational or program development and community change.

SIDEBAR

FULLY PARTICIPATORY EVALUATION

Participant-directed evaluation holds the potential to engage individuals in meaningful studies of issues in their organization or community while simultaneously teaching them evaluative knowledge and skills. Consider, for example, a group of American Indian teenagers who, with the

support of a native evaluator, conducted an evaluation of the perceived outcomes of their reservation’s health programming. The evaluator was responsible for teaching the teenagers how to develop an interview protocol, how to conduct interviews, and how to record and analyze qualitative data. The teenagers then interviewed every one of the community’s elders. After analyzing the data, the young people made a formal presentation to the tribal council. The council received good information about the elders’ perspectives; the youth learned the skills of an evaluator (King & Stevahn, 2013, p. 32).

Degrees of collaboration and coresearcher participation vary along a continuum from the traditional solo fieldworker operating alone, or teams of professional researchers working together, sometimes in the same site, often in a multisite design, in contrast to completely shared processes in which “coresearchers,” that is, people in the setting studied, own the inquiry from design through data collection to analysis and reporting. Along the middle of the continuum are various degrees of partial and periodic (as opposed to continuous) collaboration.

Overt Versus Covert Observations

A traditional concern about the validity and reliability of observational data has been the effects of the observer on what is observed. People may behave quite differently when they know they are being observed versus how they behave naturally when they don’t think they’re being observed. Thus, the argument goes, covert observations are more likely to capture what is really happening than are overt observations where the people in the setting are aware they are being studied.

Researchers have expressed a range of opinions concerning the ethics and morality of conducting covert research, what Mitchell (1993) called “the debate over secrecy” (p. 23). Historically, one end of the continuum is represented by Edward Shils (1959), who absolutely opposed all forms of covert research including “any observations of private behavior, however technically feasible, without the explicit and fully informed permission of the person to be observed.” He argued that there should be full disclosure of the purpose of any research project and that even participant observation is “morally obnoxious . . . manipulation” unless the observer makes explicit his or her research questions at the very beginning of the observation (Shils, 1959; quoted in Webb et al., 1966, p. vi).

At the other end of the continuum is the “investigative social research” of Jack Douglas (1976). Douglas argued that conventional anthropological field methods have been based on a consensus view of society that views people as basically cooperative, helpful, and willing to have their points of view understood and shared with the rest of the world. In contrast, Douglas adopted a conflict paradigm of society that led him to believe that any and all covert methods of research should be considered acceptable options in a search for truth.

The investigative paradigm is based on the assumption that profound conflicts of interest, values, feelings, and actions pervade social life. It is taken for granted that many of the people one deals with, perhaps all people to some extent, have good reason to hide from others what they are doing and even to lie to them. Instead of trusting people and expecting trust in return, one suspects others and expects others to suspect him. Conflict is the reality of life; suspicion is the guiding principle. . . . It’s a war of all and no one gives anyone anything for nothing, especially truth. . . .

All competent adults are assumed to know that there are at least four major problems lying in the way of getting at social reality by asking people what is going on and that these problems must be dealt with if

one is to avoid being taken in, duped, deceived, used, put on, fooled, suckered, made the patsy, left holding the bag, fronted out, and so on. These four problems are (1) misinformation, (2) evasions, (3) lies, and (4) fronts. (Douglas, 1976, pp. 55, 57)

Just as degree of participation in fieldwork turns out to be a continuum of variations rather than an all or none proposition, so too is the question of how explicit to be about the purpose of fieldwork. The extent to which participants in a program under study are informed that they are being observed and are told the purpose of the research has varied historically from full disclosure to no disclosure, with a great deal of variation along the middle of this continuum (Junker, 1960). Discipline-based ethics statements (e.g., American Psychological Association, American Sociological Association) now generally condemn deceitful and covert research. Likewise, institutional review board (IRB) procedures for the protection of human subjects have severely constrained such methods. They now refuse to approve protocols in which research participants are deceived about the purpose of a study, as was commonly done in early psychological research. One of the more infamous examples was Stanley Milgram’s New Haven experiments aimed at studying whether ordinary people would follow the orders of someone in authority by having these ordinary citizens administer what they were told were behavior modification electric shocks to help students learn, shocks that appeared to the unsuspecting citizens to go as high as 450 volts despite the screams and protests heard from supposed students on the other side of a wall. The real purpose of the study, participants later learned, was to replicate Nazi prison guard behavior among ordinary American citizens (Milgram, 1974).

IRBs also refuse to approve research in which people are observed and studied without their knowledge or consent, as in the infamous Tuskegee Experiment. For 40 years, physicians and medical researchers, under the auspices of the U.S. Public Health Service, studied untreated syphilis among black men in and around the county seat of Tuskegee, Alabama, without the informed consent of the men studied, men whose syphilis went untreated so that the progress of the disease could be documented (Jones, 1993). Other stories of abuse and neglect by researchers doing covert studies abound. In the late 1940s and early 1950s, schoolboys at the Walter E. Fernald State School in Massachusetts were routinely served breakfast cereal doused with radioactive isotopes, without permission of the boys or their guardians, for the dissertation of a doctoral student in nutritional biochemistry. In the 1960s, the U.S. Army secretly sprayed a potentially hazardous chemical from downtown Minneapolis rooftops onto unsuspecting citizens to find out how toxic materials might disperse during biological warfare. Native American children on the Standing Rock Sioux Reservation in the Dakotas were used to test an unapproved and experimental hepatitis A vaccine without the knowledge or approval of their parents. In the 1960s and 1970s, scientists tested skin treatments and drugs on prisoners in a Philadelphia County jail without informing them of potential dangers. Unfortunately, the stories of abuses and deceits make for a mountain of cases of treating those researched as subjects to be manipulated in service to the good (or might one say god) of science.

Doctoral students frustrated by having their fieldwork delayed while they await IRB approval need to remember that they are paying for the sins of their research forebears, for whom deception and covert observations were standard ways of doing their work. Those most subject to abuse were often the most vulnerable in society—children, the poor, people of color, the sick, people with little education, women and men incarcerated in prisons and asylums, and children in orphanages or state correctional schools. Anthropological research was commissioned and used by colonial administrators to maintain control over indigenous peoples. Protection of human subjects’ procedures is now an affirmation of our commitment to treat all people with respect. And that is as it should be. But the necessity for such procedures comes out of a past littered with scientific

horrors, for which those of us engaging in research today may still owe penance. At any rate, we need to lean over backward to be sure that such history is truly behind us—and that means being ever vigilant in fully informing and protecting the people who honor us by agreeing to participate in our research, whether they be homeless mothers (Connolly, 2000) or corporate executives (Collins, 2001a).

However, not all research and evaluation falls under IRB review, so the issue of what type and how much disclosure to make remains a matter of debate, especially where the inquiry seeks to expose the inner workings of cults and extremist groups or those whose power affects the public welfare, for example, corporations, labor union boards, political parties, and other groups with wealth and/or power. For example, Maurice Punch (1985, 1989, 1997), formerly of the Nijenrode Business School in the Netherlands, has written about the challenges of doing ethnographic studies of corruption in both private and public sector organizations, notably the police.

One classic form of deception in fieldwork involves pretending to share values and beliefs in order to become part of the group being studied. Sociologist Richard Leo carefully disguised his liberal political and social views, instead feigning conservative beliefs, to build trust with police and thereby gain admission to interrogation rooms (Allen, 1997, p. 32). Sociologist Leon Festinger and colleagues (1956) infiltrated a doomsday cult by lying about his profession and pretending to believe in the cult’s prophecies. Sociologist Laud Humphreys (1970) pretended to be gay to gather data for his dissertation on homosexual encounters in public parks. Anthropologist Carolyn Ellis (1986) pretended to be just visiting friends when she studied a Chesapeake Bay fishing culture. Her negative portrayals made their way back to the local people, many of whom were infuriated. She later expressed remorse about her deceptions (Allen, 1997).

Some covert research is conducted by engaging with a system to find out how it operates in practice (compared with how those in control say it operates). This is especially possible in public settings open to anyone (as opposed to inside organizations or groups). Here are some classic examples:

• To study the availability of condoms, trained observers can document how easy it is for teenagers to purchase condoms from pharmacies. Observers would also record where condoms are located and displayed in the store.

• To study compliance with laws that prohibit discrimination in housing, interracial couples can attempt to rent apartments advertised as available.

• To study hospital emergency room procedures, observers can document what medical issues and what types of patients get priority treatment.

• To study hand-washing behaviors in a public restroom, those being observed are not aware they are being observed and are not identified in the covert research (Hays & Singh, 2012, p. 226).

Evaluation Issues and Applications

In traditional scholarly fieldwork, the decision about the extent to which observations would be covert was made by researchers balancing the search for truth against their sense of professional ethics. In program evaluation research, the information users for whom the evaluation is done have a stake in what kind of methods are used, so the evaluator alone cannot decide the extent to which observations and evaluation purposes will be fully disclosed. Rather, the complexities of program evaluation mean that there are several levels at which decisions about the covert–overt nature of evaluation observations must be made. Sometimes only the funders of the program or of the

evaluation know the full extent and purpose of observations. On occasion, program staff may be informed that evaluators will be participating in the program, but clients will not be so informed. In other cases, an evaluator may reveal the purpose and nature of program participation to fellow program participants and ask for their cooperation in keeping the evaluation secret from program staff. On still other occasions, a variety of people intimately associated with the program may be informed of the evaluation, but public officials who are less closely associated with the program may be kept “in the dark” about the fact that observations are under way. Sometimes the situation becomes so complex that the evaluator may lose track of who knows and who doesn’t know, and of course, there are the classic situations where everyone involved knows that a study is being done and who the evaluator is—but the evaluator doesn’t know that everyone else knows.

A Canadian evaluation studied the performance of immigration officers whose job was to identify individuals who should not be admitted into Canada without special documentation. The evaluation included sending actors to border posts and airports to play the roles of American travelers whose situation should call for an immigration interview.

Everything went fine until word got out. We [found] that immigration officers were two to three times more effective after knowing that they were observed than before. That allowed the evaluation to identify some of the weaknesses in the system—which were due not so much to information systems as they were to human resources. (Gauthier, 2013, p. 1)

In undertaking participant observation of the community leadership program mentioned earlier, my two evaluation colleagues and I agreed with the staff to downplay our evaluation roles and describe ourselves as “educational researchers” interested in studying the program. We didn’t want the participants to think that they were being evaluated and therefore worry about our judgments. Our focus was on evaluating the program, not the participants, but to avoid increasing participant stress, we simply attempted to finesse our evaluation role by calling ourselves “educational researchers.”

Our careful agreement on and rehearsal of this point with the staff fell apart during introductions (at the start of the six-day retreat) when the program director proceeded to tell the participants—for 10 minutes—that we were just participants and they didn’t have to worry about our evaluating them. The longer he went on reassuring the group that they didn’t have to worry about us, the more worried they got. Sensing that they were worried, he increased the intensity of his reassurances. While we continued to refer to ourselves as “educational researchers,” the participants thereafter referred to us as evaluators. It took a day and a half to recover our full participating roles as the participants got to know us on a personal level as individuals.

Trying to protect the participants (and the evaluation) had backfired and made our entry into the group even more difficult than it otherwise would have been. However, this experience sensitized us to what we subsequently observed to be a pattern in many program situations and activities throughout the week and became a major finding of the evaluation: staff overprotection of and condescending attitudes toward participants.

Based on this and other evaluation experiences, I recommend full and complete disclosure. People are seldom really deceived or reassured by false or partial explanations—at least not for long. Trying to run a ruse or scam is simply too risky and adds to evaluator stress while holding the possibility of undermining the evaluation if (and usually when) the ruse becomes known. Program participants, over time, will tend to judge evaluators first and foremost as people, not as evaluators.

SIDEBAR

ETHICAL CHALLENGES IN QUALITATIVE RESEARCH

The relationship between informed consent and covert research serves as the ultimate illustration of ethical dilemmas in data collection. On the one hand, the distinction between overt and covert research is often unclear. . . . On the other hand, to make written informed consent mandatory would mean the end of much “street-style” ethnography. . . .

As much as we appreciate proper practice on consent, it does raise issues about how and if we may be able to study phenomena such as crime, elites and private groups. If we accept that what we find out is a result of how we find it, then this is a good illustration of the close but at times not unproblematic relationship between research ethics and knowledge production. A great moral responsibility rests with decision makers here.

There are no standard answers to these dilemmas, and this was the very argument why they have been found too important to be left to researchers alone. We need to be prepared for all these challenges, which demand that we put them on the agenda from the very start of our projects and ask ourselves how they relate to our own particular project.

But, what should we do? A good piece of advice is always to invite experienced researchers with particular knowledge in research ethics and in your field to discuss matters with you.

—Anne Ryen (2011, pp. 418–419)

Ethics and Qualitative Research

The nature of the questions being studied in any particular evaluation will have a primary effect on the decision about who will be told that an evaluation is under way. In formative evaluations where staff members and/or program participants are anxious to have information that will help them improve their program, the quality of the data gathered may be enhanced by overtly soliciting the cooperation of everyone associated with the program. Indeed, the ultimate acceptance and usefulness of formative information may depend on such prior disclosure and agreement that a formative evaluation is appropriate. On the other hand, where program funders have reason to believe that a program is corrupt, abusive, incompetently administered, and/or highly negative in impact on clients, it may be decided that an external, covert evaluation is necessary to find out what is really happening in the program. Under such conditions, my preference for full disclosure may be neither prudent not practical. On the other hand, Whyte has argued that “in a community setting, maintaining a covert role is generally out of the question” (Whyte, 1984, p. 31).

Confidentiality

Finally, there is the related issue of confidentiality. Those who advocate covert research usually do so with the condition that reports conceal names, locations, and other identifying information so that the people who have been observed will be protected from harm or punitive action. Because the basic researcher is interested in truth rather than action, it is easier to protect the identity of informants or study settings when doing scholarly research. In evaluation research, however, while the identity of who said what may be possible to keep secret, it is seldom possible to conceal the identity of a program, and doing so may undermine the utility of the findings.

Evaluators and decision makers will have to resolve these issues in each case in accordance with their own consciences, evaluation purposes, political realities, and ethical sensitivities. We’ll return

to discussion of ethics in qualitative research in the final chapter. Before moving on, though, let’s consider the special ethical issues raised by research and evaluation involving Internet communities.

Ethical Issues in Qualitative Research on Internet Communities

Research on the Internet raises new opportunities to engage in covert research—and new ethical challenges. It is easy to join online groups without disclosing one’s true identity or research purpose. That doesn’t make it ethical; it just makes it easy. Exhibit 6.3 presents ethical guidance for qualitative research on Internet communities.

EXHIBIT 6.3 Seven Ethical Issues in Qualitative Research on Internet Communities

1. Intrusiveness. Different ethical issues arise in passive analysis of Internet postings versus active participant observation by researcher involvement in the community through posting communications and interacting. Under what circumstances and for what purposes, if any, is covert participation in an Internet community ethical?

2. Respect for privacy. What is the perceived level of privacy of the Internet community being studied? Is it a closed group requiring registration or open to anyone? What are the group’s norms and expectations regarding degree of privacy?

3. Sensitivity to vulnerability. How vulnerable is the Internet community? Internet support groups for victims of sexual abuse, survivors of cancer, or people living with AIDS patients may feel highly vulnerable, so extra sensitivity and precautions are necessary.

4. Potential harm. Consider how research or evaluation, and subsequent publication of findings, might harm individuals or the Internet community as a whole.

5. Internet informed consent. Consider whether informed consent is required or can be waived and, if required, how it will be obtained.

6. Confidentiality. How can the identity of participants be protected, especially when verbatim quotes are used in the inquiry?

7. Intellectual property rights. Some participants in an Internet community may prefer publicity to anonymity or, for other reasons, want to maintain control over their postings and communications.

—Adapted from Eysenbach and Till (2001) and Berry (2004)

Researchers lurking on Internet communities may be regarded with suspicion, and they may interfere with the functioning of the community. Consider this reaction by a participant in a health support group who quit when she learned that a researcher was monitoring the group:

When I joined this, I thought it would be a support group, not a fishbowl for a bunch of guinea pigs. I certainly don’t feel at this point that it is a safe environment, as a support group is supposed to be, and I will not open myself up to be dissected by students or scientists. (King, 1996, p. 120)

Online, virtual, and digital research and evaluation are rapidly evolving and will continue to evolve as new technologies and applications emerge. If you are studying virtual communities or

networks, it will be important to stay up-to-date on innovative data gathering and reporting possibilities as well as corresponding ethical challenges (see, e.g., Boellstorff, Nardi, & Pearce, 2012; Ess, 2014; Eynon, Fry, & Schroeder, 2008; Fielding, Lee, & Blank, 2008; Gosling & Johnson, 2013).

MODULE

45 Variations in Duration of Observations and Site Visits: From Rapid Reconnaissance to Longitudinal Studies Over Years

Another important dimension along which observational studies vary is the length of time devoted to data gathering. In the anthropological tradition of field research, a participant-observer would expect to spend six months at a minimum, and often years, living in the culture being observed. The fieldwork of Napoleon Chagnon (1992) among the Yanomami Indians in the rain forest at the borders of Venezuela and Brazil spanned a quarter-century. To develop a holistic view of an entire culture or subculture takes a great deal of time, especially when, as in the case of Chagnon, he was documenting changes in tribal life and threats to the continued existence of these once-isolated people. The effects of his long-term involvement on the people he studied became controversial (Geertz, 2001; Tierney, 2000a, 2000b), a matter we shall take up later. The point here is that fieldwork in basic and applied social science aims to unveil the interwoven complexities and fundamental patterns of social life—actual, perceived, constructed, and analyzed. Such studies take a long time.

Educational researcher Alan Peshkin constitutes a stellar example of a committed fieldworker who lived for periods of time in varied settings to study the intersections between schools and communities. He did fieldwork in a Native American community; in a high school in a stable, multiethnic, midsized city in California; in rural, east-central Illinois; in a fundamentalist Christian school; and in a private, residential school for elites (Peshkin, 1986, 1997, 2000b). To collect data, he and his wife, Maryann, lived for at least a year in and with the community that he was studying. They shopped locally, attended religious services, and developed close relationships with civic leaders as well as teachers and students.

In contrast, evaluation and action research typically involve much shorter durations in keeping with their more modest aims: generating useful information for action. To be useful, evaluation findings must be timely. Decision makers cannot wait for years while fieldworkers sift through mountains of field notes. Many evaluations are conducted under enormous pressures of time and limited resources. Thus, the duration of observations will depend to a considerable extent on the time and resources available in relation to the information needs and decision deadlines of primary evaluation users. Later in this chapter, we’ll include reflections from an evaluator about what it was like being a part-time, in-and-out observer of a program for eight months but only present 6 hours a week out of the program’s 40-hour weeks.

On the other hand, sustained and ongoing evaluation research may provide annual findings while, over years of study, accumulating an archive of data that serves as a source of more basic research into human and organizational development. Such has been the case with the extraordinary work of Patricia Carini at the Prospect School in North Bennington, Vermont. Working with the staff of the school to collect detailed case records on students of the school, she established an archive with as much as 12 years of detailed documentation about the learning histories of individual students and the nature of the school programs they experienced. Her data included copies of the students’ work (completed assignments, drawings, papers, projects), classroom observations, teacher and parent observations, and photographs. Any organization with an internal evaluation information system can look beyond quarterly and annual reporting to building a knowledge archive of data to document development and change over years instead of just months.

Participant observations by those who manage such systems can and should be an integral part of this kind of knowledge-building organizational data system that spans years, even decades.

Rapid Appraisal Studies: Qualitative Inquiries Built for Speed

On the other end of the time continuum from long-term ethnographic studies are rapid appraisal site visits and short-term studies that involve observations of a single segment of a program, sometimes for only an hour or two. Evaluations that include brief site visits to a number of program locations may serve the purpose of simply establishing the existence of certain levels of program operations at different sites. Chapter 1 presented just such an observation of a single two-hour session of an Early Childhood Parent Education program in which mothers discussed their child-rearing practices and fears. The site visit observations of some 20 such program sessions throughout Minnesota were part of an implementation evaluation that reported to the state legislature how these innovative (at the time) programs were operating in practice. Each site visit lasted no more than a day, often only a half-day.

Sometimes an entire segment of a program may be of sufficiently short duration that the evaluator can participate in the complete program. The leadership retreat we observed lasted six days, plus 3 one-day follow-up sessions during the subsequent year.

Rapid appraisal studies, also called rapid reconnaissance and rapid assessment, are aimed at quick, low-cost data collection to answer specific, urgent questions. The purpose is to obtain a narrow, but in-depth understanding of the conditions and needs of the targeted group in a specific area to inform immediate decisions. Qualitative methods are especially appropriate because, being open, naturalistic, and emergent, they make it possible to get into the field quickly. Applications include the following.

1. Update a needs assessment, situation analysis, or project proposal, and establish an up-to- date baseline: Often months, sometimes years, pass between the time a project is originally formulated and the time when the implementation actually begins. Approval and funding allocation processes can take a long time. The original needs assessment and/or situation analysis on which the proposal was based can become badly out of date. Conducting a rapid appraisal can be used to update the situation based on field visits, purposeful interviews, and retrieving any new documentation.

2. Launch a new initiative: Quick site visits to engage local people at the start of a project can serve to officially launch an initiative while gathering data about baseline perceptions and conditions and short-term priorities, building relationships, and documenting immediate concerns.

Example: When I became codirector in the field for the Caribbean Agricultural Extension Project, our purpose was to develop extension improvement plans in eight Caribbean countries in the Windward and Leeward Islands. By the time the project was funded and personnel recruited, the original needs assessment and project proposal was three years out of date. We also needed to begin engaging local extension and agriculture personnel in the participating countries. With the Caribbean Agricultural Research Development Institute and University of the West Indies, we formed multidisciplinary teams to conduct a rapid reconnaissance of each country’s extension and agricultural situation. Each five-person team included two agricultural scientists (e.g., agronomist, soil scientists, plant specialist, etc.), a social scientist, an extension specialist, and an evaluator. The teams spent one week in each country visiting small farmers’ fields, interviewing key knowledgeables, examining documents and reports, and

assessing needs and capacities. Reports were produced by the teams before they left each country so that a briefing could be held with relevant officials and project partners prior to departure. These rapid appraisals served to update the project baseline, renew the situation analysis, and launch the project. Four separate teams (two countries per team) worked simultaneously so that all updated appraisals were completed in two weeks.

3. Crisis management: A rapid appraisal may be needed when a project falls into crisis. Such crises are precipitated by project staff turnover, changes in the government, or emergencies (floods, droughts, civil unrest).

Example: In 2013, Al-Qaeda began assassinating young women engaged in the national polio vaccination campaign in Pakistan. This was in retaliation for the U.S. Central Intelligence Agency having used a fake polio vaccination campaign as a way to locate Osama bin Laden. Pakistani and international humanitarian and health agencies had to halt the vaccination campaign and conduct a rapid appraisal of the changed situation to strategize about how to reconfigure the polio vaccination campaign (Khan, 2013).

4. Solve problems: Expert observers are skilled at rapid appraisal. They bring specialized knowledge to bear quickly to solve problems. But to do so, they have to get into the field where they can see what is going on.

Example: Sri Lanka was experiencing a failing irrigation system that threatened their food security. Irrigation experts sponsored by the International Irrigation Management Institute conducted a rapid assessment and found inadequate drainage, lack of maintenance of systems, improper water management, excessive seepage, lack of demonstration projects, poor irrigation plots, improper operation, and lack of communication in headquarters (Chambers & Carruthers, 1986).

5. Add independent assessment to controversial or highly visible findings: When the findings of a study enter public dialogue, those who are unhappy with the findings will commonly attack the study’s methods. Since no study is perfect, flaws will be detected and highlighted. A return to the field to augment a study’s findings with a rapid appraisal can help sort out the weight of evidence and validity of conclusions.

Example: Controversy arose on the H1N1 virus vaccination campaign in 2009 in Martha’s Vineyard, Massachusetts, where towns registered to receive the vaccine independently of the local hospital and physicians’ offices. When the planned islandwide vaccination clinic was postponed twice due to delays in vaccine delivery, finger-pointing ensued and the cause of the problem became a matter of contention and visibility. An independent qualitative rapid assessment was able to provide credible and timely data on root causes (Stoto, Nelson, & Klaiman, 2013).

Quality and Utility of Rapid Appraisal Methods and Applications

Rapid appraisals respond to increasing demands for “real-time” data. Where decisions have to be taken with imperfect data, some data are better than none, and timely findings are better than reports that come in after a decision has been taken. While speed can contribute to utility, the quality and selectivity of rapid recon data can raise credibility questions. Rapid appraisal teams need to be carefully selected to be credible, represent diverse perspectives and skills, be politically and culturally sensitive, have the capacity to gather and analyze data quickly, communicate

effectively, and work together smoothly under pressure. (For resources and applications, see Bamberger et al., 2011, pp. 67–72; Beebe, 2008; Hargreaves, 2013; Social Impact, 2013; U.S. Agency for International Development, 2010, 2011.)

The critical point of this module is that the length of time during which observations take place depends on the purpose of the study and the questions being asked, not some ideal about what a typical participant observation must necessarily involve. Field studies may be massive efforts with a team of people participating in multiple settings to do comparisons over several years. At times, then, and for certain studies, long-term fieldwork is essential. At other times and for other purposes, as in the case of short-term formative evaluations, it can be helpful for program staff to have an evaluator provide feedback based on just one hour of onlooker observation at a staff meeting, as I have also done.

My response to students who ask me how long they have to observe a program to do a good evaluation follows the line of thought developed by Abraham Lincoln during one of the Douglas –Lincoln debates. In an obvious reference to the difference in stature between Douglas and Lincoln, a heckler asked, “Tell us, Mr. Lincoln, how long do you think a man’s legs ought to be?”

Lincoln replied, “Long enough to reach the ground.”

Fieldwork should last long enough to get the job done—to answer the research or evaluation questions being asked and fulfill the purpose of the study. That said, short site visits pose special challenges. I’ve devoted this chapter’s rumination (MQP Rumination # 6) to the problems associated with short-term fieldwork (also sometimes known as “quick-and-dirty” site visits).

MQP Rumination # 6 (A Long One)

Evaluation’s Site Visit Scandal: Poor-Quality Fieldwork, Worse Ethics

I am offering one personal rumination per chapter. These are issues that have persistently engaged, sometimes annoyed, occasionally haunted, and often amused me over more than 40 years of research and evaluation practice. Here’s where I state my case on the issue and make my peace.

Over the years I’ve heard more complaints from program staff and evaluators about wasteful and useless site visits than any other aspect of evaluation. What kinds of complaints? Little things like that site visits are ineptly conceived, planned, staffed, timed, and conducted. The stories I hear lead me to believe that site visit quality is too often poor in more developed countries and too often outrageous in the developing world. The problem strikes me as huge, so be prepared; this is a long rumination.

What Are Site Visits?

A “site” is the location of an organization, program, project, community activity, meeting, or event. In short, a site is where something is going on that is of interest and importance to a funding or oversight agency (e.g., a philanthropic foundation, an international agency, or a

government administrative unit) or is relevant for the inquiry of a researcher or student doing fieldwork. The “visit” involves a person or team going to the site to observe what is happening, interview key people, review relevant documents, and write a report. Here are examples, both international and domestic. The World Bank is funding a five-year mother- and-infant nutrition project in Bangladesh, a well-digging project in Burkina Faso, or an agricultural development project in the Andes. Halfway through the project, a three-person team is selected by the World Bank to visit the project for a few days to assess progress. A philanthropic foundation is funding an adult literacy program in a rural community and sends a program officer out for a day to observe some classes and talk with participants. The U.S. Department of Transportation and several state agencies are funding driver education programs aimed at reducing use of cellular phones while driving; a team of national and state staff go for two-day site visits to observe four such programs in different states. In essence, evaluation site visits involve fieldwork on location to report independently on how a project is being implemented and what it is accomplishing.

Midterm and end-of-project evaluation teams may well be the most common form of formal evaluation activity in the world. There are thousands of such reviews commissioned by scores of funding agencies every year. Many are well done by competent site visitors (evaluators, researchers, program experts) who produce credible reports that are helpful both to people at the site and to the agency that has commissioned the site visit. So what’s the problem? Many site visits are organized poorly, staffed inadequately, and conducted badly (poor-quality fieldwork), leading to useless reports. How and why does this occur? Here’s the tip of the iceberg.

Site Visit Problems: Some Examples

• Contracting processes to hire independent international site visitors are often mired in bureaucratic solicitation procedures that create delays in getting teams assembled and into the field. Once funds are finally released, the procurement timelines are unmanageably short. I regularly see the solicitations seeking evaluators to conduct site visits, many such requests every month. I receive phone calls and e-mails asking if I’m available. But who’s available to pick up and leave on two weeks’ notice to spend a month doing site visits in remote areas of Eritrea? Or India? Or Ecuador? Except in unusually fortunate cases, the least qualified evaluators are the ones who are available on short notice, including novices and those who don’t have much work because they simply aren’t very good. International evaluation colleague Ricardo Wilson-Grau has dubbed the contract solicitation process “tendering madness,” and he has his own ruminations about it, which includes this astute observation, with which I heartily agree:

Tendered evaluations commit the mortal sin of focusing on the formal written scope of work rather than on the flesh and blood. Even the most sophisticated checklists for selecting the best tenders, and which include, of course, the evaluators’ qualifications, do not emphasize what matters most to the success of an evaluation: the evaluator.

I posed the question of why this problem is so pervasive to an international development aid site visit contractor with 25 years’ experience. He told me, “Truthfully, at the end of the day the main people available are retired, alcoholic, burned out, cynical-to-the-core former aid workers and foreign service officers who want to escape their spouses and pick up a few bucks while drinking their way through the assignment.” Exaggeration? Hyperbole? Rant? As I probed his perspective, based on his years of administering, organizing, managing, and

supervising a large number and variety of site visits, it became clear that he knew firsthand that of which he spoke—for he was, himself, a near-retirement, alcoholic, burned-out, cynical-to-the-core aid administrator. He confessed that he didn’t really care about the quality of the work of people hired as site visitors. “No one pays any attention to site visit reports anyway,” he explained. “It’s all part of the game, just window-dressing to give the appearance of some kind of accountability.” So he hired people like himself who followed regulations, did the paperwork right, didn’t make waves, and made good drinking buddies.

• I asked a highly experienced international colleague if he thought drunks conducting site visits was a widespread problem. He launched into his own war stories, including an especially embarrassing incident in an indigenous community that struggled with alcoholism where he was the last to recognize that a person assigned to his team, feigning illness, was actually drunk. The local program people had to point out the problem to him. Then he added, “And don’t forget to mention the sexual predators and harassers.”

• From program directors and staff I hear complaints about site visits being planned and imposed with no input from them as to timing, purpose, or process. The logistics are a nightmare. The site visit teams are ill prepared, haven’t done their homework, take up lots of staff time, expect VIP treatment, and are condescending, arrogant, rude, and, too often, incompetent, corrupt, or both. Junior or novice evaluators are often assigned tasks that should be done by senior evaluators (who may be off carousing instead of working).

• Domestically, philanthropic foundations and governmental agencies too often send program officers and administrative staff into the field to conduct site visits with little or no training, a vague scope of work, and little advance notice to sites and programs being visited.

Those complaints are somewhat abstract, so let me illustrate with some concrete examples.

Examples of the Underbelly of Site Visits

A session at the 2005 joint AEA/CES (American Evaluation Association/Canadian Evaluation Society) International Evaluation Conference in Toronto on evaluation site visits featured many horror stories documenting that external site visit evaluators hired by international donors often lack minimum evaluation competence and expertise to carry out their assignments. Alexey Kuzmin (2005) did his doctoral dissertation on evaluation capacity building. In the course of his fieldwork, he turned up a number of instances of unscrupulous evaluators engaged in unethical practices while conducting site visits. In one case, an American evaluation site visitor to a small program arrived late one day, left early the next, never visited the program, and refused to take the documents offered by the program. Sometime later, the program director received an urgent demand from the funder for required documentation (the very documentation that the evaluator was supposed to have taken, analyzed, and presented to the funder), which it took considerable expense to get to the funder by express courier. Subsequently, the program was denied additional funding for lack of an evaluation having been completed.

In another instance, an external evaluator arrived, one without any substantive expertise in the program’s area of focus. He spent a couple of days with the program and then disappeared, leaving an unpaid hotel bill and expenses that the program had to cover because it had made arrangements for his visit. A couple of months later, the program director received a draft evaluation report for her comments, but the deadline for sending comments

had already passed. The evaluation contained numerous errors and biased conclusions based on the evaluator’s own prejudices. The program director spent a considerable amount of time writing a response to the evaluation, but the evaluation, with its errors and negative judgments, had already been posted on a prominent agency website. The program had to resort to expensive legal action to have the evaluation removed.

A third example concerns a very affable evaluator of a program undergoing an external midterm review. The American evaluation team leader was a charming man who gave a great deal of attention to the program director. In a private conversation with her, he talked in depth about an organizational development model he had recently implemented in another country. It gradually became clear that the evaluator was offering positive evaluation findings if the director hired him as a consultant to return and implement his model in her agency. He said, “I really want to say good things about your program. I want to support it, but I need to hear some things from you before I do that.” The program director, feeling quite vulnerable, contacted a lawyer for advice about how to protect herself and her agency, advice she heeded, but found the whole experience traumatic.

A more recent example involved an evaluator who refused the full services of a translator during a three-day site visit. His scope of work included interviewing program participants. He said that he would use the translator to ask questions in the participants’ African language, but he didn’t want the responses translated. He explained, “I’m an expert at reading body language. I’ve been all over the world. I can tell by watching people whether they are having good program experiences. Translations can’t be trusted. I trust what I see.” The program director was flabbergasted but afraid of alienating him, so she deferred.

In a similar instance, the program director contacted the donor agency that had sent the site visit evaluator and got an appropriate response. The site visit was terminated. But the program director felt great trepidation about complaining to the donor agency about an inept evaluator, fearing that it would be interpreted as resistance to being accountable. The power dynamics are palpable.

Student Site Visits

Site visit problems are not always (or even usually) so dramatic. Nor are site visits just done by professionals. Students often conduct site visits to local agencies as part of a course, an academic paper assignment, or thesis work. Seldom do the professors who assign such site visits prepare or train students. They just send them off to get some experience in the real world. I’ve heard story after story from program directors about students who were attentive, engaged, sensitive, and a delight to involve—and I’ve heard stories about students who were insensitive, inappropriate, and clueless.

An evaluation colleague told me about observing a graduate student do a site visit while she was also on site for an evaluation she was conducting. “I remember the student sitting in one of the classrooms for quite a long time, more than a couple of hours. I circled back to this room and asked her how it was going. I was especially interested in what she was learning about the culture of the school, which was the focus of her observations. I asked, so what are you learning? She yawned, looked bored, and said, “Not much,” and added that she would probably leave soon. I noticed that she had only a half-page of notes. The next time I looked for her, she was gone. Imagine my surprise when I found out that she had written up her observations as a chapter in a book on school innovation. The methods description in that chapter did not correspond to the degree and nature of fieldwork I knew had actually occurred.”

Anecdotal Evidence

These anecdotes are sufficient, I hope, to illustrate the nature and variety of site visit problems. I have many more I could share, including many positive ones. MQP Rumination # 1 in Chapter 1 (pp. 31–33) was about valuing anecdotes as data. They are illuminative and illustrative, not representative or generalizable. What I’ve reported here are outlier (hopefully) anecdotes to illustrate the nature of the problem, not the scope of the problem, which is hard to determine. In discussing these examples with a former graduate student, Donna Podems, who is a full-time independent evaluation consultant working globally, she commented,

I have been doing site or field visits for 20 years. It makes up a bulk of the work I do. I have had a range of experiences with site visits, some great, some awful, but the majority of my site visits are good. Not always great. Rarely bad. But good.

That strikes me as a fair and balanced picture.

But the patterns of ineptness and ethical lackadaisicalness in the negative anecdotes are sufficient, in my opinion, to raise concern. Certainly, I know of agencies that work hard to ensure quality site visits and evaluators who bring knowledge, skill, and competence to the work of site visits. In hopes of further enhancing sensitivity to the nature of the problem, let me invite you to a short stroll down memory lane.

Site Visit Ethics: Tainted History

Potemkin Village Site Visits

The phrase Potemkin village refers to a fake village, built only to deceive and falsely impress. According to legend, in 1787, Grigory Potemkin, governor of Crimea, erected fake settlements along the banks of the Dnieper River to fool Russian empress Catherine II and her entourage, including several foreign ambassadors, during their visit to the region. Potemkin wanted to show that the region, which had been devastated by war, had been rebuilt and was prospering under his governance. He constructed portable village facades to give the appearance of development. As the queen’s barge passed by, Potemkin had his men dressed up as peasants and looked like they lived and worked in the flourishing village. Once the barge floated down river, the fake peasants would take down the phony facades and rebuild at a new site during the night, repeating the deception at various spots along the route.

The phrase Potemkin village has since come to be used generally to designate and denigrate any fake creation built to deceive people into thinking that conditions are better than they really are. For observers engaged in fieldwork, the story raises the question of whether what is being observed is real and typical. For program evaluation site visits in particular, which are often short in duration and planned by the people whose program is being evaluated, the potential for making things appear rosier than they are is a serious validity concern. Observers need to be able to exercise some control over where they go, what they see, and who they talk to in order to ensure that they are not being manipulated into observing only what those in authority want them to observe.

True or False

From a research and evaluation perspective, the Potemkin village story also symbolizes the problem of finding out what the truth is. Historians debate whether the Potemkin village deception actually happened, and if it did, whether the empress was in on the plot as a way of impressing the foreign ambassadors accompanying her. This last possibility is boosted by the fact that Potemkin and the empress were lovers, though it must be acknowledged, it is not unprecedented for lovers to deceive each other. Whether the original story is true or false, the story has endured and serves as a cautionary tale for observers to look behind the facades offered by those in positions of authority who want to make a positive impression. Cultivate multiple sources. Examine a situation from diverse perspectives. Dig beneath the surface to find out what’s really going on. Triangulate.

The Infamous Theresienstadt Site Visit

Near the end of World War II, rumors began to circulate about the Nazi extermination of Jews. The Nazis wanted to refute those rumors, so they took one camp, called Theresienstadt, in what was then Czechoslovakia, and turned it, if ever so briefly, into a model town. They shot a movie there to prove how well they treated the Jews and invited the Red Cross to inspect it. This was 1944. In preparation for the Red Cross inspection, a beautification plan was implemented by the Nazi administration. They painted buildings, planted flowers, opened stores, and put up a bandstand in the town square. They built a kindergarten in a small park, near a children’s home. They opened a coffee shop. They turned the concentration camp into a pleasant little town. And they made it a lot less crowded. Just before the Red Cross delegation arrived, the Nazis shipped 7,500 people off to the Auschwitz death camp, reducing crowding and creating more open spaces.

On June 23, 1944, the Red Cross delegates came for a one-day inspection. The Red Cross delegates went exactly where the Nazis took them—and only where they took them. They didn’t question a single prisoner. The Swiss head of the delegation took pictures showing how happy the children looked—and included them in his report. The final report of the Red Cross

delegation reported that Theresienstadt looked like a normal provincial town where the “elegantly dressed women all had silk stockings scarves and stylish handbags.” The delegates also wrote that Theresienstadt was a final destination camp and that people who come there were not sent elsewhere—because that’s what the inspection team was told. In fact, by the time of the visit, some 68,000 people had already been shipped from there to the death camps (CBS News, 2007).

Back to the Present

Think Theresienstadt is just yesterday’s news? I recently had a lengthy exchange with an evaluator who was disturbed by having just come from a midterm site visit of a large and important but controversial project funded by an international agency in an Asian developing country. All sites visited were selected by the host government. Only project participants selected by the host government were interviewed and always with project and/or government people present. Most requested documents and data were “not available.” The team had to produce and hand in their report in the country before they departed. It is not clear what happened to the report. On the midterm review team of four people, only one person had any evaluation background, knowledge, or training—the person who was telling me about the experience—and that person was the only one who saw any problem with the way the review was organized and the only one who complained, to no avail. I can’t reveal details of the project without breaking confidentiality, but I can tell you that the project involved issues of life and death for the intended beneficiaries and huge potential embarrassment to the host country. Every aspect of the site visit was managed and manipulated to produce findings that would portray the government’s policies in a positive light. The evaluator complained to me about the corruption of the process but admitted having signed off on the report because “otherwise I wouldn’t have been paid and my expenses wouldn’t have been reimbursed.” (At the end of this rumination, I’ll suggest some guidelines to help avoid or at least ameliorate this kind of situation.)

The Perspective From the Site

Programs on the receiving end of being “visited” (what a euphemism!) for evaluation recognize the high stakes and act accordingly. A colleague summed up the situation succinctly:

The site does not have direct power over a site visit as the visit is almost always externally mandated and contracted. The information collected will potentially be used to make decisions about the site or intervention. So a site visit is usually like a high stakes evaluation without any real accountability, affecting continuation, expansion, contraction, or even termination. So sites try to protect themselves, which is the absolutely rational thing to do. The site prepares for the visit by sweeping up messes, cleaning, priming community storytellers, arranging with counterparts or counterpart proxies to drop by for support, filing data in surprisingly obscure places, and even going so far as to arrange a preemptive invitation by a well-known, high-status expert on the pretext of providing advice to the project, but making sure the site visitor knows. The truth is that everyone is gaming the system on a site visit.

But sometimes, sometimes, an external visitor is competent, attentive, and sincere, and appears to have something to contribute and wants to do so. Learning can occur. Mutual understanding can emerge. Good things can happen. One such visitor came to a long-term project where I was M&E [Monitoring and Evaluation] team leader and got the keys to the kingdom, listened well, offered insightful feedback, and left it a better place. He is still remembered.

Reflections From a Veteran Site Visitor

In discussing site visits with veteran evaluators, I collected many stories and reflections, more than I can include in this rumination. Most reported that, on balance, their experiences were positive. But all had run into some situations that gave rise to concerns. Doug Horton, an international evaluator with more than 40 years’ experience in the field, told me he’d never witnessed the worst excesses reported here but he had his own list of problematic encounters. Here are his reflections:

To Party or to Evaluate? That Is the Question

Sometimes it isn’t the evaluators who want to drink, but the program team being evaluated or local collaborators or beneficiaries. A common strategy of program teams that organize site visits is to plan a “show-and-tell” event that combines a presentation of the program’s activities and a celebration with local partners or beneficiaries. In such cases, the evaluators may find that instead of presenting an opportunity for data gathering, the site visit becomes a party!

In one case that an international evaluator relayed to me, the evaluation team drove several hours to a remote community where 30 heads of household (mainly men) were lined up in the town square, ready to great the team. After formal greetings, the evaluators were informed that they had about an hour before lunch, to observe the program’s activities, meet with people, and conduct interviews. During this hour, the gender specialist found it impossible to meet with women, who were all involved in preparing the lunch! After an hour, an array of local dishes was spread out on tables in the local school, and the lunch was accompanied by speeches (generally extolling the virtues of the program being evaluated), music, and drink.

After lunch, the festivities continued, severely limiting the possibility of “serious fieldwork,” until the climax, when the evaluation team leader was presented with a live rooster, which he was expected to take back to the capital city—presumably to prepare a chicken soup to recover from the drinking.

In this case, the evaluators decided that it would be too rude to stop the party and focus on the evaluation, so they participated in the festivities and did their best to gather information on the sidelines. The following morning, while some of the program team members were nursing hangovers, the evaluators called a meeting with the program leaders to clarify the purposes and appropriate activities for future site visits and emphasize the importance of using the limited time available for gathering information that would be of use for program improvement and providing accountability to sponsors.

How to Stop the Speeding Train?

When confronted with an evaluation, program teams often try to give the evaluators so much detailed information that they will never be able to “see the forest for the trees.” This is sometimes referred to as the mushroom strategy, since the program team is attempting to cultivate the evaluation the way one grows mushrooms: by burying them in bullshit and keeping them in the dark.

A variant of the mushroom strategy commonly used by those who plan site visits is to pack so much in and keep the evaluators moving around so much that they don’t have time to reflect on their preliminary findings or adjust their work plan. A common feature of international site visits is that the organization or program being evaluated organizes such a “dense” series of site visits—often in several parts of the country and sometimes even in different countries—that there is precious little time left for the evaluators to review their notes (if they were able to take any), meet for joint discussions and reflections on their observations, or begin working on their report. In cases like this—which are very common—an important job for the evaluators is to renegotiate the schedule of site visits to focus on the collection of the most essential information from the

most essential sites, and to carve out time for evaluation team members to discuss and reflect on their findings as they evolve and adjust the evaluation plan as needed. (Horton, personal e-mail communication, July 18, 2014. Used with permission.)

Reframing Site Visits

A highly experienced evaluation colleague has been “wrestling with the site visit ball-and- chain for some time.” While the term site visit can be understood to mean something very formal, “a very planned dog-and-pony show by program staff to demonstrate the greatness of a project, we’re trying very hard in a large, multisite evaluation to avoid those situations by reframing site visits as case study data collection opportunities.” The evaluation team was working to avoid site visits becoming “program song and dance routines, but we’re running up against a program culture of expectations that are hard to change.” She went on to describe arriving at a site to conduct a planned two-hour interview with the leadership of a project, only to be confronted with a formal PowerPoint presentation and told that they could ask questions for a half-hour after the presentation. They were able to negotiate more time for an interview, but the program staff still felt a strong need to make a formal presentation, an illustration of the high-stakes politics and pressures that can surround site visits.

Site Visit Standards

Most site visits may well be conducted appropriately, professionally, and effectively. Certainly many are. Some of my most memorable experiences have been conducting site visits with engaged and insightful colleagues. Going into the field to see what is happening is essential. On the other hand, some proportion of site visits is conducted insensitively, incompetently, inappropriately, and unproductively. How many is not and cannot be known. My sense is that the numbers are sufficiently large and the poor practices are sufficiently widespread to constitute a significant problem. As a starting point toward reducing the problem, then, I propose 10 standards for the conduct of site visits.

Draft Standards for Site Visits

1. Competence: Ensure that site visit team members have skills and experience in qualitative observation and interviewing. Subject matter expertise, like an economics doctorate (to pick a random example), does not suffice. Nor does simply being available for the assignment.

2. Knowledge: Where the purpose of the site visit is evaluative, ensure that at least one team member, preferably the team leader, has evaluation knowledge and credentials. I apologize for being redundant, but this deserves emphasis: Subject matter expertise does not suffice. Nor does simply being available for the assignment. Evaluation is a profession grounded in a body of knowledge, standards, specific ethical principles, and skills, including cultural competence. It’s a radical suggestion, I realize, but teams conducting evaluation site visits should include a professional evaluator.

3. Preparation: Site visitors should know something about the site being visited. Whether this knowledge is procured through background materials, briefings, prior experience, or some combination thereof, one of the most common complaints from people at the site is that site visitors display a blinding ignorance of place, context, and what goes on at the site. Those who commission, fund, and otherwise engage site visitors have an obligation to ensure some reasonable level of preparation.

4. Site participation: People at the site should be engaged in planning and preparation for the site visit to minimize disruption to program activities and services to program beneficiaries. Logistics, costs, scheduling, access to documents, and the nature of the inquiry should be planned and transparent, with reasonable consideration of how the inquiry may be intrusive.

5. Do no harm: Site visits are not benign. The stakes can be high. People can be put at risk. Programs can be put at risk. Good intentions, naïveté, and general cluelessness are not excuses. Apologies after the fact are insufficient. Be alert to what can go wrong. Take responsibility. Be accountable. Commit as a team that you will, first and foremost, do no harm. (This goes for those who commission evaluations as well as those who conduct them.)

6. Credible fieldwork: While those at the site should be involved and informed, they should not control the information collection in ways that undermine, significantly limit, or corrupt the inquiry. The site visitors, for example, should determine the sample of activities observed and people interviewed, insofar as possible, and should be able to arrange confidential interviews to enhance data quality.

7. Neutrality: A site visitor should not have a preformed position on the intervention or the intervention model. Openness to what the data reveal is essential but cannot be assumed.

8. Debriefing and feedback: Before departing from the site visit, some appropriate form of debriefing and feedback should be provided to key people at the site. In addition to providing highlights of the findings, insofar as possible and appropriate, those at the site should know when they will receive a report, either oral or written or both. If no reporting will be shared, why not? What follow-up will occur?

9. Site review: Those at the site should have an opportunity to respond in a timely way to site visitors’ reports, to correct errors and provide an alternative perspective on findings and judgments, if necessary. Site visit reports, often done in haste, may be infused with site visitors’ biases and preconceptions. The fact that they’re independent and external doesn’t make them neutral or accurate. Triangulation and a balance of perspectives should be the rule.

10. Follow-up: The agency commissioning the site visit should do some minimal follow-up (a brief survey or phone call) to assess the quality of the site visit from the perspective of the locals on-site. And the site visit team should know in advance that such a follow-up assessment will take place. A philanthropic foundation in Minnesota routinely has its own program officers conduct due diligence site visits. The executive director sends out a short follow-up survey that comes directly to her—and that she reads and responds to. The results have turned up a number of problems that have informed staff development on how to conduct a site visit appropriately.

Rethinking Site Visits

These proposed standards accept site visits as an ongoing method for conducting external evaluations. But the notion that a valid, meaningful, and useful evaluation can result from a short visit by an external team with limited knowledge of context and little time to collect data is hugely problematic. The widespread use of site visits as a central form of accountability evaluation needs to be rethought. But that’s a different rumination, for another forum. So while my preferred solution is eliminating ritualistic site visits because the method is

inherently flawed, the proposed standards are aimed at some modest improvements and preventing the worst abuses.

But 10 standards may be too many. In that regard, one principle may well suffice, the standard offered by Leslie Goodyear et al. (2014) as the final sentence in their important book Qualitative Inquiry in the Practice of Evaluation:

The responsibility of a qualitative evaluator is to leave the setting enriched for having conducted the evaluation.

MODULE

46 Variations in Observational Focus and Summary ofDimensions Along Which Fieldwork Varies

Why so many options? Why so many variations?

Why so many choices? Why so many decisions?

Because it’s a messy world. One size doesn’t fit. —Halcolm

Module 44 discussed how observations vary in the extent to which the observer participates in the setting being studied, the tension between insider and outsider perspectives, and the extent to which the purpose of the study is made explicit. Module 45 discussed variations in the duration of the observations. A major factor affecting each of these other dimensions is the scope or focus of the study or evaluation. The scope can be broad, encompassing virtually all aspects of the setting, or it can be narrow, involving a look at only some small part of what is happening.

Parameswaran (2001) wanted to interview young women in India who read Western romance novels. Thus, her fieldwork had a very narrow focus. But to contextualize what she learned from interviews, she sought “active involvement in my informants’ lives beyond their romance reading.” How did she do this?

I ate snacks and lunch at cafes with groups of women, went to the movies, dined with them at their homes, and accompanied them on shopping trips. I joined women’s routine conversations during break times and interviewed informants at a range of everyday sites, such as college grounds, homes, and restaurants. I visited used-book vendors, bookstores, and lending libraries with several readers and observed social interactions between library owners and young women. To gain insight into the multidimensional relationship between women’s romance reading and their experiences with everyday social discourse about romance readers, I interviewed young women’s parents, siblings, teachers, bookstore managers, and owners of the lending libraries they frequented. (p. 75)

The tradition of ethnographic fieldwork has emphasized the importance of understanding whole cultural systems. The various subsystems of a society are seen as interdependent parts, so that the economic system, the cultural system, the political system, the kinship system, and other specialized subsystems could only be understood in relation to each other. In reality, fieldwork and observations have tended to focus on a particular part of the society or culture because of specific investigator interests and the need to allocate the most time to those things that the researcher considered most important. Thus, a particular study might present an overview of a particular culture but then go on to report in greatest detail about the religious system, kinship system, or technology system of that culture.

In evaluating programs, a broad range of possible foci makes choosing a specific focus challenging. One way of thinking about focus options involves distinguishing various program processes sequentially: (a) processes by which participants enter a program (the outreach, recruitment, and intake components), (b) processes of orientation to and socialization into the program (the initiation period), (c) the basic activities that make up program implementation over

the course of the program (the service delivery system), and (d) the activities that go on around program completion, including follow-up activities and client outcomes over time. It would be possible to observe only one of these program components, some combination of components, or all of the components together. Which parts of the program and how many are studied will clearly affect issues such as the extent to which the observer is a participant, who will know about the evaluation’s purpose, and the duration of observations.

Chapter 5 discussed how decisions about the focus and scope of a study involve trade-offs between breadth and depth. The very first trade-off comes in framing the research questions to be studied. The problem is to determine the extent to which it is desirable and useful to study one or a few questions in great depth or to study more questions but each in less depth. Moreover, in emergent designs, the focus can change over time.

Sensitivity of the Inquiry: Potential for Controversy

The degree to which fieldwork is inquiring into matters that are sensitive or potentially controversial will affect how the inquiry is conducted. Occasionally, I have been asked to help prepare university students for cross-cultural experiences in which they will be doing fieldwork on some topic of interest. Most have become accustomed to Western media norms where no questions are off-limits. They propose research on politics, sexual practices, gender relationships, sexual orientation, family power dynamics, income distribution, and religion as if these were straightforward topics for open study anywhere in the world. Quite the contrary. Most people in most cultures in the world do not openly and easily discuss these matters with each other, much less with strangers. Such obliviousness is potentially insulting and sometimes dangerous. Ultimately, engaging in fieldwork requires building trusting relationships with the people in the setting being studied. Inquiry into sensitive and potentially controversial issues requires delicacy, time, and judgment. Key informants who understand the local context will be crucial to learning how to approach sensitive issues.

Dimensions Along Which Fieldwork Varies: An Overview

We’ve examined 10 dimensions that can be used to describe some of the primary variations in fieldwork (Modules 44–45). Those dimensions are summarized in Exhibit 6.4. Each is a continuum. These dimensions can be used to help design observational studies and make decisions about the parameters of fieldwork. They can also be used to organize the methods section of a report or dissertation in order to document how research or evaluation fieldwork actually unfolded.