Week 1 Assignment
To print, please use the print page range feature within the application.
2 Designs for Description
In this chapter you will learn:
1 About cross-sectional, time-series, longitudinal, case study designs and the criteria for selecting each.
2 How to interpret findings from cross-sectional and time-series designs. 3 The value of case studies and how to judge their quality. 4 About meta-analysis and how it is used to combine results of several studies. 5 The differences between quantitative and qualitative research. 6 About focus groups as study designs. 7 Observational data and its limitations.
After formulating a tentative model to guide the research, investigators need a research plan. In it they outline how they intend to collect data on each variable and how they plan to analyze the relationships among the variables. When you study tables, graphs, or other quantitative presentations, you may never think of the decisions and actions required to gather and organize the data they illustrate. Yet these decisions and actions determine the value of the research, and investigators should conduct each step carefully.
Research Methodology Research methodology is a structured set of steps and procedures for completing a research project.1
The quality of a set of data is determined by the research methodology, even though little discussion of the methodology is included in a final report. For this reason, it is altogether too easy for readers or listeners to underestimate the importance of research techniques and to take for granted the accuracy of reported findings.
The data collection and analysis of research methodology include the following steps:
1 deciding when and how often to collect data 2 developing or selecting measures to “operationalize” each variable 3 identifying a sample or test population 4 choosing a strategy for contacting subjects 5 planning the data analysis 6 presenting the findings
Page 1 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Research designs are plans that guide decisions about when and how often to collect data, what data to gather, from whom and how to collect data, and how to analyze data. The term research design has a general and a specific meaning.
• General meaning: The general meaning of research design refers to the plan for the study’s methodology. The design should indicate the purpose of the study and demonstrate that the plan will answer the research question(s) and is consistent with the study’s purpose. Frequently, research designs are described as blueprints for the final research product.
• Specific meaning: The specific meaning of research design refers to the type of study. Common types of studies are cross-sectional studies, time-series studies, case studies, and experiments. These types of studies or designs dictate when and how often to collect the data and how much control an investigator will exert over the research environment.
Types of Studies Cross-sectional studies, time-series studies, and case studies place researchers in an environment where they have little, if any, control over the events. In general, all these studies use what is referred to as “observational data.”2 In contrast, experimental studies allow the researcher to control the research environment. The experimental setting and subjects are selected carefully, and the investigator decides who will be exposed to a treatment or intervention, at what intensity, and for how long. Only with experimental data can we clearly test whether one variable causes a change in another. Accordingly, we have labeled experimental designs as designs for explanation and will discuss them later in the chapter. Yet, there are many phenomena that we might think of as important explanatory variables, for which random assignment is impossible. One example is the unemployment rate across different cities. We cannot randomly assign some cities to one level of unemployment and other cities to a different level. Another example is the occurrence of a particular gene across people. A researcher cannot randomly assign a gene associated with cancer to some participants in a study. Therefore, when studying such phenomena as explanatory variables, we are limited to observational data and accordingly limited in our ability to make causal claims. With observational data, we can at best test whether one variable relates to another in a manner that may suggest causality.
Thus, we have labeled and categorized cross-sectional studies, time-series studies, and case studies as designs for description. These three designs for description may be used separately or they may be combined. Cross-sectional studies are often combined with time-series studies, and case studies may incorporate features of cross-sectional or time-series studies.
Descriptive designs provide a wealth of information that is easy to understand and interpret. They may identify problems and suggest solutions. Such studies can be undertaken to answer questions such as, How many? How much? How efficient? How effective? How adequate? The designs are used frequently to produce the data needed for planning, monitoring, and evaluating. For instance, administrators can combine findings from descriptive studies with other information and decide what, if any, action to take. Consider once again the problem of prison suicides. With stricter enforcement of drunk-driving laws, reports of prison suicides have increased.3 These reports led investigators to collect data on how many prison suicides were committed each year and who committed suicide. For each reported suicide, investigators noted whether the prisoner had been incarcerated for a violent crime, a nonviolent crime, or being intoxicated or was being held in isolation. Upon analysis
Page 2 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
investigators found that suicides tended to occur among males incarcerated for drunkenness or kept in isolation.
The analysis did not tell prison officials why one male prisoner chose to commit suicide and another did not. But the data did tell them how widespread the problem of prison suicide was and what characterized the inmates who were more likely to commit suicide.
What could prison officials do with the information? They could identify how many inmates in their institution were at risk of committing suicide. In other words, how many males did they have incarcerated for drunkenness or placed in isolation each day? Was the number of such incarcerations steadily increasing? Did the number go up on weekends or during certain times of the year?
Officials could propose and evaluate solutions to prevent those prisoners at risk of committing suicide from doing so and determine the feasibility of implementing these solutions. If a solution involves separate facilities or increased staffing, how large a facility is needed to take care of anticipated peak needs? What changes, if any, need to be made in the current staff work schedules? If a solution is tried, its effectiveness can be monitored. Did the rate of suicides drop? If a suicide occurred, had the inmate been identified as being at risk? If identified, had he received the available treatment for a potential suicide?
The number of possible questions goes on. Data are needed to answer them. Other examples would lead to different questions requiring different data. Administrators require data to carry out their tasks. Administrators who understand how data were collected, whether or not they were involved in the collection, will be better prepared to use the data effectively and efficiently.
The administrator who wants to know what caused a particular event or outcome will find descriptive designs somewhat limited. Nevertheless, all three descriptive designs discussed in this chapter can eliminate untenable explanations and furnish valuable leads. As King, Keohane, and Verba state in their discussion of this topic: “Good description is better than bad explanation.”4 And, if the designs are planned and analyzed carefully, they may produce reasonable estimates of a treatment’s effect on a dependent variable. Thus, in some specific circumstances, descriptive designs may suggest causality.5
Designs to Find Relationships and Show Trends
Cross-Sectional Designs
A cross-sectional design collects data on all relevant variables at one time. A researcher may collect data from various sources—such as surveys, forms, or a database—and create a unique database. For example, to study factors associated with traffic fatalities, a researcher may collect data from several sources for each state on the number of fatalities, road conditions, traffic density, arrests and penalties for various traffic offenses, and so on.
Two analogies are often used to describe cross-sectional designs. In one analogy, the design is viewed as a physical “cross section” of the population of interest. In the other analogy, the design is seen as a “snapshot.” Both analogies underscore the static, time-bound nature of the design. The cross-sectional design depicts what exists at one point in time. Clearly, events may change markedly at a later time, even in the next time interval.
Cross-sectional designs should be used to
Page 3 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
• answer questions of how many? How much? Who? How efficient? How effective? How adequate?
• gather information on people’s knowledge, attitudes, and behavior • stimulate exploratory research and identify hypotheses for further research
Cross-sectional designs are particularly suited for studies that involve collecting data
• on many variables • from a large group of subjects • from subjects who are dispersed geographically
Using Surveys With Cross-Sectional Studies
We normally think of cross-sectional studies in conjunction with surveys, whereby an individual or a representative of an organization answers a questionnaire or fills out a form. Analysts then compile the data and analyze the variables. Imagine a survey to answer the research question, “How do the careers of executives working in the public and nonprofit spheres compare?” The model may have included variables such as gender, age, professional education, and professional experience. An advantage of well-designed, well-documented, and carefully implemented cross-sectional designs is that researchers with different interests and models often can work with data from a single cross- sectional study. This particular set of survey data could then be analyzed by other researchers whose primary interest is how gender, age, or academic preparation is related to career outcomes. The use of data by researchers who did not participate in their collection is called secondary data analysis.
The media often sponsor surveys on current issues. Interviewers ask respondents about their knowledge, opinions, or behavior. Then the particular magazine, newspaper, television network, or other media outlet reports the percent of respondents in the various categories: how many are familiar with an issue; how many agree or disagree with a stated opinion; how many act in a particular fashion. The reports may indicate how responses vary according to demographic characteristics: Do women feel differently from men? Do people who live on the East and West Coasts behave differently from citizens who live in other areas of the United States? And so on. Undoubtedly, as you read this, you also will begin to note the many studies conducted and reported with the goal of better understanding ourselves and our society.6
Some surveys are conducted on a regular basis. Over 100 U.S. federal agencies use cross-sectional designs to collect data. The U.S. Census Bureau, a major source of demographic and economic data, regularly surveys individuals, governments, and businesses. These data, which constitute official statistics, may provide a snapshot of the state of the nation as they report the status of its people and of the natural and economic environments. The data guide the decisions of public agencies, nonprofit organizations, and businesses. For example, data on age distribution help estimate demands for services. With knowledge of how many young children are in a community, planners can predict how many children will enter public schools each year. Child care providers can develop business plans based on the number of children eligible for care, where they live, and their characteristics. Nonprofits may use the data to support the need for a proposed service.
Assembling Data in Cross-Sectional Studies
Page 4 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
A researcher can create a cross-sectional database by combining data from different sources, as previously mentioned. A specific strategy is to collect data from records. Employee files, volunteer applications, client records, budgets, and grant applications are all potential data sources. Employee files can yield detailed information on employee characteristics, which can be linked to performance evaluations, patterns of absenteeism, or turnover data. Applications for volunteer positions provide similar information. In both cases the information may identify gaps in employee skills or volunteer interests that can help in designing training or recruiting strategies. The possibilities are almost endless and depend on the needs of an organization and the imagination of a researcher.
Cross-sectional studies may provide data on just one variable (e.g., the educational level of the population), but their greatest value is in describing the relationships among several variables. When the data are analyzed, cases are divided into different groups based on values of the independent variables. Basing a cross-sectional study on a carefully developed model helps to ensure that appropriate information is collected and provides a guide for data analysis. Although such studies can provide useful information and be effective in testing hypotheses, they may suffer from less-than- careful research techniques. For example, if models are not thought out or surveys are not adequately tested in advance, important questions may be omitted or misunderstood by those completing the survey.
Limitations of Surveys
Although surveys can be very use full for collecting data, they unfortunately are often poorly designed and implemented. An investigator may attempt to obtain information from too many people, be unable to make return calls or send follow-up mailings, or be plagued with a low response rate. Questionnaires may be constructed without sufficient care given to purpose, design, and question wording. Investigators should conduct a pilot study, that is, a small study to test the adequacy of a data collection instrument or procedure. They may fail to do so; the survey instruments might also ignore relevant variables. We suspect that these problems have become worse with the ease of constructing online surveys. Respondents, especially those contacted by persons wanting data on public and nonprofit agencies and their administrators, may suffer from “respondent fatigue.” A 20- minute survey, which may seem short to its designer, can be burdensome to an administrator who receives several such requests, has limited interest in the topic, or has to take additional time to think about an answer or gather data. The result may be a low response rate. If the investigator has no way of contacting nonrespondents, this problem is compounded.
In some instances, surveys may produce the wrong data or have little or no usable data. They may have excluded critical variables. For example, a survey of state government employees received 33,000 responses and took 550 hours of staff time to process. All that could be done with the data was to report the answers to each question. The survey included only one independent variable, and that variable turned out to be unusable. The question asking respondents to indicate where they worked was phrased in such a way that the analysts could not tell whether a person who answered “Personnel” was referring to the state office of personnel or another agency’s personnel office. Such problems are preventable. Administrators should not begin collecting data without critically evaluating the survey items and data collection process to see whether the survey will be able to answer the research questions. An important safeguard is to conduct a pilot study.
Analyzing Data in Cross-Sectional Studies
Page 5 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
The data produced from cross-sectional designs can be analyzed in any number of ways using a variety of statistical tools. In this chapter we will use only frequencies and percentage to illustrate how cross-sectional data can be analyzed. Example 1.3 in the previous chapter, drawn from a cross- sectional design, was implemented to learn the placement record of state-funded job-training programs.7 Prior to the study, administrators had no data on the success of training programs in placing participants. The data on job-training program participants were collected from state government files. The dependent variable was employment status at the end of the training program. The independent variables were type of training program, age, gender, race, and educational attainment.
For each type of training program, analysts examined placement outcomes by age, gender, race, and educational attainment. They found that participants in on-the-job training programs were most likely to be employed. Males were more likely to be employed than females; employment status varied with age but did not vary with race or educational attainment. Tables similar to Table 2.1 were produced. Administrators could study the detailed tables to learn whom the programs served and how well a program did in placing its various client groups.
Table 2.1 Age and Employment Status of Participants in On-the-Job Training Programs
Age Employed after Training 16–21 years 40 (51%) 22–24 years 37 (74%) 25–54 years 84 (70%) > 54 years 3 (75%)
Example 2.1 comes from a cross-sectional study conducted for the U.S. Internal Revenue Service to
1 determine satisfaction with online or electronic filing (E-filing) as well as ideas for product improvement
2 identify reasons for not filing online 3 compare perceptions of online filers and other taxpayers 4 determine ease of communicating with the IRS
The study, typical of cross-sectional studies, gathered data on many variables. The example here, An Application of a Cross-sectional Design, is from one part of the analysis. We chose an example that illustrates how cross-sectional designs can guide marketing decisions. We assume that the research team generated ideas (variables) to explain why some taxpayers choose to use E-file and others do not. We included the IRS table, which considered both how users varied in their perception of E-filing and what attributes of filing taxes were important to them.
Example 2.1 Application of a Cross-Sectional Design
Page 6 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Problem: The Internal Revenue Service (IRS), the U.S. tax collection agency, compares taxpayer satisfaction with its products and services. One study compares online filers (users of E-file), lapsed users (those who discontinued using E-file), and nonusers. Comparison variables include demographic characteristics, ease of communicating with the IRS, perceptions of E-file features, and perceptions of the IRS. Are attitudes about E-file attributes associated with product use?
Design: A telephone survey of 1,000 randomly sampled employed taxpayers between the ages of 18 and 74.
Findings: From 977 people who provided information.
Table 2.2 IRS Survey Results
Type of User Agreed That Current Users
(599) Lapsed Users (106)
Nonusers (272)
Having accurate return is really important 95% 92% 91% is true of E-file 69 53 39 Being assured return is private/secure is really important 92 92 87 is true of E-file 55 34 26 Easy to use/little hassle is really important 78 75 73 is true of E-file 65 30 32 Being inexpensive is really important 71 70 60 is true of E-file 53 45 40 Getting return to IRS is really important 71 58 46 is true of E-file 82 67 61 Getting refund faster is really important 61 44 42 is true of E-file 78 62 54 E-file is a better way to file your federal
income taxes 63 26 27
The analysts concluded that attitudes were related to usage. They noted that while accuracy, security, and ease of use are important to all taxpayers, nonusers and lapsed users “have NOT gotten the message of E-file’s benefits in three areas they actually care a lot about—Accuracy, Privacy/Security, and Ease of Use.”
Page 7 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Source: Adapted from “Findings from the 2005 Taxpayer Satisfaction Study,” U.S. Department of Treasury, Internal Revenue Service, July 2005. Publication 4241, Catalog 37303Q. Available at http://www.unclefed.com/IRS-Forms/2005/p4241.pdf.
Limits of Cross-Sectional Designs
Cross-sectional studies generally are inappropriate if investigators want to learn why something happened. As noted previously, one cannot demonstrate with cross-sectional, observational data that a treatment, intervention, or other independent variable caused a given outcome. Except in the case of “experimental research designs,” investigators cannot exert the needed amount of control over the intervention or over the environment surrounding the study. Investigators may be unable to rule out alternative explanations as to why something happened. For instance, in the preceding job-training program example, investigators cannot eliminate the possibility that participants in on-the-job training programs were more likely to be employed than nonparticipants because of observable—and unobservable—factors associated with self-selecting into job-training programs.8 This kind of concern is the major peril of using observational data, and cross-sectional observational data in particular. Nevertheless, a major use of cross-sectional designs is to uncover relationships that can be studied further in experimental studies.
Studies on the effects of estrogen are a good example of how cross-sectional research can inform and motivate experimental studies. In 1985 a noted medical journal published two cross-sectional studies on estrogen.9 One study examined data on 1,234 postmenopausal women living in a Boston suburb. The researchers found that estrogen users in this sample were twice as likely as nonusers to experience heart disease. In another study, researchers surveyed 121,964 postmenopausal nurses and found that estrogen users were one-third as likely to experience heart disease. Design features may have led to the different findings. The studies had different sample populations. One study used a mailed questionnaire to collect data; the other relied on personal interviews and physical examinations. One study collected data over a four-year period; the other over an eight-year period. The results of both studies were plausible theoretically.10
Statistical association is insufficient evidence that a treatment or program causes an observed outcome. Experiments are designed to produce evidence of causality and the effect that a treatment has on a dependent variable. To see whether hormones were actually beneficial, the Women’s Health Initiative recruited and randomly assigned 27,347 women to take either hormones or a placebo. The pills looked the same and the participants did not learn whether they had received hormones until the end of the study. The study was discontinued after 11 years when women who had been taking hormones were found to be at greater risk for heart disease.11
Time-Series Studies
Time-series studies collect and present data on a single unit or set of subjects. The data are collected on the same variable(s) at frequent, closely spaced regular intervals over a relatively long period. The data can depict both short-term changes and long-term trends in a variable. Most of us are familiar with time series that regularly report indicators of some aspect of the nation’s economic or social climate. Such time series include consumer price indexes, the unemployment rate, and crime rates.
Time series are suited for situations in which an administrator wants to do the following:
Page 8 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
1 establish a baseline data measure 2 describe changes over time 3 keep track of trends 4 forecast future trends 5 evaluate the impact of a program or policy
Time-series data are neither hard to gather nor hard to arrange for analysis. The data may be gathered by investigators or taken from existing databases. The data come from one or more units—such as a state, county, or office—and may be collected on the first workday of every month, the fourth Thursday of every November, or any other interval appropriate to the study. An accompanying account of events should be kept and consulted to help explain unexpected patterns: For example, a drop in swimming pool usage may be associated with an unusually cold or rainy summer.
Administrators find time-series data presented in graphs or tables easy to interpret and to combine with other information. They use time series to monitor programs and activities under their jurisdiction. For example, administrators in a social services department may decide to reallocate resources within the department based on the number of clients served, the number of services offered, and the length of service time.
A time series may or may not include an explicit independent variable. Time, however, is an implied independent variable. The data are frequently presented in a graph with time along the horizontal axis and the dependent variable along the vertical axis. The reader’s attention focuses on the dependent variable and its changes or variations over time. Of course, the passage of time itself is not a variable. Rather, events and actions that take place during the passage of time may correlate to the variable graphed in the time series. However, using time allows us to understand the trend, up or down, in the variable of interest. Then investigators can ask questions about what is happening over time to cause the trend.
Looking at a Time-Series Study
Let us consider an example. Figure 2.1 shows two time-series lines—one represents the unemployment rate and the other the property crime rate. Property crimes are burglary, larceny-theft, motor vehicle theft, and arson. The data help illustrate several features of time series and show how to examine the relationship between two time series.12
Note that this example reports U.S. data for the entire country; the employment data are collected from a sample of U.S. households, and the property crime statistics are from individual law enforcement agencies. You could have done the same analysis on a single state, county, or city.
Page 9 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Table 2.3 Variations Within a Time Series
Variation Definition Long-term trends General movement of a variable, either upward or downward, over a
number of years Cyclical variations Changes in a variable that occur within a long-term trend; cycles recur
in one- to five-year intervals Seasonal variations Fluctuations traceable to seasonally related phenomena; e.g., holidays,
weather Irregular (or random)
fluctuations Changes that cannot be attributed to long-term trends, cyclical
variations, or seasonal variations
Now, what do you look for in this time series? Your eye should scan each line to identify the overall patterns and noticeable dips and rises in the line. You look for four types of variations within a time series that reports yearly or monthly data.13 These variations and their definitions are included in Table 2.3.
Long-Term Trends in the Data
Look first at the changes in the property crime rate:
• 1980–1984: Property crime rate decreases. • 1985–1991: Property crime rate increases. • 1992–2000: Property crime rate decreases. • 2001: Property crime rate increases. • 2002–2013: Property crime rate decreases.
Page 10 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Variables neither increase nor decrease indefinitely. Changes in social, economic, and environmental conditions may lead to a change in direction. Since 1991 the U.S. crime rate, including property crimes, has steadily dropped except for a small increase in 2001. Explanations for the decline have included the state of the U.S. economy, the increasing age of the U.S. population, and changes in the criminal justice environment (e.g., more police and more people in prison).14 The time series for the property crime rate from 1984 through 2004 shows two distinct long-term trends. From 1985 through 1991 the property crime rate increased. Since 1991 it has decreased. The only deviation was a 1 percent increase in 2001. Without 2002 data, the observer cannot tell whether the 2001 increase signified a change in the trend’s direction or an irregular fluctuation. The 2002 data indicate that the increase was an irregular fluctuation.
Cyclical Variations in the Data
Cyclical variations are regularly occurring fluctuations within a long-term trend that last for more than a year; frequently, cyclical patterns recur in one- to five-year intervals. In some cases cycles may be very regular. For example, the percentage of Americans voting reaches a peak every four years during presidential elections. A complete cycle is from “peak to peak” or “valley to valley.” The unemployment data from 1980 to 2004 have two cycles of approximately 10 years each. The data show long stretches when the unemployment rate kept decreasing. Between 1980 and 2012, despite increasing steeply at some points, periods of increasing unemployment rarely lasted more than three years. We can assume that political interventions brought about the change in direction. The cyclical variations, as illustrated by the unemployment rate, are more common than evenly spaced cycles. That is, up-and-down movements reoccur throughout a time series but not in a regular pattern of equal time periods.
Seasonal Variations in the Data
Seasonal variations describe changes that occur within the course of a year. Data must represent time intervals that recur within a year, such as days, weeks, months, or quarters. Seasonal variations include fluctuations traceable to weather, holidays, or similar seasonally related phenomena. The observed fluctuations occur within a single year and recur year after year. If we had included monthly data in Figure 2.1, you would have seen several fluctuations in any given year. Such fluctuations are considered seasonal variations only if a similar pattern is seen year after year. An administrator may use seasonal information to decide how to staff public facilities—for example, how many staff to hire for city parks and for how long to hire them (for 10, 12, or 14 weeks). A jail administrator finding that the number of people in jail is highest on Monday mornings can plan ahead for the increased demand. Ignorance of seasonal demands may result in erroneous conclusions. Imagine the disastrous consequences if a merchant were to assume that his December sales of toys marked a business upswing that would carry through to January and February. Think of the problems of the jail administrator who orders food for the week based on the number of inmates on Monday morning.
Page 11 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Irregular Fluctuations in the Data
Irregular fluctuations are variations not associated with long-term trends, cyclical variations, or seasonal variations. The irregular fluctuations may be the result of nonrandom or random movements. A nonrandom movement is brought about by a condition or set of conditions that can be identified and explain the variation. The conditions may be inferred from records of concurrent events. For example, negative publicity about how police handle crime reports may explain a drop in reported crimes, or a community may be struck by a natural disaster, which impacts employment. Another example is the increase in the unemployment rate from 4.9 percent in the fourth quarter of 2007 to 10.1 percent in the fourth quarter of 2009. If such factors are identified—for example, the United States experiencing a major recession in 2008–2009—we are likely to conclude that the variation was nonrandom. Otherwise, we may assume the variation is random. Random movements are unexplained variations that most often are relatively minor. However, many analysts would consider the large increase in the unemployment rate during that time to be more than minor.
The 1 percent increase in the property crime rate in 2001 was an irregular fluctuation that seems relatively inconsequential and random. A Google search suggested that the increase was largely ignored. Yet one item illustrated the hazards of prematurely assuming that a one-year shift marks the beginning of a longer trend. A research firm summarized the 2001 crime statistics and claimed confirmation for its prediction that beginning in 2000 the crime rate would increase. In fairness to the firm, it looked at more than the one data source. The researchers assumed that a weakening U.S. economy, increased spending on fighting terrorism, and the anticipated release of inmates reaching the end of their sentences would cause an increase in crime.15
For most administrative purposes, one simply needs to know the types of fluctuations and to recognize evidence of their occurrence in a table or a graph. Otherwise, you may misinterpret the ups and downs in a graphed time series and erroneously attribute changes in a time series to a specific event or administrative action.
It is important to draw a graph of the time series and analyze it visually. We will often see graphs with more than one variable, as is the case with Figure 2.1. However, for administrative purposes the clarity of time series diminishes if several variables are examined together. In Figure 2.1 the graph included two variables: property crime rate and unemployment rate. You could scan the graph visually and note where the two varied together and where they went in different directions. Between 1980 and 1988, the two rates seemed to diverge. From 1992 through 2006, they both appear to be declining. After 2006, they again appear to diverge. If more variables were placed on the graph, it would become increasingly difficult to follow and interpret the data.
Forecasting With Time-Series Data
Time-series data are used frequently in forecasting or in evaluating the effectiveness of a policy. In both cases statistical techniques take into account the fluctuations of a variable. For the most part, these techniques are beyond the scope of this text. Nevertheless, here are some helpful comments about forecasting:
1 Quantitative methods of forecasting that depend on time-series data work best for short- term forecasts (e.g., forecasting for up to two years). A parks director or jail administrator may use quantitative techniques to forecast park usage or jail occupancy. This information
Page 12 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
may help in making staffing decisions or ordering supplies. But forecasts made for longer periods are less likely to be accurate. A major problem is correctly forecasting a change in direction of the long-term trend.
2 Assuming that existing long-term trends may continue unchanged can work reasonably well, but the forecaster must be aware that rates of change eventually will vary and long- term trends will change direction. For example, a town whose population has grown by 10 percent each decade will at some time experience slower growth. An unanticipated slowdown has serious implications if public facilities are planned around the assumption of a much larger population.
3 Identifying long-term trends, cycles, and any seasonal variation is important in assessing the impact of a program or policy. Without knowing how the variable has been changing over time, one may mistakenly conclude that a change in a time series is the result of the program or policy.
4 Qualitative techniques are important for long-range forecasting. An administrator who wants to identify the types of demands that her agency will face in 10 years may wish to use focus groups or interviews to obtain opinions from citizens and experts.
Longitudinal Designs
Longitudinal designs collect information on the same cases or comparable cases for two or more distinct time periods. Time series are also longitudinal designs. However, we treated time series in a preceding section because of their specific characteristics and common usage.
Panel designs refer to longitudinal designs that gather data on the same individuals each time.16
Typically the cases are individuals, but cases can consist of families, organizations, or some other unit of analysis. The design combines some characteristics of the cross-sectional design and the time- series design. Because these studies can reveal which individual cases change between data collections, the researcher is able to determine the changes that take place within the group and establish a time order for variables. For example, a training director can organize unemployment data as a time series and note changes in the unemployment rate. However, unless a panel design is used, she cannot tell how individuals in this population have changed. That is, some people employed at one time may be unemployed at a later time; some unemployed at the earlier time may be employed later. Since a panel design follows the same individual cases over time, shifts by individuals from one condition (employed) to another (unemployed) can be measured.
Until the late 1960s, poverty was considered to be a relatively permanent characteristic of individuals. Repeated cross-sectional surveys showed the same proportion of Americans living in poverty, and so it was assumed that the same people were living in poverty each time. These studies showed almost no change from one year to the next in the distribution of income. However, a study of American families, the Panel Survey of Income Dynamics, found that roughly one-third of those who were poor one year were not poor the next and had not been poor the year before. The number of poor was stable because the number coming into poverty and the number leaving just about equaled each other.17
The way that information about the number of elderly receiving institutional care was obtained provides another example of the importance of panel studies. Federal, state, and local administrators have faced an increasing burden of providing care for elderly Americans. Cross-sectional samples of older persons taken at various times had consistently shown that about 5 percent of people age 65 and
Page 13 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
older were in nursing homes, hospitals, or other long-term care facilities. This often was interpreted to mean that any person 65 and older has only a 5 percent chance of being institutionalized. But researchers tracked a panel of individuals over several years and found that many people enter and leave long-term care facilities several times in their later years. At any moment, only 5 percent may be in nursing homes, hospitals, or long-term care facilities, but over a period of years at least 20 percent of those age 65 and older will spend some time in one or more of these places.18
Framingham, Massachusetts, Heart Study
Another example of a large-scale panel study is the Framingham, Massachusetts, Heart Study.19 In this study 5,209 residents of the town of Framingham between the ages of 30 and 62 were recruited to submit to a detailed medical examination and lab tests every two years. The information from this study is being used to issue advice about diet, exercise, and other factors related to health conditions. In 1971, 5,124 children and spouses of the original participants were recruited for the offspring study. In 2001, researchers began the third generation study, consisting of 3,900 of the original participants’ grandchildren. These follow-up studies have allowed further investigations of risk factors for health problems and identification of genes that contribute to various diseases. The Framingham Heart Study is continuing—new exams of participants and offspring took place in 2014, collaborative studies are continuing, and medical history updates are being collected on a regular schedule.20
A major problem of panel studies is in obtaining an initial representative sample of respondents willing to be interviewed at set intervals over an extended period of time. Panel members also may drop out of the study for one reason or another. This is called subject attrition or experimental mortality. Another difficulty is that when people are the cases, the repeated interviews and observations may influence their behavior. When interviewed repeatedly over a period of months or years, people may change the way they answer questions in order to be consistent from one time to the next.21
Cohorts
When studies follow groups of cases, we often refer to these groups as cohorts. A cohort consists of cases having experienced the same significant event at a specific time period. They may be individuals, organizations, or some other unit of analysis.22 The term cohort may refer to birth cohort: those cases born in a specific year or period. Cohorts may also be defined by the year they entered the study or by the occurrence of or exposure to a particular event such as the year of having graduated from college or having fought in a particular war. In a panel study, investigators obtain information on the same individuals each time. But a study using cohorts may be conducted in any number of ways. Note that not all studies involving cohorts are longitudinal studies. A different sample may be taken from the cohort group each time data are gathered, or a researcher may survey members of a cohort, such as residents of New Orleans, and compare the responses of those who relocated after two hurricanes with those who returned.23
Meta-Analysis Meta-analysis is a systematic technique to locate, retrieve, review, summarize, and analyze a set of existing quantitative studies.24 It is also defined as “the statistical synthesis of data from separate but similar (that is, comparable) studies, leading to a quantitative summary of the pooled results.”25
Page 14 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Researchers conduct meta-analysis to draw general conclusions from several empirical studies on a given program or policy; develop support for hypotheses that merit further testing; and identify characteristics of a program, its environment, or its clients that are associated with effectiveness.
Imagine you want to learn whether early childhood education programs for disadvantaged children, such as Head Start, are successful. You probably will begin by reviewing the literature. Assume you find 10 recent research articles examining the effectiveness of different programs. You identify similar and dissimilar findings among these studies. As you classify the findings, you may compare program clients, note how cases were chosen for analysis, note how program success is defined and measured, and record any unique features in the program or the research. If the findings are dissimilar, you may look for an explanation as to why they differ. Differences in program features or in the conduct of research may account for dissimilar findings. If the findings are similar, you may ask whether the program would seem as effective if the research methodology were different. Or you may question whether the program would be as effective with other groups or in other locations.
Proponents of meta-analysis argue that simple reviews of literature such as described here are ineffective for arriving at conclusions about research results, such as the effectiveness of Head Start. A researcher cannot summarize a large number of studies effectively, much less understand and interpret how the studies are integrated. If a subset of studies is selected for closer analysis, the reviewer’s selection of studies may be colored by his own biases. In reviewing even 10 studies, the researcher may be unable to avoid inconsistent judgments. He may focus on sampling problems in one study and errors in data analysis in another.26
Meta-analysis, on the other hand, enables investigators to review a large body of literature and to integrate its findings. Meta-analysis requires the researchers to focus on the same specific components of every study analyzed. If you read a description of a meta-analysis, you should be able to repeat the researchers’ steps and to come to the same conclusions. Unlike literature reviews, meta-analysis uses quantitative procedures in synthesizing the results of several studies.27
Conducting a Meta-Analysis
In conducting a meta-analysis, investigators record the same information from each study. Their goal is to identify hypotheses that are supported in study after study. In doing this they have to eliminate alternative explanations, such as chance, which may account for the observed relationships. The analysts record the statistical information reported on the dependent variables of interest and the statistical relationships between independent and dependent variables. They also record information on the study itself, such as the dates of data collection and of publication; who the research subjects were; how the subjects were selected; the research design; evidence that research involved sound measures; and the type of publication (e.g., book, academic journal, or unpublished paper).
Once the data are collected, the investigators integrate the findings to create one database. Integrating study findings involves quantifying specific dependent variables. Analysts also identify
relevant independent variables and incorporate them into a common database. To illustrate the meaning of “integrating the findings,” we consider two examples:
1 In one meta-analysis, analysts examined 261 citizen surveys. They identified survey questions that asked people to rate specific urban services such as trash collection and categorized the questions by the type of service being rated. Neither the question wording nor the possible responses were identical. The analysts ignored variations in question wording
Page 15 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
and created a common scale so that responses to similar questions from different cities could be combined. After the data were combined, the investigators observed that citizens in cities throughout the United States gave the highest ratings to arts programs and public safety and the lowest ratings to planning.28
2 The second meta-analysis examined studies on survey response rates. Analysts identified 115 studies to evaluate strategies to improve the response rate for mail surveys published between 1940 and 1987, with response rate as the dependent variable. Across the studies, the analysts identified 17 different independent variables. Each of these variables had been tested in at least three studies. After detailed analysis, the investigators identified two factors that increased response rate to mail surveys: including a cover letter with an appeal to respond and keeping the questionnaire short (fewer than five pages).29
Meta-analysis allows one to “see” a body of research rather than focus on the quality of individual studies. A researcher may be skeptical if a hypothesis is supported only in poor-quality studies. On the other hand, findings may have even greater credibility if a hypothesis is supported no matter the quality of a study’s methodology. Information on the dates of data gathering and publication help identify trends and changes in them. For example, the investigators in the response-rate study found that beginning in the mid-1970s, preliminary notification of subjects improved response rate.
Limitations of Meta-Analysis
A major problem in meta-analysis is locating a set of studies. At a minimum the topic must have been of interest long enough for a research history to develop. Little or no research may exist on a current “hot” research topic. The researcher conducting a meta-analysis wants to include all appropriate studies or a representative sample of such studies in his analysis. Consequently, he must do a relatively exhaustive literature search. If he selects articles from a few journals or from a short time period, he runs the risk of working with a biased sample.
Each article selected does not have to include exactly the same dependent variables, nor must it study the same relationships. Of the 261 cities whose citizen surveys were analyzed, 70 percent rated their police. Less than 25 percent rated animal control or street lighting. The independent variables used in the response-rate study varied from study to study.
Critics of meta-analysis point to the problems introduced by biased article selection. Let’s go back to our early childhood education example in which 10 publications were studied. First, we mentioned the problem of bias introduced if the studies analyzed do not represent the larger body of studies conducted on early childhood education. A critic may raise the related question of the “file drawer.” The argument goes as follows: If many studies of a subject are done, chance alone will cause some hypotheses to be supported and the research reporting them to be published. Analysts who include only published studies may have a biased sample because many other unpublished studies on the same topic may be stashed in researchers’ file drawers. Researchers put studies aside when preliminary data fail to support the hypotheses and the studies seem to be going nowhere. Social scientists in recent years have discussed the reasons why researchers should publish null findings, but doing so is still arguably rare.30 A statistical solution to this problem requires the analyst to calculate how many unpublished papers have to exist for the findings to be contradicted.31
Another criticism of meta-analysis has been termed the “apples and oranges” problem. The critics argue against combining dissimilar studies. Consider the citizen-survey data in which findings based
Page 16 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
on different question wording and different response patterns were grouped together to create a variable. Gene V. Glass, a major proponent of meta-analysis, argues that these critics want replication of research, not comparison of similar findings. Meta-analysis is intended to mine information from related, but not identical, studies. Furthermore, he argues that combining different studies is not much different from combining responses from different subjects.32
Meta-analysis may seem deceptively simple. Yet it is time-consuming to identify the appropriate studies and to conduct the data analysis. Understanding the quantitative procedures used to synthesize results requires statistical knowledge. Readers who wish to try their hand at meta-analysis should be familiar with the statistics discussed in this text. Then they should review one or more of the books included in the Recommended for Further Reading section at the end of this chapter. They should read several meta-analysis studies for examples. Given the amount of research literature available and the difficulty and expense of researching some topics, a good meta-analysis may be a very efficient alternative to large-scale original research and increasingly feasible to conduct given advances in technology and data access.
Qualitative Research and Designs to Fill in the Details Quantitative studies typically involve many cases and many variables that can be measured in a predetermined, specific way. The data are numeric and can be summarized numerically. Since an important goal of quantitative studies is to compare cases on several variables, factors unique to individual cases are not included and information about context is often ignored. For example, investigators use the designs described thus far to obtain information on a standard set of items for a large number of cases, as with cross-sectional studies, or information on as few as a single case for many time periods, as with time series.
While using quantitative research designs is important, these designs are not suitable for obtaining detailed information about the context in which events or behaviors occur, nor do they allow flexibility in the type of data obtained from case to case in the same study. To obtain this detailed information, investigators use more qualitative and fewer quantitative approaches. Qualitative research methods have long been important in basic disciplines as well as in applied areas such as administration. Studies using a qualitative research approach typically obtain more in-depth, detailed information on fewer cases than do studies using more quantitative designs. Two important qualitative designs, described later, are the case study and the focus group.
Qualitative Research
Qualitative research methods have received increased attention in recent years from both practitioners and researchers.33 Qualitative research produces information or data difficult or impossible to convert into numbers. The qualitative study is defined by its extensive use of such information, its preference for developing full information on relatively few cases, and its consideration of the unique features of each case. Researchers may draw on both quantitative and qualitative methods in conducting any one study.
Qualitative studies may include information on the unique features and the environment of each case. They describe specific features of each individual, organization, jurisdiction, or program. Qualitative studies may involve extensive fieldwork; the researcher goes to where the cases are located and obtains information on them in their natural setting.34 In this way the researcher does not
Page 17 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
attempt to manipulate any aspect of the situation being studied but reports it as it is. Nevertheless, the qualitative researcher’s background and personality influence data collection and interpretation. The researchers use their experiences and insights to design a study and to interpret the findings. A researcher’s interactions with subjects affect what he is told and what information he is given.
In qualitative studies the researcher usually works with a flexible design. Although the studies may have a clearly defined methodology and plan of action, the researcher usually has great flexibility. He may alter the design as the research progresses. Typically, he uses several sources of information because multiple sources give a fuller picture of a case and its setting and help to verify other information.
Researchers using qualitative techniques need different skills than those required when using quantitative designs. An interviewer in a quantitative study receives a list of questions that she asks every respondent. All other interviewers would use the same set of questions and ask them in the same way. In a qualitative study, the interviewer may have a suggested set of questions but asks them as the situation dictates. Based on the response to one question, the interviewer asks another question; she needs to ask the question, listen, interpret, and phrase a proper follow-up question.35
The researcher using qualitative methods must be able to record information accurately, write clearly, differentiate trivial from important details, and draw appropriate conclusions from the information. Since data from qualitative studies tend to be descriptions, observations, and responses to interview questions, a large amount of information is obtained. To make sense out of it may take time and effort. Rather than doing statistical analysis of numerical data as in quantitative studies, the researcher looks for words, themes, and concepts in the analysis of qualitative data. These become the data from which the researcher draws conclusions, answers research questions, and develops additional hypotheses.36
Case Studies
Case studies examine in some depth persons, decisions, programs, or other entities that have a unique characteristic of interest. For example, a case study may be designed to study women in nontraditional jobs, a new approach to budgeting, or a high school health clinic. Authors of one text on the design of research state that case studies are essential for description and thus fundamental for social science. They also maintain that without good description explanation is pointless.37 Although case studies may be either qualitative or quantitative, here we only present the case study as a qualitative design.38
Except for studies that involve a single case within an administrator’s jurisdiction, administrators seldom initiate or participate in the design of case studies. Even the single case study may be a by- product of someone else’s research needs. For example, some state and local administrators seek out university students to conduct case studies. The studies give the students a “real-world” experience, and the administrators learn more about their agency and its programs.
Case studies are the preferred research strategy if an investigator wants to learn the details about how something happened and why it may have happened. Administrators may conduct a case study to investigate the following:
• a program or a policy that has had remarkable success (or one that has failed) • programs or policies that have unique or ambiguous outcomes • situations in which actors’ behavior is discretionary
Page 18 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Although historical events can be the focus of a case study, the case should be contemporary so that the investigator has access to information. Ideally the investigator will have direct access to the people, program, and practice details involved. Whereas analysts implementing cross-sectional or time-series designs may never contact program administrators, employees, or clients, an investigator conducting a case study cannot be so detached.
Information From Multiple Data Sources
One of the hallmarks of a case study is the combination of several different sources of information. The sources of information used in case studies include documents, archival information, interviews, direct observation, participant observation, and physical artifacts.
The inclusion of information from multiple sources is a major strength of case studies. First, each data collection strategy affects the types of questions a researcher can answer. For example, from direct observation an investigator learns how people behave. From interviews he hears their explanations of their behavior. The two sources of information give a more complete picture than either piece alone.
Second, the investigator can corroborate information gained from one source with information gathered from another source. For example, staff in a crisis intervention center may report a high level of community support. The claim will have greater credibility if an independent review of agency records confirms extensive community support.
Requiring information from multiple sources also is a drawback of case studies. Typically, different information sources are studied using different research techniques. For example, interviewing skills are needed for face-to-face meetings with subjects, questionnaire design skills are needed to conduct a mail or electronic survey, and content analysis skills are needed for archival research. Most of us are skilled in only one or two research techniques. Thus a case study may require a research team or else suffer from unevenness. Administrators may find that they and their staff lack both the training and the time needed to do effective case studies.
Each research technique and data collection strategy takes time to design, pretest, and carry out. Incorporating multiple data sources and employing different techniques and possibly multiple researchers demands time, expertise, and energy. Consequently, researchers who conduct case studies often find that studying multiple cases is impractical because of the effort and resources required.
Focusing on the Components of a Case
The case study may focus on the case as a whole (all of its components) or just on certain components. For example, to study a state pretrial release program, the investigator may look at the program in its entirety: why it was developed, how it was initially organized, how its organization was changed since it began, why those changes were made, how defendants are chosen for pretrial release, and how much discretion staff and judges have in making pretrial release recommendations. Alternatively, the investigator may focus on the implementation of the program in one location or in a few locations. In the first instance, the program is the case. In the second, each specific location’s program could be a separate case.
Most administrative case studies seem to focus on components. A case study may focus on one provision of a program rather than evaluating the entire program. For example, in a study of a pretrial release program, “successful” defendants who showed up for scheduled court appearances were compared with “unsuccessful” defendants who failed to appear. This is a limited case study. Limiting
Page 19 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
the study to the analysis of a single component does not take advantage of the strength of the case study design. A more valuable analysis would be to examine the circumstances of what promoted success, perhaps by exploring work, education, family support systems, and other variables in an in- depth manner.
Case studies may be conducted on a single case or on a set of similar cases. Each approach (using a single case study of a program or comparing two or more programs through case studies of each) would be valuable in determining the effectiveness of the program. An individual case study of a phenomenon can provide research depth. Multiple cases of a phenomenon provide research breadth. Ideally, researchers would like to have both depth and breadth. In reality, limited resources mean researchers often have to choose between the two, striking a balance that is appropriate for the task at hand. In some cases, this means either sacrificing some contextual information to include more cases or sacrificing having some more examples in order to explore just one or two in much greater detail.
Because of the potential value of case studies and the rich details they provide, you may want more information about them. If you read a case study, you may gain greater insight into how to approach or solve a problem. Practically speaking, administrators’ direct experience with case studies may be limited to their professional reading. Case studies may alert you to new management techniques, programs to solve chronic problems, or strategies to improve the quality of agency or community life. If the case study is well done, you should have a good idea of whether you could implement a similar solution. Even if the case study documents a failure or otherwise does not apply, it may spur you to think more about your work environment and responsibilities. A creative administrator may read a case study, contact the researcher or program administrator to learn more details, and then be prompted to implement a new program or strategy.
What Makes a Quality Case Study?
Robert Yin carefully distinguishes between the case study as a type of design on the one hand and the data collection typically conducted in a case study on the other.39 He also implies that most people incorrectly assume that any qualitative study is a case study. Although case studies tend to be qualitative, Yin recommends an approach for case studies that follows the scientific method. This approach entails the researcher stating a problem; formulating a research question, objective, or hypothesis; identifying the case to be studied; planning the data collection; collecting the data; analyzing the data; and writing a report.
You may wonder whether a particular case study is anything more than a good story.40 Sometimes it may not be. The study is only as good as the objectivity and training of the investigators.
In certain instances, particular case studies are intended to be only preliminary or exploratory. An investigator conducting an exploratory case study will have a research question but appropriately may decide to forgo formulating initial hypotheses. Instead, he will develop hypotheses from the case study once it is completed and draw generalizations from later studies. The exploratory case study serves as the basis for establishing new research questions, new hypotheses, and a continuing research agenda. Then, a separate case study will serve to thoroughly “test” that hypothesis.
Case-study researchers need to be particularly careful to follow sound research practice.
• They must decide what kind of and how much data would support a hypothesis before collecting the data. Otherwise, the natural wish of researchers to have the data support their hypotheses may bias the collection and interpretation of the evidence.
Page 20 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
• They should also be careful about who conducts the case study. A researcher with limited knowledge of the organization and limited resources may not contribute much new information or insight. Yet a researcher who works within the organization may reflect the organization’s biases.
• As part of the design, researchers must decide what constitutes a case. Doing so may not be easy.41 Deciding on what is the case to be studied, for example, may be a problem if the case involves a program. Consider an agency program to aid abused children. The agency may have a program for abused children specifically, but it may also have services for abused mothers and for families with a potential for abuse. Should these services be included in the case study? The program may have evolved from an earlier program. At what point should the case start? Should the case study include the board of directors, clients, or others in the community? The investigator must answer these questions in order to limit, or set boundaries around, the case study. An ill-defined case can distort data, just as can a case without predetermined criteria for judging whether a hypothesis is supported. A poorly defined case can also allow the researcher to wander from her original research question.
As an administrator, you may want to evaluate the quality of the design so you can determine whether you are reading more than just a good story.
1 Look for evidence that the investigator had a specific, answerable research question. 2 Assess whether the investigator had a model before starting the study. 3 Determine whether the investigator decided the specific boundaries of the case. 4 Be sure that the case study’s procedures from design through implementation are
thoroughly documented, including any potential biases on behalf of the researcher.
Case studies may involve the investigator intensely in the case and require that the researcher interpret qualitative information. Consequently, case studies are hard to replicate, and great care must be taken to document what was done, how, and why. Transparency of a case study is important, as in any form of research, so that the strengths and limitations of the work are evident.
Focus Groups Focus-group methods use group interviews to obtain qualitative data. Researchers have long used group interviews to save time and money by getting a number of people together to provide information.42 Focus-group procedures have evolved in recent years and include a set of characteristics that distinguishes them from other group techniques. Although focus-group research is qualitative, often it is used in conjunction with more quantitative procedures. Survey researchers use focus groups to generate and test the items to be used on a questionnaire or survey instrument. Focus groups also are used to elaborate on data collected in surveys.
Focus-Group Discussions
Focus groups are semi-structured discussions by small groups of participants about a common topic or experience.43 They are useful in obtaining information that is difficult to obtain with other methods. Investigators use a focus-group interview to get in-depth information about and reactions to a
Page 21 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
relatively small number of topics or questions—typically fewer than 10 questions and often around five or six.
A study using focus groups usually includes several group interviews. Each group is relatively homogeneous with respect to its participants’ background characteristics. This homogeneity helps to ensure that individuals will not be afraid to express their feelings about the issue at hand.44
Homogeneity also helps guarantee that the group has a common experience to discuss, whether it is opinions about marketing material, crime victims’ perceptions of police response, or need for a new service. What distinguishes focus groups is the presence of group interaction in response to researchers’ questions.
The focus-group discussion is led by a moderator who should be an experienced interviewer skilled at group facilitation. The moderator also needs to be familiar with the questions and the purpose of the study. The moderator asks the questions and guides the discussion to ensure the participation of all members of the group. The hallmark of focus groups is the explicit use of group interaction to produce data and insights that would be less accessible otherwise.
Focus groups constitute an accepted research method.45 Not every group using discussion techniques is a focus group, however. Focus groups have specific defining characteristics; they are created by research teams for well-defined purposes. Focus groups rely on the strengths of qualitative methods, including exploration and discovery; understanding things in depth and in context; and interpreting why things are the way they are and how they got that way. To serve these purposes, focus groups, like other qualitative methods, require a great deal of openness and flexibility.46
Although the overall process of focus-group research usually is highly structured, the interviewing must be flexible. The information obtained is qualitative and usually voluminous.
The researcher must:
• identify themes • find answers to questions • summarize the discussion of the group members
Having well-designed questions based on a clearly defined purpose will facilitate the analysis and use of the data. Additionally, the researcher must do the following:
• carefully plan the session prior to assembling the group(s) and the interviewing • clarify the purpose of the study and discuss it with colleagues • be clear about what information is needed, how it will be gathered (audio or video
recordings, notes, transcription) and why, who will use it and how, and its confidentiality or anonymity status
• make available a written plan for the entire project, including a schedule and budget, which should be developed in advance
Focus Groups in Public Administration
Focus groups are used in the public sector in many ways. A manager may find focus groups useful for needs assessments—to identify the services clients need, where they are best delivered, and which are not appropriate, and to develop a better understanding of concerns as perceived by relevant participants. A public organization might use focus groups to address such questions as: What do various citizen groups perceive as the most important problems of the jurisdiction? Why?47
Page 22 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Focus groups aid in program design and planning when the objective is to develop ways to deal with a problem or situation. What are the possibilities for dealing with a problem? What options are feasible? How would the client group receive them? Focus groups can be used to evaluate existing programs and to answer such questions as: How well is a program working? Are clients dissatisfied? Why? What changes will be acceptable? How well do clients think that they would respond to a change if one were instituted? As these situations suggest, the manager would probably want to get more explanation and discussion concerning responses to questions than generally would be possible in a large-scale survey.
Focus groups have been used with:
• citizens to assess the quality of city services and to identify priorities for future activities • groups of clients of rehabilitation programs to assess current services and explore which
types of programs are most effective • top-level city administrators to obtain input to a planned survey of citizens • employees to find out about and improve their working environment48
There is widening interest in participatory focus groups, wherein participants are involved in data analysis and interpretation.49 Although focus groups often are used in exploratory studies and as adjuncts to other forms of research, they also can be used as the major procedure without being supplemented by quantitative methods.50
A health needs assessment in a rural county of North Carolina is an example of the use of focus groups in combination with other methods. Administrators and researchers chose focus groups as a nonthreatening way to obtain perceptions of health, health-related behaviors, and service delivery in the county. The researchers were advised specifically not to use surveys: “These people have been surveyed to death. They’re tired of being asked if they are poor.”51
Case Study: County Focus Group
The county was fragmented by its geography, numerous neighborhoods, and variety of agencies and schools. Researchers conducted 40 focus groups ranging in size from 3 to 37 individuals, greatly exceeding the optimal size of 8 usually recommended.52 The sessions were not recorded. A two-person team of facilitators conducted the sessions, with one team member moderating and the other taking notes. The focus groups provided important data, involved residents in the project, and helped legitimize the research and its resulting interventions. Members of the groups were asked six questions addressing health problems and potential solutions. Participants were also given a copy of the questions to provide an opportunity for a written, private response. This proved to be useful. The focus-group data were combined with statistics on disease rates; mortality; causes of death and injuries; and with demographic, economic, and social data supplied by state and federal agencies to plan new services and to improve existing services and facilities. Cultural differences among residents and differences in values between the more traditional, longtime residents and the younger health care professionals were identified in the process.53
Page 23 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Summary In this chapter we have discussed designs that guide studies describing the occurrence of a variable or the relationship between variables. These research designs help a researcher to decide when to make observations and how many observations to make. The design selected may depend on the nature of the dataset or the problem-solving skills of the administrator.
Cross-sectional and time-series designs are particularly effective and efficient. Either singly or in combination, they provide valuable information to administrators, legislators, and the public. The data derived from both designs may be organized to communicate information quickly through graphs or tables. Cross-sectional designs show relationships among variables of interest at one point in time. Cross-sectional designs often call for the collection of many pieces of data. Innumerable investigators may access, manipulate, and analyze the resulting database according to their individual interests.
Time-series designs demonstrate long-term, cyclical, and seasonal trends and identify irregular fluctuations in the occurrence of a variable. A time-series design requires an investigator to collect data on a measure at regular intervals. To distinguish between random and nonrandom irregular fluctuations, a historical record of events that can affect the occurrence of the variable is needed. Time-series designs help a researcher to describe a variable over time. They may forecast changes in a variable and assist in making operational decisions. For example, knowing the seasonal variations in arrests allows a court administrator to make staffing assignments.
With longitudinal designs, investigators follow individual cases and obtain information on them for several time periods. These designs allow investigators to measure the changes taking place within a group as well as to measure the change in a group characteristic over time.
Meta-analysis allows researchers to assemble a set of similar studies, use their data to form a single dataset, and determine what, if any, general hypotheses have been supported consistently. The major difficulty in performing meta-analysis is to identify a representative set of studies. It behooves the researcher to search through many sources to identify appropriate published and unpublished research. The conclusions reached through meta-analysis require thorough statistical analysis in order to provide evidence either supporting a hypothesis or arguing that it could have occurred by chance.
Several qualitative approaches provide useful information for administrators. Case studies provide detail that shows how something happened and why it happened. Case studies usually include information on the natural surroundings of events. One of the strengths of case studies is that they can involve multiple sources of data. Because of this requirement for multiple data sources, we suspect that most administrators and their staff have neither the time nor the resources to conduct case studies.
Nevertheless, administrators may be interested in case-study findings and how to use them should a particular case study turn out to be more than just a good story. To determine the quality of a case study, the administrator looks for evidence that the investigator had a research question and model before collecting data, the case was clearly defined—which can at times be difficult—and the case- study procedures were documented thoroughly.
Focus-group interviewing is used to obtain detailed information from a small group of individuals. A moderator asks a well-developed set of questions and leads the discussion in the focus group. The responses of the participants to the questions and to each other’s comments provide data difficult to obtain with other methods. Focus groups often are used to supplement more quantitative studies, such as those using cross-sectional designs, and have many uses in the public sector.
Page 24 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Figure 2.2 classifies the various designs discussed in this chapter. Three of the major types—cross- sectional designs, longitudinal designs, and qualitative research—in turn include more than one subtype. Those are also illustrated in the diagram.
Notes 1 Sometimes the term research methodology refers to the theoretical study of research methods. 2 See the following for a more extensive discussion of observational data and designs producing observational
data: Paul Rosenbaum, Design of Observational Studies (New York: Springer, 2009); William Cochran, Planning and Analysis of Observational Studies, ed. Lincoln Moses and Frederick Mosteller (New York: John Wiley & Sons, 1983).
3 M. Specter, “Suicide Rate of Jail Inmates Rising Sharply,” Washington Post, February 18, 1985, sec. A1, 18. For recent data, see Bureau of Justice Statistics at http:www.bjs.gov.
4 Gary King, Robert Keohane, and Sidney Verba, Designing Social Inquiry: Scientific Inference in Qualitative Research (Princeton, NJ: Princeton University Press, 1994), 44.
5 We discuss the issue of causality in cross-sectional and time-series designs in Chapter 3. Hellevik discusses causal analysis with survey data in Introduction to Causal Analysis: Exploring Survey Data by Crosstabulation, Contemporary Social Research Series, no. 9 (London: Allen & Unwin, 1984). Robert K. Yin discusses using case studies to infer causality in Applications of Case Study Research, 3rd ed. (Thousand Oaks, CA: Sage, 2011) and in Case Study Research: Design and Methods, 5th ed. (Thousand Oaks, CA: Sage, 2013).
6 For example, see ICPSR: Inter-university Consortium for Political and Social Research at http://www.icpsr.umich.edu. Local governments often conduct surveys of residents to assess opinions regarding services and other matters. The Belle County Survey accompanying this text is such a survey. The data are in Belleco.xls and Belleco.sav. The data in the County Data File, also accompanying the text, are cross-sectional.
7 B. Braddy, An Evaluation of CETA Adult Training Programs in North Carolina Division of Employment and Training (Raleigh, NC: Department of Political Science and Public Administration, May 1983). Unpublished manuscript.
Page 25 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
8 See this article reviewing numerous evaluations of job-training programs: Robert J. LaLonde, “Employment and Training Programs.” Available at http://www.nber.org/chapters/c10261.pdf. Several of these evaluations compare participants with job training to those receiving no training.
9 See J. C. Bailar III, “When Research Results Are in Conflict,” New England Journal of Medicine, October 24, 1985, 1080–1081. Summary of research results reported in “Studies Reach Opposite Conclusions about How Estrogen Pills Affect Heart,” Raleigh News & Observer, October 24, 1985, p. 1A.
10 Certain biochemical chemical research suggested that the estrogen improves cholesterol levels, decreasing the probability of heart disease. Other biochemical research also suggested that estrogen increases blood clotting, increasing the probability of heart disease.
11 “WHI Study of Younger Postmenopausal Women Links Estrogen Therapy to Less Plaque in Arteries.” Available at www.nhlbi.nih.gov/news/press-releases/2007/whi-study-of-younger-postmenopausal-women- links-estrogen-therapy-to-less-plaque-in-arteries.
12 A similar graph for the years 1960 through 1980 accompanied Nicholas D. Kristol’s article, “Scholars Disagree on Connection between Crime and the Jobless,” Washington Post, August 7, 1982, which discusses further the relationship between the two variables.
13 Time-series analysis may involve much smaller intervals (e.g., hours and days). These intervals will show analogous patterns. For the sake of clarity we have focused on longer time intervals; we assume that readers who are dealing with other intervals will be able to interpret their data.
14 Steven D. Levitt, “Understanding Why Crime Fell in the 1990s: Four Factors That Explain the Decline Plus Six That Do Not,” Journal of Economic Perspectives 18 (Winter 2004): 163–190.
15 Rosemary J. Erickson and Kristi M. Balzar, Summary and Interpretation of Crime in the United States, 2001 Uniform Crime Report, Federal Bureau of Investigation, Released October 28, 2002 (San Diego: Athena Research Corp., 2003).
16 Scott Menard, Longitudinal Research, 2nd ed. (Thousand Oaks, CA: Sage, 2002), 2. In practice, panel designs may be referred to as “longitudinal” designs. Someone using the data will want to check the documentation. Those interested in health and social aspects of aging will find valuable information in the following: Nathalie Huguet, Shayna D. Cunningham, and Jason T. Newsom, “Existing Longitudinal Data Sets for the Study of Health and Social Aspects of Aging,” in Longitudinal Data Analysis: A Practical Guide for Researchers in Aging, Health, and the Social Sciences, ed. Jason T. Newsom, Richard N. Jones, and Scott M. Hofer (New York: Routledge, 2012), chapter 1, 1–42. The authors provide extensive detail on 12 major datasets.
17 Julian Simon and Paul Burstein, Basic Research Methods in Social Science, 3rd ed. (New York: Random House, 1985), 161–162. See the original study by Mary Jo Bane and David T. Ellwood, “Slipping Into and Out of Poverty: The Dynamics of Spells” (Harvard University, 1983, mimeo.); revised and reissued as Working Paper no. 1199, National Bureau of Economic Research, Cambridge, MA, September 1983. See a follow-up study: June A. O’Neill, Laurie J. Bassi, and Douglas A. Wolf, “The Duration of Welfare Spells,” The Review of Economics and Statistics 69, no. 2 (May 1987): 241–248. See also David C. Ribar, Marilyn J. Edelhoch, and Qiduan Liu, “Food Stamp Participation among Adult-Only Households,” Southern Economic Journal 77, no. 2 (October 2010): 244–270.
18 Morton Hunt, Profiles of Social Research (New York: Russell Sage Foundation, 1985), 209. 19 Up-to-date information on the study and its design is at http://www.clinicaltrials.gov/ct/show/NCT00005121.
Currently, the web site for this study is at http://www.framinghamheartstudy.org. To locate articles reporting analysis of the Framingham data, consult the Science Citation Index and the Social Science Index.
20 See http:www.framinghamheartstudy.org/participants/fhs-news.php.
Page 26 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
21 Robert F. Boruch and Robert W. Pearson discuss in detail the advantages and disadvantages of panel designs. See “Assessing the Quality of Longitudinal Surveys,” Evaluation Review 12 (1988): 3–18. Menard’s Longitudinal Research, chapter 4, also covers the issues in panel designs. See http://www.Philanthropy.iupui.edu for extensive information on the Philanthropy Panel Study (PPS). This study obtained data on the same households every two years beginning in 2001.
22 Norvelle Glenn, Cohort Analysis, 2nd ed. (Thousand Oaks, CA: Sage, 2005). 23 See Glenn, Cohort Analysis, for information on analyzing panel data. 24 David Cordray and Robert Fischer, “Synthesizing Evaluation Findings,” in Handbook of Practical Program
Evaluation, ed. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (San Francisco: Jossey-Bass, 1994), 202.
25 Robert F. Boruch and Anthony Petrosino, “Meta-Analyses, Systematic Reviews, and Evaluation Syntheses,” in Handbook of Practical Program Evaluation, 3rd ed., ed. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (San Francisco: John Wiley & Sons, 2010), chapter 22, 531–553.
26 For more details on the limitations of traditional reviews of the literature, see Frederic M. Wolf, Meta- Analysis: Quantitative Methods for Research Synthesis, Quantitative Applications in the Social Sciences, no. 59 (Beverly Hills, CA: Sage, 1986), 10–11; and John E. Hunter, Frank L. Schmidt, and Gregg B. Jackson, Meta-Analysis: Cumulating Research Findings Across Studies, Studying Organizations, no. 4 (Beverly Hills, CA: Sage, 1981), 129–130. Geoff Cumming, Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (New York: Routledge, 2012), includes three chapters on meta-analysis. He discusses history, approaches, and tools of meta-analysis. See pages 181–262.
27 Cordray and Fischer, “Synthesizing Evaluation Findings,” 200–206. See also Julia H. Littell, Jacqueline Corcoran, and Vijayan Pillai, Systematic Reviews and Meta-Analysis (New York: Oxford University Press, 2008).
28 Thomas I. Miller and Michelle A. Miller, “Standards of Excellence: U.S. Residents’ Evaluations of Local Government Services,” Public Administration Review (November–December 1991): 503–514.
29 Francis J. Yammarino, Steven J. Skinner, and Terry L. Childers, “A Meta-Analysis of Mail Surveys,” Public Opinion Quarterly (Winter 1991): 613–639.
30 Sharon Paynter and Maureen Berner, “Organizational Capacity of Social Service Organizations,” Journal of Health and Human Services Administration 37, no. 1 (Summer 2014): 111–145. The authors searched the literature for factors reported to relate to the success of large nonprofit organizations. They then applied these to small volunteer nonprofit organizations and found that the results did not hold.
31 Robert Rosenthal, Judgment Studies: Design, Analysis, and Meta-Analysis (New York: Cambridge University Press, 1987), 223–225.
32 Gene V. Glass, Barry McGaw, and Mary Lee Smith, Meta-Analysis in Social Research (Beverly Hills, CA: Sage, 1981), 220.
33 For example, see Michael Q. Patton, How to Use Qualitative Methods in Evaluation (Newbury Park, CA: Sage, 1987); and Peter J. Haas and J. Fred Springer, Applied Policy Research: Concepts and Cases (New York: Garland Publishing, 1996).
34 John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 4th ed. (Thousand Oaks, CA: Sage, 2014).
35 Sharon L. Caudle, “Using Qualitative Approaches,” in Handbook of Practical Program Evaluation, ed. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (San Francisco: Jossey-Bass, 1994), 69–95. Also see Herbert J. Rubin and Irene S. Rubin, Qualitative Interviewing: The Art of Hearing Data, 3rd ed. (Thousand Oaks, CA: Sage, 2012).
Page 27 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
36 To learn more, see Matthew B. Miles, A. Michael Huberman, and Johnny Saldaña, Qualitative Data Analysis: A Methods Sourcebook, 3rd ed. (Thousand Oaks, CA: Sage, 2013).
37 King, Keohane, and Verba, Designing Social Inquiry, 44. 38 Robert Wood Johnson Foundation, Qualitative Research Guidelines Project, Using Qualitative Methods in
Health Care Research. Available at http://www.qualres.org; Robert E. Stake, The Art of Case Study Research (Thousand Oaks, CA: Sage, 1995).
39 Yin, Applications of Case Study Research, 3rd ed., chapter 1. Also see Yin, Case Study Research: Design and Methods, 5th ed., which is especially recommended for gaining information on how to do case studies.
40 For a view of how managers obtain knowledge through the use of stories, see Ralph P. Hummel, “Stories Managers Tell: Why They Are as Valid as Science,” Public Administration Review 51 (January–February 1991): 31–41. See also Richard A. Krueger, “Using Stories in Evaluation,” in Handbook of Practical Program Evaluation, 3rd ed., ed. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (San Francisco: John Wiley & Sons, 2010), chapter 18, 404–423.
41 See Charles C. Ragin and Howard S. Becker, What Is a Case? Exploring the Foundations of Social Inquiry (Cambridge, England: Cambridge University Press, 1992). Also see Yin, Case Study Research: Design and Methods, 5th ed., chapter 2.
42 Robert K. Merton, “The Focused Interview and Focus Groups,” Public Opinion Quarterly 51 (1987): 550 –566; Robert K. Merton, Marjorie Fiske, and Patricia Kendall, The Focused Interview, 2nd ed. (Glencoe, IL: Free Press, 1990); Richard A. Krueger and Mary Anne Casey, Focus Groups: A Practical Guide for Applied Research, 5th ed. (Thousand Oaks, CA: Sage, 2015), 7–15.
43 Ralph S. Hambrick Jr. and James H. McMillan, “Using Focus Groups in the Public Sector,” Journal of Management Science and Policy Analysis 6 (Summer 1989): 44.
44 Hambrick and McMillan, “Using Focus Groups in the Public Sector,” 48. 45 David Morgan, The Focus Group Guidebook (Thousand Oaks, CA: Sage, 1997), 29. 46 Morgan, The Focus Group Guidebook, 31. 47 Hambrick and McMillan, “Using Focus Groups in the Public Sector,” 44–45. 48 These and other examples are cited in Hambrick and McMillan, “Using Focus Groups in the Public Sector,”
46–47. Also see Christopher McKenna, “Using Focus Groups to Study Library Utilization,” Journal of Management Science and Policy Analysis 7 (Summer 1990): 316–329.
49 Krueger and Casey, Focus Groups. 50 David Morgan and Richard Krueger, “When to Use Focus Groups and Why,” in Successful Focus Groups:
Advancing the State of the Art, ed. David Morgan (Newbury Park, CA: Sage, 1993), 3–19. Also see Debra L. Dean, “How to Use Focus Groups,” in Handbook of Practical Program Evaluation, ed. Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (San Francisco: Jossey-Bass, 1994), 341.
51 Thomas Plaut, Suzanne Landis, and June Trevor, “Focus Groups and Community Mobilization: A Case Study from Rural North Carolina,” in Successful Focus Groups: Advancing the State of the Art, ed. David Morgan (Newbury Park, CA: Sage, 1993), 205.
52 Plaut, Landis, and Trevor, “Focus Groups and Community Mobilization,” 206. 53 See “Focus Group Report,” an example of a focus group study accompanying this text.
Terms for Review
research methodology research designs
Page 28 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
cross-sectional design longitudinal design time-series designs long-term trends cyclical variations seasonal variations random variations nonrandom variations quantitative studies qualitative studies panel design cohort case study focus group meta-analysis observational data experimental data observable variables unobservable variables
Questions for Review The following questions should indicate whether you have a basic competency in this chapter’s material.
1 What is the value of a research design? 2 a List the advantages and disadvantages of cross-sectional, time series, longitudinal
studies, and case studies. b When should an investigator use a panel design instead of a time-series design? c Under what conditions would a focus group be the best method to use?
3 Select one of the following topics: automobile accidents, water quality, drug abuse, homeless persons, single-parent families, management information systems, personnel training, strategic management, or outsourcing. For the selected topic, pose a research question appropriate to each of the following: cross-sectional design, a panel design, a time- series design, a case study, a focus group.
4 Data have been collected annually on air quality in Smokey for the past 15 years.
Explain why the data can be analyzed using a time-series design. What types of trends or variations should a researcher look for? How might a researcher distinguish random variations from nonrandom variations?
5 The Metro Hospital collected data on nurses every three years from 1996 to 2007. Beginning with 2007, the data were collected every year. What limitations would an analyst encounter in studying nursing trends from 1996 to the present?
Page 29 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
6 Explain why public agencies are unlikely to conduct case studies. 7 Distinguish between units and components in conducting a case study. 8 For one of the topics in Question 3, generate a list of questions that might be asked of
members of a focus group. 9 Why might government agencies use focus groups more now than in the past? What
objections might managers have to the use of focus groups? 10 Compare literature reviews to meta-analyses, and discuss why researchers would invest
time in conducting a meta-analysis instead of original research.
Problems for Homework and Discussion
1 For each of the following studies, identify the variables; state the implied hypothesis(es); identify the research design, and briefly evaluate its appropriateness. (Note: A study may modify a common design or combine features from more than one design.)
a A random audit of Unemployment Insurance (UI) sampled eight UI payments per week. If a payment error was found, auditors determined the dollar amount and classified the error by type, source, and cause. Errors were categorized as overpayment with fraud, overpayment without fraud, or underpayment. Sources of errors were the claimant, employer, or agency. Causes were identified by law or regulation violation.
b To evaluate the effectiveness of Head Start, an early education program, children who had been in Head Start and who were in the first through third grades were given cognitive tests.
c Before the adoption of the Magnuson-Moss Warranty Act, the Federal Trade Commission collected data from 4,300 respondents who had purchased a major durable good the previous year. Each respondent rated the performance and servicing of products purchased during the year. The 4,300 members of the sample were randomly selected from a national consumer mail panel. To evaluate the act, the FTC later asked 8,000 respondents drawn from the same national consumer mail panel the same questions.
d To develop a statistical base on private foundations, data were gathered from tax records on selected foundations’ resources and expenses in 1987, 1994, 1998, and annually beginning in 2001.
e To assess training needs and how the government could work best with a private agency to meet them, the private agency invited two groups of its clients to participate in discussions led by a moderator from the nearby university.
2 Follow the instructions that appear after Table 2.4, which shows the percentage of seniors in high school who think that smoking a pack a day is a serious health risk.
Table 2.4 Percent of High School Seniors Who Believe That Smoking One or More Packs of Cigarettes A Day Poses a Serious Health Risk.
Year Percent Year Percent
Page 30 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Year Percent Year Percent 1975 51.3 1995 65.6 1976 56.4 1996 68.2 1977 58.4 1997 68.7 1978 59.0 1998 70.8 1979 63.0 1999 70.8 1980 63.7 2000 73.1 1981 63.3 2001 73.3 1982 60.5 2002 74.2 1983 61.2 2003 72.1 1984 63.6 2004 74.0 1985 66.5 2005 76.5 1986 66.0 2006 77.6 1987 68.6 2007 77.3 1988 68.0 2008 74.0 1989 67.2 2009 74.9 1990 68.2 2010 75.0 1991 69.4 2011 77.7 1992 69.2 2012 78.2 1993 69.5 2013 78.2 1994 67.6 2014 78.0
Draw a graph to illustrate this trend over time. Comment on the variations found in this dataset. What policy recommendations can be made based on these data? (Data from Institute for Social Research, University of Michigan, Monitoring the Future Project)
3 An administrator for a state employment commission wants to study seasonal variations in the unemployment rate so she can schedule staff vacations and conferences at times when the demand for services (as measured by the unemployment rate) is lowest. The data are shown in Table 2.5.
Table 2.5 Unemployment Rate
a Graph the data. b Comment on any trends you notice in these data. c What times of year would you prefer for staff vacations and conferences?
4 For each of the following problems, suggest a research design and justify your choice:
Page 31 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
a Identify revenues generated by a county sales tax first adopted in 1980 and increased periodically since then.
b Learn whether change in state penalties for drunk driving was associated with fewer drinking-related traffic fatalities.
c Identify what computer hardware and software are used by local governments and how the governments use them.
d See whether consolidating purchasing by Middletown decreased costs of buying supplies.
e See whether agency managers who attended a decision-making seminar used the skills taught.
f Determine whether police personnel involved in a wellness program took fewer sick days and had fewer claims against the department’s health insurance plan after enrolling in the wellness program.
5 A state has 10 high school health clinics. These clinics have been opened within the last four years to serve the physical and mental health needs of high school students. Some of the clinics are located within a high school, and others are within a block of the school. The state department of education has decided to do a case study of these clinics to evaluate their performance and to see whether similar clinics should be established throughout the state. (In answering the following questions you may make your own assumptions to fill in specific details about the clinics.)
a Identify the units and components that could be the subjects of the case study. What information would you want on the units? On the components?
b List possible data sources you would use. Indicate the type of information you would want from each data source.
c Write a memorandum discussing why a case study would be a valuable research strategy.
6 Look at Example 1.3 in Chapter 1. Assume that you are a management analyst in the department administering the job-training program. Write a memorandum to the program manager outlining what actions you would recommend based on these data. Remember that conducting further research can be a recommended action.
7 Locate a study or article that you would classify as a meta-analysis, and answer the following questions concerning it. What was the topic? How many studies were reviewed? Was a specific hypothesis investigated in the meta-analysis? What did the authors of the meta-analysis conclude from the study?
Working With Data
1 Access the County Data File accompanying this textbook. Analyze the data to see how income, population density, and the number of active primary care physicians varies by region of the state. For each region, find the low, high, and average for each variable for each region. Report your findings in a table.
Page 32 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
2 Cross-sectional data can be reported in tables showing differences in percents or means. Access the County Data File accompanying this textbook. From the income variable, create three categories of counties, for example, high income, medium income, and low income.
a For each county income category, determine the mean values for the number of individuals served in drug and alcohol facilities and the crime index variables. Do the data show that these items vary across low, medium, and high income counties? What evidence supports your observations? Write a paragraph reporting your findings.
b Categorize the counties into two groups, a high and a low category according to the variables (1) number of individuals served in drug and alcohol facilities and (2) the crime index. For each income category, determine what percent of the counties fall into the high category for each variable and what percent fall into the low category. (This can be set up as a table with two rows and three columns, with one column for each income level.) In this analysis, do the data show that the items vary by income? Do these findings support the findings you had in 2(a)? Write a paragraph reporting your findings.
c Of the two options, that used in 2(a), comparing the means, and the one used in 2 (b), comparing percents, which way of presenting this information would you recommend using in a report on this issue? Justify your choice.
3 Access the County Data File accompanying this textbook. Test the hypotheses that you formulated as part of the Chapter 1 exercise. Do the data support your hypotheses? Cite evidence to support your answer.
4 Access the County Data File accompanying this textbook. Draw a scatterplot of the crime index as the dependent variable against population density. In a short paragraph, describe the relationship. What can you conclude?
5 Access the document Focus Group Report accompanying this textbook. Create one additional question that you would have asked if you were involved as a researcher. Explain your choice of question.
Recommended for Further Reading A good place to find information on time series, especially as a forecasting tool, is in management science texts.
One widely available text is D. R. Anderson, D. J. Sweeney, and T. A. Williams, Quantitative Methods for Business, 11th ed. (Mason, OH: Thomson South-Western, 2013), chapter 6 “Time Series Analysis and Forecasting.” Also, D. W. Williams, “Forecasting Methods for Serial Data,” in Handbook of Research Methods in Public Administration, ed. G. L. Miller and M. L. Whicker (New York: Marcel Dekker, 1999), pp. 301–352.
For a discussion of various types of longitudinal designs, see Scott Menard, Longitudinal Research, 2nd ed. (Thousand Oaks, CA: Sage, 2002).
Robert K. Yin, Case Study Research: Design and Methods, 5th ed. (Thousand Oaks, CA: Sage, 2014) is an excellent starting point for information on case studies. This chapter’s section on case studies used this and earlier versions of Yin’s book. Another book, Applications of Case Study Research (Newbury Park, CA: Sage, 2012), by the same author, provides detailed examples of case studies.
Page 33 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...
Dawson Hancock and Bob Algozzine, Doing Case Study Research: A Practical Guide for Beginning Researchers (New York, NY: Teachers College Press, 2006) is a short, readable text covering all steps in the process of conducting a case study.
For more information on focus groups, see the references listed at the end of Chapter 7, especially Richard Krueger and Mary Anne Casey, Focus Groups: A Practical Guide for Applied Research, 5th ed. (Thousand Oaks, CA: Sage, 2015). For a more advanced and analytical presentation, see Edward E. Fern, Advanced Focus Group Research (Thousand Oaks, CA: Sage, 2001).
The reader interested in meta-analysis should review G. V. Glass, B. McGaw, and M. L. Smith, Meta-Analysis in Social Research (Beverly Hills: Sage, 1981). T. D. Cook, et al., eds., Meta-analysis for Explanation: A Casebook (New York: Russell Sage Foundation, 1992) has four chapters on the method and four studies. Also highly recommended is Morton Hunt, How Science Takes Stock: The Story of Meta-Analysis (New York: Russell Sage Foundation, 1997). Hunt discusses how agencies of the American national government began using meta-analysis more frequently, especially in areas of medical research. Geoff Cummings, Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (New York: Routledge, 2012), has three chapters on meta-analysis. He discusses small- and large-scale studies and different tools.
The following book uses journal articles to illustrate some qualitative methods. It is brief and inexpensive: Larry S. Luton, Qualitative Research Approaches for Public Administration (New York: M. E. Sharpe, 2010).
Page 34 of 34Research Methods for Public Administrators
3/3/2021file:///C:/Users/Lamar%20Holmes/Desktop/VitalSource%20Bookshelf_%20Research%20Methods...