615 assignment

Evaluation and Program Planning 56 (2016) 50–55

Measuring infrastructure: A key step in program evaluation and planning

Carol L. Schmitta,*, LaShawn Glasgowa, S. Rene Lavinghouzeb, Patricia P. Riekerc, Erika Fulmerb, Kelly McAleera, Todd Rogersa

aRTI International, Public Health Research Division, USA bCenters for Disease Control and Prevention, Office on Smoking and Health, USA cBoston University and Harvard Medical School, USA

A R T I C L E I N F O

Article history: Received 10 December 2015 Received in revised form 22 March 2016 Accepted 23 March 2016 Available online 26 March 2016

Keywords: Infrastructure Infrastructure measurement Program evaluation Program sustainability Public health

A B S T R A C T

State tobacco prevention and control programs (TCPs) require a fully functioning infrastructure to respond effectively to the Surgeon General’s call for accelerating the national reduction in tobacco use. The literature describes common elements of infrastructure; however, a lack of valid and reliable measures has made it difficult for program planners to monitor relevant infrastructure indicators and address observed deficiencies, or for evaluators to determine the association among infrastructure, program efforts, and program outcomes. The Component Model of Infrastructure (CMI) is a comprehensive, evidence-based framework that facilitates TCP program planning efforts to develop and maintain their infrastructure. Measures of CMI components were needed to evaluate the model’s utility and predictive capability for assessing infrastructure. This paper describes the development of CMI measures and results of a pilot test with nine state TCP managers. Pilot test findings indicate that the tool has good face validity and is clear and easy to follow. The CMI tool yields data that can enhance public health efforts in a funding-constrained environment and provides insight into program sustainability. Ultimately, the CMI measurement tool could facilitate better evaluation and program planning across public health programs.

ã 2016 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier .com/ locat e/e valprogplan

1. Background

A comprehensive approach to tobacco prevention and control— including state and community interventions; mass-reach health communication interventions; cessation interventions, surveil- lance, and evaluation; and infrastructure, administration, and management—has contributed to a significant decline in U.S. adult smoking rates over the past 50 years (CDC, 2014; U.S. DHHS, 2014a). Despite a considerable drop in U.S. adult cigarette smoking prevalence from 43% in 1965 to 17.8% in 2013 (Jamal et al., 2014), disparities remain in smoking among population subgroups, and many current smokers are using multiple tobacco products (Campaign for Tobacco-Free Kids, 2015; U.S. DHHS, 2014a). Moreover, tobacco use is still the leading cause of premature death in the United States—a fact that undergirds the Surgeon General’s recent call for accelerating the national movement to further reduce tobacco use (U.S. DHHS, 2014a). To plan and

* Corresponding author at: RTI International, 701 13th Street, NW, Suite 750, Washington DC 20005-3967, USA.

E-mail address: cschmitt@rti.org (C.L. Schmitt).

http://dx.doi.org/10.1016/j.evalprogplan.2016.03.007 0149-7189/ã 2016 Elsevier Ltd. All rights reserved.

implement interventions that respond effectively to the Surgeon General’s call, state tobacco prevention and control programs (TCPs) require fully functioning infrastructure (CDC, 2014; Lavinghouze, Snyder, Rieker, & Ottoson, 2013; Lavinghouze, Snyder, & Rieker, 2014).

Broadly speaking, infrastructure provides the foundation for planning, delivering, and evaluating public health programs (U.S. DHHS, 2014b); a strong, functioning infrastructure provides the foundation on which to build policies, systems, and organizational capacities that are optimally responsive to public health threats (Lavinghouze et al., 2013). Given its significance to public health programs, infrastructure needs to be defined in clear, practical, actionable, and evaluable terms so that it can be measured and examined. This is the premise under which Lavinghouze et al. (2014) developed the Component Model of Infrastructure (CMI). The model, shown in Fig. 1, is based on case studies of state TCPs, a literature review of diverse public health program infrastructure articles (e.g., asthma, diabetes, oral health), and theories from other disciplines such as sociology, organizational development, and economics (Lavinghouze et al., 2014). The CMI defines infrastructure as five synergistic core components:

Fig. 1. Component Model of Infrastructure.

C.L. Schmitt et al. / Evaluation and Program Planning 56 (2016) 50–55 51

� Networked partnerships involve strategic collaborations and multilevel relationships among the state TCP and organizational stakeholders at the national, state, and local levels. Although they fill different roles, networked partners work to ensure the accomplishment of all activities necessary to achieve common public health goals.

� Multilevel leadership refers to individuals who provide direction for a program or enable resources and processes to support program direction. Leaders and champions can be identified at all levels, including those “above” the program in the health department or other organizational unit where the program is located, within the program in roles other than the program manager or director, among lateral agency partners, and in local programs. Leadership at all levels is necessary to ensure functioning program infrastructure and progress toward health goals.

� Engaged data involves identifying, collecting, and employing data to promote action. Data can be used in a manner that engages staff, partners, decision makers, and local programs to act. Data should not merely be collected and summarized, but also used actively to promote public health goals. Training, technical assistance, and follow-through are necessary to ensure the proper use of data.

� Managed resources refers to leveraging funds from diverse sources and recruiting and supporting staff with the skills and knowledge to plan and implement best practices. A functional infrastructure requires resources beyond financing; it includes mobilizing an adequate number of staff and partners who reflect the diversity of the communities served and have a variety of technical, program, and administrative skills. Staff, partners, and local programs must also have the necessary training and skills to implement the TCP efficiently and effectively.

� Responsive strategic plans are dynamic and evolve in response to contextual influences, such as changes in scientific evidence,

priorities, funding levels, and external support. In addition, the planning process is collaborative and includes viewpoints from multiple stakeholders (Ebbesen, Heath, Naylor, & Anderson, 2004). The process fosters shared ownership and responsibility for the goals and objectives among the state program, partners, and local programs. Responsive plans and planning are devel- oped and implemented collaboratively with diverse stake- holders.

The model also includes three supportive components (strate- gic understanding, operations, and contextual influences) (CDC, 2014; Lavinghouze et al., 2014). The supportive components are important for program planning and evaluation and are critical to implementing functioning infrastructure (Lavinghouze et al., 2014). However, the core components are the foundation of the CMI and include indicators more readily operationalized for measurement.

Although CMI is an evidence-based framework that can inform TCP efforts to develop and maintain their infrastructure (Laving- houze et al., 2013, 2014), measures of its constructs are still being developed and tested; this is needed to advance the model’s utility for program and evaluation planning and to determine its predictive capability (Lavinghouze et al., 2014). CMI is distinct in that it specifically refers to functioning program infrastructure, as opposed to the wider public health system infrastructure (Baker et al., 2005; Handler, Issel, & Turnock, 2001; Lavinghouze et al., 2013; Roper, Baker, Dyal, & Nicola, 1992). Previous attempts to measure program infrastructure have not been based on a comprehensive conceptualization of infrastructure; for example, measures that only address limited aspects, such as partnerships or funding (Farrelly, Pechacek, & Chaloupka, 2003; Granner & Sharpe, 2004). Additionally, previous efforts neither fully considered the dynamic contexts that affect infrastructure measures (e.g., changes in staffing and funding) nor sufficiently took into account

52 C.L. Schmitt et al. / Evaluation and Program Planning 56 (2016) 50–55

influences outside of the immediate program (e.g., support from leaders external to the program and the agency in which it is housed) (Ebbesen et al., 2004). Developing measures based on the CMI, a framework that reflects the multicomponent, complex, and layered nature of infrastructure, can help address previous measurement limitations. This paper describes the development and pilot testing of the CMI measurement tool, an important step toward further elucidating, and eventually leveraging, linkages between program infrastructure and public health outcomes and impact (Meyer, Davids, & Mays, 2012). Our pilot test marks progress toward integrating infrastructure assessment into pro- gram planning and evaluation efforts and suggests that the CMI tool has practical applications beyond tobacco control.

2. Methods

The measurement tool was developed to capture each of the CMI core components. Tool development involved three stages: (1) identifying key constructs to be covered in the instrument, (2) adapting existing survey items and drafting new items as necessary, and (3) engaging experts and stakeholders to help validate the tool. Identification of key constructs involved reviewing primary source data used to develop the CMI. As part of this work, we analyzed qualitative data from case studies of public health programs and interviews with public health program managers (Lavinghouze et al., 2014). We identified prominent themes and constructs important to program and evaluation planning under each element of the CMI and verified our assumptions with CMI developers.

To move from framework constructs to the development of specific survey items, we conducted targeted literature searches and reviewed existing chronic disease, capacity-related tools such as the Strength of Tobacco Control Index (Stillman, Schmitt, Clark, Trochim, & Marcus, 2016) and the Cancer Plan Self-Assessment Tool (CDC, 2012). Relevant validated items in existing tools were adapted for use in the CMI tool. Additionally, several new items were drafted to address key constructs which lacked existing relevant survey items. New items were based on CMI source data, input from CMI developers, and feedback from evaluation, public health, and tobacco prevention and control experts.

Table 1 Example survey items by CMI core component.

CMI Core Component Examples of Characteristics

Networked Partnerships � Diversity beyond specific focus (integration and coordination) � Nurtured beyond fundee relationship � Extend program’s reach � Facilitate progress on health achievements and implementation of strategies

Multilevel Leadership � Occurs at multiple levels (above, below, within, and � Identification, development, and nurturing of ch � Concept of ownership of programs at multiple le

Responsive Plans/Planning � Dynamic, evolving, responsive, flexible � Shared ownership � Education and recruitment tool � Progress yardstick

Managed Resources � Diversified funding streams, leveraging, integrati coordination � Staff expertise nurtured and sustained � Staff and partners continue to grow through trai financial acumen, and technical assistance

Engaged Data � Use of data to increase program visibility, attract p understand the public health burden and public h achievements, drive program directionand plannin

Source: Author.

The final step of the measurement tool development process included expert review of the draft instrument. This occurred during an in-person roundtable hosted by the Centers for Disease Control and Prevention (CDC) Office for Program Planning and Evaluation and involved 22CDC evaluators from across the agency. Participants worked in small groups to complete sections of the tool from the perspective of state program respondents and provided feedback related to the clarity of the items, usefulness of data captured, and potential analysis challenges. RTI survey methodologists also reviewed the draft measurement tool and made recommendations for item wording and response categories. Ultimately, our development efforts yielded a 49-item survey divided into five sections representing the CMI core components (Table 1).

To pilot test the CMI measurement tool, we conducted telephone interviews in January-February 2014 with a purposive sample of 9 TCP managers (Idaho, Kentucky, Massachusetts, Minnesota, Nebraska, New Mexico, North Carolina, Utah, and West Virginia) representing 8 of the 10 U.S. Department of Health and Human Services regions (U.S. DHHS, 2015). The purpose of the pilot test was to assess the user-friendliness of the CMI tool, the extent to which respondents perceived the definitions of each infrastructure component as clear and relevant, and the degree to which the questions validly represented each infrastructure component. Tobacco control experience of the respondents ranged from 2 to 20 years. We obtained oral participant consent, and the study was reviewed by RTI International’s institutional review board. Respond- ents received a copy of the measurement tool prior to the call. Each interview took approximately 90 minutes. After completing each section of the survey, respondents were asked to describe any difficulty they had in understanding or responding to the questions. At the conclusion of the survey, respondents were asked to provide their overall impressions of the tool and the extent to which they thought the survey elicited responses that accurately and compre- hensively described their program infrastructure.

Our project team debriefed after each interview to share interview experiences and discuss respondent feedback. If neces- sary, we revised the tool for subsequent interviews based on respondentfeedback. Thisprogressive analysis allowedusto identify constructs that required additional explanation and to fill data gaps with each successive interview using the revised instrument.

Number of Items

Example Survey Item

24 How many [voluntary health organizations] are represented on your state tobacco control coalition? How many of these are active members?

lateral) ampions vels

5 Does your program have the support of a key leader or champion from other state and local government agencies? [yes, no, don’t know; brief example of support provided]

9 What is the status of your program’s sustainability plan? [current written plan, developing or updating plan, no plan or planning underway]

on,

ning,

9 Have there been any staff changes (new hires, resignations) during the past contract year? [yes, no; number of new staff, number of lost staff]

artners, ealth g

14 Thinking about data on subpopulations, would you say that your program has [all of what it needs, most of what it needs, some of what it needs, none of what it needs]?

C.L. Schmitt et al. / Evaluation and Program Planning 56 (2016) 50–55 53

Our analysis focused on reviewing interviewer and respondent feedback to assess user-friendliness of the tool, the extent to which respondents perceived the definitions of each infrastructure component as clear and relevant, and the degree to which the questions validly represented each infrastructure component. In addition, we analyzed interview notes to identify facilitators and barriers to completing the interview and examined responses to assess data quality.

3. Findings and conclusions

Respondents generally found the CMI tool to be user-friendly and specifically noted that it was clear and easy to follow. We did not receive any questions about the definitions of infrastructure components, which were read to respondents at the start of corresponding survey sections, which led us to conclude that the definitions are clear. The majority of comments focused on the extent to which the tool’s focus on infrastructure was relevant and meaningful to respondents.

Eight TCP managers provided general feedback at the end of the survey. One of the eight respondents reiterated the sensitive nature of some items (e.g., leadership and champion support questions) and noted that some items require more thought than others (e.g., identifying top partners). The remaining seven respondents indicated that the tool’s focus on infrastructure was relevant and meaningful to them and that they understood the conceptual relationship between infrastructure and program planning, implementation, and outcomes. For example, the tool was described as “thought-provoking” and a “timely” resource that could facilitate TCP strategic planning. One respondent particularly appreciated the tool’s alignment with CDC’s Best Practices for Comprehensive Tobacco Control Programs (CDC, 2014). Another respondent reflected on the components of infrastructure that were measured and the role that infrastructure plays in ensuring implementation of effective tobacco control interventions. This respondent attributed his or her program’s success to the well- established partnerships built over two decades and the ability to retain knowledgeable, skilled staff. One manager felt it was important for funders to know that “it takes hard work” to develop and maintain a functioning infrastructure. Finally, TCP managers were very interested in seeing their responses in relation to other programs’ data and encouraged the project team to develop a dashboard report to summarize survey responses.

As we developed the tool, we found that it could be challenging for potential users to condense their program’s infrastructure into multiple-choice answers. All were able to complete the survey items, but most program managers (67%) wanted to provide additional context related to the composition and functions of their statewide coalitions beyond what the multiple-choice response options allowed. These respondents were more comfort- able when they could provide additional background information; thus, we added open-ended questions to the survey to aid in the interpretation of findings. Inductive coding may aid in the synthesis of responses to open-ended items. We will explore the most efficient strategies for analyzing and presenting qualitative responses in future CMI data collection efforts.

Other respondents wanted to provide additional information before selecting a response. For instance, one TCP with decades of experiencefounditchallengingtoselectoneresponsetodescribethe frequency of the program’s interactions with the state tobacco control coalition. After noting that interactions could be at least daily when the legislature was in session, this manager concluded that the baseline rate of contact was several times or more each month. In several instances, TCP managers felt it necessary to describe the types of skills and expertise their programs needed after having selected the response Most of what it needs to achieve its goals.

Three respondents did note that some relevant items (e.g., those indicating the need for additional support from leadership) raised were sensitive issues; thus, we restricted the level of detail requested in these items. Although all participants described a champion within the health department, they did express concern about answering this question if there was no such champion. In at least one case, a participant noted that state law prohibited government agency representatives from voting on coalition issues, which could be perceived as critical of the state’s government; in several cases, participants described a lack of political support after key political allies had left office. This, too, could be construed as critical of the current state legislature.

4. Future directions

TCPs require a fully functioning infrastructure to achieve their goals (CDC, 2014; Lavinghouze et al., 2014). The CMI is an evidence- based framework that can inform TCPs’ efforts to strengthen and maintain their infrastructure and facilitate program planning and evaluation. We developed measures of CMI core components to facilitate ongoing infrastructure assessment and monitoring and to evaluate the model’s applicability and predictability. Pilot-test findings suggest that the CMI-based measurement tool is user- friendly and face-valid. We enhanced the accuracy of reporting by including open-ended questions that allowed respondents to qualify multiple choice responses. Feedback obtained during the pilot test indicates that assessment of these program infrastructure components is relevant and meaningful to TCP managers and accurately reflects their program infrastructure. These results provide “proof of concept” that TCP infrastructure can be operationally defined and measured at the state level.

Efforts to improve public health program effectiveness through planning and evaluation include the understanding of complex adaptive models such as program infrastructure as portrayed in the CMI. The CMI is a practical and actionable evidence-based model useful for program planning and evaluation (Lavinghouze et al., 2014). The model provides a framework that can facilitate the development of program guidance documents, best practices for infrastructure implementation, funding announcements, and technical assistance. It can also serve as the basis for surveillance and evaluation efforts and for educating about the public about the need for strong, functioning program infrastructure in public health (Lavinghouze et al., 2014). Creating a valid measurement tool is the next logical step toward furthering the use of the CMI for program planning and evaluation.

Although promising, these findings are based on a limited, purposive sample of state representatives. Future work will include developing measurement methods to support ongoing assessment and monitoring of program infrastructure to evaluate the predictive validity of CMI measures. We modified the CMI measurement tool based on pilot test findings, and CDC intends to administer the tool to TCPs in all 50 states and the District of Columbia. The data collected from that effort will allow for a more robust examination of the applicability of CMI measurement in all states and will allow us to examine the psychometric properties of CMI measures. A state and national baseline description of TCP infrastructure will also allow evaluators to begin examining the association of CMI measures and program outcomes.

We believe the CMI approach to measuring infrastructure is generalizable to other public health programs because the model is built on work from multiple public health programs (Lavinghouze et al., 2014), and the CMI measures in this study were modified after input from evaluators from diverse CDC program areas. Collecting similar information from different state public health programs would allow researchers and practitioners to examine more fully non-tobacco program infrastructure and explore the relationships

54 C.L. Schmitt et al. / Evaluation and Program Planning 56 (2016) 50–55

between infrastructure, funding, and public health impacts. Importantly, an applied understanding of infrastructure can provide the basis for strategic investments to ensure that public health programs have the infrastructure needed to address the increasingly complex public health challenges of the 21st century.

5. Lessons learned

We contend that the best planned program can be more successfully implemented in the context of a robust infrastructure. The CMI tool provides program planners with a practical way to assess the elements of functioning infrastructure available and necessary not only for implementing interventions but also for subsequent outcome or impact evaluations. The CMI also provides a concrete way to communicate the value of an abstraction like infrastructure and to help shape the thinking of stakeholders and funders. Table 1 provides a listing of the five components and some basic questions that can be used as a checklist for an initial assessment of infrastructure. Although this brief description is not definitive, it offers program planners and evaluators a framework and a preliminary tool to use as a building block until a fully validated tool and evidence of its utility become available. Forward thinking and planning about infrastructure development with the CMI gives program managers an additional means for leveraging and sustaining public health interventions, especially when funding amounts and staffing levels are uncertain.

Conflicts of interest and source of funding

No conflicts of interest are declared. Carol Schmitt, LaShawn Glasgow, Kelly McAleer, and Todd Rogers were supported through a contract with the Centers for Disease Control and Prevention (CDC) Office on Smoking and Health (OSH) (contract # 200-2011-F- 39606: Scientific, Technical and Operational Services for Epidemi- ology, Surveillance, and Evaluation). Patricia P. Rieker was supported under contract by CDC’s OSH.

References

Baker, E. L., Potter, M. A., Jones, D. L., Mercer, S. L., Cioffo, J. P., Green, L. W., et al. (2005). The public health infrastructure and our nation’s health. Annual Review of Public Health, 26, 303–318.

Campaign for Tobacco-Free Kids (2015). Broken promises to our children. The 1998 State tobacco settlement 15 years later. Campaign for Tobacco-Free Kids.

Centers for Disease Control and Prevention (CDC). (2012). Cancer Plan Self- Assessment Tool. Available at http://www.cdc.gov/cancer/ncccp/pdf/ cancerselfassesstool.pdf.

Centers for Disease Control and Prevention (CDC) (2014). Best practices for comprehensive tobacco control programs—2014. Atlanta, GA: U.S. Department of Health and Human Services, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.

Ebbesen, L. S., Heath, S., Naylor, P. J., & Anderson, D. (2004). Issues in measuring health promotion capacity in Canada: a multi-province perspective. Health Promotion International, 19(1), 85–94.

Farrelly, M. C., Pechacek, T. F., & Chaloupka, F. J. (2003). The impact of tobacco control program expenditures on aggregate cigarette sales: 1981–2000. Journal of Health Economics, 2(5), 843–859.

Granner, M. L., & Sharpe, P. A. (2004). Evaluating community coalition characteristics and functioning: a summary of measurement tools. Health Education Research, 19(5), 514–532.

Handler, A., Issel, M., & Turnock, B. (2001). A conceptual framework to measure performance of the public health system. American Journal of Public Health, 91 (8), 1235–1239.

Jamal, A., Agaku, I. T., O’Connor, E., King, B. A., Kenemer, J. B., & Neff, L. (2014). Current cigarette smoking among adults—United States, 2005–2013. MMWR Weekly Report, 63(471), 1108–1112.

Lavinghouze, R., Snyder, K., & Rieker, P. (2014). The component model of infrastructure: a practical approach to understanding public health program infrastructure. American Journal of Public Health, 104(8), e14–24.

Lavinghouze, R., Snyder, K., Rieker, P., & Ottoson, J. (2013). Consideration of an applied model of public health program infrastructure. Journal of Public Health Management Practice, 6, E28–E37.

Meyer, A. M., Davids, M., & Mays, G. P. (2012). Defining organizational capacity for public health services and systems research. Journal of Public Health Management Practice, 8(6), 535.

Roper, W., Baker, E., Dyal, W., & Nicola, R. (1992). Strengthening the public health system. Public Health Report, 107(6), 609–615.

Stillman, F.A., Schmitt, C.L., Clark,P. I., Trochim, W.M. K., & Marcus, S.E. (2016). The strength of tobacco control index. Monograph 17. Evaluating ASSIST. Available at: http://cancercontrol.cancer.gov/brp/tcrb/monographs/17/m17_2.pdf.

U.S. Department of Health and Human Services (U.S. DHHS) (2014a). The health consequences of smoking—50 years of progress. A report of the surgeon general. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.

U.S. Department of Health and Human Services (U.S. DHHS). (2014b). Healthy People 2020—public health infrastructure objectives. Available at http://healthypeople. gov/2020/topicsobjectives2020/objectiveslist.aspx?topicid=35.

U.S. Department of Health and Human Services (U.S. DHHS). (2015). Regional Health Administrators. Available at http://www.hhs.gov/ash/rha/.

Carol L. Schmitt, PhD, has over 15 years of experience in conceptualizing, measuring, and evaluating intervention program components at the local, state, national, and international levels, in addition to designing and testing data collection instruments. Much of her recent work has focused on developing methods and measures for evaluating state and community initiatives, primary in tobacco control, obesity, and health disparities. In addition to her research skills, Dr. Schmitt currently serves as a Director of Public Health in RTI’s Center for Health Policy Science and Tobacco Research.

LaShawn M. Glasgow, DrPH, is an evaluator in RTI International’s Public Health Research Division with expertise in qualitative research and evaluation methods. Dr. Glasgow leads evaluation planning, data collection, analysis, and the development of evaluation technical assistance tools for national, state and community-level chronic disease prevention and control programs. Prior to joining RTI, Dr. Glasgow worked as an evaluator at the Pennsylvania Department of Health and with community-based health and social service programs.

S. René Lavinghouze, MA, has over 25 years in designing, conducting and managing evaluations focusing in prevention programs and is currently the Evaluation Team Lead in the Office on Smoking and Health at CDC. Her focus is on the study of program infrastructure as the foundation that supports program capacity, implementation, and sustainability. She is lead author of the Component Model of Infrastructure (CMI), a practical, systems-approach model that supports program implementation and enables outcome measurement, links infrastructure to capacity, and facilitates the likelihood of sustainable public health achievements. She was the lead evaluation scientist in the Division of Oral Health at CDC for 8 years and also worked in HIV/AIDS prevention in CDC/DHAP for 4+ years. Prior to coming to CDC, she worked at a private evaluation and organizational service firm for over 7 years and has worked in academic and local health department settings. She received her Master’s in Community Psychology at Georgia State University.

Particia P. Rieker, PhD, is a faculty member at Boston University, Department of Sociology and Harvard Medical School, Department of Psychiatry, Boston, MA. She publishes widely and her research focuses on the determinants of health outcomes.

Erika Fulmer, MPA, a Health Scientist with the Centers for Disease Control and Prevention (CDC), has 16 years of professional experience in behavioral health sciences research and evaluation. Specializing in tobacco control program evaluation and outcome assessment, Ms. Fulmer has led numerous public health initiatives at the national, state, and local levels. In addition to tobacco control and prevention, she has worked on a range of public health evaluation issues, including the prevention of obesity, diabetes, cancer, heart disease and stroke; substance abuse treatment; tuberculosis control and HIV/AIDS prevention. Her work has included designing and implementing evidence-based strategic performance measurement systems, planning national program and policy evaluations, providing evaluation technical assistance to national, state, and local public health agencies, and participating in the development and implementation of public health monitoring systems. She received her Master of Healthcare Administration at the University of North Carolina, Chapel Hill.

Kelly McAleer, MSPH, is a public health analyst in RTI International’s Public Health Research Division. Ms. McAleer has experience evaluation and quantitative data, and assists on projects centered around mixed methods evaluations of federally funded programs. Prior to joining RTI, Ms. McAleer worked for the North Carolina Department of Health in the Asthma program, as well being a CDC Public Health Prevention Service Fellow. She received her Master of Science in Public Health in Epidemiology from the University of South Florida.

C.L. Schmitt et al. / Evaluation and Program Planning 56 (2016) 50–55 55

Todd Rogers, PhD, has over 30 years’ experience designing, conducting, and evaluating large-scale public health interventions. He has many peer-reviewed publications and conference presentations on tobacco control program design and

evaluation. He earned MS and PhD degrees from the Pennsylvania State University, and was a Fellow in at the Stanford University School of Medicine. He has served as a research scientist at Stanford University (1987–97), the Public Health Institute (1998–2011), and RTI International (2011–current).