615 assignment

jsbfg0001

programevaluation.pdf

Home >Human Resource Management homework help >615 assignment

PROGRAM EVALUATION

Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity

Report to Congressional Committees

November 2014

GAO-15-25

United States Government Accountability Office

Highlights of GAO-15-25, a report to congressional committees

November 2014

PROGRAM EVALUATION Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity

Why GAO Did This Study To improve federal government performance and accountability, GPRAMA aims to ensure that agencies use performance information in decision making and holds them accountable for achieving results. The Office of Management and Budget (OMB) has encouraged agencies to strengthen their program evaluations– systematic studies of program performance–and expand their use in management and policy making. This report is one of a series in which GAO, as required by GPRAMA, examines the act’s implementation. GAO examined federal agencies’ capacity to conduct and use program evaluations and the activities and resources, including some related to GPRAMA, agencies found useful for building that capacity.

GAO reviewed the literature to identify the key components and measures of evaluation capacity. GAO surveyed the PIOs of the 24 federal agencies subject to the Chief Financial Officers Act regarding their organizations’ characteristics, expertise, and policies, and their observations on the usefulness of various resources and activities for building evaluation capacity. All 24 responded. GAO also interviewed OMB and Office of Personnel Management (OPM) staff about their capacity-building efforts.

What GAO Recommends GAO is not making recommendations.

OMB staff provided technical comments on a draft of this report that were incorporated as appropriate. OPM provided no comments.

What GAO Found In a governmentwide survey of agency Performance Improvement Officers (PIO), GAO found uneven levels of evaluation expertise, organizational support within and outside the organization, and use across the government. The Government Performance and Results Act of 1993 (GPRA) is a key component of the enabling environment for federal evaluation capacity, having established a solid foundation of agency performance reporting and leadership commitment to using evidence in decision making. However, only half the agencies reported congressional interest in or requests for program evaluation studies.

Eleven of the 24 agencies reported committing resources to obtain evaluations by establishing a central office responsible for evaluation of agency programs, operations, or projects, although only half these offices were reported to have a stable source of funding. Seven agencies reported having a high-level official responsible for oversight of evaluation. A quarter of agencies reported having agency-wide policies or guidance concerning key issues in study design, evaluator independence and objectivity, report transparency, or implementing findings. Two-thirds of the agencies reported evaluation coverage of less than half their performance goals. Over a third reported using evaluations to a moderate or greater extent as evidence in support of budget or policy changes or program management. Those agencies with centralized evaluation authority reported greater evaluation coverage and use of the results in decision making.

Since the GPRA Modernization Act of 2010 (GPRAMA) was passed, 2 to 4 agencies established a central evaluation office or leader. Half the agencies reported increased efforts to improve their evaluation capacity through hiring, training, conference participation, and consulting experts, but 4 to 5 reported declines in hiring and conference participation. About half reported increased use of evaluations as supporting evidence for management and policy decisions.

About a quarter of PIOs were not familiar with their agencies’ various capacity- building activities but many of those that did respond rated hiring, professional networking, consulting with experts, reviewing progress on priority goals, and holding goal leaders accountable under GPRAMA most useful for building capacity to conduct evaluations. They rated engaging program staff in evaluation design, conduct, and reporting, and the GPRAMA priority goal review and accountability provisions most useful for building capacity to use evaluation.

Based on our survey results, GAO observes that

• Promoting information sharing in professional networks and engaging program managers and staff in evaluation studies and priority goal reviews offer promise for building capacity in a constrained budget environment.

• Engaging congressional and other stakeholders in evaluation planning might increase their interest in and adoption of evaluation recommendations.

• Congressional committees can communicate their interest in evaluation by consulting with agencies on their strategic plans and priority goals, reviewing agency annual evaluation plans to ensure they address issues that will inform congressional decision making, and requesting evaluations to address specific questions of interest.

View GAO-15-25. For more information, contact Nancy Kingsbury, (202) 512-2700, [email protected]

http://www.gao.gov/products/GAO-15-25�

Page i GAO-15-25 Program Evaluation

Letter 1

Background 3 Federal Agencies’ Capacity to Conduct and Use Program

Evaluation Is Uneven 8 Some Agencies Improved Their Evaluation Capacity after

GPRAMA Was Enacted 17 Some Agencies Reported that Professional Networking, Hiring,

Engaging Program Staff, and Some GPRAMA Provisions Were Useful for Building Evaluation Capacity 20

Concluding Observations 28 Agency Comments 29

Appendix I Methodology for the Survey of Performance Improvement Officers in CFO Act Agencies 31

Appendix II Results from 2014 Survey of Performance Improvement Officers 33

Appendix III GAO Contacts and Staff Acknowledgments 56

References 57

Related GAO Products 60

Figures

Figure 1: Number of Agencies Citing Evaluation Evidence to Support Various Decisions 16

Figure 2: Agencies Reporting Change since 2010 in Efforts to Improve Their Capacity to Conduct Evaluations 18

Figure 3: Agencies Reporting Change since 2010 in Citing Evaluation as Supporting Evidence in Decisions 19

Figure 4: PIOs’ Views on Usefulness of Activities and Resources for Improving Agency Capacity to Conduct Credible Evaluations 21

Contents

Page ii GAO-15-25 Program Evaluation

Figure 5: PIOs’ Views on Usefulness of Activities and Resources for Improving Agency Capacity to Use Evaluations in Decision Making 25

Abbreviations AAAS American Association for the Advancement of Science AEA American Evaluation Association CFO Chief Financial Officer COO Chief Operating Officer CAP Cross-Agency Priority goal GAO Government Accountability Office GPRA Government Performance and Results Act GPRAMA GPRA Modernization Act HHS Department of Health and Human Services OMB Office of Management and Budget OPM Office of Personnel Management PIC Performance Improvement Council PIO Performance Improvement Officer

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Page 1 GAO-15-25 Program Evaluation

441 G St. N.W. Washington, DC 20548

November 13, 2014

The Honorable Thomas R. Carper Chairman The Honorable Tom Coburn, M.D. Ranking Member Committee on Homeland Security and Governmental Affairs United States Senate

The Honorable Elijah E. Cummings Ranking Member Committee on Oversight and Government Reform House of Representatives

The federal government faces a number of significant fiscal, financial management, and performance management challenges in responding to the diverse and increasingly complex issues it seeks to address. The reporting requirements of the Government Performance and Results Act of 1993 (GPRA) were intended to provide both congressional and executive decision makers with more objective information on the relative effectiveness and efficiency of federal programs and spending.1 Although GPRA helped improve the availability of agency performance information, federal managers reported limited use of performance data for decision making.2

The GPRA Modernization Act of 2010 (GPRAMA, or the act) aims to ensure that agencies use performance information in decision making and holds them accountable for achieving results and improving government performance.3 The Office of Management and Budget (OMB), too, has encouraged agencies to strengthen their program evaluations—systematic studies of program performance—and expand their use of evidence and evaluation in budget, management, and policy decisions to improve government effectiveness. However, in our 2013

1Pub. L. No. 103-62, 107 Stat. 285 (Aug. 3, 1993). 2GAO, Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges, GAO-13-518 (Washington, D.C.: June 26, 2013). 3Pub. L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011).

http://www.gao.gov/products/GAO-13-518�

Page 2 GAO-15-25 Program Evaluation

survey of federal managers, we found that their use of performance information had stagnated since our 2007 survey and that, among other things, their use was hindered by inadequate staff expertise in performance measurement and analysis as well as a widespread lack of program evaluations.4

This report is one of a series responding to GPRAMA’s mandate that we examine implementation of the act.5 In this report, we assess federal agencies’ current evaluation capacity and identify how some GPRAMA provisions and other activities have contributed to its improvement. Specifically, our objectives were to learn:

1. What are the key elements and extent of agency evaluation capacity—that is, the ability to obtain and use evaluations in decision making?

2. What progress, if any, has been made since 2010 across the government in improving evaluation capacity?

3. What activities, if any, especially those related to GPRAMA provisions, have agencies found useful in building their evaluation capacity?

To answer our first objective, we reviewed published domestic and international research and commentary on key components of organizational capacity for program evaluation, including GAO reports and recommendations of national and international evaluation organizations. We identified the key organizational characteristics, expertise, and policies believed either to be required for or to indicate the ability to obtain credible evaluations of agency programs and policies and to use the results in management and policy decisions. We also identified strategies used or proposed for building an organization’s evaluation capacity.

To answer all three objectives, we surveyed the Performance Improvement Officers (PIO) of the 24 executive branch agencies covered

4GAO-13-518 and Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making, GAO-13-570 (Washington, D.C.: June 26, 2013). 5In GAO-13-518 we summarize our work in 2013 on implementation of the act. Other reports we issued pursuant to this mandate are listed at the end of this report.

http://www.gao.gov/products/GAO-13-518�

http://www.gao.gov/products/GAO-13-570�

http://www.gao.gov/products/GAO-13-518�

Page 3 GAO-15-25 Program Evaluation

by the Chief Financial Officers (CFO) Act of 1990, as amended.6 The survey questionnaire was designed to obtain information on their agencies’ elements of evaluation capacity as described above, and their observations and perceptions of the usefulness of various resources and activities for building their agencies’ capacity to produce evaluations and use the results in decision making. We administered the web-based survey from May through June 2014, receiving responses from all 24 agencies. Throughout this report except where specifically noted, when we refer to agencies, we are referring to both cabinet departments and independent agencies. (More information on the survey is in appendix I. The survey questions and summarized results are in appendix II.) In addition, we reviewed examples of agency evaluation plans and policies that the survey respondents provided. We also interviewed OMB and Office of Personnel Management (OPM) staff about their capacity- building efforts, reviewed agency guidance and memorandums, and attended related interagency information-sharing forums.

We conducted this performance audit from September 2013 to November 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Program evaluations are systematic studies that use research methods to address specific questions about program performance. Evaluation is closely related to performance measurement and reporting. Whereas performance measurement entails the ongoing monitoring and reporting of program progress toward preestablished goals; program evaluation typically assesses the achievement of a program’s objectives and other aspects of performance in the context in which the program operates. In particular, evaluations can be designed to isolate the causal impacts of programs from other external economic or environmental conditions in order to assess a program’s effectiveness. Thus, an evaluation study can provide a valuable supplement to ongoing performance reporting by

631 U.S.C. § 901(b). The 24 CFO Act agencies, generally the largest federal agencies, are listed in appendix I.

Background

Page 4 GAO-15-25 Program Evaluation

measuring results that are too difficult or expensive to assess annually, explaining the reasons why performance goals were not met, or assessing whether one approach is more effective than another.7

Evaluation can play a key role in program planning, management, and oversight by providing feedback on both program design and execution to program managers, legislative and executive branch policy officials, and the public. In our 2013 survey of federal managers, we found that while only about a third had recent evaluations of their programs or projects, the majority of those who had evaluations reported that they contributed to understanding program performance, sharing what works with others, and making changes to improve program management or performance.8

GPRAMA made changes to agency performance management roles, planning and review processes, and reporting intended to ensure that agencies used performance information in decision making and were held accountable for achieving results and improving government performance.9 The act required the 24 CFO Act agencies and OMB to establish agency and governmentwide cross-agency priority goals, review progress on those goals quarterly, and report publicly on their progress and strategies to improve performance, as needed, on a governmentwide performance website. It also encouraged a more detailed and comprehensive understanding of those strategies by requiring agencies to identify and coordinate the program activities, organizations, regulations, policies, and other activities—both internal and external—that contribute to each agency priority goal. GPRAMA, along with related OMB guidance, established and defined performance management responsibilities for agency officials in key management roles: the Chief Operating Officer (COO), the PIO, and a goal leader responsible for

7GAO, Designing Evaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: January 2012). 8GAO-13-570. 9For additional information on the GPRAMA requirements, see our web page on leading practices for results-oriented management at www.gao.gov/key_issues/managing_for_results_in_government. For more information on these roles, see GAO, Managing for Results: Enhanced Goal Leader Accountability and Collaboration Could Further Improve Agency Performance, GAO-14-639 (Washington, D.C.: July 22, 2014).

GPRAMA Established an Expectation That Evidence Would Have a Greater Role in Agency Decision Making

http://www.gao.gov/products/GAO-12-208G�

http://www.gao.gov/products/GAO-13-570�

http://www.gao.gov/key_issues/managing_for_results_in_government�

http://www.gao.gov/products/GAO-14-639�

Page 5 GAO-15-25 Program Evaluation

coordinating efforts to achieve the cross-agency and agency priority goals.

The PIO role was created in 2007 by executive order.10 GPRAMA established the role in law and specified that it be given to a “senior executive” at each agency who reports directly to the agency’s COO or to its deputy agency head. The PIO is to advise the head of the agency and the COO on goal setting, measurement, and reviewing progress on the agency priority goals. OMB guidance gave PIOs a central role in promoting agency use of evaluation and other evidence to improve program performance, describing their roles as

• “. . . driving performance improvement efforts across the organization, by using goal-setting, measurement, analysis, evaluation and other research, data-driven performance reviews on progress, cross-agency collaboration, and personnel performance appraisals aligned with organizational priorities.”

• “Help components, program office leaders and goal leaders to identify and promote adoption of effective practices to improve outcomes, responsiveness and efficiency, by supporting them in . . . securing evaluations and other research as needed . . . and creating a network for learning and knowledge sharing about successful outcome- focused, data-driven performance improvement methods across all levels of the organization and with delivery partners.”11

The act also charged the Performance Improvement Council (PIC), the Office of Personnel Management (OPM), and OMB with responsibilities to improve agency performance management capacity. The PIC is an interagency council that was created by executive order, but GPRAMA established it in law and specified that it would be chaired by the OMB Deputy Director for Management and that membership would include the PIOs from all 24 CFO Act agencies, as well as any others. The PIC’s duties include facilitating agencies’ exchange of successful practices and the development of tips and tools to strengthen agency performance management, and assisting OMB in implementing certain GPRAMA

10Executive Order No. 13,450, Improving Government Program Performance, (Nov. 13, 2007). 11OMB, Preparation, Submission, and Execution of the Budget. Part 6. Strategic Plans, Annual Performance Plans, Performance Reviews, and Annual Program Performance Reports, OMB Circular No. A-11 (Washington, D.C.: August 2012, updated July 2014).

Page 6 GAO-15-25 Program Evaluation

requirements. The PIC holds “principals only” and broader meetings open to other agency staff, has formed several working groups that focus on issues relating to implementing GPRAMA and related guidance, and provides a networking forum for staff from different agencies who are working on similar issues. In 2012 through 2014, OMB and the PIC supported several interagency forums on evaluation and evidence that were open to all federal agency staff.

The act charged OPM with (1) identifying key skills and competencies needed by federal employees for developing goals, evaluating programs, and analyzing and using performance information for improving governmental efficiency and effectiveness; (2) incorporating those skills and competencies into relevant position classifications; and (3) working with agencies to incorporate these skills and competencies into agency training. OPM identified core competencies for performance management staff, PIOs, and goal leaders and published them in a January 2012 memorandum.12 OPM identified relevant existing position classifications that are related to the competencies and worked with the PIC Capacity Building working group to develop related guidance and tools for agencies. In December 2012, the PIC released a draft Performance Analyst position design, recruitment, and selection toolkit. OPM worked with the Chief Learning Officers Council and the PIC Capacity Building working group to develop a website—the Training and Development Policy wiki—that lists some resources for personnel performance management and implementing GPRAMA. OPM is currently conducting pilot studies through 2015, in collaboration with the Chief Human Capital Officers Council, of how to build staff capacity in several competencies identified as mission critical across government, including data analysis. OPM officials also noted that they make databases, such as the Federal Employee Viewpoint Survey, available to agencies for their staff to use in program evaluations.13

12OPM, “Memorandum for Chief Human Capital Officers: Government Performance and Results Act Modernization Act of 2010 Functional Competencies” (Washington, D.C.: Jan. 3, 2012). 13The Federal Employee Viewpoint Survey measures employees’ perceptions of whether, and to what extent, conditions characterizing successful organizations are present in their agencies. See http://www.fedview.opm.gov.

http://www.fedview.opm.gov/�

Page 7 GAO-15-25 Program Evaluation

OMB has taken several steps to help agencies develop evaluation capacity by issuing guidance, promoting the exchange of evaluation expertise through the PIC, and working selectively with certain agencies. Since 2009, OMB has issued several memorandums urging efforts to strengthen the use of rigorous impact evaluation, and demonstrate the use of evidence and evaluation in budget submissions, strategic plans, and performance plans.14 In May 2012, OMB encouraged agencies to designate a high-level official responsible for evaluation who could develop and manage the agency’s research agenda and provide independent input to agency policymakers on resource allocation and to program leaders on program management. In July 2013, the Directors of OMB, the Domestic Policy Council, the Office of Science and Technology Policy, and the Chairman of the Council of Economic Advisers, jointly issued a memorandum encouraging agencies to adopt an “evidence and innovation agenda”: applying existing evidence on what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery. In particular, the memorandum encouraged agencies to exploit existing administrative data to conduct low-cost experiments, and implement outcome-focused grant designs and research clearinghouses to catalyze innovation and learning.15

OMB staff established an interagency group to promote sharing of evaluation expertise, and organized a series of workshops and interagency collaborations. The workshops addressed issues such as potential procedural barriers to evaluation (e.g., the Paperwork Reduction Act information collection reviews) and promising practices for collecting evidence (e.g., developing a common evidence framework). OMB staff facilitated the collaboration of staff from the Department of Education and the National Science Foundation in developing common standards of evidence for reviewing research proposals, and another group of agencies in developing a common framework of standards for reviewing completed evaluations.

14OMB, Increased Emphasis on Program Evaluations, M-10-01. Memorandum for the Heads of Executive Departments and Agencies (Washington, D.C.: Oct. 27, 2009); and Use of Evidence and Evaluation in the 2014 Budget, M-12-14. Memorandum for the Heads of Executive Departments and Agencies (Washington, D.C.: May 18, 2012). 15OMB, Next Steps in the Evidence and Innovation Agenda, M-13-17. Memorandum for the Heads of Executive Departments and Agencies (Washington, D.C.: July 26, 2013).

OMB Efforts to Improve Agency Evaluation Capacity

Page 8 GAO-15-25 Program Evaluation

Studies of organization or government evaluation capacity have found that it requires analytic expertise and access to credible data as well as organizational support both within and outside the organization to ensure that credible, relevant evaluations are produced and used. Our survey found levels of evaluation expertise, support, and use uneven across the government. For example, 7 of the 24 agencies have central leaders responsible for evaluation; in contrast, 7 agencies reported having no recent evaluations for any of their performance goals.

To address our first objective and guide our assessment of agency evaluation capacity, we reviewed the research and policy literature on evaluation capacity, including assessments of agencies in Canada and the United Kingdom, and guidance from the American Evaluation Association (AEA) and the United Nations Evaluation Group.16 While the details vary, these frameworks commonly emphasize three general categories of elements of organizational, especially national, evaluation capacity:

• An enabling environment supporting the use of evidence in management and policymaking: credible information and statistical systems, legislation or policies to institutionalize monitoring and evaluation, public interest in evidence of government performance, and senior leadership commitment to transparency, accountability, and managing for results.

• Organizational resources to support the supply and use of credible evaluations: a senior evaluation leader; an evaluation office with clearly defined roles and responsibilities, a stable source of funding, and independence; an evaluation agenda, policies and tools to ensure study credibility and utility; staff expertise and access to experts; and collaboration with program managers and stakeholders.

• Evaluation results and use: evaluation quality and credibility; coverage of the agency’s key programs or goals; transparent reporting and public dissemination of reports; recommendation follow-up; and the use of evaluation results in program management, policy making, and budgeting.

16The reference list names the sources we reviewed.

Federal Agencies’ Capacity to Conduct and Use Program Evaluation Is Uneven

An Agency’s Evaluation Capacity Depends on Both Policy Makers’ Requests for Information and the Agency’s Ability to Produce Credible, Relevant Information

Page 9 GAO-15-25 Program Evaluation

To learn about federal agencies’ evaluation capacity, we surveyed the PIOs or their deputies at the 24 CFO Act agencies because of the central role GPRAMA and OMB assigned them to promote agency performance assessment and improvement efforts.17 Our 2012 survey of PIOs found that they held senior leadership positions and that most of them were involved in the central aspects of agency performance management to a large extent.18 Although the PIO position was created in 2007, only one of the initial PIOs continued to hold this position at the time of our 2014 survey. Half had started serving in this position within the past 2 years. Many of our survey respondents held key senior leadership positions in their agencies: 8 PIOs served as the agency’s Chief Financial Officer, another 4 as Assistant Secretary or Deputy for Administration or Management. Seventeen reported to their agency’s COO, 2 to the agency’s administrator or commissioner, and 3 to the agency’s CFO. In order to report on the policies and practices of offices throughout these agencies, we encouraged the PIOs to consult with others when completing the survey and several indicated that they did so.

GPRA represents a central component of the enabling environment for U.S. government evaluation capacity by providing, for over 20 years, a statutory framework for performance management and accountability across the government. Accordingly, most PIOs reported that their senior leadership demonstrated a commitment to using evidence in management and policy making through agency guidance (17), internal agency memorandums (12), congressional hearings (9), and speeches (8). Other avenues offered in comments included budget justifications (10) and town hall meetings or videos for agency managers and staff (2).

Moreover, as we have noted previously, GPRA has produced a solid foundation of generating and reporting performance information. Three- quarters of the agencies (18) said that reliable performance data are available on outcomes for all their priority goals, 3 more said data are

17The number of agency PIOs who responded to survey questions depended on skip instructions contained in the survey, and some PIOs chose not to answer certain questions. See appendix II for the actual number of respondents for each question. 18GAO, Managing for Results: Agencies Have Elevated Performance Management Leadership Roles, but Additional Training Is Needed, GAO-13-356 (Washington, D.C.: Apr. 16, 2013).

Our Survey Respondents

Most Federal Agencies Reported Access to and Commitment to Using Evidence but Fewer Reported Congressional Requests for Program Evaluation

http://www.gao.gov/products/GAO-13-356�

Page 10 GAO-15-25 Program Evaluation

available for more than half their priority goals. (One of the independent agencies was exempt from developing priority goals.)

However, our survey respondents indicated that congressional interest in and requests for program evaluation are not widespread. Although the federal government has long invested in evaluation, about half the agencies (13) reported having explicit agency-wide authority to use appropriated funds for evaluation. Some pointed to specific legislative authorities, while one PIO commented, “Evaluation is considered inherent to responsible management and programs use appropriated fund[s] for this purpose.” Less than half the agencies (10) indicated that they had congressional mandates to evaluate specific programs. However, one- third (7) indicated that they had neither explicit agency-wide authority nor a program-specific requirement to conduct evaluations. The importance of this is that in a prior study agency evaluators told us that not having explicit evaluation authority represented a barrier to the use of program funds for evaluation.19

Our survey asked the PIOs about the agency resources and policies committed to obtaining credible, relevant evaluations. Their responses indicated uneven levels of development across the agencies. About half the agencies (11) reported committing resources to obtain evaluations by establishing a central office responsible for evaluating agency programs, operations, or projects. However, less than a third of agencies have an evaluation plan or agency-wide policies or guidance for ensuring study credibility.

About one-third of the agencies (7) reported having assigned responsibility to a single high-level official to oversee their evaluation studies. Although agencies do not need a central evaluation leader in order to conduct credible evaluations, establishing such a position with clear responsibilities sends a message about the importance of evaluation to agency managers. Almost all these individuals (6) were responsible for setting these agencies’ evaluation agendas but only half (3) were responsible for following-up evaluation recommendations. Similar numbers of departments and independent agencies reported

19GAO, Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research, GAO-11-176 (Washington, D.C.: Jan. 14, 2011), p. 19.

Half of Federal Agencies Report Committing Resources to Obtain Credible, Relevant Program Evaluations

Central Evaluation Leadership and Resources

http://www.gao.gov/products/GAO-11-176�

Page 11 GAO-15-25 Program Evaluation

having such officials with titles such as Chief Evaluation Officer, Chief Strategic Officer, and Assistant Secretary.

According to AEA guidance, a central evaluation office can promote an agency’s evaluation capacity and provide a stable organizational framework for planning, conducting, or procuring evaluation studies. All the agencies with a single official responsible for overseeing evaluations also reported having a central office responsible for evaluating agency programs, operations, or projects, but only about half the agencies in total (11) had a central office. The central offices could have other responsibilities as well, such as strategic planning. Most of these offices were said to be independent of program offices in making decisions about evaluation design, conduct, and reporting and to have access to analytic expertise through external experts or contractors, but about half were reported to have a stable source of funding (6). Funding generally came through regular appropriations, although two agencies reported having evaluation set-asides—that is, the ability to tap a percentage of operating divisions’ appropriations for evaluation. A larger proportion of independent agencies (5 of 9) than departments (6 of 15) reported having central offices.

As discussed earlier, having analytic expertise is a critical element of evaluation capacity. Most agencies with a central office responsible for evaluations (7—8 of 11) reported that the evaluation staff had training and experience to a great or very great extent in each of the following areas: research design and methods, data management and statistical analysis, performance measurement and monitoring, and translating evaluation results into actionable recommendations. Slightly fewer reported that central evaluation office staff had great or very great subject matter expertise (5). Three survey respondents also volunteered that their staff had additional expertise, including economic analysis, geographical information systems and Lean cost reduction analysis.

Organizations, whether government agencies or professional societies, develop written policies or standards in order to provide benchmarks for ensuring the quality of their processes and products. AEA has published guides for the individual evaluator’s practice and for developing and

Few Agencies Have Policies to Ensure Credible and Relevant Evaluations

Page 12 GAO-15-25 Program Evaluation

implementing U.S. government evaluation programs.20 About one-quarter of agencies reported having agency-wide written policies or guidance for key issues addressed in those guides:

• selecting and prioritizing evaluation topics; • consulting program staff and subject matter experts; • ensuring internal or external evaluator independence and objectivity; • selecting evaluation approaches and methods; • ensuring completeness and transparency of evaluation reports; • timely, public dissemination of evaluation findings and

recommendations; or • tracking implementation of evaluation findings.

A few more agencies, but less than half, reported having policies on ensuring quality of data collection and analysis, which could apply to research as well as program evaluation. Central evaluation leadership was not required to adopt evaluation policies; as only about half of the agencies with agency-wide evaluation policies had a central evaluation office. Agencies provided us with examples of guidance on information quality or scientific integrity as well as program evaluations specifically.21

We, along with OMB and AEA, have all noted that developing an evaluation agenda is important for ensuring that an agency’s often scarce research and evaluation resources are targeted to the most important issues and can shape budget and policy priorities and management practices.22 Less than a third of the agencies (7) reported having an agency-wide evaluation plan. Most such plans were reported to cover multiple years and programs across all major agency components. Senior agency officials and program managers were said to have been consulted in developing all these plans, but few agencies reported consulting

20American Evaluation Association, An Evaluation Roadmap for a More Effective Government (Oct. 2013); Guiding Principles for Evaluators (2004) accessible at www.eval.org. 21For an example of a comprehensive set of evaluation policies, see U.S. Agency for International Development, Evaluation: Learning from Experience. USAID Evaluation Policy (Washington, D.C.: Jan. 2011). http://www.usaid.gov/evaluation. 22GAO-11-176; OMB, Evaluating Programs for Efficacy and Cost-Efficiency, M-10-32. Memorandum for the Heads of Executive Departments and Agencies (Washington, D.C.: July 29, 2010); American Evaluation Association (2013).

http://www.eval.org/�

http://www.usaid.gov/evaluation�

http://www.gao.gov/products/GAO-11-176�

Page 13 GAO-15-25 Program Evaluation

congressional stakeholders or researchers. All but 1 of the 7 agencies that had a plan also had a central evaluation office.

Because we found in a previous report that stakeholder involvement facilitates the use of evaluation studies, we asked whether stakeholders were consulted in designing and conducting evaluation studies, either formally or informally.23 Almost all the PIOs reported consulting senior agency officials (20) and program managers (21) and three-quarters consulted researchers, but few (5) reported consulting congressional staff, less than local program providers or regulated entities.

Agency evaluation offices are located at different organizational levels, which we have previously found affects the scope of their program and analytic responsibilities as well as the range of issues they consider. In a previous study, we found that evaluators in central research and evaluation offices described having a broader and more flexible choice of topics than did evaluators in program offices.24 In our 2014 survey, half the federal agencies (12) reported that some agency components (such as an administration or bureau) had a central office responsible for evaluation and that the number of such components ranged from 1 to 12 within a department or independent agency. These offices generally existed in addition to, rather than instead of an agency-wide office responsible for evaluation; as a result, 10 agencies had neither type of office. As might be expected, component offices were less likely than central offices to be considered independent of program offices (6 of 12 agencies reported that all or many of their offices had independence in decision making), but 10 of 12 reported that all or many of these offices had access to external experts, and, like the central offices, few reported having a stable source of funds.

About half the agencies with component central offices for evaluation reported that the evaluation staff had training and experience to a great or very great extent in research design and methods, data management and statistical analysis, performance measurement and monitoring, and translating evaluation results into actionable recommendations. These were slightly lower than the ratings for the central office staff’s training. As one might expect, staff were characterized as having great to very great

23GAO-13-570. 24GAO-11-176, p. 20.

Agency Components’ Resources and Policies

http://www.gao.gov/products/GAO-13-570�

http://www.gao.gov/products/GAO-11-176�

Page 14 GAO-15-25 Program Evaluation

subject matter expertise more often in component offices (9 of 12) than in central evaluation offices (5 of 11).

Only a few PIOs (2 to 4) reported that many or all component central offices for evaluation had written evaluation policies or guidance for any of the issues we listed. More often PIOs (2 to 6) reported not knowing if they had those specific policies.

To assess the results or outcomes of agency evaluation activity, our survey asked the PIOs about the characteristics of the evaluations they produced and their use in decision making. In line with the level of resources they committed to evaluation, the availability and use of program evaluations were uneven across the 24 federal agencies. Even though agencies may not have many evaluations, more than a third report using them from a moderate to a very great extent to support several aspects of program management and policy making.

Because agencies use the term “program” in different ways, we chose to assess agencies’ evaluation coverage of key programs and missions by the proportion of performance goals for which evaluations had been completed in the past 5 years or were in progress. The number of performance goals may vary across agencies but, per OMB guidance, they are supposed to be specific, near-term, realistic targets that an agency seeks to influence to advance its mission, and publicly reports. Only four agencies reported full evaluation coverage of their performance goals. Two-thirds of the agencies reported evaluation coverage of less than half their performance goals; including 7 that reported having evaluations for none of their performance goals. Evaluation coverage was greater in agencies that established centralized authority for evaluation. Three of the 4 agencies with full coverage of their performance goals had both a central evaluation leader and central evaluation office, while all 7 agencies with no coverage had neither. Interestingly, 2 of the 7 agencies that reported having no evaluations of their performance goals did report having component evaluation offices, so they might have had some evaluations that simply did not address topics considered key to advancing their mission.

GAO guidance notes that strong evaluations rely on sufficient and appropriate evidence; document their assumptions, procedures, and

Agencies Report Moderate Use of Evaluations in Managing Programs, Setting Policy, or Allocating Resources

Page 15 GAO-15-25 Program Evaluation

modes of analysis; and rule out competing explanations.25 Thus, transparent reporting of data sources and analyses are critical for ensuring that evaluations are considered credible and trustworthy. About half the PIOs (10) reported that their evaluation reports are transparent to a great or very great extent in describing the data sources used and the analyses performed forming the basis of conclusions; another 7 indicated that they did not know or did not respond to the question.

According to the evaluation capacity literature, timely, public dissemination of evaluation findings is important to support government accountability for results to the legislature and the public and to ensure that findings are available to inform decision making. Half the agencies (11) reported publicly disseminating their evaluation results by posting reports to a searchable database on their websites; fewer reported presenting findings at professional conferences (9), sending a notice and link to the report through electronic mailing lists (7), or conducting webinars on findings for the policy community (6). A couple of the PIOs commented that they post some, but not all, reports on the agency website. Of the 11 agencies posting evaluation reports to a website, half reported that they did so within 3 months of completion, although 1 indicated it can take from 6 months to a year. In addition, a few agencies sponsor research clearinghouses that review evaluations of social interventions and provide the results in searchable databases on their websites to help managers and policy makers identify and adopt effective practices.

If program evaluations or any form of performance information are to lead to performance improvement, they must be acted on. Seven agencies reported that they had procedures for obtaining management’s response to evaluation recommendations, 8 for obtaining follow-up action on those recommendations. In their comments, a few PIOs noted that they had policies for responding to reports or recommendations from GAO or the Inspector General. Another PIO reported that a number of internal briefings are held to ensure management awareness of evaluation findings as well as a cross-agency research utilization committee composed of staff from program, public affairs, and congressional and intergovernmental relations offices that decides on the appropriate level of publicity effort for the report.

25 GAO-12-208G.

Responding to or Using Evaluation Findings

http://www.gao.gov/products/GAO-12-208G�

Page 16 GAO-15-25 Program Evaluation

Over a third of the agencies (9 to 10) reported that evaluations were used to a moderate or greater extent to support policy changes, budget changes, or internal proposals for change in resource allocation or management, or to award competitive grants (figure 1). Five agencies reported using evaluation to support all these activities to a moderate or greater extent on average. In comments, PIOs described a variety of ways in which evaluation evidence could be used in awarding competitive grants: reviewing the merit of research proposals, evaluating grantee prior performance and outcomes, assessing credit worthiness, and allocating tiered evidence-based funding, which varies the level of funding based on the extent and quality of the evaluation evidence supporting a program’s effectiveness.

Figure 1: Number of Agencies Citing Evaluation Evidence to Support Various Decisions

Note: Survey items were abbreviated.

Agencies with centralized evaluation authority, independence, and expertise reported greater evaluation use in management and policy making, demonstrating its importance. More than half of the 7 agencies that reported great use of evaluation had a senior evaluation leader or a central evaluation office. Moreover, the agencies whose central offices were independent of the program office, those with access to external

Page 17 GAO-15-25 Program Evaluation

experts or contractors, and those whose staff were rated as having great or better expertise in research methods and subject matter reported greater use of evaluation in decision making.

GPRAMA was enacted in January 2011, revising existing GPRA provisions and adding new reporting requirements. Around the same time, OMB increased its outreach to agencies to encourage them to conduct program evaluations. We assessed change in agency evaluation capacity in this period through survey questions about when an office started conducting evaluations and whether the frequency of certain activities had changed. While organizational changes in evaluation capacity were few during this period, half the agencies reported a greater use of evaluation in decision making since 2010.

Organizational evaluation capacity has grown some since 2010. One-third of the agencies have a high-level official responsible for oversight of the agency’s evaluation studies, and 2 of those 7 positions were created after 2010, both in 2013. In fact, in its May 2012 memorandum, OMB encouraged agencies to designate a high-level official responsible for evaluation who can

• “Develop and manage the agency’s research agenda; • Conduct or oversee rigorous and objective studies; • Provide independent input to agency policymakers on resource

allocation and to program leaders on program management; • Attract and retain talented staff and researchers, including through

flexible hiring authorities such as the Intergovernmental Personnel Act; and

• Refine program performance measures, in collaboration with program managers and the Performance Improvement Officer.”26

In addition, 4 of 11 agencies with a central office responsible for evaluation reported that this office started conducting evaluations after 2010. One agency added both a central leader and a central office in 2013; 3 others just added a central office. Of the 12 agencies that reported having evaluation offices in their major components, most existed before GPRAMA was enacted, but 5 agencies have established new component evaluation offices since then.

26OMB M-12-14, p. 4

Some Agencies Improved Their Evaluation Capacity after GPRAMA Was Enacted

A Few Agencies Centralized Their Evaluation Responsibilities after GPRAMA Was Enacted

Page 18 GAO-15-25 Program Evaluation

Presumably in response to greater administration attention to program evaluation, half the agencies reported that efforts to improve their capacity to conduct credible evaluations had increased at least somewhat since GPRAMA was enacted in January 2011. About half the PIOs reported increases in staff participation in evaluation conferences and knowledge sharing forums, hiring staff with research and analysis expertise, training staff in research and evaluation skills, and consultation with external research and evaluation specialists. Nine agencies reported increases in all these activities. Most of the remaining agencies reported no change in training or consultation with specialists (4 to 5), or decreases in hiring or participating in conferences (4 to 5) in this period. These decreases may reflect federal budget constraints and the general decline in federal hiring in recent years.

Figure 2: Agencies Reporting Change since 2010 in Efforts to Improve Their Capacity to Conduct Evaluations

Note: Survey items were abbreviated.

In line with the increases reported in capacity building activities and organizational resources, about half the agencies reported that their use of evaluation as supportive evidence had increased at least somewhat since 2010 (only a few reported great increases). About half the PIOs reported that the use of evaluation had increased for implementing changes in program management or performance, designing or

Half the Agencies Reported Increasing Their Capacity Building Activities after GPRAMA Was Enacted

Half the Agencies Reported Increasing Their Use of Evaluations as Evidence in Decision Making after 2010

Page 19 GAO-15-25 Program Evaluation

supporting program reforms, sharing what works or other lessons learned with others, allocating resources within a program, or supporting program budget requests. The rest reported that their use of evaluation evidence remained about the same in this period, with none reporting a decline in use of evaluation as evidence. Eight agencies reported increased use in all these activities, and an equal number reported that their use remained the same on all. Since, in a separate question, 5 agencies either provided no opinion or reported little or no current use of evaluation evidence to support budget, policy, or program management, we conclude that this group has continued to make little or no use of evaluations since 2010.

Figure 3: Agencies Reporting Change since 2010 in Citing Evaluation as Supporting Evidence in Decisions

Note: Survey items were abbreviated.

Page 20 GAO-15-25 Program Evaluation

Our survey asked the PIOs how useful various activities or resources were for improving their agency’s capacity to conduct credible evaluations. Several PIOs did not answer these questions, in part because they were not familiar with such activities. Many of those who did respond found that hiring, professional networking, consulting with experts, and training as well as some of the GPRAMA accountability provisions were very useful for improving capacity to conduct evaluations. Our survey also asked about the usefulness of various activities or resources for improving an agency’s capacity to use evaluations in decision making. Again, several agencies did not respond, but most of those that did reported that engaging program staff, conducting quarterly progress reviews, and holding goal leaders accountable for progress on agency priority goals were very useful in improving agency capacity to make use of evaluation information. Some other GPRAMA-related activities were not found as useful for enhancing evaluation use. In addition, agencies had not taken full advantage of available technology to disseminate evaluation results, thus potentially limiting their influence on decision making.

Our survey asked the PIOs about the usefulness of 14 different actions or resources for improving their capacity to conduct evaluations, drawn from the literature and some GPRAMA provisions related to building agency capacity. About a third of the respondents indicated either that they had no opinion or did not respond to these questions, similar to the number not responding or reporting no change in the use of capacity-building activities since 2010. About two-thirds of agencies (15) reported hiring staff with research and analysis expertise, and 11—nearly half of the PIOs—thought it was very useful for improving agency capacity to conduct credible evaluations. Almost half the agencies used special hiring authorities, such as the Presidential Management Fellows, Intergovernmental Personnel Act, or American Association for the Advancement of Science (AAAS) fellows program, and generally found them useful for improving agency evaluation capacity. Other agency- specific means of obtaining staff were mentioned in comments—for example, an Evaluation Fellowship Program at the Centers for Disease Control and Prevention. Figure 4 summarizes agencies’ reports on the usefulness of the full range of activities and resources posed for building capacity to conduct evaluations.

Some Agencies Reported that Professional Networking, Hiring, Engaging Program Staff, and Some GPRAMA Provisions Were Useful for Building Evaluation Capacity

PIOs Reported Hiring, Professional Networking, and Consulting with Experts Were Very Useful in Improving the Capacity to Conduct Evaluations

Page 21 GAO-15-25 Program Evaluation

Figure 4: PIOs’ Views on Usefulness of Activities and Resources for Improving Agency Capacity to Conduct Credible Evaluations

Note: Survey items were abbreviated.

The PIO survey respondents also gave high marks to professional networking for building staff capacity. Two-thirds of the PIOs reported that staff participation in professional conferences or evaluation interest groups for knowledge sharing was useful, with 9 PIOs citing these activities as very useful in improving agency capacity to conduct credible evaluations. Examples mentioned included the Association for Public Policy Analysis and Management research conference and an Evaluation Day conference sponsored by the U.S. Department of Health and Human Services (HHS) Office of the Assistant Secretary for Planning and Evaluation.

Page 22 GAO-15-25 Program Evaluation

The exchange of evaluation tips and leading practices through the PIC or other network was considered moderately useful for capacity building by a third of the PIOs. PIOs provided examples of information-sharing networks besides the PIC, such as OMB’s Evaluation Working Group, which holds governmentwide meetings on government performance topics; Federal Evaluators, an informal association of evaluation officials across government; Washington Evaluators, a local affiliate of the American Evaluation Association; and the National Academy of Public Administration. Some agencies have established informal networks to share information internally, such as HHS and the U.S. Department of Labor. Also mentioned were communities of practice that engage both public and private sectors but are focused on a specific domain—for example, the Organisation for Economic Co-operation and Development’s EvalNet, which focuses on international development, and the Environmental Evaluators Network.

Consultation with external experts for conceptual or technical support was rated as very useful for improving the capacity to conduct evaluations by most using it (9 of 15). However, this did not apply to other forms of external consultation. Seven agencies reported having an annual or multi- year evaluation agenda, and 3 of them reported consulting with congressional or other external stakeholders on their plan. These 3 found consultation useful to varying degrees for building their agency’s evaluation capacity to conduct evaluation.

Training in specific skills and knowledge—for example, types of evidence, assessing evidence quality, report writing, and communication—is frequently cited in the evaluation literature as a way to build organizational or individual evaluation capacity. Besides asking about participating in professional conferences and networks, our survey asked about the usefulness of training in evaluation skills—for example, describing program logic models, choosing appropriate evaluation designs, and collecting and analyzing data. Half the agencies reported engaging in internal or external training—whether delivered in a classroom, online, or in webinars. Half the agencies using internal training reported that it was very useful for improving capacity to conduct credible evaluations. PIOs who reported on agency experience with external evaluation training were less enthusiastic, but still considered the training useful for developing evaluation skills overall.

OMB, in addition to encouraging agencies to conduct evaluations through guidance, sponsored a number of governmentwide open forums on performance issues. About half the PIOs reported a range of opinions on

PIOs Reported Training Is Needed to Build Skills

Page 23 GAO-15-25 Program Evaluation

the usefulness of the OMB forums on the Paperwork Reduction Act, procurement, data sharing, and related rules and procedures to help improve agency capacity for conducting credible evaluations.

Nevertheless, 7 or more of the agencies identified training or guidance in several skills as still needed to a great or very great extent to improve their agencies’ capacity to conduct credible evaluations. These skills included: translating evaluation results into actionable recommendations—a requirement for getting evaluation results used— data management and statistical analysis, and performance measurement and monitoring. Few reported that more training in research design and methods or subject matter expertise was greatly needed. Our survey asked what other types of training or guidance might be needed to improve agency capacity. A few PIOs commented that training is needed in preparing statements of work for evaluation contracts, data analytics and visualization of information, and learning how to effectively use evidence and evaluation information.

Our survey asked about several activities and resources related to GPRAMA provisions linked to creating an enabling environment for agency evaluation capacity. Majorities of PIOs stated that conducting quarterly progress reviews on their priority goals, and holding goal leaders accountable for progress on those goals, were moderately to very useful in improving their agency’s ability to conduct credible evaluations. In response to GPRAMA provisions to improve agency performance management capacity, the PIC and OPM developed a Performance Analyst position design, recruitment, and selection toolkit to assist agencies’ hiring. Seven PIOs reported that their agencies used the toolkit, and 3 did not find it useful for building agency evaluation capacity.

About a third of the PIOs reported that their agencies made an effort to incorporate the core competencies that OPM identified for performance management staff into internal agency training. However, 2 of the 7 agencies did not find the effort useful for improving staff evaluation capacity. The competencies primarily address general management skills and define planning and evaluating fairly simply—as setting and monitoring progress on performance goals—so they do not address some of the specific analytic skills PIOs reported were still needed for conducting evaluations. GAO previously recommended that OPM, in coordination with the PIC and the Chief Learning Officer Council, identify

PIOs Reported Some GPRAMA–Related Activities Useful for Building the Capacity to Conduct Evaluation

Page 24 GAO-15-25 Program Evaluation

performance management competency areas needing improvement and work with agencies to share information about available agency training in those areas.27 OPM agreed with those recommendations and has embarked on a 2-year pilot program to test how to build capacity in several mission critical competencies identified across government, such as strategic thinking, problem solving, and data analysis, to ensure that both program staff and management can use evaluation and analysis of program performance.

OMB senior officials also engaged with agency officials on the Performance Improvement Council to collaborate on improving program performance. Eight of the 14 agencies that responded considered the exchange of evaluation tips and leading practices through the PIC or other networks as at least moderately useful for improving their evaluation capacity. For example, the PIC developed a guide to best practices for setting milestones and a guide and evaluation tool to help agencies set their agency priority goals.

Previously, we found that experienced evaluators emphasized three basic strategies to facilitate evaluation’s influence on program management and policy: demonstrate leadership support of evaluation for accountability and program improvement, build a strong body of evidence, and engage stakeholders throughout the evaluation process.28 Accordingly, our survey asked the PIOs how useful various activities or resources were for improving their agency’s capacity to use evaluations in decision making. Several did not answer these questions because they did not use the particular activity or resource or had no opinion. The PIOs who responded mainly cited engaging program staff, conducting quarterly progress reviews, and holding goal leaders accountable for progress on agency priority goals as very useful for improving agency capacity to make use of evaluation information in decision-making.

Over two-thirds of the PIOs responded that involving program staff in planning and conducting evaluation studies was useful for improving agency use of evaluation; 11 saw it as very useful. Engaging staff throughout the process can gain their buy-in on the relevance and

27GAO-13-356. 28GAO-13-570.

PIOs Reported Engaging Program Staff and Holding Goal Leaders Accountable Were Very Useful for Building Capacity to Use Evaluations in Decision Making

http://www.gao.gov/products/GAO-13-356�

http://www.gao.gov/products/GAO-13-570�

Page 25 GAO-15-25 Program Evaluation

credibility of evaluation findings; providing program staff with interim results or lessons learned from early program implementation can help ensure timely data for program decisions. Majorities of PIOs affirmed that other forms of program staff engagement were also very useful: providing program staff and grantees with technical assistance on evaluation and its use and agency peer-to-peer presentations of evaluation studies to discuss methods and findings

Figure 5: PIOs’ Views on Usefulness of Activities and Resources for Improving Agency Capacity to Use Evaluations in Decision Making

Note: Survey items were abbreviated.

As mentioned earlier, majorities of PIOs viewed the new GPRAMA activities of conducting quarterly reviews and holding goal leaders accountable as moderately to very useful for improving agency capacity to conduct credible evaluations. Majorities of the responding PIOs also viewed those same activities as moderately to very useful for improving agency capacity to use evaluations in decision making. However, another GPRAMA provision—coordinating with OMB and other agencies to review progress on cross-agency priority (CAP) goals—met with a range

Page 26 GAO-15-25 Program Evaluation

of opinions. Equal numbers reported that it was moderately to very useful, somewhat useful, or not useful at all for improving an agency’s use of evaluation. Because the 14 CAP goals for this period cover 5 general management improvement and 9 cross-cutting but specific policy areas, some of the 24 PIOs may have been more involved than others in those reviews.

Other activities potentially useful for improving the capacity to use information from evaluations rely on leveraging resources. A third of the PIOs reported that exchanging leading practices, tips, and tools for using evidence to improve program or agency performance through the PIC or other network was moderately or very useful in improving agency capacity to use evaluation results in decision making. Many of the same networks named as helping to improve their capacity to conduct credible evaluations were also named with regard to improving capacity to use evaluations in decision making. These included the Environmental Evaluators Network, Federal Evaluators, the National Academy of Public Administration, and the OMB Evaluation Working Group.

Seven agencies reported having an agency-wide annual or multi-year evaluation plan or agenda of planned studies, and 6 PIOs reported consulting with congressional and other external stakeholders on that plan. However, these consultations were not viewed as useful for improving their agency’s capacity to use evaluations in decision making. The absence of consultation may miss an opportunity to ensure that evaluations will address the questions of greatest interest to congressional decision makers and will be perceived as credible support for proposed policy or budget changes. In previous work, we found that dialog between congressional committees and executive branch agencies was necessary to achieve a mutual understanding that would allow agencies to provide useful information for oversight.29

Previously we found that a key strategy for promoting the use of evaluation findings was to make them digestible and usable and to proactively disseminate them. Our survey posed various options that agencies could take to publicly disseminate their evaluation findings. Half

29GAO, Program Evaluation: Improving the Flow of Information to the Congress, GAO/PEMD-95-1 (Washington, D.C.: Jan. 30, 1995), and Managing for Results: A Guide for Using the GPRA Modernization Act to Help Inform Congressional Decision Making, GAO-12-621SP (Washington, D.C.: June 15, 2012).

Agencies May Not Be Taking Full Advantage of Technology to Disseminate Evaluation Results

http://www.gao.gov/products/GAO/PEMD-95-1�

http://www.gao.gov/products/GAO-12-621SP�

Page 27 GAO-15-25 Program Evaluation

the respondents reported posting evaluation reports in a searchable database on their websites, and half of them viewed this practice as moderately to very useful for improving their agency’s capacity to use evaluations in decision making. However, 3 did not find the practice useful. Electronic mailing lists are more proactive than posting a report to a website and permit tailoring the message to different audiences. A third of all respondents disseminated evaluation reports by electronic mailing lists, which most saw as somewhat to very useful for facilitating the use of evaluations in decision making. Tailoring messages for particular audiences—for example, federal policy makers, state and local agencies, and local program affiliates—may, however, increase the applicability and use of evaluation findings by these other audiences.

GPRAMA requires OMB to provide quarterly updates on agency and cross-agency priority goals on a central, government-wide website, Performance.gov, to make federal program and performance information more accessible to the Congress and the public. In our survey, PIO reviews were mixed about the utility of this website to improve agency capacity to use evaluations in decision making. Almost half the agencies found the practice somewhat to moderately useful for improving the agencies’ use of evaluation findings in decision making, but one-fourth of the agencies did not. In 2013, GAO reviewed Performance.gov and recommended that OMB work with the General Services Administration and the PIC to clarify specific ways that intended audiences could use the website and specify changes to support these uses.30 OMB staff agreed with our recommendations, and Performance.gov continues to evolve. Currently each agency has a home page that provides links to the agency’s strategic plan, annual performance plans and reports, and other progress reviews.

Data.gov is a federal government website that provides descriptions of datasets generated or held by the federal government in order to increase the ability of the public to locate, download, and use those datasets. A third of the PIOs reported that sharing databases in public repositories such as Data.gov for researchers and the public to use helped in improving agency capacity to use evaluations in decision making, but 1 thought it was not useful. However, a third of PIOs stated that the agency

30GAO, Managing for Results: Leading Practices Should Guide the Continued Development of Performance.gov, GAO-13-517 (Washington, D.C.: June 6, 2013).

http://www.gao.gov/products/GAO-13-517�

Page 28 GAO-15-25 Program Evaluation

did not use this vehicle. Vehicles such as Data.gov and Performance.gov are primarily intended to improve government transparency and expand information’s use by the Congress and the public, but they can also help support agency requests for budget and policy changes to improve government performance.

Although OMB and several agencies have taken steps since 2010 to expand federal evaluation efforts, most agencies demonstrate rather modest evaluation capacity. Those with centralized evaluation authority reported greater evaluation coverage and use in decision making, but additional effort will be required to expand agencies’ evaluation capacity beyond those that already possess evaluation expertise. In addition to hiring and training staff and consulting experts, promoting information sharing through informal and formal evaluation professionals’ networks offers promise for building agencies’ capacity to conduct evaluation in a constrained budget environment. Engaging program staff, regularly reviewing progress on agency priority goals, and holding goal leaders accountable can help build agency use of evaluation in decision making, as our survey results show. While timely, public dissemination of performance and evaluation results may not directly influence agency decision making, it is important to support government transparency and accountability for results to the Congress and the public.

Directly engaging intended users (for example, involving program staff in planning and conducting evaluations and holding regular progress reviews) was strongly associated with increasing evaluation use in internal agency decision making. In contrast, few agencies reported consulting congressional and other external stakeholders in conducting their evaluation studies or developing their evaluation agendas. However, some program reforms require program partners and legislators to take action. Engaging congressional and other stakeholders in evaluation planning might increase their interest in evaluation as well as their adoption of evaluation findings and recommendations.

In the absence of explicit authority or congressional request, agencies may be reluctant to spend increasingly scarce funds on evaluation studies that are perceived as resource intensive. A stable source of evaluation funding could help maintain a viable evaluation program that produced a steady stream of information to guide program management and policy making. Even so, only a quarter of the agencies in our survey reported that their evaluation offices had a stable source of funding. Congressional appropriators could direct the use of program or agency funds for

Concluding Observations

Page 29 GAO-15-25 Program Evaluation

evaluating federal programs and policies. As we have noted before, congressional committees can also communicate their interest in evaluation in a variety of ways to encourage agencies to produce credible, relevant studies that inform decision making:31

• consult with agencies on proposed revisions to their strategic plans and priority goals, as GPRAMA requires them to do every 2 years, to ensure that agency missions are focused, goals are specific and results-oriented, and strategies and funding expectations are appropriate and reasonable;

• request agency evaluations to address specific questions about the implementation and results of major program or policy reforms, in time to consider their results in program reauthorization; and

• review agencies’ annual evaluation plans or agendas to ensure that they address issues that will inform budgeting, reauthorization, and ongoing program management.

We requested comments on a draft of this report from the Director of the Office of Management and Budget, whose staff provided technical comments that we incorporated as appropriate, and from the Director of the Office of Personnel Management, who provided none.

We are sending copies of this report to other interested congressional committees, and the Director of the Office of Management and Budget and the Director of the Office of Personnel Management. In addition, the report will be available on our web site at http://www.gao.gov.

31GAO-12-621SP, and GAO-13-570.

Agency Comments

http://www.gao.gov/�

http://www.gao.gov/products/GAO-12-621SP�

http://www.gao.gov/products/GAO-13-570�

Page 30 GAO-15-25 Program Evaluation

If you or your staff have any questions about this report, please contact me at (202) 512-2700 or by e-mail at [email protected]. Contacts for our Office of Congressional Relations and Office of Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Nancy Kingsbury, Ph.D. Managing Director Applied Research and Methods

mailto:[email protected]�

Appendix I: Methodology for the Survey of Performance Improvement Officers in CFO Act Agencies

Page 31 GAO-15-25 Program Evaluation

We administered a web-based questionnaire from May 2, 2014, to June 19, 2014, on federal agency evaluation capacity resources and activities to the Performance Improvement Officers (PIO) or their deputies at the 24 agencies covered by the Chief Financial Officers Act of 1990 (CFO act).1 We received responses from all 24 agencies (listed at the end of this appendix.) The survey gave us information about agencies’ evaluation resources, policies, and activities, and the activities and resources they have found useful in building their evaluation capacity. (The survey questions and summarized results are in appendix II.) We sent respondents an e-mail invitation to complete the survey on a secure GAO web server. Each e-mail contained a unique username and password. During the data collection period, we sent follow-up e-mails and, if necessary, called nonresponding agencies on the telephone.

Because this was not a sample survey, it has no sampling errors. In practice, however, any survey may introduce nonsampling errors that stem from differences in how a particular question is interpreted, the availability of sources of information, or how the survey data are analyzed. All can introduce unwanted variability into the survey results. We took a number of steps to minimize these nonsampling errors. A social science survey specialist designed the questionnaire, in collaboration with our staff who had subject matter expertise. In addition, we pretested the questionnaire in person with PIOs at three federal agencies to make sure that the questions were relevant, clearly stated, easy to comprehend, and unbiased. We also affirmed that data and information the PIOs would need to answer the survey were readily obtainable and that answering the questionnaire did not place an undue burden on them.

Additionally, a senior methodologist within our agency independently reviewed a draft of the questionnaire before we administered it. We made appropriate revisions to its contents and format after the pretests and independent review. When we analyzed data from the completed survey, an independent analyst reviewed all computer programs used in our analysis. Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire; thus, we did not key the data into a database, avoiding data entry errors.

1The CFO Act agencies are listed at 31 U.S.C. § 901(b).

Appendix I: Methodology for the Survey of Performance Improvement Officers in CFO Act Agencies

Page 32 GAO-15-25 Program Evaluation

Additionally, in reviewing the agencies’ answers, we confirmed that the PIOs had correctly bypassed inapplicable questions (such as questions we expected them to skip). We concluded from our review that the survey data were sufficiently reliable for the purposes of this report.

The 24 agencies subject to the CFO Act include

• Agency for International Development • Department of Agriculture • Department of Commerce • Department of Defense • Department of Education • Department of Energy • Department of Health and Human Services • Department of Homeland Security • Department of Housing and Urban Development • Department of the Interior • Department of Justice • Department of Labor • Department of State • Department of Transportation • Department of the Treasury • Department of Veterans Affairs • Environmental Protection Agency • General Services Administration • National Aeronautics and Space Administration • National Science Foundation • Nuclear Regulatory Commission • Office of Personnel Management • Small Business Administration • Social Security Administration.