Analyze the Power of Administrative Rulemaking

Loading... Accessibility Information and Tips Back 1 article(s) will be saved.

The link information below provides a persistent link to the article you've requested.

Persistent link to this record: Following the link below will bring you to the start of the article or citation. Cut and Paste: To place article links in an external web document, simply copy and paste the HTML below, starting with "<a href"

To continue, in Internet Explorer, select FILE then SAVE AS from your browser's toolbar above. Be sure to save as a plain text file (.txt) or a 'Web Page, HTML only' file (.html). In FireFox, select FILE then SAVE FILE AS from your browser's toolbar above. In Chrome, select right click (with your mouse) on this page and select SAVE AS

Record: 1

Title:

The quality and use of regulatory analysis in 2008.

Authors:

Ellig J; Mercatus Center at George Mason University, Arlington, VA, USA. [email protected] McLaughlin PA

Source:

Risk analysis : an official publication of the Society for Risk Analysis [Risk Anal] 2012 May; Vol. 32 (5), pp. 855-80. Date of Electronic Publication: 2011 Nov 07.

Publication Type:

Journal Article

Language:

English

Journal Info:

Publisher: Blackwell Publishers Country of Publication: United States NLM ID: 8109978 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1539-6924 (Electronic) Linking ISSN: 02724332 NLM ISO Abbreviation: Risk Anal Subsets: MEDLINE

Imprint Name(s):

Publication: 2002- : Malden, MA : Blackwell Publishers Original Publication: New York : Plenum Press, c1981-

MeSH Terms:

Government Agencies* Cost-Benefit Analysis ; Information Services ; United States

Abstract:

This article assesses the quality and apparent use of regulatory analysis for economically significant regulations proposed by federal agencies in 2008. A nine-member research team used a six-point (0-5) scale to evaluate regulatory analyses according to criteria drawn from Executive Order 12866 and Office of Management and Budget Circular A-4. Principal findings include: (1) the average quality of regulatory analysis, though not high, is somewhat better than previous regulatory scorecards have shown; (2) quality varies widely; (3) biggest strengths are accessibility and clarity; (4) biggest weaknesses are analysis of the systemic problem and retrospective analysis; (5) budget or "transfer" regulations usually receive low-quality analysis; (6) a minority of the regulations contain evidence that the agency used the analysis in significant decisions; (7) quality of analysis is positively correlated with the apparent use of the analysis in regulatory decisions; and (8) greater diffusion of best practices could significantly improve the overall quality of regulatory analysis. (© 2011 Society for Risk Analysis.)

Entry Date(s):

Date Created: 20111109 Date Completed: 20120822 Latest Revision: 20120502

Update Code:

20210210

DOI:

10.1111/j.1539-6924.2011.01715.x

PMID:

22059696

Persistent link to this record (Permalink):

https://proxy1.ncu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=mdc&AN=22059696&site=eds-live

Cut and Paste:

<A href="https://proxy1.ncu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=mdc&AN=22059696&site=eds-live">The quality and use of regulatory analysis in 2008.</A>

Database:

MEDLINE Complete

The Quality and Use of Regulatory Analysis in 2008. 

This article assesses the quality and apparent use of regulatory analysis for economically significant regulations proposed by federal agencies in 2008. A nine‐member research team used a six‐point (0–5) scale to evaluate regulatory analyses according to criteria drawn from Executive Order 12866 and Office of Management and Budget Circular A‐4. Principal findings include: ( 1) the average quality of regulatory analysis, though not high, is somewhat better than previous regulatory scorecards have shown; ( 2) quality varies widely; ( 3) biggest strengths are accessibility and clarity; ( 4) biggest weaknesses are analysis of the systemic problem and retrospective analysis; ( 5) budget or "transfer" regulations usually receive low‐quality analysis; ( 6) a minority of the regulations contain evidence that the agency used the analysis in significant decisions; ( 7) quality of analysis is positively correlated with the apparent use of the analysis in regulatory decisions; and ( 8) greater diffusion of best practices could significantly improve the overall quality of regulatory analysis.

Keywords: Benefit‐cost; cost‐benefit analysis; regulation; regulatory impact analysis; regulatory process; regulatory reform

Since 1974, all presidents have issued executive orders requiring regulatory agencies to analyze the anticipated effects of proposed regulations. President Obama's Executive Order 13563 reaffirms the principles and review processes in Executive Order 12866, which has guided regulatory analysis since 1993.([[ 1]])

Scholars, decisionmakers, interest groups, and advocates spill much ink debating whether and how agencies should do regulatory analysis. Some view regulatory analysis as an imperfect but necessary tool for understanding regulation's effects.([ 3]) Others see regulatory analysis as an attempt to "stack the deck" against new regulations, arguing that costs are easier to measure than benefits (pp. 35–36).([[ 4]]) Yet others regard regulatory analysis as a tool for crafting "smart" regulations that do more good than harm.([[ 6]]) Some view the whole enterprise as a fundamentally immoral attempt to put prices on public values that cannot be assigned monetary worth (pp. 61–62).([ 4])

Nonetheless, many scholars agree that regulatory analysis is here to stay.([ 3], [ 6], [ 8], [ 9], [10], [11], [12]) Executive Order 13563 provides further evidence of this. Yet as former Office of Information and Regulatory Affairs (OIRA) Administrator Sally Katzen noted, "we still do not know whether the agencies are implementing CBA [cost‐benefit analysis] appropriately and whether the way the agencies use CBA produces better regulatory decisions" (pp. 1314���1315).([11])

This article takes up that challenge. We apply a 12‐point qualitative framework to evaluate regulatory analyses of "economically significant" rules that were reviewed by OIRA in 2008 and proposed in the Federal Register.4 The evaluation criteria are drawn from Executive Order 12866 and Office of Management and Budget (OMB) Circular A‐4, the 2003 guidance document on best practices in regulatory analysis.([13])

Our evaluation yields numerous insights into the quality and use of regulatory analysis. Principal findings include: ( 1) the average quality of regulatory analysis is not high; ( 2) quality varies widely; ( 3) the biggest strengths in the analyses are accessibility and clarity; ( 4) the biggest weaknesses are analysis of the systemic problem and retrospective analysis; ( 5) budget or "transfer" regulations receive much lower quality analysis than other regulations; ( 6) the agency claimed to use the analysis in significant decisions for a minority of the regulations; ( 7) quality of analysis is positively correlated with the apparent use of the analysis in regulatory decisions; and ( 8) greater diffusion of best practices could significantly improve the overall quality of regulatory analysis.

2. EXISTING LITERATURE AND OUR APPROACH

Several strands of scholarly literature assess the quality of federal regulatory analysis. Some assessments are in‐depth case studies, whereas others apply quantitative scoring methods to numerous regulatory analyses.

2.1. Case Studies

Case studies suggest that even extensive economic analyses of regulations can have significant flaws. They also find that regulatory impact analysis has had marginal effects on some regulations, but rarely if ever drives major decisions.

McGarity presents five case studies from the Reagan administration.([14]) Some are success stories, but they also reveal shortcomings in monetizing benefits, gathering reliable cost data, explaining large uncertainties in estimates, or identifying a wide range of regulatory options. Similarly, Fraas credits good Regulatory Impact Analyses (RIAs) with improvements in the Environmental Protection Agency's (EPA) rules removing lead from gasoline and banning asbestos, but he also finds analytical shortcomings and notes that they did not serve as "blueprints" for the entire EPA decision.([15]) Case studies of 12 EPA rules issued between 1985 and 1995 in Morgenstern (p. 458) reveal that economic analysis always helped reduce costs, and it increased the benefits of five rules.([16]) Nevertheless, economics had little effect on decisions. The analyses exhibited a "considerable range" in quality (p. 456).([16]) Posner performed the only published study we have seen that assesses the quality of regulatory analysis for budget or "transfer" regulations that define how the federal government will spend or collect money.([17]) He concludes that agencies rarely perform analysis for these regulations and presents several cases of inadequate analysis.

Several validation studies judge the quality of RIAs by comparing their predictions to the actual results of the regulation revealed by retrospective analysis. Validation studies find that agencies tend to overstate benefits and costs more frequently than they understate them. OMB concludes that agencies tend to overstate benefit‐cost ratios, whereas independent scholars conclude there is no systematic bias in the ratios. All of these studies find that benefits, costs, and benefit‐cost ratios are inaccurate (over‐ or underestimated) more frequently than they are accurate.([18], [19], [20])

Harrington et al. present the most recent collection of case studies.([10]) They assembled multiple studies of three "relatively sophisticated" RIAs issued during the G. W. Bush administration. Despite their sophistication, the analyses had significant flaws. The authors' recommendations for improvement are: consider meaningful alternative policy options, use baselines that reveal choices and tradeoffs, include a checklist of practices that should be in an analysis, and explain deviations from this list. Noting that some regulatory analyses are prepared after key decisions have been made, they also call on EPA to prepare a preliminary regulatory impact analysis six months before the agency's final review of proposed and final regulations.

2.2. Quantitative Scoring

Quantitative approaches usually employ a "yes/no" checklist to assess whether RIAs include certain elements. Early Government Accountability Office evaluations of health, safety, and environmental regulations found that RIAs frequently failed to include key elements recommended in OMB's guidance.([[21]]) Robert Hahn co‐authored a series of papers that evaluate the quality of analysis for health, safety, and environmental regulations across three administrations—Ronald Reagan, G. H. W. Bush, and Bill Clinton—using a "yes/no" checklist based on OMB guidance.([23], [24], [25], [26]) The regulatory analyses of a large sample of environmental regulations covered an average of approximately 30 of 76 items on Hahn's scorecard, or 40% (p. 74).([27])

Belcore and Ellig score the quality of analysis for all economically significant regulations issued by the Department of Homeland Security between 2003 and 2007.([28]) They found that the quality of analysis is generally low but improved over time. They also found that quality tends to be lower for rules issued subject to tight legislative deadlines, or where Congress gave the department little discretion.

Fraas and Lutter assess 13 of the most important rules issued by EPA between 2005 and 2009, after OMB issued new and more detailed guidance in the form of Circular A‐4.([29]) Scores ranged from three points to a maximum possible nine, with an average of 5.25 (58%). Fraas and Lutter maintain that the quality of analysis is generally higher for rules issued under legislation that requires agencies to consider costs or net benefits.

Shapiro and Morrall examine 100 economically significant regulations adopted between 2002 and 2010.([30]) They assign each analysis a score of between zero and six points based on six OMB criteria. Scores ranged from zero to six, with an average of 3.85 (64%). They find that the quality of analysis is unrelated to the size of the rule's net benefits, but rules with lower political salience have higher net benefits.

Both the qualitative and quantitative literature reveal some general patterns. Some RIAs are relatively high quality, but many lack key information, and even the best ones could be improved. Analysis has never dictated decisions but has sometimes influenced decisions on the margin, and occasionally these margins involve large benefits or costs.

2.3. Our Approach

We develop a scoring system to evaluate the quality and use of regulatory analysis for a relatively large number of regulations. Our approach differs from most previous evaluations in several ways.

• 1

• It is the first project that evaluates the regulatory analyses accompanying all economically significant regulations proposed in a given year. Most previous evaluations focus on health, safety, and environmental regulations.

• 2

• We focus on proposed regulations, rather than final regulations. We seek to gauge the quality of analysis at the earliest possible point, when it arguably has the best chance of affecting decisions. Of course, many decisions have already been made by the time a rule is proposed,([31]) but we remain optimistic that better analysis may sometimes lead to better decisions (pp. 18–19).([32]) In any case, the analysis accompanying the proposed rule is usually the first comprehensive regulatory analysis available to evaluate.

• 3

• We opt for a qualitative evaluation of how well the analysis was performed, rather than an objective "yes/no" checklist of analytical issues and approaches covered.

• 4

• Our approach assesses whether the agency actually claims to use regulatory analysis to guide decisions. We also evaluate whether the agency makes a commitment to conducting retrospective analysis to assess the actual outcomes of the rule in the future.

Although we seek to evaluate the quality and use of regulatory analysis, one might also interpret this study as an evaluation of the quality of regulations themselves, for a couple of reasons. First, regulatory analysis and regulations are often jointly produced, with lawyers and economists working together on every step of a rulemaking, possibly making the quality of the analysis correlated with quality of the regulation. Second, the quality of regulatory analysis is likely correlated with the quality of regulations because the outputs of the regulatory analysis should serve as inputs to regulatory decisions. For example, one important component of regulatory analysis is the consideration of alternative approaches to achieve the desired outcomes. If a low‐quality analysis fails to consider alternatives, how can an agency be confident that its regulatory approach represents the best one, however "best" may be defined?

Although our results may provide a perspective on the quality of regulations, we caution against interpreting our study as a comprehensive evaluation of regulatory quality. We can rate the quality of regulatory analysis because a number of components are required by executive order or statute. These components are familiar to most economists and can easily be assessed by any economist with the right training. Conversely, we have not attempted to define a method of rating the quality of regulations themselves. Although such an endeavor may be worthwhile, it is beyond the scope of this article.

3. EVALUATION PROTOCOL

3.1. What Was Evaluated?

Evaluations were performed for 45 economically significant proposed regulations whose OIRA reviews were completed in 2008.5 The research team read the preamble to each proposed rule and the accompanying RIA. In some cases, agencies produced a risk assessment or additional analysis in technical support documents that we also considered. We included the Regulatory Flexibility Analysis, which assesses the effects on small entities, to the extent that this analysis had content relevant to our evaluation criteria.

This approach is broader than just reading the RIA document or section of the Federal Register notice explicitly labeled "Regulatory Impact Analysis." It is necessary because agencies organize the content differently in different rules. Sometimes the RIA is a separate document only referenced or summarized in the Federal Register preamble.([33]) Alternatively, the entire RIA may constitute a separate section of the preamble.([34]) In one case, the RIA for a proposal was in the preamble of the Notice of Proposed Rulemaking (NPRM) for a related regulation published the same day.([35]) Some parts may be in a separate regulatory analysis section of the preamble, and other parts, such as environmental impact analysis or risk assessment, may be in other sections of the preamble that discuss justifications for the regulation.([36], [37], [38]) Reading all of this material allowed us to give the agency credit when due, regardless of where the analysis appears.

3.2. Scoring System

We evaluate regulatory analysis on the basis of 12 criteria, grouped into three categories:

• 1

• Openness: How easily can a reasonably informed, interested citizen locate the analysis, understand it, and verify the underlying assumptions and data?

• 2

• Analysis: How well does the analysis define and measure the outcomes or benefits the regulation seeks to accomplish, define the systemic problem the regulation seeks to solve, identify and assess alternatives, and evaluate costs and benefits?

• 3

• Use: How much did the analysis appear to affect decisions in the proposed rule, and what provisions did the agency make for tracking the rule's effects in the future?

Fig. 1 lists the 12 criteria. Appendix A lists detailed questions considered under each criterion. Appendix B presents a cross‐walk chart that maps OMB's November 2010 "Regulatory Impact Analysis Checklist" into our scoring criteria. The Openness and Analysis criteria, numbered 1–8, are straightforward interpretations of provisions in Executive Order 12866 and Circular A‐4.

Graph: 1 Regulatory analysis assessment criteria.

The Use criteria deserve further explanation. The first Use criterion asks whether the analysis seemed to affect decisions. To score this criterion, we assess whether the agency claimed to use information about the regulation's expected outcomes, the systemic problem, or benefits or costs of alternatives to make decisions. The second Use criterion asks whether the agency made its decisions fully cognizant of the net benefits of alternatives. We do not expect the analysis to dictate the decision via a rigid rule, such as "regulate only when monetized benefits exceed monetized costs."Section 1 of Executive Order 12866 explicitly instructs agencies to regulate only when the benefits "justify" the costs, unless the law requires another approach. Thus, we look to see whether the agency either selected the alternative that maximized net benefits or clearly explained why some other alternative was preferable to the one that maximized net benefits.

Searching the Federal Register notice for documentation of use will not identify any undocumented, "behind the scenes" influence of the analysis. We may also overestimate the effects of analysis in situations where the agency reached decisions, then crafted or cited the analysis to support those decisions.([31]) The actual influence of economists and economic analysis in rulemaking (as opposed to the influence documented in the Federal Register notice) likely differs across agencies and even across rules within agencies. At one extreme, economists can have a lot of influence when a regulation is drafted, although it may not be documented in the proposed regulation or preamble (pp. 6–7).([32]) At the other extreme, economists and their analyses may be ignored entirely. Two points in between are: ( 1) the economic analysis has no effect, but the agency writes it to support the rule, or ( 2) the economic analysis has some effect that is documented in the notice. Our evaluation method identifies these latter kinds of cases. Because we cannot easily distinguish between the two on the basis of claims in the NPRM or RIA, our method only assesses whether the agency seemed to use the analysis. By examining the documentation of use, we at least identify where analysis is likely to have influenced rulemaking and offer a starting point for future research into the matter.

Criteria 11 and 12 assess the extent to which the RIA or preamble to the regulation make provisions for retrospective analysis. The executive orders governing regulation offer scant guidance on this, but Executive Order 13563 reiterates the requirement in Executive Order 12866 that each agency have a plan for retrospective review of regulations. A recent edition of OMB's annual report on the benefits and costs of federal regulation declared, "we recommend that serious consideration be given to finding ways to employ retrospective analysis more regularly, in order to ensure that rules are appropriate, and to expand, reduce, or repeal them in accordance with what has been learned" (p. 43).([39]) The Government Performance and Results Act (GPRA) Modernization Act of 2010 requires the federal government and agencies to identify high‐priority goals; specify the programs, activities, tax expenditures, and regulations that contribute to each goal; and regularly evaluate the contributions.([40]) An agency can lay the groundwork for compliance with the law by establishing goals and measures, identifying data, and committing to retrospective analysis in the preamble to the regulation. Agencies have in fact done these things for some regulations.6([41], [42], [43], [44])

For each criterion, evaluators assigned a score ranging from 0 (no useful content) to 5 (comprehensive analysis with potential best practices). Thus, each analysis has the opportunity to earn between 0 and 60 points. In general, the research team used the guidelines in Table I for scoring. Because the Analysis criteria involve many discrete issues, we developed a series of subquestions for each of the four Analysis criteria (listed in Appendix A), and awarded a 0–5 score for each subquestion. These scores were then averaged to calculate the score for the individual criterion.

I What Do the Scores Mean?

5	Complete analysis of all or almost all aspects, with one or more "best practices."
4	Reasonably thorough analysis of most aspects and/or shows at least one "best practice."
3	Reasonably thorough analysis of some aspects.
2	Some relevant discussion with some documentation of analysis.
1	Perfunctory statement with little explanation or documentation.
0	Little or no relevant content.

Compared to an objective checklist, our qualitative approach provides a richer and potentially more accurate evaluation of the actual quality of the analysis. As OIRA notes: "Objective metrics can measure whether an agency performed a particular type of analysis, but may not indicate how well the agency performed this analysis" (p. 19).([45]) For example, rather than just asking whether the analysis considered alternatives or counting the number of alternatives considered, we give an analysis a higher score if it considered a wider range of alternatives. Instead of just asking whether the agency named a market failure, we assess whether the agency provides a coherent theory and plausible evidence that the market failure exists, awarding a higher score on the basis of how convincing the evidence is. The qualitative approach also encourages agencies to find the best way to do analysis that can inform decisions, instead of treating regulatory analysis as a "check the box" compliance exercise.

A qualitative evaluation can be more subjective, less transparent, and harder to replicate. Several aspects of our research design seek to minimize these drawbacks. We designed the evaluation process to achieve a common, intersubjective understanding of which practices deserve which kind of score, and evaluators took notes justifying each score.7 The entire nine‐member research team underwent extensive training in which we evaluated several of the same proposed regulations and accompanying RIAs, compared scores, and discussed major differences until we achieved a consensus on scoring standards. For questions that were particularly difficult to evaluate, we developed written guidelines describing practices that would justify various scores in most cases. Each analysis was scored by one of the authors of this article and another team member, with discussion to achieve consensus when scores differed significantly. Each author also reviewed the other's scores and notes, and then discussed and resolved differences to ensure that all documents were evaluated as consistently as possible on all questions.

In addition, we subjected the scores to ex post statistical analyses to test whether our research design produced a high degree of interrater reliability. Interrater reliability is the degree to which raters agree with each other about their subjective evaluations of a given object. The Cohen kappa index is the most commonly used statistical measure of interrater reliability in social sciences.([[48]]) Other typical tests include Spearman's rho and Pearson's chi‐squared, both of which test the independence of the ratings.

The goal of our interrater reliability testing was to ascertain whether our evaluation system yields consistent agreement among raters trained in the system. First, we created agreement matrices using the prediscussion scores for all questions together and for each individual question. These scores reflected each rater's evaluation before any discussion and deliberation about differences in ratings. Appendix C reports these agreement matrices. The first matrix, in Table A.I, uses score data for all criteria. Each subsequent table shows the agreement matrix for a specific scoring criterion. The first scorer's rating dictates vertical location whereas the second scorer's rating controls the horizontal location. Thus, each matrix that corresponds to a particular criterion—Tables A.II through A.XIII—has 45 observations of score pairs.

A well‐designed system would show substantial agreement between scorers, regardless of whom the scorers were or which regulation was scored. Such agreement would produce density along the diagonal in the agreement matrices, and that is precisely what we observe. At the bottom of each agreement matrix, we list the count and percentage of score pairs that are in perfect agreement or disagreement by different numbers of points. Table A.I reveals considerable agreement between raters: 42.4% of all ratings (229 of 540) were in perfect agreement, and another 36.7% (198 of 540) exhibited a difference of only one point. About 15% (79 of 540) showed a difference of two points, and only 6.3% (34 of 540) had a difference of more than two points. Tables A.II–A.XIII show similar results for the agreement distributions for each individual question. No particular question stands out as an egregious generator of disagreement among the raters.

Cohen's kappa for the entire sample is 0.4784, which, according to the rules of thumb put forth by Landis and Koch, indicates moderate agreement.([50]) Of course, this kappa is calculated using prediscussion ratings. After discussions, there was 100% agreement, and Cohen's kappa equaled 1. Table II shows the results of Spearman's rho tests for the entire sample and for each individual question. The values of rho range from 0.414 to 0.713, and the null hypothesis that the two ratings are independent is strongly rejected for each question. We also tested independence by calculating Pearson's chi‐squared (not reported), finding similar results.

II Analysis of Interrater Reliability

Criterion	Spearman's rho	p‐Value
All	0.621	0.000
1	0.414	0.005
2	0.675	0.000
3	0.713	0.000
4	0.504	0.000
5	0.447	0.002
6	0.586	0.000
7	0.646	0.000
8	0.591	0.000
9	0.639	0.000
10	0.504	0.000
11	0.421	0.004
12	0.465	0.001

The tests indicate that our rating system would likely produce statistical agreement for any set of raters, assuming they underwent the same training. Scores for each regulation on each criterion, as well as notes justifying the scores, are available at http:/www.mercatus.org/reportcard.

3.3. An Example

To illustrate how the evaluation protocol works, Table III reproduces scores and notes for a regulation that received a middling score on criterion 5, outcomes. "Outcomes are not what the program itself did but the consequences of what the program did" (p. 15).([51]) We intentionally employ the broader term "outcomes" rather than "benefits" because some regulations seek to achieve goals that do not necessarily meet the economist's definition of a social benefit. We ask merely whether the regulatory analysis articulates, measures, and justifies an outcome that affects citizens' quality of life, regardless of whether the goal increases net social benefits.

III Outcome Discussion in Labor Department's Cranes and Derricks Proposed Rule

Criterion	Score	Comment
How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them?	3	–
Does the analysis clearly identify ultimate outcomes that affect citizens' quality of life?	5	Workplace safety—reduced fatalities and accidents.
Does the analysis identify how these outcomes are to be measured?	5	Number of fatalities and accidents avoided.
Does the analysis provide a coherent and testable theory showing how the regulation will produce the desired outcomes?	1	Asserts only that "OSHA analysis" shows that an indicated number of fatalities would be eliminated. The text of the rule does a better job explaining several published articles that identify major causes of crane accidents.
Does the analysis present credible empirical support for the theory?	2	Some examples of recent accidents are presented, and the preamble to the rule explains how the proposed rules would have prevented those accidents. It is not clear if these are typical or generalized examples.
Does the analysis adequately assess uncertainty about the outcomes?	2	Uncertainty is acknowledged, and several benefit estimates are offered. However, the sensitivity discussion is cursory and does not provide much in‐depth analysis on how injuries and fatalities would likely be affected.

The regulation in Table III is an Occupational Safety and Health Administration (OSHA) regulation intended to improve safety around cranes and derricks at construction sites. The analysis identified workplace safety outcomes and explained how to measure them, earning a five on each of these questions. However, the analysis does not provide much documented theory or evidence that the regulations would reduce fatalities and accidents; the reader is simply assured that "OSHA analysis" proves this is so. An explicit theory, rather than just an assertion, and documentation of evidence supporting the theory would have earned this analysis a higher score on these two questions. Although the RIA acknowledges uncertainty about benefits, it provides little analysis showing how the uncertainties would affect estimates of injuries and fatalities.

3.4. Caveats

Three significant caveats accompany our findings. First, we evaluate the quality of regulatory analysis and its apparent use in decisions, but we do not evaluate whether the proposed rule is economically efficient, fair, or otherwise good public policy. This article is an assessment of how agencies conduct and claim to use regulatory analysis, not a policy analysis of the regulations themselves.

Second, we evaluated whether the RIA and preamble to the proposed rule make a reasonable effort at covering the major elements of regulatory analysis. We did not seek to replicate the results, produce our own analysis, or verify the underlying data and studies. Commenters on this article who have in‐depth experience with particular regulations have usually told us that we have been too lenient. For example, an EPA air pollution regulation proposed in 2008 scores fairly high for its analysis of uncertainty regarding the size of benefits, but Fraas documents significant shortcomings in the EPA's uncertainty analysis of benefits of air quality regulations.([52]) A high‐scoring analysis may thus still have flaws and inaccuracies because of poor underlying data or theories that turn out to be wrong. Authors of previous regulatory scorecards have also noted this drawback (p. 196, p. 3).([[24], [29]]) Nevertheless, as Dudley (p. 8) notes: "Benefit‐cost analysis isn't perfect, but it's the best we have."([ 3]) The strength of a scorecard approach is its ability to rank numerous regulatory analyses according to consistent criteria. As we shall see, the approach identifies significant differences in the quality of analysis across different regulations.

Third, we give each criterion the same weight. Of course, some criteria, such as whether the agency identified a systemic problem or whether it analyzed a broad array of alternatives, may have more policy impact than whether the RIA is clearly written for the average citizen. The results below often break out score data by our three categories of criteria or by individual criteria, so that readers who want to focus on particular criteria or groups of criteria can do so. For example, readers who are concerned solely about the quality of regulatory analysis can ignore our evaluations of Use and focus on criteria 1–8, which assess Openness and Analysis. We calculated Spearman's rho and Kendall's tau‐b to assess whether inclusion of the Use criteria substantially alters the ranking of the regulations. The rankings with and without the Use criteria are highly correlated—ρ= 0.959 and τb= 0.861—with p‐values of 0.000. Readers who believe some individual criteria should be omitted or weighted more heavily are welcome to download our Excel spreadsheet with a full set of score results for every regulation, and conduct similar tests.8

4. SCORES AND RANKINGS

4.1. Summary Statistics

Both the average and median score were 27 of 60 possible points, or 45%. The best analysis received 43 points (72%), and the worst received only seven points (12%). Fig. 2 shows the distribution of scores.

Graph: 2 Distribution of analysis scores.

In general, the documents score higher on Openness than on the other two categories. The average score on the Openness criteria was 11 of 20 possible points, compared to 8.5 points for Analysis, and 7.7 points for Use.

4.2. Best and Worst Analyses

Table IV lists scores for all 45 regulations, along with their Regulation Identifier Numbers and the name of the issuing department. The best initial analysis in 2008 was for the Department of Transportation's (DOT) proposed Corporate Average Fuel Economy regulation, followed by the EPA's National Ambient Air Quality Standards (NAAQS) for Lead, and Housing and Urban Development's (HUD) proposed revisions to the Real Estate Settlement Procedures Act. The three worst analyses come from the Social Security Administration, Department of Veterans' Affairs, and Department of Defense.

IV Scores for 45 Economically Significant Regulations from 2008

Proposed Rule	RIN	Department	Total	Openness	Analysis	Use
Car and Light Truck Corporate Average Fuel Economy 2011–2015	2127‐AK29	DOT	43	15	16	12
National Ambient Air Quality Standards for Lead	2060‐AN83	EPA	42	14	16	12
Real Estate Settlement Procedures Act	2502‐AI61	HUD	41	15	16	10
Class Exemption for Provision of Investment Advice, Proposed Rule	1210‐AB13	Labor	40	15	15	10
Congestion Management Rule for LaGuardia Airport	2120‐AI70	DOT	39	13	13	13
Large Aircraft Security Program	1652‐AA53	DHS	38	15	13	10
US VISIT Biometric Exit System	1601‐AA34	DHS	38	9	15	14
Fiduciary Requirements for Disclosure in Participant‐Directed Plans	1210‐AB07	Labor	37	15	11	11
Notice of Class Exemption for Provision of Investment Advice	1210‐ZA14	Labor	37	12	14	11
Effluent Limitations Guidelines and Standards for Construction	2040‐AE91	EPA	37	14	14	9
Electronic Prescriptions for Controlled Substances	1117‐AA61	DOJ	36	14	12	10
Migratory Bird Hunting	1018‐AV62	Interior	35	14	12	9
Nondiscrimination in State/Local Government Services	1190‐AA46	DOJ	35	14	9	12
Nondiscrimination by Public/Commercial Facilities	1190‐AA44	DOJ	34	14	9	11
Railroad Tank Car Transportation of Hazardous Materials	2130‐AB69	DOT	33	10	13	10
HIPAA Code Sets	0958‐AN25	HHS	33	15	10	8
Family and Medical Leave Act of 1993	1215‐AB35	Labor	33	18	10	5
Congestion Mgt. for John F. Kennedy Airport and Newark Airport	2120‐AJ28	DOT	30	10	8	12
Cranes and Derricks in Construction	1218‐AC01	Labor	30	14	9	7
Refuge Alternatives for Underground Coal Mines	1219‐AB58	Labor	28	12	8	8
Integrity Management Program for Gas Distribution Pipelines	2137‐AE15	DOT	28	7	11	10
State‐Specific Inventoried Roadless Area Management	0596‐AC74	USDA	28	11	12	5
Energy Conservation Standards for Fluorescent Lamps	1904‐AA92	Energy	27	6	11	10
Alternative Energy Production on the OCS	1010‐AD30	Interior	27	8	10	9
Standardized Risk‐Based Capital Rules (Basel II)	1557‐AD07	Treasury	27	9	9	9
Changes to the Outpatient Prospective Payment System	0938‐AP17	HHS	27	13	7	7
Hospital Inpatient Prospective Payment Systems	0938‐AP15	HHS	27	14	6	7
Oil Shale Management–General	1004‐AD90	Interior	26	9	9	8
HIPAA Electronic Transaction Standards	0938‐AM50	HHS	25	12	8	5
Employment Eligibility Verification	9000‐AK91	FAR	24	13	7	4
Teacher Education Assistance Grant Program	1840‐AC93	ED	23	10	4	9
Abandoned Mine Land Program	1029‐AC56	Interior	21	10	4	7
Maximum Operating Pressure for Gas Transmission Pipelines	2137‐AE25	DOT	21	11	7	3
Federal Perkins Loan Program	1840‐AC94	ED	21	10	2	9
Revisions to Medicare Advantage and Prescription Drug Benefits	0938‐AP24	HHS	19	8	6	5
Prospective Payment System for Long‐Term Care Hospitals	0983‐AO94	HHS	17	9	2	6
Medicare Program: Revisions to Physician Fee Schedules	0938‐AP18	HHS	17	6	4	7
Medicaid Program Premiums and Cost Sharing	0938‐AO47	HHS	17	10	3	4
State Flexibility for Medicaid Benefit Packages	0938‐AO48	HHS	16	9	4	3
Proposed Hospice Wage Index for Fiscal Year 2009	0938‐AP14	HHS	16	9	3	4
Prospective Payment System for Skilled Nursing Facilities	0938‐AP11	HHS	14	7	2	5
Schedule of Fees for Consular Services	1400‐AC41	State	13	7	4	2
CHAMPUS/TRICARE	0720‐AB22	Defense	12	7	4	1
Post‐9/11 GI Bill	2900‐AN10	VA	10	6	2	2
Time and Place for a Hearing Before an Administrative Law Judge	0960‐AG61	SSA	7	4	0	3
Average			27.3	11.04	8.5	7.7

1 Note: Regulations in italics are budget or "transfer" regulations.

4.3. Average Scores by Regulation Type

The 15 regulations in italics in Table IV are budget or "transfer" regulations. These regulations outline how the federal government will spend money, set fees, or administer spending programs. Most of these regulations score poorly. Calculating average scores by type of regulation reveals a big discontinuity, as Table V shows. Average scores for most types of regulations range between 30 and 35 points. Transfer regulations, however, average just 17 points. Transfer regulations score lower on all the three categories of criteria, but the biggest difference is in the Analysis category, where transfer regulations score only about one‐third the points of other types of regulations.

V Average Scores by Regulation Type

Type	Number of Regulations	Average Score	Openness	Analysis	Use
Civil rights	2	34.5	14.0	9.0	11.5
Economic	10	34.2	13.4	11.4	9.4
Security	3	33.3	12.3	11.7	9.3
Environment	9	31.8	11.2	11.6	9.0
Safety	6	29.3	11.3	10.0	8.0
Transfer	15	17.1	8.6	3.5	4.9

This finding is consistent with OMB's observation that agencies do not usually estimate the social benefits and costs of transfer regulations (p. 19).([39]) Posner documents the same phenomenon.([17]) It is not obvious why transfer regulations should receive different analytical treatment, for as OMB notes (p. 19), transfer regulations generate substantial social costs via mandates, prohibitions, and price distortions.([39]) Our results on transfer regulations illustrate a more general point: the data from this project can provide a starting point for analyzing a variety of factors that might influence the quality of regulatory analysis, such as the nature of the regulation, politics, legislative mandates, or deadlines. (See Shapiro and Morrall and Belcore and Ellig (pp. 38–41) for similar examples.)([[30]])

4.4. Agency Average Scores

Table VI lists average scores for each agency. HUD's one regulation earned it the highest agency average. EPA placed second, and Homeland Security placed third. Scores decline relatively smoothly as one moves down the list, except for the 7.7‐point gap that separates Health and Human Services (HHS), ranked 13th, from State, ranked 14th.

VI Agency Average Scores

Agency	Number of Regulations	Average Score	Openness	Analysis	Use
1. HUD	1	41.0	15.0	16.0	10.0
2. EPA	2	39.5	14.0	15.0	10.5
3. DHS	2	38.0	12.0	14.0	12.0
4. DOJ	3	35.0	14.0	10.0	11.0
5. Labor	6	34.2	14.3	11.2	8.7
6. DOT	6	32.3	11.0	11.3	10.0
7. USDA	1	28.0	11.0	12.0	5.0
8. Interior	4	27.3	10.3	8.9	8.3
9. Treasury	1	27.0	9.0	9.0	9.0
10. Energy	1	27.0	6.0	11.0	10.0
11. Federal acquisition	1	24.0	13.0	7.0	4.0
12. Education	2	22.0	10.0	3.0	9.0
13. HHS	11	20.7	10.2	5.0	5.5
14. State	1	13.0	7.0	4.0	2.0
15. Defense	1	12.0	7.0	4.0	1.0
16. Veterans affairs	1	10.0	6.0	2.0	2.0
17. Social security	1	7.0	4.0	0	3.0

Most of the agencies in the top half of the list produced more than one economically significant regulation in 2008. All of the agencies in the bottom half produced just one, except for HHS (11 regulations) and Education (two regulations). Whether this pattern reflects economies of scale or mere coincidence remains to be seen.

We caution the reader against drawing strong inferences about agencies' analytical abilities on the basis of these scores for one year. Most departments produced small numbers of regulations, and many consist of diverse agencies that may not all produce the same quality of analysis. Generalizations about different agencies' abilities would require either a larger data set spanning more years or in‐depth case studies.

4.5. Average Scores by Criterion

Average scores on individual criteria reveal where regulatory analysis in practice is generally strongest and weakest. The criterion with the highest average score in Table VII is criterion 1, Accessibility. This is not surprising, as making documents accessible to the public via the Internet is relatively easy to do regardless of the quality of the analysis itself. The two lowest scoring criteria are both related to retrospective analysis: establishing measures and goals to track the regulation's effects in the future (criterion 11) and gathering data for such assessment (criterion 12).

VII Ranking of Scores on Individual Criteria

Criterion	Including Transfer Regulations	Excluding Transfer Regulations
Accessibility	3.53	3.30
Clarity	2.93	3.50
Some use of analysis	2.44	2.63
Outcome definition	2.36	3.10
Model documentation	2.33	2.83
Alternatives	2.29	2.93
Data documentation	2.24	2.63
Net benefits	2.20	2.93
Benefit‐cost analysis	2.09	2.60
Systemic problem	1.80	2.40
Retrospective data	1.73	2.03
Measures and goals	1.36	1.53
Overall average score	27.31	32.43

The other low‐scoring criterion is identification of the market failure or other systemic problem the regulation is supposed to solve. This low score is puzzling because Section 1 of Executive Order 12866 leads off by stating that each regulation must identify the problem it seeks to address and assess the significance of that problem. The analyses that score low on this criterion either simply assert a reason for the regulation, with no accompanying theory or evidence, or mention no explicit rationale at all beyond implementing a statute. Such weaknesses are disturbing. It is hard to have confidence that a regulation really will solve a problem, or that the agency has selected the best option for solving a problem, if the agency cannot articulate the problem, cite convincing evidence that the problem exists, and explain its root cause.

Given the lower average scores of transfer regulations, it is no surprise that average scores on individual criteria are generally higher when transfer regulations are excluded. But even excluding transfer regulations, the average score on this criterion is only 2.4 points. We can identify more than a few examples of prescriptive regulations that scored a 1 or 2 on identification of the systemic problem. These include Treasury's risk‐based capital rules for banks, Interior's abandoned mine land program and oil shale management rules, DOT's maximum operating pressure for gas transmission pipelines, and Federal Acquisition's employment eligibility verification rules.

5. USE OF REGULATORY ANALYSIS

Different scholars offer different conclusions about whether economic analysis actually has much influence on regulatory decisions. Hahn and Tetlock conclude that few RIAs have much effect.([27]) Officials interviewed by West claim that decisionmakers often make up their minds before the analysis is done.([31]) Williams, on the other hand, suggests that regulatory analysis can affect decisions behind the scenes, even if the agency does not explicitly explain in its Federal Register notice (pp. 6–7).([32]) Our scoring on the Use criteria offers another perspective on this question.

5.1. Do Agencies Claim to Use Regulatory Analysis?

Table VI shows that criterion 9, Use of Analysis, has the third highest average score. An agency can earn points on this criterion even if statutorily prohibited from considering some factors, such as costs or net benefits. For example, when setting NAAQS, "[a]ccording to the Clean Air Act, EPA must use health‐based criteria in setting the NAAQS and cannot consider estimates of compliance cost."([42]) However because health is one of the key benefits of air quality standards, the EPA received two points on criterion 9 for using the health analysis to inform its decision.

Criterion 10, Net Benefits, receives a lower average score when transfer regulations are included (2.20 points) than when they are excluded (2.93 points). One might argue that net benefits are irrelevant when a regulation "merely" transfers money, but surely most federal expenditures are supposed to achieve some type of public benefit that could often be measurable. To achieve a good score on this criterion, the agency does not have to select the alternative that maximizes net benefits. Rather, the agency must demonstrate that it was cognizant of net benefits and weighed them against other factors when making its decision. If the RIA calculates net benefits of multiple alternatives but the preamble to the proposed rule clearly states the justification for choosing an alternative that did not maximize net benefits, the agency can still score well on this criterion. We score the Net Benefits criterion this way to avoid imposing the value judgment that agencies "ought" to choose the alternative that maximizes net benefits. Instead, we evaluate whether decisionmakers considered net benefits and then determined what weight net benefits should have in the decision.

Figs. 3 and 4 show that the scores on these two criteria have a somewhat bimodal distribution. About 10 regulations earned a score of 4 or 5. For more than 20 regulations, the agency seems to have used little or nothing of the analysis. The remaining regulations show some apparent use of the analysis, but not substantial use. We infer from this that agencies sometimes claim regulatory analysis had a significant effect on the regulation, but more often they claim a marginal effect or make no claim at all.

Graph: 3 Breakdown of criterion 9.

Graph: 4 Breakdown of criterion 10.

The really low scores in the Use category are on the two retrospective analysis criteria. Only four regulations earned a 3 or better on criterion 11, Measures and Goals, and only 10 regulations earned a 3 or better on criterion 12, Retrospective Data. Few economically significant regulations include any substantial plans for retrospective analysis of either costs or benefits. Seventeen years after passage of the GPRA required agencies to develop goals and measures for their major programs, this is disappointing news indeed. Because economically significant regulations are the ones with the largest impact, surely most of them are related to an agency's fundamental mission and strategic goals.

5.2. Correlation of Quality and Use

Because we evaluated both the quality and the apparent use of regulatory analysis, we can test to see whether there is any correlation between the two. Table VIII shows regression results using all 45 regulations; Table IX excludes transfer regulations. Both tables reveal that there is a tighter and more significant correlation between Use (criteria 9–12) and the Analysis score (criteria 5–8) than between Use and the total Quality score (criteria 1–8). In other words, agencies are likely to claim they used the analysis if it is more thorough, even if it is more difficult to find, less thoroughly documented, or harder to read.

VIII Use of Analysis Versus Quality, 45 Regulations (Tobit Regressions)

Dependent Variable	Constant	Quality Score (Criteria 1–8)	Analysis Score (Criteria 5–8)	Chi‐Squared	Pseudo R‐Squared
Criteria 9–12 (All Four Use Criteria)	1.48	0.32	–	27.27**	0.12
	[1.36]	[6.12]***	–	–	–
Criteria 9–12 (All Four Use Criteria)	3.20	–	0.54	34.77***	0.15
	[4.49]***	–	[7.24]***	–	–
Criteria 9 and 10 (Some Use and Net Benefits)	1.69	–	0.35	22.37***	0.11
	[2.69]***	–	[5.35]***	–	–
Criteria 11 and 12 (Measures and Goals and Retrospective	1.28	–	0.21	10.99***	0.06
Data)	[2.24]**	–	[3.51]***	–	–
Criterion 9 (Some Use of Analysis)	1.43	–	0.12	6.88***	0.04
	[3.36]***	–	[2.71]***	–	–
Criterion 10 (Net Benefits)	−0.02	–	0.26	29.46***	0.18
	[−0.05]	–	[6.26]***	–	–
Criterion 11 (Measures and Goals)	0.36	–	0.11	7.70***	0.06
	[0.97]	–	[2.85]***	–	–
Criterion 12 (Retrospective Data)	0.68	–	0.12	9.37***	0.07
	[1.91]*	–	[3.22]***	–	–

• 2 t‐statistics in brackets.
• 3 ***Significant at the 1% level.
• 4 **Significant at the 5% level.
• 5 *Significant at the 10% level.

IX Use of Analysis Versus Quality, 30 Nontransfer Regs (Tobit Regressions)

Dependent Variable	Constant	Quality Score (Criteria 1–8)	Analysis Score (Criteria 5–8)	Chi‐Squared	Pseudo‐R2
Criteria 9–12 (All Four Use Criteria)	4.35	0.209	–	5.14**	0.04
	[2.09]**	[2.37]***	–	–	–
Criteria 9–12 (All Four Use Criteria)	3.58	–	0.51	13.73***	0.10
	[2.57]**	–	[4.17]***	–	–
Criteria 9 and 10 (Some Use and Net Benefits)	2.31	–	0.30	5.47**	0.04
	[1.63]	–	[2.44]**	–	–
Criteria 11 and 12 (Measures and Goals and Retrospective	0.90	–	0.24	4.58**	0.04
Data)	[0.72]	–	[2.21]**	–	–
Criterion 9 (Some Use of Analysis)	0.67	–	0.18	4.56**	0.04
	[0.71]	–	[2.19]**	–	–
Criterion 10 (Net Benefits)	1.44	–	0.14	3.73*	0.04
	[1.75]*	–	[1.98]*	–	–
Criterion 11 (Measures and Goals)	0.15	–	0.12	3.69*	0.04
	[0.72]	–	[1.98]*	–	–
Criterion 12 (Retrospective Data)	0.63	–	0.12	3.28*	0.03
	[0.77]	–	[1.85]*	–	–

• 6 t‐statistics in brackets.
• 7 ***Significant at the 1% level.
• 8 **Significant at the 5% level.
• 9 *Significant at the 10% level.

Table VIII shows that there is a positive and statistically significant correlation between the quality of the analysis and every subcomponent of the Use score. When the sample is confined to nontransfer regulations, however, the relationship is somewhat less extensive, as Table IX shows. Taken together, the Use criteria are still highly correlated with quality of the analysis. Quality of analysis is also correlated with the sum of criteria 9 and 10 (Some Use of Analysis and Net Benefits), and with the sum of criteria 11 and 12 (Measures and Goals and Retrospective Data). But when the regressions are run using individual criteria, the quality of analysis is only marginally significant for criteria 10–12. For nontransfer regulations, it seems that the principal source of correlation between quality and apparent use is criterion 9, which measures whether the agency claimed the RIA affected decisions in the regulation. For nontransfer regulations, good analysis might not be correlated with consideration of net benefits, nor does it necessarily imply that the agency will provide for retrospective analysis.

Nevertheless, there is some evidence that agencies claim they used the analysis in regulatory decisions when the analysis is better. Perhaps improving the quality of analysis improves the odds that decisionmakers will find it useful. Or perhaps when decisionmakers are willing to use regulatory analysis, better regulatory analysis gets produced. It is also possible that, by the time a proposed rule and the accompanying analysis emerge from several iterations of revision within the agency and the OIRA review process, quality and use are mutually interdependent. Finally, the correlation may be driven by other factors, such as statutory requirements that agencies either must or must not consider various aspects of regulatory analysis when making decisions.

Even if most agencies treat RIAs as a mere compliance exercise, it is interesting to note that agencies are more likely to claim that their analysis influenced their decisions when the analysis is better. Clearly, the relationship between quality of analysis and agencies' claimed use of analysis is an area ripe for further research.

6. BEST PRACTICES

Our qualitative evaluation method identifies which analyses have done a particularly good job according to the various criteria. Table X compares the average score on each criterion with the highest score any analysis achieved on that criterion. On most criteria, only a handful of analyses earned the highest score of 5. No analysis earned a score of 5 for criterion 8, Benefit‐Cost Analysis, but at least one earned a 5 on each subquestion under Benefit‐Cost Analysis. Clearly, more widespread adoption of existing best practices could substantially improve the quality of most regulatory analyses.

X Best Practices Not Widely Shared

Criterion	Average Score	Highest Score Achieved	No. Earning Highest Score
1. Accessibility	3.53	5	12
2. Data documentation	2.24	5	1
3. Model documentation	2.33	5	3
4. Clarity	2.93	5	3
5. Outcome definition	2.36	5	2
6. Systemic problem	1.80	5	1
7. Alternatives	2.29	5	1
8. Benefit‐cost analysis	2.09	4	3
9. Some use of analysis	2.44	5	2
10. Considered net benefits	2.20	5	2
11. Measures and goals	1.36	5	1
12. Retrospective data	1.73	5	1

7. CONCLUSIONS

Regulatory analysis is supposed to inform regulatory decisions, not simply justify them after the fact or merely fulfill a requirement to clear a rule through OIRA. Because proposed regulations usually reflect a great deal of up‐front work and are supposed to represent the agency's preferred approach to problem solving, we evaluated the quality of regulatory analyses accompanying proposed regulations. This allows us to assess whether the analysis publicly disclosed closest to the time when initial decisions are made is comprehensive and reliable enough to inform those decisions. In addition, we evaluated whether the agency claims to use regulatory analysis to inform its decisions, now and in the future. This allows us to assess whether the quality of regulatory analysis is correlated with its apparent use.

Our findings on quality are generally consistent with prior literature. Previous regulatory scorecard literature finds that analyses earn an average of 40–64% of the total possible points, with higher scores for more recent years.([[27], [29]]) The average for all regulations we assessed was 27.3 of 60 possible points, or 46%. Excluding transfer regulations, the average was 32.4 points, or 54%. Along with Shapiro and Morrall and Fraas and Lutter, our figures may suggest that the quality of regulatory analysis has improved somewhat since Hahn's seminal scorecards.([[30]])

Qualitative scoring allows us to distinguish between better and worse implementation of economic analysis. The scores clearly indicate that every aspect of regulatory analysis is done at least somewhat well by someone in some agency on some regulation, but no single analysis comes close to doing everything well. Substantial improvements in regulatory analysis could occur across the board if federal agencies had the incentives to mobilize and spread know‐how that already exists.

Our results also suggest that regulatory analysis is perhaps more widely used than previous research has shown. Agencies claimed that some aspect of the analysis affected some major aspect of the regulatory decision in about 10 rules, or 22% of the total. Moreover, the apparent use of analysis is positively correlated with quality of analysis. Agencies are more likely to claim they used the RIA when the RIA is better—though which way the causation runs remains to be seen.

This article reports just the first steps in a multifaceted, ongoing research project. One extension would be to evaluate regulations issued in 2009, to assess whether there was much difference in the quality of regulatory analysis during the last year of the Bush administration and the first year of the Obama administration. Evaluations similar to ours could also be used to assess whether President Obama's Executive Order 13563 has any effect on the quality of regulatory analysis.

Finally, the data can be used to analyze other factors that might affect the quality of regulatory analysis, such as deadlines, politics, statutory requirements, court decisions, or institutional factors unique to particular agencies. Some of the literature cited in Section 1 found that these types of factors affected the quality of regulatory analysis.([[30]]) Other research is also suggestive. McLaughlin, for example, finds that "midnight" regulations issued late in an outgoing administration's term receive shorter review at OIRA.([53]) McLaughlin and Ellig report that midnight regulations, transfer regulations, and regulations with statutory deadlines all have lower quality analysis, and the latter two types of regulations also receive shorter review times at OIRA.([54]) This suggests that the quality of analysis varies systematically with institutional factors. The evaluations reported in this article are the first step in testing these types of hypotheses.

ACKNOWLEDGMENTS

This article reports on the first stage of an ongoing project initiated by the authors at the Mercatus Center at George Mason University. Although one of the authors (McLaughlin) is no longer affiliated with the Mercatus Center, his part in the evaluation of RIAs was performed when he was a research fellow at the Mercatus Center and before his employment with the U.S. Department of Transportation. It is a revised version of a longer working article available online at http://mercatus.org/publication/quality&#8208;and&#8208;use&#8208;regulatory&#8208;analysis&#8208;2008._SP_(_sp_[55]_SP_)_sp_ We acknowledge the substantial contributions of the other individuals who served on the research team that evaluated these regulatory analyses: Mark Adams, David Bieler, Katelyn Christ, Christina Forsberg, Stefanie Haeffele‐Balch, Gabriel Okolski, and Kevin Rollins. We thank Mohamad Elbarasse for research assistance, and Rick Belzer, Susan Dudley, Art Fraas, Randy Lutter, John Morrall, Marcus Peacock, Richard Williams, and two anonymous peer reviewers for helpful comments. The views and opinions expressed by the authors do not necessarily state or reflect those of the U.S. government, the U.S. Department of Transportation, or the Federal Railroad Administration, and shall not be used for advertising or product endorsement purposes.

Appendices

APPENDIX A: MAJOR FACTORS CONSIDERED WHEN EVALUATING EACH CRITERION

Note: Regardless of how they are worded, all questions involve qualitative analysis of how well the RIA addresses the issue, rather than "yes/no" answers.

Openness

• 1 How easily were the RIA, the proposed rule, and any supplementary materials found online?

• How easily can the proposed rule and RIA be found on the agency's website?

• How easily can the proposed rule and RIA be found on Regulations.gov?

• Can the proposed rule and RIA be found without contacting the agency for assistance?

• 2 How verifiable are the data used in the analysis?

• Is there evidence that the RIA used data?

• Does the RIA provide sufficient information for the reader to verify the data?

• How much of the data are sourced?

• Does the RIA provide direct access to the data via links, URLs, or provision of data in appendices?

• If data are confidential, how well does the RIA assure the reader that the data are valid?

• 3 How verifiable are the models and assumptions used in the analysis?

• Are models and assumptions stated clearly?

• How well does the RIA justify any models or assumptions used?

• How easily can the reader verify the accuracy of models and assumptions?

• Does the RIA provide citations to sources that justify the models or assumptions?

• Does the RIA demonstrate that its models and assumptions are widely accepted by relevant experts?

• How reliable are the sources? Are the sources peer‐reviewed?

• 4 Was the Regulatory Impact Analysis comprehensible to an informed layperson?

• How well can a nonspecialist reader understand the results or conclusions?

• How well can a nonspecialist reader understand how the RIA reached the results?

• How well can a specialist reader understand how the RIA reached the results?

• Is the RIA written in "plain English"? (Light on technical jargon and acronyms, well organized, grammatically correct, and direct language used.)

Analysis

• 5 How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them?

• How well does the RIA identify ultimate outcomes that affect citizens' quality of life?

• How well does the RIA identify how these outcomes are to be measured?

• Does the RIA provide a coherent and testable theory showing how the regulation will produce the desired outcomes?

• Does the analysis present credible empirical support for the theory?

• Does the analysis adequately assess uncertainty about the outcomes?

• 6 How well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?

• Does the analysis identify a market failure or other systemic problem?

• Does the analysis outline a coherent and testable theory that explains why the problem (associated with the outcome above) is systemic rather than anecdotal?

• Does the analysis present credible empirical support for the theory?

• Does the analysis adequately assess uncertainty about the existence and size of the problem?

• 7 How well does the analysis assess the effectiveness of alternative approaches?

• Does the analysis enumerate other alternatives to address the problem?

• Is the range of alternatives considered narrow or broad?

• Does the analysis evaluate how alternative approaches would affect the amount of the outcome achieved?

• Does the analysis adequately address the baseline—what the state of the world is likely to be in the absence of further federal action?

• 8 How well does the analysis assess costs and benefits?

• Does the analysis identify and quantify incremental costs of all alternatives considered?

• Does the analysis identify all expenditures likely to arise as a result of the regulation?

• Does the analysis identify how the regulation would likely affect the prices of goods and services?

• Does the analysis examine costs that stem from changes in human behavior as consumers and producers respond to the regulation?

• Does the analysis adequately address uncertainty about costs?

• Does the analysis identify the approach that maximizes net benefits?

• Does the analysis identify the cost‐effectiveness of each alternative considered?

• Does the analysis identify all parties who would bear costs and assess the incidence of costs?

• Does the analysis identify all parties who would receive benefits and assess the incidence of benefits?

Use

• 9 Does the proposed rule or the RIA present evidence that the agency used the Regulatory Impact Analysis?

• Does the proposed rule or the RIA assert that the RIA's results affected any decisions?

• How many aspects of the proposed rule did the RIA affect?

• How significant are the decisions the RIA affected?

• 10 Did the agency maximize net benefits or explain why it chose another option?

• Did the RIA calculate net benefits of one or more options so that they could be compared?

• Did the RIA calculate net benefits of all options considered?

• Did the agency either choose the option that maximized net benefits or explain why it chose another option?

• How broad a range of alternatives did the agency consider?

• 11 Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future?

• Does the RIA contain analysis or results that could be used to establish goals and measures to assess the results of the regulation in the future?

• In the RIA or the proposed rule, does the agency commit to performing some type of retrospective analysis of the regulation's effects?

• Does the agency explicitly articulate goals for major outcomes the rule is supposed to affect?

• Does the agency establish measures for major outcomes the rule is supposed to affect?

• Does the agency set targets for measures of major outcomes the rule is supposed to affect?

• 12 Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so?

• Does the RIA or proposed rule demonstrate that the agency has access to data that could be used to assess some aspects of the regulation's performance in the future?

• Would comparing actual outcomes to outcomes predicted in the RIA generate a reasonably complete understanding of the regulation's effects?

• Does the agency suggest it will evaluate future effects of the regulation using data it has access to or commits to gathering?

• Does the agency explicitly enumerate data it will use to evaluate major outcomes the regulation is supposed to accomplish in the future?

• Does the RIA demonstrate that the agency understands how to control for other factors that may affect outcomes in the future?

APPENDIX B: CROSS‐WALK OF 2010 OMB REGULATORY IMPACT ANALYSIS CHECKLIST WITH OUR EVALUATION C...

OMB Checklist	Our Evaluation Criteria
Does the RIA include a reasonably detailed description of the need for the regulatory action?	Criterion 6: How well does the analysis demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
Does the RIA include an explanation of how the regulatory action will meet that need?	Criterion 5: How well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them?
Does the RIA use an appropriate baseline (i.e., best assessment of how the world would look in the absence of the proposed action)?	Criterion 7, question D: Does the analysis adequately assess the baseline—what the state of the world is likely to be in the absence of further federal action?
Is the information in the RIA based on the best reasonably obtainable scientific, technical, and economic information and is it presented in an accurate, clear, complete, and unbiased manner?	Criterion 2: How verifiable are the data used in the analysis
	Criterion 3: How verifiable are the models or assumptions used in the analysis?
	Criterion 4: Was the analysis comprehensible to an informed layperson?
	Criterion 3 includes an assessment of whether the models and assumptions are based on peer‐reviewed or otherwise reliable publications. However, the evaluation does not assess the quality of the underlying science.
Are the data, sources, and methods used in the RIA provided to the public on the Internet so that a qualified person can reproduce the analysis?	Criterion 1 takes the first step by assessing how easily the RIA itself can be found on the Internet.
	Criteria 3 and 4 include an assessment of how easily the reader could find the underlying data, sources, and methods from information or links provided in the RIA or the Federal Register notice.
To the extent feasible, does the RIA quantify and monetize the anticipated benefits from the regulatory action?	Criterion 5, question 2: How well does the analysis identify how the outcomes are to be measured?
To the extent feasible, does the RIA quantify and monetize the anticipated costs?	Multiple questions under Criterion 8 (Benefits and Costs) assess how well the analysis identifies, quantifies, and monetizes costs.
Does the RIA explain and support a reasoned determination that the benefits of the intended regulation justify its costs (recognizing that some benefits and costs are difficult to quantify)?	Criterion 8, question F: Does the analysis identify the approach that maximizes net benefits?
	Criterion 8, question G: Does the analysis identify the cost‐effectiveness of each alternative considered?
Does the RIA assess the potentially effective and reasonably feasible alternatives?	Criterion 7: How well does the analysis assess the effectiveness of alternative approaches?
Does the preferred option have the highest net benefits (including potential economic, public health and safety, and other advantages; distributive impacts; and equity), unless a statute requires a different approach?	Criterion 10: Did the agency maximize net benefits or explain why it chose another option?
Does the RIA include an explanation of why the planned regulatory action is preferable to the identified potential alternatives?	Criterion 9: Does the proposed rule or RIA present evidence that the agency used the regulatory analysis?
	Criterion 10: Did the agency maximize net benefits or explain why it chose another option?
Does the RIA use appropriate discount rates for the benefits and costs that are expected to occur in the future?	Considered under Criterion 5, question 2: How well does the analysis identify how the outcomes are to be measured?, as well as several questions about measurement and comparison of benefits and costs under Criterion 8 (Benefits and Costs).
Does the RIA include, if and where relevant, an appropriate uncertainty analysis?	Criterion 5, question E: Does the analysis adequately assess uncertainty about the outcomes?
	Criterion 6, question D: Does the analysis adequately assess uncertainty about the existence and size of the problem?
	Criterion 8, question E: Does the analysis adequately address uncertainty about costs?
Does the RIA include, if and where relevant, a separate description of the distributive impacts and equity (including transfer payments and effects on disadvantages or vulnerable populations)?	Criterion 8, question H: Does the analysis identify all parties who would bear costs and assess the incidence of costs?
	Criterion 8, question I: Does the analysis identify all parties who would receive benefits and assess the incidence of benefits?
Does the analysis include a clear, plain‐language executive summary, including an accounting statement that summarizes the benefit and cost estimates for the regulatory action under consideration, including the qualitative and nonmonetized benefits and costs?	Criterion 4: Was the analysis comprehensible to an informed layperson?
Does the analysis include a clear and transparent table presenting (to the extent feasible) anticipated benefits and costs (qualitative and quantitative)?	Criterion 4: Was the analysis comprehensible to an informed layperson?
Goals and measures to assess results of the regulation in the future—No content.	Criterion 11: Does the proposed rule establish measures and goals that can be used to track the regulation's results in the future?
Provisions for gathering data to assess results of the regulation in the future—No content.	Criterion 12: Did the agency indicate what data it will use to assess the regulation's performance in the future and establish provisions for doing so?

APPENDIX C: EVALUATION OF INTERRATER RELIABILITY

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Graph

Footnotes

1 "Economically significant" regulations are defined as regulations that have an economic impact exceeding $100 million or that adversely affect in a material way the economy, a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or state, local, or tribal governments or communities (Sec. 3[f][1]). Economically significant regulations require an extensive Regulatory Impact Analysis that assesses the need, effectiveness, benefits, costs, and alternatives for the proposed regulation (Sec. 6[a][3][C]).

2 Reginfo.gov lists 48 proposed, economically significant regulations whose OIRA reviews were concluded in 2008. Three RIAs could not be found at the time these evaluations were performed, leaving us with 45 regulations to evaluate.

3 For readers who are still skeptical about the value of including the two retrospective analysis criteria, we calculated Spearman's rho and Kendall's tau‐b to assess whether inclusion of these criteria substantially alters the ranking of the regulations. The rankings with and without the Use criteria are highly correlated—rho = 0.981 and tau‐b = 0.920—with p values of 0.000.

4 The term "intersubjective" refers to subjective interpretations that different individuals can share because they have commonly understood meanings. Social scientists most commonly use the term to denote economic agents' ability to understand the interpretations and meanings of other economic agents, or the social scientist's ability to understand the interpretations and meanings of the economic agents who are the subject of study.(46, 47) We think it applies equally well here, when colleagues share similar subjective understandings of what constitutes better and worse analyses.

5 The spreadsheet is available at http://www.mercatus.org/reportcard.

REFERENCES

Obama B. Improving regulation and regulatory review—Executive Order 13563. Presidential Documents. Federal Register, 2011 ; 76 (14): 3821 – 3823.

Clinton WJ. Regulatory planning and review—Executive Order 12866. Presidential Documents. Federal Register, 1993 ; 58 (190): 51734 – 51744.

Dudley SE. Lessons learned, challenges ahead. Regulation, 2009 ; 32 (2): 6 – 11.

Ackerman F, Heinzerling L. Priceless: On Knowing the Price of Everything and the Value of Nothing. New York : New Press, 2004.

Parker RW. Grading the government. University of Chicago Law Review, 2003 ; 70 (4): 1345 – 1486.

6 Graham J. Saving lives through administrative law and economics. University of Pennsylvania Law Review, 2008 ; 157 (1): 395 – 540.

7 Viscusi WK. Fatal Trade‐Offs: Public and Private Responsibilities for Risk. Oxford : Oxford University Press, 1992.

8 Sunstein CR. The Cost‐Benefit State. Chicago : American Bar Association, 2002.

9 Revesz RL, Livermore MA. Retaking Rationality: How Cost‐Benefit Analysis Can Better Protect the Environment and Our Health. New York : Oxford University Press, 2008.

Harrington W, Heinzerling L, Morgenstern R. Reforming Regulatory Impact Analysis. Washington, DC: Resources for the Future Press, 2009.

Katzen S. Cost‐benefit analysis: Where should we go from here? Fordham Urban Law Journal, 2006 ; 33 (4): 1313 – 1317.

Hahn RW, Sunstein CR. A new executive order for improving federal regulation? Deeper and wider cost‐benefit analysis. University of Pennsylvania Law Review, 2002 ; 150 (5): 1489 – 1552.

Office of Management and Budget. Circular A‐4, 2003. Available at http://www.whitehouse.gov/omb/circulars%5fa004%5fa&#8208;4/, Accessed on April 27, 2010.

McGarity T. Reinventing Rationality. Cambridge : Cambridge University Press, 1991.

Fraas A. The role of economic analysis in shaping regulatory policy. Law and Contemporary Problems, 1991 ; 54 (4): 113 – 125.

Morgenstern R. Economic Analysis at EPA. Washington, DC : Resources for the Future, 1997.

Posner EA. Transfer regulations and cost‐effectiveness analysis. Duke Law Journal, 2003 ; 53 (3): 1067 – 1110.

Harrington W, Morgenstern R, Nelson P. On the accuracy of regulatory cost estimates. Journal of Policy Analysis and Management, 2000 ; 19 (2): 297 – 332.

Validating Regulatory Analysis : Report to Congress on the Costs and Benefits of Federal Regulations and Unfunded Mandates on State, Local, and Tribal Entities. Washington, DC : Office of Information and Regulatory Affairs, 2005.

Harrington W. Grading Estimates of the Benefits and Costs of Federal Regulation: A Review of Reviews. Discussion Paper 06–39. Washington, DC : Resources for the Future, 2006.

Air Pollution : Information Contained in EPA's Regulatory Impact Analyses Can be Made Clearer. Washington, DC: Government Accountability Office, 1997.

Regulatory Reform: Agencies Could Improve Development, Documentation, and Clarity of Regulatory Economic Analyses. Washington, DC : Government Accountability Office, 1998.

Hahn RW, Burnett J, Chan YI, Mader E, Moyle P. Assessing regulatory impact analyses: The failure of agencies to comply with Executive Order 12866. Harvard Journal of Law and Public Policy, 2000 ; 23 (3): 859 – 871.

Hahn RW, Dudley P. How well does the government do cost–benefit analysis? Review of Environmental Economics and Policy, 2007 ; 1 (2): 192 – 211.

Hahn RW, Lutter RW, Viscusi WK. Do Federal Regulations Reduce Mortality? Washington, DC : AEI‐Brookings Joint Center for Regulatory Studies, 2000.

Hahn RW, Litan R. Counting regulatory benefits and costs: Lessons for the U.S. and Europe. Journal of International Economic Law, 2005 ; 8 (2): 473 – 508.

Hahn RW, Tetlock PC. Has economic analysis improved regulatory decisions? Journal of Economic Perspectives, 2008 ; 22 (1): 67 – 84.

Belcore J, Ellig J. Homeland security and regulatory analysis: Are we safe yet? Rutgers Law Journal, 2008 ; 40 (1): 1 – 96.

Fraas A, Lutter R. The Challenges of Improving the Economic Analysis of Pending Regulations: The Experience of OMB Circular A‐4. Washington, DC: Resources for the Future, 2010.

Shapiro S, Morrall J. The Triumph of Regulatory Politics: BCA and Political Salience. Rutgers University, Working paper, 2011.

West W. Inside the black box: The development of proposed rules and the limitations of procedural controls. Administration & Society, 2009 ; 41 (5): 579 – 599.

Williams R. The Influence of Regulatory Economists in Federal Health and Safety Agencies. Mercatus Center at George Mason University, Working paper, 2008.

Department of Transportation. Passenger Car and Light Truck Corporate Average Fuel Economy 2011–2015. Regulatory Impact Analysis, 2008.

Department of Labor. Class exemption for provision of investment advice. Proposed Rules. Federal Register, 2008 ; 73 (164): 49895 – 49920.

Department of Labor. Class exemption for provision of investment advice. Notices. Federal Register, 2008 ; 73 (164): 49924 – 49932.

Environmental Protection Agency. Effluent limitations guidelines and standards for the construction and development point source category. Proposed rules. Federal Register, 2008 ; 73 (230): 72561 – 72614.

Department of Health and Human Services. Proposed hospice wage index for fiscal year 2009. Proposed rules. Federal Register, 2008 ; 73 (85): 24000 – 24035.

Department of Health and Human Services. Proposed changes to the hospital inpatient prospective payment systems and fiscal year 2009 rates. Proposed Rules. Federal Register, 2008 ; 73 (84): 23528 – 23938.

2009 Report to Congress on the Benefits and Costs of Federal Regulations and Unfunded Mandates on State, Local, and Tribal Entities. Washington, DC : Office of Information and Regulatory Affairs, 2009.

GPRA Modernization Act of 2010, Pub. L. 111–352, 124 Stat. 3866 (January 4, 2011).

Department of Homeland Security. Air/Sea Biometric Exit Project. Regulatory Impact Analysis, 2008.

Environmental Protection Agency. Proposed Lead NAAQS, Regulatory Impact Analysis, 2008.

Department of Transportation. Pipeline Safety: Integrity Management Program for Gas Distribution Pipelines. Regulatory Impact Analysis, 2008.

Department of Transportation. Congestion Management Rule for New York LaGuardia Airport. Regulatory Impact Analysis, 2008.

2008 Report to Congress on the Benefits and Costs of Federal Regulations and Unfunded Mandates on States, Local, and Tribal Entities. Washington, DC : Office of Information and Regulatory Affairs, 2008.

Schütz A. Common‐sense and scientific interpretation of human action. Philosophy and Phenomenological Research, 1953 ; 14 :(1): 1 – 38.

Lavoie D. Hermeneutics, subjectivity, and the Lester‐Machlup debate: Toward a more anthropological approach to empirical economics. Pp. 167 – 184 in Samuels WJ (ed). Economics As Discourse: An Analysis of the Language of Economists. Norwell, MA : Kluwer Academic Publishers ; 1990.

Siegel S, Castellan NJ. Nonparametric Statistics for the Behavioral Sciences, 2nd Ed. New York : McGraw‐Hill, 1988.

Perreault WD, Laurence LE. Reliability of nominal data based on qualitative judgments. Journal of Marketing Research, 1989 ; 26 (2): 135 – 148.

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977 ; 33 (1): 159 – 174.

Hatry HP. Performance Measurement. Washington, DC : Urban Institute Press, 1999.

Fraas AG. Uncertainty in EPA's Analysis of Air Pollution Rules: A Status Report. Resources for the Future Discussion Paper 10–04, February, 2010.

McLaughlin PA. The consequences of midnight regulations and other surges in regulatory activity. Public Choice, 2011 ; 147 (3): 395 – 412.

McLaughlin PA, Ellig J. Does OIRA review improve the quality of regulatory impact analysis? Evidence from the final year of the Bush II administration. Administrative Law Review, 2011 ; 63 (SE forthcoming): 102 – 123.

Ellig J, McLaughlin PA. The Quality and Use of Regulatory Analysis in 2008. Mercatus Center at George Mason University, Working paper, 2010.

~~~~~~~~

By Jerry Ellig and Patrick A. McLaughlin

Reported by Author; Author

Source: Risk analysis : an official publication of the Society for Risk Analysis, 2012 May, Vol. 32 Issue 5, p855 Item: 22059696

	

	
		Back