1,000 word essay aligned with the Drummond checklist critiquing the quality of the assigned article

· Does not always lead to choices that maximize public health and limited resources

· Is complicated and not as easily understood as traditional randomized clinical trials

· Represents a relatively new field

18 of 36

· Are the results helpful?

· Are the methods appropriate?

· Are the results valid?

· Do valid results apply to my decision context?

28 of 36

· Indicates compliance with established quality standards

· Refers to clear description of the model structure, equations, parameters, and assumptions

32 of 36

· Lay summary, accessible to any interested reader

· Model type and intended application; funding sources; model structure; model inputs, outputs, and components; and validation methods, results, and limitations

· Detailed technical document, allowing expert evaluation and potential recreation

35 of 36

· Is well defined and in answerable form

· Clearly identifies the alternatives being compared

· Identifies the perspective from which the comparison is made

17 of 42

· Is an active PE policy intervention worth it?

· Will a smoking-cessation intervention do any good?

· How much does it cost to run our syringe-exchange program?

· What are the costs and outcomes of the school wellness policy?

33 of 42

36 of 42

From the perspective of (a) both the Ministry of Health and the Ministry of Community and Social Services budgets and (b) patients incurring out-of-pocket costs, is a chronic home care program preferable to the existing program of institutionalized, extended care in designated wards of general hospitals?

41 of 42

· May be a deciding factor of comparative cost-effectiveness

· Should justify any departure from societal perspective and explain any expected impacts on results

· Should be provided for each competing alternative

· Who did what to whom, where, and how often

· Helps readers judge relevance to a given setting

· Allows readers to detect cost or consequence omissions

· Provides enough information for program duplication

18 of 35

· Represents the full range of potential alternatives

· Covers all feasible and practical options, regardless of data availability

· Includes any extendedly dominated options

· Explores the appropriateness of a "do nothing" option

· Compares to the current practice if only one comparison is made

27 of 35

· Are valid effectiveness estimates included for each alternative?

· Are the estimates convincing?

· Is the evidence produced by a single study?

· Does it represent the field's broader evidence?

· Did the authors justify use of a single source?

34 of 35

· Helps demonstrate the evaluation's effectiveness

· Thoroughly covers underlying methods, even if cited from previous works

· Evaluates causal inference from quasi-experimental or observational data

· Avoids reliance on randomized controlled trial data for effect size

· Studies often conduct identification, measurement, and valuation simultaneously.

· Quality evaluation considers each step separately.

15 of 40

· Was the full range of costs and consequences included?

· Could an expert assess the completeness?

· Were all the relevant perspectives covered?

· Were both capital and operating costs included?

22 of 40

· Were the resource utilization sources described and justified?

· Were any identified items omitted?

· Were there any special circumstances that complicated measurement?

· E.g., overhead costs paid from shared resources

28 of 40

· Were sources identified for all values?

· Were approximations made for any absent market values (as for volunteer labor)?

· Were consequences valued appropriately for the question (e.g., conversion of health effects to monetary value for cost-benefit analyses)?

33 of 40

· Were measurement-strategy-related variations in health state valuation described and justified?

· State-to-preference mapping method

· Preference elicitation details

· Were costs and consequences discounted to their present value? Were the discount rates justified?

40 of 40

· Were the comparisons properly chosen?

· Were both strongly dominated and extendedly dominated alternatives included?

· Were the calculations for the final, best incremental cost-effectiveness ratio well documented?

· Were appropriate statistical analyses performed?

· Were the proper distributions used?

· How were missing data addressed?

· Were alternative statistical techniques used to accurately model the incremental cost-effectiveness ratio uncertainty from trial data?

14 of 36

· Were all of the uncertain parameters identified?

· Were parameters with no assumed uncertainty identified?

· Was any lack of uncertainty justified (e.g., known with absolute certainty)?

21 of 36

· If an arbitrary range of plausible values was used for important parameters, was empirical justification provided for it and the distribution shape?

· Were easy-to-fit distributions (e.g., triangular) with limited statistical justification used?

· Was unavoidable lack of uncertainty evidence for any important variable clear in results discussion?

27 of 36

· Was justification made for any departure from a probabilistic approach?

· If a probabilistic approach was used, was there an analysis of the expected value of perfect information?

· Did the study use any other forms of sensitivity analyses (e.g., scenario analyses on model uncertainty)?

35 of 36

· Were the main results sensitive to the sensitivity analyses results?

· Was heterogeneity within subgroups assessed?

· Implications for generally cost-ineffective interventions with potential value for high-risk groups

· Important for evaluating an intervention's potential health disparities effects

· Necessary for critical review of economic evaluations

· Foundation upon which modern methodological advice is built

· Considered standard practice

12 of 83

· Lack of transparency and quality are barriers to utilization of economic evaluation studies.

· Transparency for lay and technical audiences helps ensure quality.

· The wide range of underlying assumptions used in modeling studies is a barrier to assessing quality.

· For comparisons of different studies of the same intervention to be meaningful, the methodologies underlying the results are comparable.

· Prescriptive reporting guidelines aid in transparent communication.

· Agreed-upon preferred analytic methods are needed.

17 of 83

· The elements should be included in every analysis.

· This still allows analysts to report alternative results using different methods while promoting comparability in the field.

· The reference case developed by the U.S. Public Health Service Panel on Cost-Effectiveness in Health and Medicine is a guiding document.

· A second panel will be updating these guidelines in 2016 after 20 years of use in the field.

23 of 83

The societal perspective should be adopted.

· Taking a narrow perspective that includes either healthcare system or patient costs and outcomes may not adequately capture the costs and outcomes of a health-promotion intervention.

· Societal perspective captures:

· All benefits and harms of the intervention in the outcome, even if they do not accrue to the intended recipients

· The costs of all resources, regardless of who pays and even if no money changes hands

29 of 83

Effectiveness estimates should incorporate benefits and harms.

· Corollary to societal perspective

· Narrowly considering only the benefits to target population could ignore harms to others.

· Example: Folic acid fortification potentially masking pernicious anemia in elderly populations

· An intervention can have unintended negative consequences for a group that was not the intended target.

35 of 83

Mortality and morbidity consequences should be combined using quality-adjusted life years (QALYs).

· Example of hypothetical cancer therapy

· Ignoring health-related quality of life (HRQoL) by focusing narrowly on life years obscures critical information on treatment effectiveness.

· Use of disability-adjusted life years (DALYs) widely accepted

· Analysts may consider reporting both QALY and DALY results when feasible.

39 of 83

Effectiveness estimates from best-designed and least-biased sources should be used.

· This highlights the importance of causally strong systematic review and meta-analysis techniques in the field.

· The utility of trial-based evaluations is limited if the effectiveness results deviate widely from the best evidence in the broader field.

45 of 83

Costs should include healthcare services, patient and caregiver time, and costs of nonhealth impacts.

· This is closely related to the adoption of the societal perspective.

· It is important to capture the costs from all of these sectors.

· Costs should be reported separately to help identify barriers to adoption of cost-effective interventions.

· Who pays is not always (or often not) who benefits from public health interventions.

50 of 83

Comparison should be made with existing practice and, if necessary, a viable low-cost alternative.

· Comparison to current practice directly relates to the decision context, but may be misleading if current practice is based on ineffective or expensive interventions.

· Anything will look cost-effective if compared to a sufficiently bad alternative.

· Status quo bias should not require us to limit the consideration of alternatives to current bad practice.

57 of 83

Discounting of costs and health outcomes should be undertaken at a real rate of 3% per year (with 5% as an alternative scenario).

· Debate continues about the "best" discount rate as well as whether health and cost outcomes should discounted at the same rate.

· But there is general consensus on uniform discount rates.

· Using a reference discount rate is critical for comparability.

· Rate may vary based on different decision-making contexts.

· Analyses should include the reference rate for broader comparability.

61 of 83

One-way and multiway sensitivity analysis should be undertaken for important parameters.

· Guideline replaced by the general practice of conducting probabilistic sensitivity analyses

· Limits of using one-way or even multiway sensitivity analyses noted

65 of 83

Comparison of the ICER should be made with those for other relevant interventions.

· Broadly applicable

· Requires consideration of the most relevant alternative

70 of 83

· Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Task Force

· Developed to be consistent with CONSORT methodology for reporting results from clinical trials

· Are largely consistent with guidelines previously reviewed

· Contain additional detail on reporting economic evaluations within context of clinical study, as opposed to when using decision-analytic model

75 of 83

· National Institute of Health and Care Excellence (NICE)

· Ensure minimum methodological standards

· Improve comparability across submissions

· World Health Organization (WHO)

· Used extensively outside of Europe and North America

83 of 83

International Society for Pharmacoeconomics and Outcomes Research

· ISPOR has more than 30 sets of guidelines and continues to provide additional guidance.

· These guidelines are:

· "Official" (i.e., intended for making reimbursement decisions)

· Technical guidance for academics to improve the quality of research

· ISPOR collaborated on a task force with the Society for Medical Decision Making, issuing a series of articles on Modeling Good Research Practices that update the technical aspects of the original reference case guidance.

· Clearly describes the model structure, equations, parameter values, and assumptions

· Enables interested parties to understand the model

· Establishes foundation upon which model evaluation is based

9 of 80

International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-SMDM

· Transparency and validation are required components of full model evaluation.

· Validation is subjecting the model to tests, such as comparing model results with events observed in reality.

· If transparency helps readers understand what a model does and how it does it, validation is the only way for readers to determine how well it works.

13 of 80

· The effort put into building a comprehensive model of disease epidemiology and related health and healthcare costs makes using the same model to evaluate multiple interventions attractive if only to realize some economy of scale.

· More importantly, using the same model to evaluate a range of interventions provides a level methodological field so that variation in ICERs across interventions should be due primarily to intervention-specific inputs and assumptions instead of variation in underlying modeling approaches.

· For these types of models, there is a distinction between validation focused on underlying methods and validation of a specific application of the model.

· Because there is no set of criteria that makes a model "valid," the concept of validation should apply to the specific application, not to the model itself.

19 of 80

1. Face validity

2. Verification (or internal validity)

3. Cross-validity

4. External validity

5. Predictive validity

31 of 80

· Was once a core component of validation in scale development

· It has fallen out of the nomenclature given its subjectivity.

· ISPOR-SMDM notes that as currently practiced, peer review is insufficiently consistent to be relied on for determining face validity.

· Included in current best practice recommendations, despite limitations

· Should include description of process

37 of 80

· Also called internal validity, consistency, or technical validity

· Is a two-step process:

· Verifying individual equations

· Verifying accurate implementation in code

· Should be aligned with best practices in software development

46 of 80

· This is also called external consistency.

· This involves:

· Examining different models that address the same problems and comparing the results

· Examining the differences and their causes

· If different models are highly dependent (i.e., use parameters from other models), the value of cross-validation decreases.

· Modelers should:

· Search for modeling analyses of the same or similar problems

· Discuss insights gained from similarities and differences in results

55 of 80

· Compares a model's results to actual event data

· Involves simulating events that have occurred, often from a clinical trial

· Could apply to the model as a whole or to components such as:

· Population creation

· Disease incidence

· Disease progression

· Outcome occurrence

· Intervention effects

65 of 80

· In a modeling context, refers to ensuring that inputs and outputs are consistent with available data

· Is important for unobservable intermediate parameters

· Commonly, matches output to external data and adjusts model parameters to achieve a fit with known outcome

· Should include in reporting

· Target

· Goodness-of-fit metric

· Search algorithm

· Acceptance criteria

· Stopping rule

72 of 80

· Involves:

· Identifying an opportunity in which a study design can be well-specified and simulated while reporting the outcome before the outcome occurs

· Comparing the predictions to what actually happened

· Is an important test for multiple-use models that are attempting to inform a broad set of specific applications

· Can be most easily done based on well-reported clinical trial designs that are reported before results are released

· Is much harder to achieve in health-promotion interventions

80 of 80

Users of models should examine validation results for:

· Rigor of the process

· Quantity and quality of sources used

· How well sources represent model's proposed use

· Model's ability to simulate sources in appropriate detail

· Closeness of results to observed outcomes

· Growing evidence of poor use of limited health resources

· Rising healthcare expenditures

· Increasing doubts about "free" market as the mechanism for health resource allocation, access to health/health outcomes

10 of 72

· Attempts to maintain the clear benefits of economic evaluation by incorporating other resource-prioritization approaches

· Is a recognized field that deals with rationing, but lacks consensus on best methods

18 of 72

· Behavioral science (achieving consensus)

· Epidemiologists/clinicians (needs assessment)

· Philosophers (social justice)

· Economists (efficiency)

· Administrators (feasibility, historical allocation)

26 of 72

Assessing Cost-Effectiveness (ACE) framework (Rob Carter et al.) is the basis of an "ideal" approach to priority setting.

· Economic theory: guidance on choices amidst resource scarcity

· Ethics: standards for judging societal welfare

· Empirical evidence and user considerations for relevance

30 of 72

· Provides theoretical rationale for better-to-worse alternative ranking

· Bases rank on an allocative efficiency assessment reflective of societal values

· Overlaps with ethics definition of societal welfare

34 of 72

· Welfare economics: favors individual preferences and role of the market

· Extra-welfarism: is health-based, conducive to government intervention and judgment by health professionals

· Decision-making approaches (DMA): focus on the decision-maker as arbiter

37 of 72

40 of 72

· Like economics, uses logical argument with explicit assumptions

· Unlike welfarism, sometimes rejects the role of community preferences in moral judgments

47 of 72

· Deontology: duty, process, and individual rights

· Consequentialism: outcomes and collective good

· Distributive justice: impartial, fair balance of competing individual claims

· Combination based on background, experience, and choice problem and setting

60 of 72

· Is commonly used but not as a purely technical approach

· Values efficiency but not at the expense of other objectives

· Strives to balance economic techniques and decision rules with due process and consultation

· Suffers from technical barriers (e.g., lack of data), can limit scope

66 of 72

· Aims to deliver services to people who will benefit

· Creates guidelines for matching patients to treatment types and intensity levels

· Focuses on evidence-based clinical and public health practices rather than efficiency-based care rationing

71 of 72

While decision-making approaches cannot ignore economic evaluation, policy makers and the public reject those which rely exclusively upon it.

· In 2010, the Assessing Cost-Effectiveness (ACE) in Prevention study issued a final report of five-year effort to evaluate cost-effectiveness of prevention investments in Australia.

· Funded by Australian National Health and Medical Council

· Led by Theo Vos and Rob Carter of University of Queensland and Deakin University

· Developed to address the challenges of aging population.

· With consideration of increasing budget pressures

· With a focus on identifying best values in prevention

14 of 93

· Focused on achieving a more efficient and fairer prevention system

· Addressing deep health inequalities facing indigenous Australians

· Allocating prevention resources to reduce inequities

· Was the largest and most rigorous evaluation of preventive strategies at the time

· Is still a model for using economic evaluation to develop broad health-promotion prioritization strategies

20 of 93

· Evaluated 123 prevention interventions and 27 treatment interventions

· Adapted the general population model to evaluate the cost-effectiveness of 21 interventions specifically for indigenous population

· More than doubled the total number of economic evaluations of health promotion and illness prevention in Australia

· Placed results in broader priority-setting context by:

· Getting guidance from stakeholders in government, health nongovernment organizations, academia, and service providers

· Having stakeholders help formulate conclusions by combining technical results with other policy-relevant considerations such as acceptability, feasibility, and equity

21 of 93

47 of 93

Large impact on population health could be achieved with a limited number of cost-effective interventions.

· Taxation of tobacco, alcohol, and unhealthy foods

· Regulation of sodium in bread, cereals, and margarine

· Improving efficiency of blood pressure and cholesterol-lowering drug prescription

· Gastric banding for severe obesity

· Intensive sun-exposure-prevention campaign

55 of 93

Technical findings were combined with second-stage filter considerations.

· The evidence varies.

· "Likely" for taxation and regulation

· "Sufficient" for treatment interventions

· "Limited" for sun-exposure campaign

· Taxation and regulation have low implementation costs but high political costs.

· Proposed changes to blood pressure and cholesterol prescribing have challenges to stakeholder acceptability.

· Government subsidies for gastric banding requires additional guidelines on best practices.

61 of 93

· 43 were either cost-saving or cost less than $10,000 per DALY prevented.

· These should only be ignored if there were serious reservations about the evidence base or insurmountable problems related to acceptability or feasibility.

· 31 provided good value compared to a threshold of $50,000 per DALY prevented.

· 38 were expensive compared to the threshold of $50,000 per DALY prevented.

· Of these, four had insufficient evidence, two were associated with more harm than benefit, and two were dominated by more cost-effective alternatives.

The authors concluded that prevention interventions are not always good value and are not always better than treating disease.

70 of 93

81 of 93

· Conducted because of differences in many key evaluation parameters

· Target disease burden

· Prevalence and distribution of harmful exposures

· Effectiveness of intervention strategies

· Models of health service delivery

· Acceptability to stakeholders

· Cost of implementation

· Identified meaningful differences in population's health priorities

· Community health gain

· Cultural security

92 of 93

· Increase tax on tobacco by 30%.

· Change to volumetric tax on alcohol.

· Implement 10% tax on unhealthy foods.

· Enact mandatory regulation of salt in bread, margarine, and cereal.

· Disinvest in prevention activities found to be not cost-effective, including:

· Current cardiovascular preventive treatment

· PSA testing for prostate cancer

· Aspirin for primary prevention of cardiovascular disease

· Most approaches for promoting fruit and vegetable intake and weight-loss programs

· School-based illegal drug interventions