Answer five questions in requirement . Each question requires more than 300 words， a total of 1,500 words

Evidence-based Management

MGMT 7250

TOPIC 6: Critically Appraised Topics (CATs)

Agenda

• Describe the purpose of a CAT • Distinguish a CAT from other review methods • Acquire scientific evidence to address a CAT question • Documenting search terms, strategies and search results • Appraise the methodological appropriateness of scientific evidence to address a CAT question • Develop evidence-based recommendations for practice

Agenda

• Describe the purpose of a CAT • Distinguish a CAT from other review methods • Acquire scientific evidence to address a CAT question

The value of CATs & acquiring the scientific evidence

CATs? A Critically Appraised Topic (CAT) provides a quick and succinct assessment of what is known (and not known) in the scientific literature about an intervention or practical issue by using a systematic methodology to search and critically appraise primary studies (CEBMa, 2018). Ø  CATs as an expedient form of a

systematic review are helpful for identifying what the science says about what works and what doesn’t.

Ø  Addressing cause and effect questions.

Background

• 2-4 paragraphs • Rationale for CAT • Why is the CAT important? to organization? Stakeholders? •  Includes the CAT question(s) •  Includes PICOC in table form, e.g.:

CAT Question(s) – Main & supplementary Qs

• Asking, which involves translating a practical issue/opportunity into an answerable question is one of the most important steps in EBP! • Mainly address cause and effect question(s)

•  IV(s) & DV •  IV driven: the effect of an intervention •  DV driven : drivers of an outcome

• Starts with the stem: what is known in the scientific/scholarly literature about……..

Activity

•  In groups of 5 • Come up with a CAT question (not one that will be used in your EBM report) • Translate the CAT question into the PICOC framework • Talk about why it is important to your organization of choice

Inclusion & exclusion criteria • Eligibility criteria • help the reviewer(s) to determine whether a study will be included in the CAT when reviewing its abstract and/or full text. • Exclusion criteria may exclude studies in a certain context

Search strategy

• This part of a CAT describes how studies were searched • Which databases were used • Which search terms were used including alternative search terms • Any search filters were applied

•  Methodological filters •  E.g., TI(meta-analy*) OR AB(meta-analy*) OR TI("systematic review") OR AB("systematic review")

• Table of search queries

Activity

• Same groups of 5 • Write down your inclusion/exclusion criteria • Think of search terms and filters you will use

e-databases

• Use ABI INFORM only for the EBMgt report • Should have access on-campus • Off-campus: try https://search-proquest-com.virtual.anu.edu.au/advanced

Study selection

• This part of the CAT describes the stages the reviewer went through to select studies Ø How you came about developing the different search queries that were tabulated in your search strategy section.

• ABI Inform deals with duplicates •  Initial screening of papers – relevance of title and abstract to CAT question(s) to check if they meet inclusion criteria

*Rule of thumb for your report – minimum of 8 papers including meta- analyses and systematic reviews.

Activity

• Same groups of 5 • Do an initial screening of the journal articles and shortlist some articles (at least 3)

Appraising the evidence

Data extraction •  Describe the key pieces of information extracted from each paper and tabulate them and rate their methodological appropriateness: •  year of publication, •  research design, •  Sample size, •  population (e.g., industry, type of employees), •  outcome measures, •  main findings, •  effect sizes, •  weaknesses, and the final level of trustworthiness should be reported clearly in structured table.

Example table in critical appraisal section

Recap Effect sizes – Cohen’s rule of thumb

https://www.statistics.com/glossary/

Critical appraisal

• Discuss the methodological appropriateness of the included studies. • Research design determines the study’s causal interpretability • Use rating system to rate methodological appropriateness, AA and A the highest ratings for methodological appropriateness.

Types of variables in scientific research • Dependant Variable (DV): of primary interest to the researcher •  Independent Variables (IV): Factors that explain the dependant variable • Mediator: variable that mediates the relationship between IV and DV • Moderator: Variable that conditions (strengthens or weakens) the relationship between IV and DV.

IV Med DV

Mod

Establishing causality – 3 steps

1.  Establish whether there is strong correlation or covariation, we usually use an effect size to infer this.

2.  Establish whether the cause precedes effect, we need to establish whether there has been before and after measurement or whether a baseline measure exists.

•  But beware of “post hoc, ergo propter hoc” fallacy 3.  Rule out alternative explanations (Theory can help explain why an

effect exists or does not as we have learned). •  Random assignment •  Control variables •  Other methods: e.g. propensity score matching

Statistical significance: p-values

• P-values comes in handy as they tell us about how likely an hypothesis is likely to be true. • P-values are probabilities that our estimated relationships or differences among variables are due to chance. •  A P-value of .05 is a commonly used threshold or level of significance (this mean that the chance that the outcomes of our study are due to chance are 5% or below) and anything above the .05 level is statistically non-significant.

Recap of methodological appropriateness rating

Activity

• Same groups of 5 •  Identify the research designs in the articles you have selected

Aggregating the evidence + applying it

Results – a) definitions

• Define key constructs

Results – b) mechanisms

• Discuss potential mechanisms linking the IV(s) to the DV(s) • Postulates of key theories

Results – c) Main findings

• This section, the longest section of the CAT, address the review question(s). •  Include rating of methodological appropriateness and effect size:

Conclusions

• Summarizes the main findings in relation to the CAT question in 2-4 sentences.

Limitations of the CAT

• All CATs have limitations • Discusses the limitations of the included studies in terms of what is NOT known. •  Nascent literature vs mature literature

• Scope of search • Critical appraisal did not assess methodological quality:

Ø population & ecological validity Ø Reliability Ø Content validity

Practical Implications & Recommendation

• How can your organization act on the CAT and its insights? • Specific recommendations for practice? • Potential boundary conditions (moderators) that need to be taking into account? • How will you take stakeholder concerns into account?

Practical significance

•  It is important to appraise just how much findings are practically important. • Effect sizes rather than p-values (Topic 4) tell us something about practical significance. • An effect may be statistically significant but not practically significant because the effect size is small. • When it comes to management interventions, it is worth investing resources where effect sizes are moderate to large.

How do you identify possible interventions or solutions in the science? •  Let’s think of an example where a meta-analysis by Griffeth, Hom, & Gaetner (2000) finds that organisational commitment has a moderate effect size on turnover (ρ = -.23). In the first instance the finding suggests the organisation would benefit from increasing commitment. •  But what how does this translate into an intervention? 1. Many research papers have a ‘practical implications’ section. This can be helpful for identifying interventions or strategies to address a problem. 2. It would also be worth acquiring additional evidence to address a question regarding solutions: Ø  “what is known in the scientific literature about drivers of commitment?”

Practical relevance

•  In evidence-based practice in management it is important to think about how findings are relevant to a local context. •  When it is not clear whether the effects generalise to the local context (i.e. have ecological validity) it is important to consider whether an effect or cause may or may not occur in the local context:

Ø Is your organisation/division/population so different from those in the study that the study's results cannot apply?

Ø How relevant is the study to what you are seeking to understand or decide? Ø How could the research potentially benefit or harm your organisation? Ø Is the intervention feasible in your setting? Ø What are your executive’s (or client’s) concerns, preferences and expectations for the outcome you are trying to prevent and the research you are offering?

Example of practical implications