Answer five questions in requirement . Each question requires more than 300 words， a total of 1,500 words

MGMT7250_TOPIC4_Scientificevidence_w.pdf

Home >Business & Finance homework help >Management homework help >Answer five questions in requirement . Each question requires more than 300 words， a total of 1,500 words

Evidence-based Management

MGMT 7250

TOPIC 4: Scientific evidence and how to acquire it

Agenda for the day

• Recap of professional evidence • What is scientific evidence and why is it important? • Scientific research designs • Acquiring scientific evidence

Managers and their professional expertise & intuitive judgements

Professional expertise in management

• Cumulative specialised knowledge and skills used to solve professional problems and issues • Differs from personal intuition and judgement which is a more instinctive line of reasoning. • Expertise is best developed through: • Prolonged practice • Highly controlled conditions and environments • Regular direct and objective feedback

Critical thinking in the context of professional expertise

Descriptive/exploratory

What is the question Descriptive question (e.g. what is, how many, to what degree….)

What are the alternative answers/claims to answers?

What do professional experts see as alternative descriptions?

What support exists for each alternative claim?

What is professional opinion on which description is more likely to be true

How good is support for each claim?

Critical appraisal of this professional opinion

Professional expertise in management

• Expertise in management is hard to develop given the nature of management decision problems, issues and opportunities. • By extension is management expertise and knowledge illusory and mythic? • Meta-analytic evidence suggests that job experience does not correlate well with job success (Schmidt et al., 1988).

Limitations (error) of professional evidence

Know the limitations of professional evidence (systematic error): •Limitations of intuitive judgement •Cognitive and social biases •Sampling bias – limits of professional expertise ØSmall n ØIdiosyncratic rather than generalisable

The limits of intuitive judgement • Bounded rationality • Decision neglect: limited search and appraisal of information, incomplete survey of alternatives, selective attention and not drawing on the full range of resources and information available to make a decision

Meta-cognitive bias

Confirmation bias

Availability bias

Illusory of causation/Patternicity

Overconfidence

Outcome bias

Optimism bias

Overgeneralisation

Primacy and recency effects

Cognitive Biases

Gender stereotyping

Attribution bias or self-serving bias

False consensus effect

Similarity bias

Conformity and Groupthink https://www.youtube.com/watch?v=BgRoiTWkBHU

Halo effect

Social Biases

How to mitigate bias?

1. Taking an evidence-based approach to decision-making and practice helps neutralise bias as suggested by research on bias (Kahneman, 2011).

2. Bayesian thinking: ØBeliefs can sometimes be too dichotomous , ie you believe or you don’t believe.

ØBayes’ Rule tries to get around this fallacy by prescribing degrees of beliefs which precludes having absolute certainty in anything.

ØHaving absolute certainty about something means that beliefs cannot be revised in light of new information.

Types of questions & professional evidence

• Different types of evidence generated through different types of research are methodologically appropriate to address different types of questions. • The questions that need to be asked to acquire evidence from professionals to identify problems and solutions tend to be (and need to be) more exploratory prompting the use of qualitative research methods

The image part with relationship ID rId4 was not found in the file.

Who do you ask?

• Knowing who to ask is important • Credibility and dependability

• The more people you ask the better

Methods to acquire professional evidence

Interviews: •Unstructured •Semi-structured *always think about Emic validity – ‘getting inside their heads’

Questionnaires: •Open-ended •Closed-ended – lower levels of systematic error if tested scales are used

Methods to acquire professional evidence

Group methods: •Delphi method •Focus groups

What is scientific evidence & why is it so important?

Critical thinking in the context of scientific evidence

Causal/Cause-and-effect

What is the question “Answerable” Causal question (What causes differences in the outcome, what would solve the problem)

What are the alternative answers/claims to answers?

Alternative causal explanations

What support exists for each alternative claim?

Which of the alternative causal explanations have scientific evidence to support them?

How good is support for each claim?

Critical appraisal of scientific evidence (How good is this scientific evidence?)

EBM in action: CISCO and scientific evidence

Case: M&A at CISCO • What works on average?

• 70% or more of M&A fail to deliver • Odds against success are great

• What are the conditions under which these results would hold? • Systematic examination of what went right and what went wrong in other M&A

• M&A between similar-sized companies rarely works • Works best with geographically proximate companies • Works best with culturally compatible firms

• Result: • Between 1993 and 1998, it acquired one firm per quarter • Successfully ‘digested’ 57 international companies

EBM in action: CISCO and scientific evidence • What was/were the research question/questions? • PICOC?

Cause and effect: Relationships between variables • Key concern: How do differences in the intervention(s) cause differences in the outcome? • To understand this, you compare the impact of intervention(s) to the impact of comparison(s)

M or A Org.

performance + or - compared to status quo?

Cause and effect: Relationships between variables • Context: How do differences in context influence the impact?

M or A Org.

performance

Size diff., cult. Diff., distance

+ or - ?

+ or - compared to status quo?

Cause and effect: Relationships between variables • Context: How do differences in context influence the impact?

M or A Org.

performance

Size diff., cult. Diff., distance

+ or - ?

+ or - compared to status quo?

Cause and effect: Relationships between variables • What did CISCO find?

M or A Org.

performance

Size difference

Cultural difference, geographic distance

- -

Cause and effect: Types of variables in scientific research • Variables: things that vary! Science is mainly about finding relationships between variables • Dependant/Outcome Variable (DV): of primary interest to the researcher • Independent/Explanatory Variables (IV): Factors that explain the dependant variable • Mediator: variable that mediates the relationship between IV and DV • Moderator: Variable that conditions (strengthens or weakens) the relationship between IV and DV.

IV Med DV

Mod

Evidence from scientific research

1. What is science?

What comes to mind when you hear the term ‘science’? Laboratory workers in white coats? The Large Hadron Collider in Geneva? Memories of exciting and seemingly dangerous chemistry experiments at school? These are widespread and popular ideas of science. So, is it any surprise that many managers and leaders, when confronted with the term ‘scientific research’, wonder what it has got to do with them or their work? Of course, science isn’t only about what might be called ‘hard science’. As a method and way of thinking about how to understand the world, science can be applied to almost anything, including organizations, management, workers and business.

The basic purpose of science is to acquire information that will help us to explain, predict and control phenomena in the world. Put differently, science distinguishes itself from other human pursuits by its power to examine and understand phenomena on a level that allows us to predict with some accuracy, if not control, the outcomes of events in the natural and human-made world1. However, for science to do so, we need trustworthy information or data, acquired in a way that minimizes bias and other misleading factors.

Still as human beings, decision makers are inclined to use the most easily accessible source of information – ourselves. What rely on what we remember from our own experiences; what we think; and what we believe to be true. Our self-centered approach to information makes does save time and effort – and probably works quite well for simple day-to-day decisions. For more complicated decisions in business and management, however, relying solely on our own experience and judgment may well lead to poor decisions. As you have learned in chapter X, we are prone to cognitive biases in our thinking, and this causes us to make mistakes in the analysis and interpretation of what our self- provided information means. Science emerged partly as a response to the twin problems of relying solely on personal information and the biases inherent in interpreting it. A core activity of science is therefore gathering objective, external information, rather than relying solely on the subjective internal knowledge in our heads. Because even the smartest people can easily be fooled into believing something that is not true, we need to put some some ‘safety measures’ into place when we acquire external information.

1.1. The scientific method One of these safety measures is ‘the scientific method’, a defining feature of science since the 17th century. The scientific method can be summarized in one sentence: Do whatever it takes to avoid fooling yourself into thinking something is true that is not, or that something is not true that is.2 It is a procedure that scientists use to ensure the trustworthiness of their findings by 1) asking a question about something they observe (how? what? when? how many? who? why? or where?), 2) formulating a hypothesis (an assumption about how things work or a prediction about

What is scientific evidence?

• Using the scientific method to use empirical evidence to test theories and assumptions, primarily about cause and effect • According to Karl Popper, the scientific method involves the method of falsification to disprove theories • Therefore, “Answerable questions”

• Research publishing in academic “peer-reviewed” journals • Scientific findings do vary in terms of their levels of trustworthiness to support a claim or address a question despite the peer review process.

Hallmarks of the scientific method

• Theoretical framework regarding relationships between variables • Hypothesising • Testing of hypotheses using evidence • Research design that enables you to infer cause and effect • Collection and (statistical) analysis of evidence based on research design to test hypotheses (e.g. is the relationship between IV and DV as hypothesized?)

The ladder of causation

(Pearl, 2018)

Counterfactuals and causality

• The “What if” • E.g. What if our organization had merged with a similar-sized organization— would our profitability have been greater? • Or: What if our organization does decide to merge with a similar-sized org— will our profitability be greater?

• Problem with this?

Counterfactuals and causality

• Problem with this: You cannot go back in time to change your organization. What you did cannot be reversed. • Solution?

Counterfactuals and causality

• The “What if” • E.g. What if our organization had merged with a similar-sized organization—would our profitability have been greater?

• Or: What if our organization does decide to merge with a similar-sized org—will our profitability be greater?

• Problem with this: You cannot go back in time to change your organization. What you did cannot be reversed. • Solution: compare other organizations that did engage in a merger (intervention) to other organizations that didn’t (comparison) • Problem with this: Other organizations might be very different from yours and those differences might lead to a different experience in terms of impact of IV on DV • Solution?

Counterfactuals and causality

• The “What if” • E.g. What if our organization had merged with a similar-sized organization—would our profitability have been greater?

• Or: What if our organization does decide to merge with a similar-sized org—will our profitability be greater?

• Problem with this: You cannot go back in time to change your organization. What you did cannot be reversed. • Solution: compare other organizations that did engage in a merger (intervention) to other organizations that didn’t (comparison) • Problem with this: Other organizations might be very different from yours and it will be difficult to identify the true cause since you don’t know which of the differences in features is causing the difference in outcome

Alternative explanations

Yes M&A No

250m Size 300m

3 years Age 6 years

3 Product quality 3.5

Profit: $20m

Profit: $30m

? ?

Ruling out alternative explanations: Random assignment • Imagine you’re a chef who wants to see the effect of adding some special seasoning to your most popular dish • How would you do it?

Random assignment: “Stirring the pot” before introducing the intervention

“Stirring the pot”

Random assignment, control variables

• Since we can’t always compare similar organizations or individuals, we could use: • Random assignment: Individuals or organizations are randomly assigned to ‘treatment’ or ’control’ group. Each individual or org has an equal chance of being in either group. • This helps ensure that on average, the two groups will be similar. • Random assignment helps wash out the effects of unmeasured individual differences explaining variation in an outcome.

• Control variables: When you can’t conduct experiments, you can account for other differences among individuals and orgs in your study by using control variables, which measure the impact of other forces (than your main intervention and comparison variables) on the outcome

Statistical inference

• Statistical methods used to infer features (e.g. causal relations) about the population based on an analysis of sample data • Need the sample to be representative • Larger samples more valid

Critical thinking: Establishing causality – 3 steps 1. Establish whether there is strong correlation, we usually use an effect

size to infer this. • Use quantiative/statistical methods to do this

2. Establish whether the cause precedes effect, we need to establish whether there has been before and after measurement or whether a baseline measure exists.

• But beware of “post hoc, ergo propter hoc” fallacy

3. Rule out alternative explanations (Theory can help explain why an effect exists or does not as we have learned). • Random assignment • Control variables • Other methods: e.g. propensity score matching

Critical appraisal of scientific evidence

Two criteria to determine the trustworthiness of scientific evidence: ØMethodological appropriateness

ØMethods appropriate for type of question or claim (ie., about effects) ØInternal validity – the extent to which legitimate causal inferences can be drawn.

ØMethodological quality ØMeasurement reliability and validity ØSampling and external validity

Scientific research designs

Research designs (1) • Systematic review - Addresses a specific question, utilises explicit and transparent methods to perform a thorough literature search and critical appraisal of individual studies, and draws conclusions about what we currently know and do not know about a given question or topic (Briner & Denyer, 2012). • Meta-analysis - Uses statistical analysis techniques in a systematic review to pool the results of the individual studies numerically in order to achieve a more accurate estimate of the effect.

• Randomised control study (true experiment) - Groups compared with each other are selected entirely at random, for example by drawing lots. This means each participant (or other unit such as a team, department or company) has an equal chance of being in the intervention or control group. In this way the influence of distorting factors is spread over both groups so the groups are as comparable as possible with each other with the exception of the intervention.

• Non-randomised control study (quasi-experiment; field experiment) - Two or more groups are compared with each other, usually comprising one group in which an intervention is carried out (experimental group) and one group where no or an alternative intervention is conducted (control group). Allocation to the groups is not randomised.

Research designs (2) • Longitudinal study: Subjects studied over many time periods

• Before and after study (pre-test, post-test) - Observations are made before and after an intervention.

• Interrupted time-series - Observations are made at multiple time points before and after an intervention, to detect whether the intervention had a significantly greater effect than an underlying secular trend.

• Comparative or case-control study - Compares people or cases in terms of a specific outcome.

• Cohort or panel study (survey) - A group of people followed over time including before and after an intervention.

• Cross-sectional study (case study, survey) - Independent variables and dependent variables are measured at the same point in time.

• Qualitative study (interviews) - Explores and tries to understand people's beliefs, experiences, attitudes, behaviour and interactions. It generates non-numerical data. The best-known qualitative research-methods include in-depth interviews, focus groups, documentary analysis and participant observation.

Cross-sectional vs longitudinal

• In cross-sectional research, the independent variable and dependent variables are measured at the same point in time which means that reverse causation is possible. • Temporal antecedence is an important characteristic of longitudinal research. • Longitudinal research includes studies that involve repeated observations (measurements) of the same variable(s) over a certain period of time, such as cohort studies or interrupted times series. • It is possible with longitudinal research to explain changes in the dependent variable or outcome over time.

Research design and the 3 steps of causality

Establishing correlation

Cause before effect Random assignment

Control variables, etc

Randomised controlled trials (experiments)

Y Y Y N

Field/Natural experiment

Y Y N Y

Longitudinal study Y Y N Y

Cross-sectional study

Y N N/Y Y

Case study N N N N

How to acquire scientific evidence

Peer-reviewed journals

• Academy of Management Journal. • Administrative Science Quarterly. • Organization Science • Strategic Management Journal. • Journal of Management. • Journal of Applied Psychology. • Psychological Bulletin. • Journal of Management Studies. • Journal of Organizational Behavior. • Journal of Human Relations. • Journal of Vocational Behavior

ABI/INFORM Business Source Elite PsycINFO ERIC Web of Knowledge Google Scholar

Online research databases

Source: CEBMa

Online databases

• There are a number of online e-databases that provide access to full- text academic journal articles including the in journals above. • Organisations and their research teams would benefit immensely to subscribe to any of the following e-databases. • Where organisations do not have access to e-databases, they can contact academics to assist with access. • Managers undertaking executive training at universities will have access to such databases. Once you know how to use these databases, searching for scientific findings can be very expedient.

Undertaking REAs & CATs

• Expedient forms of systematic reviews • A Critically Appraised Topic (CAT) provides a quick and succinct assessment of what is known (and not known) in the scientific literature about an intervention or practical issue by using a systematic methodology to search and critically appraise primary studies • Rapid Evidence Assessments (REAs) are more extensive in scope than CATs, and need to be undertaken by groups. • Organisations can either commission REAs which are an underutilised input to decision and policy- making.

Using PICOC for search terms

• Start with the two most important PICOC terms – the intervention and outcome. • Coming up with a couple of alternative terms for the outcome and intervention may also be useful. With the example just used ‘performance’ may be used instead of ‘productivity’ for example. • Then when you have some search terms it is important to pre-test them in one database. • If the yield is too big, it is important to refine your search to increase specificity. If your search yields very few papers, consider dropping a search term.

Search strategies

1. Shotgun • Use your search terms to generate a list of articles

2. Snowball • Snowballing backward: Snowballing backwards is where you begin with a more recent publication and discover which publications were used by the author by looking at the bibliography of books or articles. • Snowballing forward: Snowballing forward is where you begin with a less recent publications and discover how often that book or article has been cited by other authors by looking at Web of Knowledge or Google Scholar.

Other tips • include the terms ‘studies’, ‘systematic review’ and ‘meta-analysis’ • tick a box if available ‘scholarly journals including peer-reviewed’. • use quotation marks (e.g. "search") to search for specific key-words. • use advanced search (access from down arrow in search bar) to exclude terms, or limit the search to a specific date range or specific publication.

• not all articles are created equal - one way to identify important articles is to look at the 'cited by...' number. The higher the number, in general, the better the article. This method is not fool proof as sometimes an article may be cited highly for the wrong reasons. Click on the 'cited by' link to view articles that cite a particular article.

• use the 'related articles' link below a specific search result to find similar articles, and the 'cite' link to show a formatted reference to the chosen article.

• use the links on the far right of the screen to access full text versions of articles. You will note not all articles have a link to a free ee full text version. You can access the original article by clicking on the article title - this will take you to the publisher's website, where you can view the abstract and buy an electronic version of the article.

Activity – Search for a meta-analysis

Purpose of this activity is to learn how to use online e-databases to search for scientific evidence to test a claim or address a question Instructions: Pick one of the following claims and search for a meta- analysis. Do the findings support the claim? Effect sizes? • Goal setting increases productivity • What are the drivers of sales performance? • Trust in teams increases performance • Open-office layouts improve team performance EBP capabilities: Acquire, Critical thinking skills.