Activity

Evidence-based Management: It’s Not an Oxymoron!

Our Journey So Far…

Defining BS

Detecting BS (logically)

Describing where and why BS thrives

Deciphering BS (statistically)

Detecting BS (methodologically)

Distributions P-values Power

Confidence Intervals Effect Sizes

Proxies & Ecological fallacy

Research Designs Strategies & Studies

“The Lingo” Threats to Validity

Toward Evidence-based Practice Evidence-based practice is about making decisions through

the conscientious, explicit, and judicious use of the best available evidence from multiple sources by

Asking: Translating practical issues/problems into answerable questions1

Acquiring: systematically searching for and retrieving the evidence2

Appraising: critically judging quality of the evidence3

Aggregating: Weighing and pulling together the evidence4

Applying: Incorporating evidence into decision-making processes5

Assessing: Evaluating decisions to increases success probabilities6

The Challenge of Practicing EBM

Evidence-based practice ignores practitioners’ professional experience ✗

Evidence-based practice is all about numbers, analytics, & statistics ✗

Managers need to make decisions quickly & don’t have time for EBM ✗

Each company is unique, so usefulness of scientific evidence is limited ✗

If you do not have high-quality evidence, then you cannot do anything ✗

Good-quality evidence gives you the direct or perfect answer ✗

False beliefs about EBM

Sources of Evidence

Scientific Literature (empirical studies)

Organization (internal data)

Practitioners (professional expertise)

Stakeholders (values and concerns)

Sources of Evidence

Scientific Literature (empirical studies)

Organization (internal data)

Practitioners (professional expertise)

Stakeholders (values and concerns)

Organizational Facts (and Errors)

Small Numbers Range Restriction

Measurement Error Confounds

Example: Using last year’s average to predict/forecast next year’s

numbers (e.g., sales)

Example: Profiling “top performers” to develop criteria to

use in personnel selection

Example: Using difference scores to indicate financial performance

(e.g., profitability)

Example: Examining the impact of training but not controlling for

trainee attributes (e.g., motivation, cognitive ability)

Organizational Facts (and Errors)

Small Numbers Range Restriction

Measurement Error Confounds

Solutions: aggregate data across units or time; use confidence intervals; form collaborations

(industry/regional groups)

Solutions: examine more than just extreme cases; apply statistical correction for range restriction

Solutions: use multiple indicators; split differences into component

parts for analysis; correct for unreliability

Solutions: use control variables; aggregate data sets across units;

study the variables over time

The Pitfall of Patterns (and Intuition)

Trust me…20 years of management experience

What Good Theory Does

ü Organizes (parsimoniously) and communicates (clearly) ü Helps identify and define key problems ü Provides a basis for making predictions ü Gives meaning to observations (i.e., aids interpretation) ü Scopes empirical investigations (i.e., important variables) ü Permits generalization beyond the focal sample

Ultimate goal of theory is to explain… Why?!? (how & when too)

Why Are Fads So Compelling? § Promise effectiveness and efficiency (fast!) § Appear simple and readily implementable § Help assuage anxieties about problems § Make users feel ‘cutting edge’ § Often offer ‘half-truths’ (things that might work

sometimes, in some contexts)

What tells you that a model is probably a fad?

Some Telltale Signs of a Fad § Too simple, straightforward

§ Easy to communicate, reduces to a very small number of factors § Promise (even guarantee) results

§ Fad auteurs are confidently didactic – no false humility or hedging § Universal applicability

§ Solutions for everyone, everywhere § Aligned with zeitgeist

§ Resonates with trends or problems du jour § Novel, but not radical

§ Question assumptions; but rediscovers/repackages older ideas § Lively, entertaining

§ Articulate, bold, memorable; buzzwords, lists, acronyms, etc. § Legitimacy via gurus and star examples

§ Supported by stories of “best companies” and the status of gurus

*Miller, Hartwick, LeBreton-Miller (2004)

Sources of Evidence

Scientific Literature (empirical studies)

Organization (internal data)

Practitioners (professional expertise)

Stakeholders (values and concerns)

EBM in Action: The CAT