Answer five questions in requirement . Each question requires more than 300 words， a total of 1,500 words

Evidence-based Management

MGMT 7250

TOPIC 7: Acquiring & appraising organisational evidence

Plan for today

• What is organizational evidence? • Acquiring it • Appraising it • Its limitations

What is organisational evidence?

What is organisational evidence?

• Data are numbers, words, figures, sounds, dates, images. • Information is data that tells us something about something or someone that can be used for some purpose— “data made meaningful” • Organisational evidence are data, facts, and figures that are transformed into meaningful information to address problems, practical issues and opportunities, and questions to identify problems and solutions. • They could be the result of an internal study conducted

Activity questions

• The Vice-Chancellor of a large University in Australia has presented you with some important data on gender composition among academics in science and engineering • What do you make of it? • Discuss in groups of 10

Activity

The importance of organisational evidence

• Evidence that is most relevant and specific to the organisational context • Richer evidence about this context

• Can be very useful for identifying patterns and testing claims and assumptions • Organisational data have a diagnostic, evaluative, predictive, and prescriptive purpose in decision making such that they can: ØHelp diagnose problems and their organisational consequences (diagnostic) ØHelp identify implementable and feasible solutions (prescriptive) ØAssess practice, processes, and decision outcomes for example whether implementing a solution has let to desired outcomes (evaluative)

ØPredict for growth opportunities and innovation (predictive)

Activity questions

• What further questions could be answered by organizational evidence? • What sort of organizational evidence would you need to answer them? • Are these exploratory, descriptive, or causal?

Critical thinking in the context of organizational evidence

Descriptive/exploratory Causal

What is the question (that is, what problem is being understood or solved?

Descriptive question (e.g. what is, how many, to what degree….)

Causal question (What causes the problem, what would solve the problem)

What are the alternative answers/claims to answers?

Alternative descriptions Alternative causal explanations

What support exists for each alternative claim?

Which of the descriptions have organizational evidence to back them?

Which of the alternative causal explanations have organisational evidence to support them?

How good is support for each claim?

Critical appraisal of organizational evidence

Critical appraisal of organizational evidence

Asking the right questions Example: Organisational commitment Organisational evidence to help identify the problem: • What actually is the organisational commitment level?

• Are there patterns or trends in commitment? • Do data show how organizational commitment is a problem?

• Do data show that low organizational commitment is causing problems?

• What are the key factors that cause low/high organisational commitment?

• How applicable are scientific findings to the organisational context?

• How relevant and applicable and trustworthy are our organizational data?

Organisational evidence to help identify the solution: • What attempts to enhance organizational commitment are currently in place and are they working?

• What else is happening that might be affecting organizational commitment?

• Are there relationships between organizational commitment and other data? Employee type? Shift? Department?

• How applicable are scientific findings to the organisational context?

• How relevant and applicable and trustworthy are our organizational data?

Types of organisational data • ‘Hard’ vs ‘soft’ • Financial & accounting • Human resources • Production • Product & service quality and quantity • Sales and marketing • Customer service

Data Science vs EBMgt

• Data Science has become a real buzzword in recent times • Data science combines statistics, data analytic methods, data mining techniques, and machine learning to make sense of data and facilitates data-driven decision-making. • The real danger of this approach is that it becomes all about the data and nothing else (neglect other sources of evidence). • Managers’ tendencies to find patterns in things (patternicity) and to selectively attend to information to confirm their beliefs (confirmation bias) can lead to misdiagnosis and decision neglect.

Acquiring organisational evidence

Where do data come from and how do you acquire it? • Data warehouse • Databases and information systems • Document and content management systems • Workflow systems • Physical records • Staff • Industry bodies

Basic quantitative data analysis

AI and machine learning

• Using computer programs to identify patterns and build models using available data • Useful, but could include biases—so should be used with care

Appraising organisational evidence

Methodological appropriateness

• Depends on the type of question • Descriptive questions: • Qualitative methods • Quantitative descriptive statistics

• Causal questions • 3 criteria for causation and research design that meets these criteria

Activity questions

• Let’s go back to the university gender equity issue • What if we were to reimagine it a little differently: The population of interest is the entire university, our sample is faculty in the STEM disciplines • Are there concerns over internal validity? What? • Are there concerns over external validity? What? • How might you over come these?

Methodological quality

• For both types of questions: • Measurement errors, reliability and validity • Sampling and sampling errors • Missing context • Misleading data visualisation

Measurement error and inaccuracy

• Measurement is the process of assigning numbers to objects in such a way that specific properties of objects are faithfully represented by the properties of numbers (Krantz, Luce, Suppes, & Tversky, 1971). • Poor measures lead to poor inferences which lead to poor decisions that may attract scrutiny. • We learned about classical test theory which posits systematic and unsystematic error can cause a score or a measure to deviate from the ‘true score’ or measure. X = T + e

• Depends on how data was collected, extracted, aggregated, summarised and interpreted

Measurement reliability and validity

• A measure is reliable when it is stable and different attempts at measuring something converge on the same result • Validity concerns whether you are accurately measuring the construct in focus

The faces of validity

Face

Content

Construct

Criterion- related

Validity of measurement

Validity of decisions

How do we validly measure:

Leadership? Performance? Innovation?

Has something been overlooked? Did we forget to measure

something?

Sampling errors

• Random sampling error • Statistical fluctuation due to chance variations • Large number of atypical subjects. • Outliers and extreme values.

• Inversely related to sample size, thus only way to manage is though large samples.

• Non-random or systematic sampling error • sampling frame error, pattern of responses are atypical of target population. • Self-selection bias. • non-response error. • Sampling error not due to chance eg. Biased selection

24

Small samples & Sampling error

• Smaller samples have more sampling error than larger samples that is largely random error. • To remedy increase sample size or aggregate data across branches and divisions, subsidiaries. • Calculate or ask for a confidence interval for the point estimate.

Missing context • Content invalid dashboards • As Muller (2018) suggests, job and organisational goals and performance are multidimensional, therefore focusing on only one aspect of these goals or performance is problematic. • Sometimes important contextual information is missing or omitted that can be very important in the interpretation of data. • A baseline measure can be missing making the interpretation of change in a data series very difficult.

Misleading data visualisations

Limitations of organisational evidence

The tyranny of Metrics: Metric fixation & dysfunction Quantitative fallacy: 1. Measure whatever can be easily measured 2. Disregard that which can’t be easily measured or to give it an arbitrary quantitative value 3. What can’t be measured easily really isn’t important 4. What can’t be easily measured really doesn’t exist.

The tyranny of Metrics: Metric fixation & dysfunction (2) According to Jeremy Muller (2018), metric fixation and dysc: 1. the belief that judgement can be replaced by measures or numerical indicators 2. the belief that the transparency of metrics signals accountability and demonstrates that an organisation is carrying out its purposes and ‘moral earnestness’ 3. the belief that what gets measured is important and gets incentivised and what doesn’t is not important.

Analysis paralysis

• As Mezias and Starbuck (2008) suggest, people sometimes try to eliminate uncertainty in data by gathering more data. • This can mean that data collection and maintenance becomes an end itself. • This is why asking and addressing questions is important in EBMgt.

Absence of a logic model

• To help avoid analysis paralysis, evidence-based practice recommends using logic models as decision aids to collect organisational data and helps focus on information and metrics relevant to the problem (Rousseau, 2012). • Logic models (also sometimes referred to as logical frameworks) can be used to understand how certain inputs result in outputs, and how these outputs create desired change outcomes, given certain assumptions and external conditions (see below).

Example of a logic model Evidence-based Project/Intervention/Program:

Goal & objectives:

INPUTS ACTIVITIES OUTCOMES

What we invest What we do Participants Why this project: short-term results

Why this project: intermediate results

Why this project: long-term results

Assess & evaluate performance and achievement of program objectives

Assumptions External Factors