Answer five questions in requirement . Each question requires more than 300 words， a total of 1,500 words

Evidence-based Management

MGMT 7250

TOPIC 8: Acquiring & appraising stakeholder evidence

Agenda for today • What is stakeholder evidence? • Acquiring stakeholder evidence • Appraising stakeholder evidence

What is stakeholder evidence?

Who are stakeholders? Those parties and individuals inside and potentially outside the organisation who will be affected by practice and decisions. These stakeholders may include: ØExecutive committees ØBoards ØSenior managers ØPolicy makers ØEmployees ØCustomers ØSuppliers ØRegulators ØCommunity ØShareholders ØUnions

Why are stakeholders important in EBP?

The subjective experiences, perceptions, emotions, and concerns (voice) of stakeholders in relation to a problem, decision, and solution (practice or intervention) affect the: • Implementation of evidence-based decisions and practice • Receptivity to change needed to support evidence-based decision-

making and practice • Decision and program assessment/evaluation to inform any needed

adjustments

Acquiring stakeholder evidence & ethical considerations

Example questions to acquire evidence from stakeholders Problem identification: • How do employees feel about and view the organizational

commitment ‘problem’? • Do they think low organizational commitment has negative

consequences? • What do managers think about the problem? • Do customers or clients or service users have a view? • How relevant and applicable and trustworthy are stakeholder

concerns evidence?

Stakeholder mapping

• Stakeholder mapping can be used to determine who the most relevant stakeholders are to a decision problem. • It is important to ascertain the

likely impact of the decision on stakeholders in terms of harms and benefits in the short, medium, and long terms, and the extent to which stakeholders can potentially impact or influence the decision.

SecondaryPrimary

Activity

Reflecting back on the issue identified on the organizational data on gender representation across academic levels. (Gender imbalance at higher levels of faculty in Engineering and Science.)

Questions: • Which stakeholders would you acquire evidence from? How can

stakeholder mapping help you with this?

Activity

Similar to acquiring qualitative evidence from professional experts: See slides for Topic 3

With a few differences: Involves asking a different set of questions:

You are more interested in understanding the actual and perceived experiences of different stakeholders and their expectations You are not that interested in the expertise of the stakeholders

Acquiring evidence from stakeholders

12Lesson 1 (TLO 7.1 )

• Natural setting: Often collect data at the site where participants experience the issue or problem under study

• Multiple sources of data: Such as interviews, observations, and documents • Inductive data analysis: “Qualitative researchers build their patterns, categories, and themes

from the bottom up, by organizing the data into increasingly more abstract units of

information.”

• Participants’ meanings: Researchers are interested in understand the meanings held by the stakeholders themselves, not the meaning held by the researchers or authors who have

written about the subject (Emic validity)

• Emergent design: The research design tends to change as the research goes along. It emerges as the study comes along.

• Interpretive: Researchers make their interpretations of what they see—this is affected by their own background and could be biased (which is why it is important to have others

interpret it too)

Some important characteristics of qualitative research as it applies to

stakeholders (Creswell, 2009: Chapter 9)

13Lesson 1 (TLO 7.1 )

• Observations • Interviews • Documents • Audio-visual material

Source: See Tables 9.2 and 9.3 in Creswell 2009 (on Wattle)

Qualitative data collection processes

14Lesson 1 (TLO 7.1 )

Methods to acquire stakeholder evidence Qualitative • Participant and non-participant

observation • Focus groups • Interviews

Quantitative • Questionnaires • Documents

Source: Figure 9.1 (Creswell, 2009)

Analysing and interpreting the data

16Lesson 1 (TLO 7.2 )

Ethical considerations

• Evidence-based decisions and practice can have material (money, time, human capital) and non-material (learning, stress, change) costs and benefits on different stakeholder groups to varying degrees. • Whether or not qualitative or quantitative research methods are

used, it is important to inform stakeholders about the purpose of your research. • However, it is important not to inform stakeholders too much as this

may evoke demand characteristics. ØDemand characteristics are where stakeholders’ responses are influenced by

their perception of what it is the research is expecting to find. • Confidentiality

Activity

Reflecting back on the issue identified on the organizational data on gender representation across academic levels. Questions: • Which research methods would you use and why? How would you

take bias into account? • What are some of the questions you would ask?

Appraising stakeholder evidence

The importance of critical appraisal

• Predominantly acquired to address non-effect questions – methodological appropriateness is not as important especially when using interviews and focus groups. • Because the interest in acquiring evidence from stakeholders is to

primarily gauge their perceptions and subjective experiences of organisational phenomena, the appraisal of systematic error may not be a central concern. • Still important to anticipate potential bias and limitations of this

source of evidence.

Trustworthiness of evidence refers to the extent to which evidence can be relied on (reliability). The following is an overview of considerations needed to establish trustworthiness of evidence gathered from qualitative (idiosyncratic) research:

Credibility: examining data, data analysis and conclusions to see whether or not a study is correct and accurate. Transferability: applying research results to other contexts and settings. Dependability: Assessing whether findings are consistent and free of cognitive and social bias. Confirmability: includes an audit trail that shows raw data (written field notes, documents and records) so another researcher can verify a study when presented with the same data.

Trustworthiness (reliability) of qualitative evidence

21Lesson 2 (TLO 7.3 )

When appraising evidence, the full scope of values and concerns of individuals involved in a study should be captured. Psychological and social considerations of participants should be

taken in order to make meaning from the studies. Appraising evidence for emic validity means 'getting inside the heads' of people who have been involved in study to see their perspectives and meanings. Triangulating evidence is a way we can ensure we are appraising evidence for emic validity.

Reminder: Emic Validity

22Lesson 2 (TLO 7.3 )

Is the sampling strategy appropriate for the research question?

Does the form of data collection have a proper rationale?

How are data recorded?

How are data organised, coded and analysed?

How reliable are these methods? Have multiple coders been used?

Are the findings being validated?

Methodological appropriateness in qualitative studies (1)

23Lesson 2 (TLO 7.3 )

What to observe: Comparing like-to-like

Since random selection may not be possible, we could choose units so as to ensure we are comparing like-for-like

Keep the key variables in mind As large a sample of units as possible (so you have more observations than explanatory variables or DVs) Avoid strong correlation between two or more explanatory variables (IVs) across (the small number of) units

Methodological appropriateness in qualitative studies (2)

24Lesson 2 (TLO 7.3 )

Avoid selection bias Avoid selection of units purely based on specific values of the dependent variable (there should be some variation in the dependent variable)

E.g. avoid studying only successes Avoid selection of units purely based on size of impact

Methodological appropriateness in qualitative studies (3)

25Lesson 2 (TLO 7.3 )

Demand characteristics

• Demand characteristics contribute to measurement error • Occurs when participants interpret the aims of the research and may

acquiesce to such aims to help the researcher find what they want to find, or may response in a way that disconfirms or falsifies what they think the researcher is intending to find. • Self-report bias • Minimised through source triangulation

Social desirability

• It can be the case in questionnaires and interviews that stakeholders respond in ways they think will cast themselves as someone who is socially responsible. • For example, in an interview stakeholders

may be asked about their views about gender diversity or ethnic diversity in the organisation and may have a proclivity to respond in ways that encourage diversity even though they actually believe to the contrary.

Systematic error in questionnaires

• Administrative errors • Measurement error –

unreliable scales • Self-selection bias • Self-report bias

https://www.youtube.com/watch?v=IG5 meEV1rUE

Triangulation is a procedure used to ensure credibility in evidence from qualitative data. Four kinds of triangulation can be used to verify and validate qualitative data, but can also be applied to quantitative data. Methods triangulation: check consistency of findings generated by different data collection methods (Mixing methods). Triangulation of sources: check consistency of different data sources within the same method. Note that Evidence-Based Management more implicitly relies on triangulation of sources and methods by recommending managers rely on more than one source of evidence in decision- making.

Analyst triangulation: using multiple analysts to review findings. In quantitative research this is akin to getting multiple raters to evaluate performance to maintain inter-rate reliability. Theory/perspective triangulation: use multiple perspectives or theories to interpret data (in research only and not in practice).

Triangulation

29Lesson 3 (TLO 7.4 )

The adoption of multi-methods is viewed as an effective way to validate the findings of qualitative research. While method triangulation can involve multiple qualitative methods, the term mixed- methods is used to specifically describe studies that combine both qualitative and quantitative research methods.

Mixed Methods triangulation

30Lesson 3 (TLO 7.4 )

Lesson 3 (TLO 7.4 )

Activity

Reflecting back on the issue identified on the organizational data on gender representation across academic levels. Questions: • How would you combine different sources to get at these questions?