Critique of two research articles

Michelle_Michy

Lecture9Interpretingandcritiquingqualitativeresearch.pdf

Home >Education homework help >Critique of two research articles

Interpreting and Critiquing Qualitative Research

Meriel Norris

PH2600

Research Methods

Seminar 1 Research ethics

Seminar 2 Research questions (FINER) - qual and quant Hypotheses - quant Variables - quant Study designs - discussed in relation to quant

Seminar 3 Validity, reliability, recruitment and levels of measurement

Methods

section of a

quantitative

paper

Applies to all papers with

human participants

Seminar 4 Central tendency & variance, SD Measures of dispersion Statistical methods and techniques Type I and type II errors One and two tailed hypotheses

Seminar 5 Understanding qualitative methods and methodology – qual

Seminar 6 Critiquing qualitative research and analysis – qual

Quant

Critiquing Qualitative Research

• In this lecture we will look at how to critically evaluate qualitative research, taking account of:

- Credibility, confirmability, transferability, dependability, trustworthiness

- Rigour - Sample size - Member checking/respondent validation - Negative case analysis - Triangulation - Saturation

Critique

• Systematic analysis and evaluation • Careful, considered (plusses and minuses) • Detailed • Concrete and evidenced • Summarise and justify your considered opinion • Consider what has been reported • Consider whether what has been reported and

what has been left out or glossed over is reasonable or whether it weakens the credibility of the paper

• Referenced

What is Qualitative Research?

• Qualitative research is based in experience

• It is about how people make sense of their world

• The meanings people attach to their experience

How can we judge if this has been achieved??????

Critical Appraisal Tool

http://www.casp-uk.net/#!casp-tools- checklists/c18f8

CASP qualitative checklist

Why use a critical appraisal tool?

http://www.casp-uk.net/#!casp-tools-checklists/c18f8

Why the CASP?

Systematic ComprehensiveGood for novices

BUT

• Deals very broadly with some of the

principles and assumptions

• Not a definitive guide

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://xplodemag.co.uk/reading-pick-book/&ei=6kWEVNmUHOyu7Aa8kYGABQ&bvm=bv.80642063,d.ZGU&psig=AFQjCNFD2libuc-5L5z3RrORi1XQ7bsQ1Q&ust=1418041141936538

Spotting a Good Qualitative Paper

• There may be a lot of BAD stuff out there

• Some of the MOST IMPORTANT qualities can be the MOST DIFFICULT to evaluate (Dixon-Woods et al, 2004)

• The subjectivity bothers people

• How do you judge the creative contribution?

Telling Good from Bad

‘The thorough assessment of qualitative research is

an interpretative act and requires informed

reflective thought rather than the simple

application of a scoring system.’

Kuper et al, (2008:687).

Undertaking a Critical Review

Start Asking Questions: Read Critically

Lorraine H. De Souza, Centre for Research in Rehabilitation, Brunel University, UK

Andrew O. Frank, Northwick Park Hospital and Institute for Medical Research,

Middlesex, UK

Disability and Rehabilitation 2007, 29(7), pp587-596

Title: does it tell you what you need to know?

• Topic

• Participants

• Method

• If you were doing a database search for papers in this field, would this paper come up if your search was based on title alone?

• If you were interested in reading about this topic would a quick scan of the title tell you whether this paper was worth investing in?

Authors

• Who are they?

• Think about what sort of assumptions, prejudices, preconceptions the authors might bring to the research

• Look (elsewhere) for information that tells you something about them (incl. experience doing qual)

• Ask yourself what you might expect to see in a section of the paper on reflexivity

Abstract • Purpose

• Participants

• Methods (incl analysis)

• Findings (results)

• Conclusions

Is there a good fit (purpose, sample, methods, analysis)?

Does it have utility – does it provide an effective and accurate summary of what was done and what was found?

Go back – check what reported in the main text, does it match?

Format set by journal

Ok but how analysed

What type of qualitative study?

Introduction

• What’s the point of the study, are you convinced?

• Is the rationale persuasive, compelling?

• Have the authors established that the study is worth doing, important, relevant, timely?

• Arguments based on relevant (recent) literature?

• Theoretical and empirical literature used to good effect?

• Strong narrative thread?

Are the Research Aims Clearly Stated?

• Does the study seek to illuminate and deepen understanding of a concrete ‘lived’ subjective experience?

• Is a qualitative approach appropriate?

• Is the research question clearly articulated?

Or…

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwjG6IC2xYLaAhWGvhQKHUYrCKoQjRx6BAgAEAU&url=http://persuasionpursuit.com/bandwagon-effect-marketing-use-sell-storm/&psig=AOvVaw1pvZmeT15K3wmfYC6qSiQm&ust=1521897844749835

Methodology

• Named, explained and justified

• If no named methodology why not, is this justified?

• Theoretical position clarified (e.g. realist / interpretivist), explained and justified?

• Would a different methodology have been more suitable?

Name it

Claim it

Frame it

Recruitment: How Were Participants Approached and Recruited?

• Is the recruitment strategy clear and appropriate ?

• Is it justified and named e.g. purposive?

• Is the sample size appropriate and justified?

• Did the participants have adequate experience/ knowledge to contribute to the research (and address this research question)?

• Who volunteered? Who refused?

• Might there have been better recruitment methods and settings?

• Look at the demographic data – what does it tell you?

Recruitment 2

For example –

How might a study of the experience of back pain be affected by recruiting from a pain clinic?

Becomes important in relation to

transferability

Data Collection

• What data were collected, and how? Justified?

• Is the method likely to elicit detailed accounts?

• Was the topic guide / SSI guide provided? – Were the questions appropriate?

– How much structuring did the interview have – consider +/- of tight and loose structuring

• How long were the interviews?

• If the authors claim saturation is this justified? (thought to apply to grounded theory)

• Was the influence of the setting/context considered? (e.g. how might interviewing in a hospital setting influence accounts? Or in a focus group compared with a one-to-one interview?)

Qualitative data analysis: 1

Phenomenological Analysis

Descriptive phenomenological research, the researchers may ‘bracket’ – to set aside/ heighten awareness of their assumptions;

Create a general or essential phenomenological description

Interpretative (hermeneutic) phenomenology – bracketing theoretically questioned, instead use assumptions as a springboard for interpretation. Would not claim to identify ‘essences’. Reflexivity expected.

Do these align with the methodology described & results?

Qualitative data analysis: 2

• Grounded theory - aim to build a theory from the data (usually a visually represented ‘box and arrow’ model, with theorised social ‘causes’ and conditions) constant comparative method

• Ethnography - more mixed – probably thematic based. May have specific perspective. Should discuss how triangulation of methods was managed

• Narrative analysis - look for story structure, turning points, rhetoric, characters, metaphors

• Content analysis - search for pre-selected words/ phrases – usually only in open questions in brief questionnaire answers (but NB there are different content analyses)

Do these align with the methodology described and data collection process & results? And the research question?

Judging the Analysis

• How well is the data analysis described? Audit trail • Do you get a firm/clear idea about the processes

involved, how decisions were taken e.g. about the development of themes? Transparency

• Who did the analysis? Analytical triangulation – appropriate?

• Was any ‘checking’ process appropriate? Peer debriefing/respondent validation – appropriate?

• Do you understand what was done? • If not, how might this impact on the

quality/trustworthiness of the findings?

Judging the Findings

• Is there adequate quotation and commentary to build a convincing case for the proposed themes?

• Appropriate evidence of interpretation (if relevant)?

• Is there adequate attention to differing experiences/ viewpoints – not only shared themes? negative case analysis/ balanced voices

• Small fragments of illustrative quotations in reports make it difficult for readers to check the trustworthiness of the analysis

• Has the researcher discussed credibility?

The concept of credibility is preferred to the quantitative concept of ‘internal validity’

– Systematic and transparent data analysis – Audit trail of decisions and changing perspectives in

relation to the data (not nec. available) – Data saturation – carrying out interviews and analysis

until no new information emerges (NB mostly a grounded theory concept) 

– Prolonged engagement  – Reflexivity – Structural coherence – Member checking/ respondent validation – Negative Case Analysis – Triangulation – Peer debriefing / review

Member Checking / Respondent Validation

• The researcher shows all or part of a study’s findings to participants to determine if they are in accord with their experiences

• May help the researchers to determine whether their interpretations make sense and are acceptable to participants

• Not always possible (e.g. when participants have limited cognitive ability)

• Not always appropriate – why?

Negative Case Analysis • Search for ‘deviant’ cases is recommended –

accounts which do not fit themes/ patterns identified in the analysis.

• This action helps to reassure reader that the quotations have not been ‘cherry-picked’ to suit the researcher’s preconceptions.

• Can indicate the limits of transferability of the themes

• Honours individual difference in experience

• Can help to build more complex theory – Example - 8 participants report that exercise makes them

feel happier and more energetic and 2 say that they feel inadequate/ unhappy when exercising

– Should the focus be on the views of 6/8? Why? Why not?

Triangulation • Are the accounts of participants/ interpretations of

researchers corroborated in any way?

• Methodological triangulation vs analytical triangulation

• Avoid viewing triangulation as a ‘check’ on ‘facts’ – NOT a check on ‘truth’.

• Also, differing interpretations by different researchers may reveal something about the influence of professional concepts and expectations – not simply ‘bias’ or ‘lack of reliability’

“The findings were checked by someone else”

What do the researchers mean?

Be careful with this term, not necessarily a good thing:

A researcher brought in to verify ‘themes’ may rely far more on PERSONAL PRECONCEPTIONS and GUESSWORK than the main fieldworker

But another person could be used to help the researcher engage in ‘critical dialogue'

Peer Debriefing and Review

Confirmability

Discussion and Conclusions

• Have the findings been placed in context of what was already known?

• Are the findings discussed in relation to the research questions?

• Does the researcher discuss the potential contribution of the study? Are the claims reasonable?

Reflexivity 1

• Recognising the influence of the researcher in the research process (dependability – being accountable)

• How experience, gender, profession, social status, ethnicity & culture influence the choices made in the study (from RQ to data collection, interpretation and discussion)

Reflexivity 2 • Participants’ accounts are not transparent windows on

feelings/experiences – but products of the social situation

• Have the researchers reflected on their role and relationship with participants, their pre-conceptions, how they handled unexpected events?

• Papers should not be ‘trashed’ just because a researcher has declared a particular cultural perspective (feminist stance) or personal involvement with the research participants

• Subjectivity (bias) should be accounted for but can’t be eliminated

• If it is not accounted for you should state why/how a lack of reflexivity may have impacted on the findings

Research Ethics: Operates Throughout

Procedural Ethics

• Usually state REC approval and informed consent

• Usually note anonymity – use of pseudonyms and blurring of other identifying details

Situational Ethics - may discuss management of distress

Relational Ethics

• Researcher has the power to shape what is understood but also to distort - your judgement

Exiting Ethics

• Publication can unintentionally disrespect the participants who choose to take part in the understanding that their lives will be better understood your judgement

Transferability (generalisability/ external validity)

• ‘Generalisability’ is valued in quantitative research but inappropriate for most qualitative studies

• ‘Transferability’ is the preferred qualitative concept – can findings be transferred to other contexts or settings? Primarily judged by the person doing the transfering! Enhanced by detailed description of findings and setting/context

• Notion of ‘vertical generalisability’ – via theoretical understandings – how much does the study help to develop theory/ new insights?

Final Comments

• Issues around trusting the researchers to have analysed all their data and to have presented the findings without bias or selection also occur in quantitative research.

• Not everything can be reported in an article (typically 4000-7000 words long). So focus your critique on what was/was not reported rather than what was/ was not done by the researchers.

• In your essay pick some of the major strengths and weaknesses.

• Support with references