Critique of two research articles

Michelle_Michy

Lecture2Experimentaldesigns.pdf

Home >Education homework help >Critique of two research articles

Introduction to experimental designs

PH2600 2019 Neil O’Connell

Learning outcomes By the end of the lecture students should be able to:

 Describe basic common experimental study designs

 Consider some of the biases that attempt we control for

 Describe the basic purpose and structure of a systematic review

A bottom line

The choice of design should arise from the research

question - not the other way around.

Experimental design - definition

In which one (or more) variable(s) is manipulated and the effect of this manipulation is observed in other variables.

It aims to control all other variables.

It allows us to infer causality

Causality

If there is change to A does a change in B result?

◦ Cause must precede the effect ◦ The cause and effect must co-vary ◦ If the cause does not occur then neither

does the effect

Inferring causation - problems

 Confounding  Regression to the mean  Natural recovery  Placebo/ non-specific effects  Hawthorne Effect (Observer)  Rosenthal Effect (Experimenter

expectancy)

Time itself is a confounder

Se ve

ri ty

Time

Se ve

ri ty

Time

Control group

 By including a group who undergo the same conditions (except…) as the experimental group we control for numerous possible confounders

 For within-subjects designs this might be a control condition

Blinding Why conceal the identity of the experimental condition?

◦ A function of placebo groups - ‘sham’ interventions

◦ Single blind

◦ Double-blind

◦ Triple Blind

◦ What confounders might blinding control for?

Who can we blind in trials of PT?

Group designs – within or between subjects

Within Group design

One group of participants receives

all experimental conditions (including

control)

Offers paired data

Between-Group Design

Different groups receive the different

experimental conditions

Offers unpaired data

Designs

 Randomised controlled experiment. Parallel, cross-over, factorial

 Controlled experiment

 Quasi experimental study

Single group pre-test post-test design

Group before Same group

after

IN T ERV

EN T IO

Time series design

IN T ER

V EN

T IO

measure measure

Basic parallel experimental design (pre test-post test)

Experimental group

INTERVENTION

CONTROL Follow up

Follow up

Control group

Pre test

Post test

SAME POPULATION

TAKE BASELINE MEASURES

INTERVENTION

CONTROL FOLLOW UP

FOLLOW UP

Pre test

Post test

How to ensure the groups are the same?

Matching groups

Use the same group for the different conditions

Randomisation

RANDOMISATION

The beauty of randomisation

 It solves all your problems (maybe)!

 In NRS you can only control for known confounders

 Successful randomisation controls for all

 Even imbalances at baseline occur at random and are unsystematic biases.

RA Fisher (1935)

“Randomisation relieves the experimenter from the anxiety of considering and estimating the magnitude of the innumerable causes by which his data may be disturbed”

Concealed allocation (because all people…)

 Think of this like protection for your randomisation

 The person admitting patients to the trial must not know the allocation schedule

 Because they might cheat (remember Bill Silverman’s nurse)

 Ask “could the allocation have been fixed?”

SAME POPULATION

TAKE BASELINE MEASURES

INTERVENTION

CONTROL FOLLOW UP

FOLLOW UP

Pre test

Post test

R A N

D O

M ISAT

IO N

Drop Out and Protocol Violation

 Participants often do not complete the treatment or withdraw from trials

 Often this is not at random, so the benefits of randomisation are lost

 Clear reporting and appropriate management of violation and dropout is vital

For example…

 In a trial of Constraint induced therapy versus usual rehab the CIT group experience high drop-out because:

◦ Unable to tolerate constraint ◦ Don’t feel they are making progress

How might that skew results?

Questions to ask

How many people dropped out and

when?

Why did they drop out?

Was there a big difference in drop-

out between groups?

How were any missing data managed?

Intention to treat versus per protocol

 Per protocol: only participants who complete the trial/ treatment to which they were randomised are analysed

 Intent to treat: (ITT) All participants who were randomised are entered into the analysis regardless of violation or withdrawal

IN T

E R V

E N

T IO

N P

E R

IO D

Repeated measures design

baseline Condition

1 measure Condition

2 measure

Order and carry over effects

 Time remains a confounder

 Treatments might have lasting effects (positive or negative)

 So each treatment might start from a different baseline

 How can we fix that?

Randomised Cross-Over Design

Hulley et al. Designing Clinical Research. 2ndEdition. Lippincott Williams & Wilkins, 2001

Factorial Design

 E.G.

 Does spinal manipulation work better than CBT or is giving both the best?

 More than one IV that may interact?

 For example 2 treatments that might be delivered together

SMT NO SMT

CBT Group 1 Group 2

NO CBT Group 3 Group 4

Quasi experimental?  An odd term – usually used to denote

that allocation was not randomised.

 Can mean a design that lacks one of these:

◦ Randomisation ◦ Control group ◦ Pre-test - post test design

Non-specific

Natural history Regression to the mean Placebo / interaction

effects Hawthorne effect

Resentful demoralisation/ “frustrebo”/ nocebo

Specific

The effect of the proposed “active

ingredient” or mechanism of the

therapy

Specific and non-specific effects

ACTIVE INTERVENTION

SHAM INTERVENTION

NO INTERVENTION

The active ingredient + non

specific interaction effects

Non-specific effects

The active ingredient

Experimenting on the individual: Single subject designs, n of 1 studies

 A-B design

 A-B-A design

 A-B-A-B design

 A-B1-A-B2-A……

 Test intervention and its withdrawal

 Can allow randomisation

 Clinically achievable

RCTs are great but at the top of the evidence based tree....

RCT

COHORT STUDIES

CASE CONTROL STUDIES

CASE STUDIES, ANECDOTE, OPINION, LAB STUDIES

Systematic reviews of RCTs

Systematic Reviews

 Each individual trial represents an estimate of the true effect of a treatment

 Seek to pool all of the available evidence for the most accurate estimate

 Take a very deliberate systematic approach

 Allows pooling of results (meta-analysis)

What it isn’t

Formulation of a specific question to be addressed – (PICO structure)

Retrieval of ALL relevant literature; systematic seach, selection of studies based upon pre-defined criteria

Tabulation of study characteristics/ assessment of study quality/ risk of bias

Synthesis/numerical aggregation of study findings

Hypothesis testing? Presentation of effect size? Narrative synthesis?

Systematic Review

Meta-Analysis - What’s the point?

 Meta-analysis measures size & consistency of treatment effect across >1 study

 By pooling data from studies we can increase the accuracy and power of our estimates

 We can estimate both the treatment effect and the consistency between studies

Meta-Analysis - What’s the point?

 Less chance of rejecting an intervention that actually works

 Less chance of being fooled by falsely positive data…although….

 Quality of original studies is vital

 Rubbish in = Rubbish out!

Meta-analysis how does it work?

 Estimates the effect size for each study

 Weights each study (usually on size & precision)

 Pools each study to generate one overall effect size

 Done properly this should be more precise than that from individual studies

Don’t get lost in the forest (plot)

The line of no effect

Confidence intervals

Point estimate

Pooled estimate of effect

WHAT EVIDENCE IS OUT THERE?

IS IT ANY GOOD, SHOULD WE TRUST IT?

DOES THE TREATMENT CONSISTENTLY SEEM TO

“WORK”

DOES IT CAUSE HARM?

HOW IMPORTANT/ LARGE ARE THE

EFFECTS?

SO A GOOD SYSTEMATIC REVIEW MIGHT TELL YOU: