SINGLE SYSTEM RESEARCH DESIGN
Kishon
RoyseCh6.pptxChapter 6:
Single System Research Designs
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
What are Single System Research Designs (SSRD)?
Used for evaluation for almost 50 years.
May be used to evaluate individuals, group work, organizational and community practice.
Both quantitative and qualitative aspects.
Quantitative: Involves use of outcome measures with documented reliability and validity.
Qualitative: Simple line graphs; inferences using visual inference; no inferential statistics; and gathering data in the real contexts of client lives.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Prerequisites for Conducting an SSRD
Select a practical and valid outcome measure that can be repeatedly assessed over time.
Assess this outcome measure over time.
Display results on graph with time on horizontal axis and outcome on vertical axis.
Make reasonable inferences. Ask:
Are data showing client improvement?
Are data demonstrating client deterioration?
Do the data depict no change?
Are the data unclear?
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Selecting Outcome Measures
Must possess:
Reliability: Consistency, easy to gather, with good inter-rater agreement.
Validity.
Sensitivity to change.
Identify agency’s mission to clarify what outcome measures should be selected.
Use the best measures available with the most empirical support subject to cost constraints.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Assessing Measures Over Time
Four variations:
Repeated assessments of some measure without any intervention.
Repeated assessments of some measure at the same time intervention begins.
Repeated assessments of some measure before intervention begins (a baseline).
Repeated assessments of some measure before and after intervention. May result in a more internally valid form of SSRD.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Notation and General Principles
A = Baseline phase; data gathered in the absence of an intervention.
B = Design phase; data gathered when a particular intervention is implemented.
C = An additional phase, indicating a different intervention than “B”.
A-B = Baseline followed by intervention.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Notation and General Principles
A-B-A = Baseline – Intervention – Removal of Intervention – Second baseline.
A-B-A-B = Alternating Baseline-Intervention-Baseline-Intervention.
B-A-B = Intervention-Removal-Reinstatement.
A-B-A-C = Baseline-Intervention-Baseline-Different Intervention.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Needs Assessments
Can gather data over time that will corroborate (or refute) the need for the program.
Document the number of burglaries in a given neighborhood over a two year period to determine need for increased police presence.
Display data on a graph for a powerful visual portrayal.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Formative Evaluations
SSRDs can be use for formative evaluations to adjust and enhance programs or interventions.
Examples:
Monitoring a point system of rewards for members of a group home to determine if it should be dropped, continued, or improved.
Keeping records of frequency of antisocial behavior in a runaway shelter, prior to and during an intervention.
Tracking the effect of training on the number of prompts used by certified nursing assistants (CNAs) in dressing nursing home patients with dementia.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Quality Assurance Studies
SSRD data useful for:
Continued-stay utilization reviews.
Systematic audits of clinical records to justify length of hospital stays.
Outpatient treatments.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Summative Evaluation Designs
Empirically-based appraisal of outcomes of an innovative program.
More robust type of SSRD needed.
B Design: No baseline. Data gathered when new program implemented.
Casual inference not usually possible, but can answer question: “Did things get better when program was implemented?”
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Summative Evaluation Designs
A-B Design: Data collected at baseline and when new program implemented.
Helps eliminate threat of existing pre-treatment trend.
Helps exclude rival hypothesis that the evaluation impacted the outcome.
Still need to be cautious about causal interpretations.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Summative Experimental Designs
Permits reasonable degree of causal inference.
Allows relative confidence that program was responsible for any observed improvements.
Threats to internal validity removed by demonstrating relationship between introduction or removal of intervention and change in outcome measure.
More rigorous than simple A-B designs.
Causal inference requires “persistent concomitant variation” between application of intervention and changes in targeted behavior.
Changes observed after, not before intervention.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Summative Experimental Designs
A-B-A is the simplest form.
Baseline, followed by intervention which is removed, with data collected before and after the removal.
If consecutive meaningful changes in outcome measures can be produced, likely that program is responsible for the change.
Multiple Baseline Design uses staggered baselines for different groups receiving an intervention.
May permit causal inference. A1B1A2B2 designs.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
External Validity
SSRD ability to generalize beyond particular participants (external validity) is limited.
Research in human services and program evaluation hardly ever uses random sampling from larger population due to practical limitations.
Problem of external validity can be overcome with replication.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Inferential Statistics
Not recommended because:
May deter people unfamiliar with statistics from using SSRDs.
Parametric statistical tests based on assumptions that SSRD data do not meet.
SSRDs contradicts assumption of independence of data.
Relying on visual data alone increases Type 2 errors (missing some valid but small effects) and reduces Type 1 errors (making exaggerated claims about effects).
Time Series Analysis may be used in evaluations with large numbers of data points.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
How to Prepare Graphs
Constructed using statistical packages or Microsoft Excel.
Use thicker lines to form the vertical and horizontal axes than the lines used to connect data points.
Use black ink only.
Use actual data points (solid or open black circles).
Separate phases using a dashed vertical line.
Label each phase with an intelligible title.
Use abbreviations sparingly.
If vertical axis has zero point, elevate it above horizontal axis so data points do not rest on it.
Make graph big enough to be easily read.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
How to Prepare Graphs
Things not to do:
Do not use colored ink or shades of gray.
Do not format using computer’s 3D features.
If submitting to a journal, do not place the figure caption on the figure itself. Use a separate figure caption page
Do not have top or right border on graph.
Do not use horizontal lines running from the vertical axis across graph
Do not connect data points between phases.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
Ethics of SSRD
Client confidentiality must be taken into account as well as the right to informed consent to participate.
Evaluation of one’s practice is a gray area.
NASW Code of ethics states informed consent should be obtained when appropriate.
Legal definition of social work in some states includes conducting evaluations, so using SSRDs could simply be considered good practice.
May not fall under definition of “research” if no intent to publish or disseminate findings.
© 2016 Cengage Learning. All rights reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.
