article critique assignment

1

Quantitative Article Review Guide

General Requirements

 Have you given a complete reference for the article (based the 6th ed. of the APA Publication Manual)?

 Have you cited any other sources that you used and added a reference list at the end of the critique for these?

 Have you checked and/or had someone else check for grammar, sentence construction, spelling and clarity?

Article Critique Part I  Describe the overall purpose of the research in your own words. [Purpose: What were they

seeking to do/understand/explore/describe]. The purpose may be stated directly or you may need to infer from the research questions or the hypotheses.

Introduction  Does the author address the importance of the problem (e.g., by providing statistics or

references on widespread/prevalent the problem is and/or that it is increasing; negative impacts or costs of the problem on the persons experiencing the problem and/or other people; showing that it is a concern of or is being addressed by professionals, the government or other important organizations)

 Does the introduction utilize theory to inform the research question and/or hypotheses? If yes, which theory/theories are used [ theories may also be called models/conceptual model/conceptual framework].

Research Question/Hypothesis/Intervention  List the specific research questions being explored.  If applicable, list any specific hypotheses tested.  If applicable, describe any intervention being investigated by the study.

Methodology  Describe the type of general design (pre-experimental, quasi-experimental and

experimental) and the specific sub-type (cross-sectional, case-control)  Eligibility [also may be referred to as inclusion criteria and exclusion criteria].  What were the major demographic characteristics of participants (e.g., age, gender,

SES, race/ethnicity, other variables relevant to the study)? Provide percentages or numbers if given.

 What procedures were used to select participants? o Describe the sampling approach. Was it random sampling? If so, what type (e.g.,

simple, systematic, stratified, cluster, etc.)? Was it non-random? If so, what type (e.g., purposive, quota, snowball, deviant, critical incident, etc.)?

o How were persons recruited for the study? o If participants were recruited from a larger study, state that and briefly mention

that larger study.  If applicable, describe how participants were assigned to different groups within the

study? (may have been assigned by the researcher; may have already been in existing groups, etc).

 What was the actual sample size?

2

o Sample size is the number of people who actually participated or from whom data were collected.

 Were Institutional Review Board (IRB) agreements obtained? [may also be referred as approval from a research ethics committee or research ethics board, human subjects committee]

 Were any incentives used? [If yes, state the amount. If there were multiple rounds of data collection, state how much for each if mentioned (e.g., $10 for each interview)]

 What were the variables under study, and how were they defined? Identify any dependent and independent variables or predictor and outcome variables.

o These are the variables that are in the study’s RQ or hypotheses or research purpose

o If your study is not looking at how one thing causes/predicts/contributes to another then you just have a variable or variables, you won’t have independent/predictor or dependent/outcome variables

 List the measures for each variable used in the study, and their respective validity and reliability (if applicable).

o Ensure you state which measure was used to measure which variable. o Provide reliability statistics and validity statistics for each (these are typically

decimals or percentages) or state that they were not provided/mentioned. You may see terms such as "validation" or "psychometric(s)".

o If an established instrument was used and was not adapted in any way for this study and that instrument was validated on the same type of population as the one in this study, then reliability and validity will have been determined already and the study’s author should report the validity and reliability and/or cite the study/studies that determined these statistics.

o If instrument is new or has been modified or is being used on different population other than in studies where reliability and validity was determined, then the reliability and validity from these previous studied will not apply, so state how the researcher indicated how they established the reliability and validity of the new or modified instrument and report statistics. If the researcher did not describe this, it is a limitation.

o If scales, indexes etc. are used the researchers should report internal consistency reliability (usually reported as Cronbach’s alpha, but sometimes split half reliability).

 How were the data collected? [Talk about the method used, e.g., direct observations, self-report questionnaires/surveys, secondary data, existing records, etc.]

o For surveys/questionnaires: state type (e.g., telephone, online) o Include if one or multiple persons collected the data and any attempts to ensure

consistency in how each collected data (e.g, supervision, training) o "Secondary data" means that the data for the study you are critiquing came from

a previous study or survey. If this is the case, mention the previous study and how data was collected for that study.

o If behaviors were directly observed, state how this was done, including length of time and frequency, and number of observers if reported.

o If these things are not mentioned be sure to state that.

 What data analysis technique(s) were used in this study?

3

o You may see "descriptive statistics" (you might see: "we conducted descriptive statistical analyses")

o Common tests of statistical significance include Pearson’s r for correlation, t- tests, Analysis of variance (ANOVA), ANCOVA, Multiple analysis of variance (MANOVA),regression, chi-square; but there are others. State the test used. State what it was used for (e.g., compare scores for two groups) to the best of your ability.

 What statistical and/or data analysis software package(s) was/were used for data analysis?

o Common ones used for quantitative data analysis include: SPSS/PASW, STATA, SAS, R, AMOS, LISREL, MPLUS, HLM, MATLAB; you may also see EXCEL but there are others

 Describe the overall findings of the study in your own words (discuss it as if you were talking to someone who had no idea what you read). Aim for 5-7 sentences maximum.

o Give a summary of the findings for each research question/ hypothesis based on the data collected. This info should be found in the Results/Findings section.

o Describe overall outcomes, (e.g., percentages, mean/median/mode). o State any group differences, relationships between variables, or effects and if

they were found to be statistically significant. o Do not give what you or the authors think should happen based on the results or

limitations or recommendations.

Article Critique PART II

Note: Most, but not all, of these may be mentioned in the Discussion section of an article as "strengths and limitations".

Question 1: Describe the strengths and limitations of the following with respect to a study’s ability to answer its research questions. [NOTE: The items listed under each category below are intended to guide your thinking; I have not listed every single thing you should think about or consider. There may be additional issues related to your specific study.]

Instrumentation/Measurement strategy (including measurement reliability and validity).  Did the study use measures whose reliability and validity was already established for

use with the population in your study, and are their validity and reliability noted to be at least acceptable as indicated by actual statistics or do they provide citations for this? If yes, this is a strength.

 If the researchers modified established measures, or used it on a population different from the one for which was validated, then authors need to establish validity and reliability and report statistics/citation. If this was done and they were good/acceptable, this is strength. If they don’t provide evidence/citation, then this is a limitation.

 If authors of the study developed their own measure, do they report acceptable reliability and validity and say how this was established? If yes, this is strength. If not, then a limitation.

4

 Conducting a pilot study before the main study when new measures are being used would be a strength as it provides an opportunity to check for and correct potential measurement error.

 Using multiple ways/measures of each outcome is a strength, especially if only self- report measures are used.

Sample (discuss issues of external validity-generalizability)  Was random sampling used? If yes, it makes sample unbiased and therefore more likely

to be representative. This is a strength. If random sampling, was it stratified? If yes, this further enhances representativeness.

 Say if the study’s context, participants and setting were described in sufficient detail so that others can determine how well findings generalize to the intended population or to another population that the reader may be interested in. If it is fair, this is a strength. If the description of context, settings, and participants is very scant, point this out as a limitation.

 Given the population identified in the research question and the characteristics of those in the sample, point out who findings might generalize to based on people/cases in the sample. This would be a strength.

 Based on the same information, point out who in the population it might not generalize to because cases/persons in the sample don’t have certain characteristics or there are very few of them in the sample. This would be a limitation.

 If non-random/non-probability sampling used, this is a limitation as the sample is more likely to be biased in some way and therefore less representative of the target population

o If a non-random sample is used, is it drawn from the target population? If yes, more likely to be representative, so this is a strength.

o If non-random, is it drawn from diverse sources? If yes, more likely to be representative, so this is a strength.

o If non-random, was quota sampling used? If yes more likely to be representative: strength

 Sample size: Generally larger samples are more representative: strength.  Did authors say if size was large enough based on doing a statistical power analysis? If

yes, then this a strength; if no, then a limitation.  Non-response rate/response rate: If the response rate was low, is a limitation for

generalizability. (Usually under 50% considered low but this a rough rule of thumb; see Rubin & Babbie).

Research design (discuss issues of internal validity): To what extent does the design allow you to conclude that the independent variable is a cause of the dependent variable.

 Describe the type of general design (pre-experimental, quasi-experimental and experimental) and the specific sub-type (cross-sectional, case-control)

 See chapters 11 and 12 for strengths and limitations for causality and controlling for prominent threats to internal validity.

 Strengths may include the use of multivariate statistics to control for influence of other variables (may be called covariates).

 Use of matching, switching replications, and multiple pretests are strengths  Look at other threats:

o Measurement bias- did they use blind raters/single blind/double procedures? If yes this is a strength (see chapter 9).

5

o Attrition, diffusion/imitation of treatment, and compensatory equalization, compensatory rivalry or resentful demoralization, infidelity. List any strategies to minimize these as strengths. Minimal attrition is a strength.

Question 2

 Describe the new knowledge you gained from this article related to social work practice (What might you do differently and/or think about in your own social work practice having read this article?)

o State based on the information from this article how you might think about or practice differently as a social worker. In other words, how could you practically apply the findings or change the way you think about this problem area/population?

 How might you see this article being helpful/informative to social work practice overall?