Using Evidence-Based Practices

ResearchinCounseling-A10-yearreviewtoinformpractice.pdf

Home >Psychology homework help > Using Evidence-Based Practices

Received 10/20/09 Revised 04/27/10

Accepted 08/27/10

Research in Counseling: A 10-Year Review to Inform Practice Dee C. Ray, Darrell M. Hull, Andi J. Thacker, Laura S. Pace, Karrie L. Swan, Sarah E. Carlson, and Jeffrey M. Sullivan

H r r h i s study reviewed 4,457 articles from 1998 to 2007 in American Counseling Association division-affiliated journals to identify research articles published in counseling; 1,139 articles (25.6%) were quantitatively research based. The authors provide details related to quantitative research publicafions, including individual journal contribution fo the research base, focus areas for research, and independent and dependent variables highlighted by researchers. One summary finding was that only 6% of counseling research articles explored effectiveness of counseling interventions.

Research publication in the field of counseling is a current subject of discussion, as demonstrated by recent attention regarding research practices by the American Counseling Association's (ACA) flagship journal. Journal of Counseling & Development (JCD, see Bangert & Baumberger, 2005; Marotta & Watts, 2007; Nilsson, Love, Taylor, & Slusher, 2007). The focus on research seems to be motivated by both internal and external sources requiring data-driven methods for choosing and evaluating counseling interventions. Researchers have noted the benefits of practice based on empirical evidence. Bangert and Baumberger (2005) reported that government fimding has increasingly required the use of interventions based on randomized experimental designs. Mate and Kelly (1997) emphasized the need for the establishment of an empirical base for the professional practice of mental health counseling. They fiarther highlighted the need for original research to promote the mental health counseling field as a developed professioa

Since the early 20th century, practitioners and researchers have been aware of the benefits of incorporating research- based facts to direct clinical services (Rubin, 2008). Rubin (2008) suggested that the practice of incorporating best practices includes a process by which one evaluates reliable evidence and appoints certain interventions as empirically sup- ported. Publications based on empirically supported evidence serve as a guide to practitioners seeking to uphold ethical obligations to provide counseling services based on rigorous research methodologies.

Perhaps, most important, counselors are guided by ethi- cal obligations to seek evidence-based practices. According to the ACA Code of Ethics (ACA, 2005), "counselors have a responsibility to the public to engage in counseling practices that are based on rigorous research methodologies" (p. 9). Therefore, if counselors are to behave ethically, they must be aware of practices that are empirically supported and the general research that is occurring in the field.

Sexton (1996) discussed the importance of research when selecting relevant and effective treatments to use with clients.

He emphasized that "counselors must be confident that the current status of outcome research is a valid and reliable source of information that can guide clinical practice" (Sexton, 1996, p. 590) and that counselors must draw upon research when determining the most effective treatment. Furthermore, Sexton discussed the importance of consistent research focused on areas of counseling that are relevant to the field. One specific aim of his research was to determine if the research litera- ture being produced met the practical needs of counselors. Although he found that a variety of research publications existed about multiple topics on which counselors can draw, he did not report on quality or relevance of research available. This omission seemed to suggest that although many different topics have been researched, the consistency of research that can contribute to a substantial knowledge pool is lacking.

Nilsson et al. (2007) published a content analysis of quantitative research published in JCD over a 10-year period. They found the greatest amount of research in the areas of academic/career, multicultural issues, symptoms/disorder, and counseling process. Although the findings of this study carried significant implications about counseling research, the authors focused on only 1 out of a possible 15 ACA-affiliated journals. In addition, Nilsson et al. reviewed research from 1991 to 2000, indicating the need for a review of more current research publication practices.

Counselor educators contribute to the evidence-based lit- erature through publication in journals that may be motivated by career productivity requirements or contribution to the field. Barrio Minton, Fernando, and Ray (2008) investigated the publication patterns of counselor educators by examining submitted manuscripts from faculty members of doctoral pro- grams accredited by the Council for Accreditation of Counsel- ing and Related Educational Programs. The authors examined articles published between January 1997 and December 2006. Findings from this study indicated that approximately half of the reviewed publications were research oriented, whereas the remaining articles focused on theory and practice (Barrio

Dee C. Ray, Andi J.Thacker, Laura S. Pace, Karrie L. Swan, Sarah E. Carison, and Jeffrey M. Suiiivan, Department of Counseling and Higher Education, and Darreil M. Huil, Department of Educational Psychology, University of North Texas, Dentón. Correspon- dence concerning this articie should be addressed to Dee C. Ray, Department of Counseling and Higher Education, University of North Texas, 1155 Union Circle, Box 310829, Dentón, TX 76203 (e-mail: [email protected]).

Journal of Counseling & Development • Summer 2011 • Volume 89 349

Minton et al., 2008). Although Barrio Minton et al. reported the frequency of published research articles, they cited as a limitation the fact that the study did not address the quality or content ofthe research publications.

In response to the need for empirically supported counsel- ing practices, JCD established a section regularly devoted to research articles as early as 1993. The editors of JCD identified a distinct need to increase the amount of relevant research articles published consistently (Herr & Fabian, 1993). As the flagship journal for ACA, JCD recognized the importance of providing counseling professionals with relevant research information. In 2003, JCD added a practical section, titled Best Practices, asserting that best-practice manuscripts con- tain information regarding eotmseling interventions centered on outcome data. JCD further emphasized that best-practice manuscripts use evidence-based data to speak to one or more aspects of what interventions work well with specific coun- selors, with specific clients, with specific problems, within a specific setting (McGowan, 2003). JCD, as well as other counseling joumals (e.g.. Professional School Counseling; Lapan, 2005), called on researchers to contribute to tbe field of counseling by submitting manuscripts that validate and contribute to relevant counseling research that will further guide best counseling practices.

The purpose ofthe present research study was to provide an overview of types of research articles being published in counseling-related joumals and, more specifically, to provide a detailed description of research content within quantitative research related to the following five questions: (a) How much research is being published in ACA division counseling jour-

Ray et al.

nais? (b) In relation to the field of counseling, what were the intentions ofthe research? (c) On what type of population, if any, did counseling research focus? (d) What were the primary dependent variables? and (e) What areas of counseling will benefit from this research?

•Method The research team consisted of one faculty member and five doctoral students in a counseling education graduate program. The faculty member (first author) held a doctoral degree in counselor education and had experience conducting multiple quantitative and qualitative published studies. The doctoral students (second, third, fourth, fifth, and sixth authors) held master's degrees in counseling and had completed at least two courses in advanced statistics and research design. In addition, the faculty member trained the doctoral students over a period of several weeks on coding designs in the collection of research.

We first reviewed the ACA website (www.counseling.org) to identify ACA divisions that publish peer-reviewed joumals accepting research articles. We identified 15 joumals for review including ACA's flagship joumal. Journal of Counseling & Development, and 14 joumals associated with AC A divisions. We included only joumals that were published over the entire span of 1998-2007, thereby excluding joumals that were in early production. The full list of joumals can be fotind in Table 1. We attempted to access each joumal issue in paper form, and if unavailable, electronic databases were searched for fiill issues.

The research team developed a coding sheet addressing several aspects of publication, including number of authors.

TABLE 1

Research Articies and Totai Articie Publications for Counseling Division Journais From 1988 Through 2007

Journal

Aduitspar) Ttie Career Development Quarterly Counseling and Values Counselor Education and Supervision Family Journal Journai tor Specialists in Group Work Journal ot Addictions and Offender

Counseling Journal of College Counseling Journal of Counseling & Development Journal of Employment Counseling Journal of Humanistic Counseling,

Education and Development' Journal of tVlental Heattti Counseling Journal of ti^ulticultural Counseling

and Development tVleasurement and Evaluation in

Counseling and Development Professional Sctiool Counseling

Totai

No. of Quantita- tive and Mixed-

iMethods Articles

14 105 56 80 64 13

3g go

167 74

26 65

150 125

1,13g

No. of Quailtatlve

Articles

3 30 24 35 12 14

5 7

3g 10

g g

6 26

243

No. of Total Research Articies

17 135 80

115 76 27

44 g7

206 84

35 74

156 151

1,382

No. of Total Articies

62 291 223 243 742 312

g3 203 543 153

231 276

237

185 663

4,457

% Research to No. of Total

Articies

27 46 36 47 10 g

47 48 38 55

15 27

84 23

"Journal ot Humanistic Counseling as of Vol. 50, Issue 1, Spring 2011.

350 Journal of Counseling & Development • Summer 2011 • Volume 89

Research in Counseling

type of research (i.e., quantitative, qualitative, or mixed meth- ods), counseling area to be benefited by research (e.g., couples counseling, counselors in training), sample description (i.e., clinical or nonclinical), sample description of ethnicity and gender, other sample descriptors reported (e.g., income, education level), research focus (intention of research; i.e., to what counseling area does it make a contribution), and an open answer for primary dependent variables and sample subpopulations of focus. For each coding variable, the research team decided on a clear definition of the variable and a de- scription of the coding choices to guide the coding process. The definitions and descriptors were included in the coding forms for quick access. Coders then independently coded the same five articles using the initial coding system. The research team met to discuss variations and challenges in coding. Open answers for primary dependent variables and subpopulations were compiled into one database for review by the team. The team then categorized dependent variables and subpopulations for fiirther coding. Coding variable definitions and descriptors were refined according to consensus. Following this process, the team rated 10 articles individually and independently on the finalized coding form to establish interrater reliability. Once rater agreement was demonstrated to be substantial (see Landis & Koch, 1977), researchers were permitted to code independently. Although ideal methods include multiple raters of single articles, the considerable number of articles located for this study limited the application of such a process. On the basis of a thorough survey of similar literature reviews, the research team decided to use the method of establishing agreement prior to independent coding of articles (Fouad & Byars-Winston, 2005; Hsieh et al., 2005; Skowron & Reine- mann, 2005; Wood, Crane, Schaalje, & Law, 2005). Joumal issues published fi-om 1998 to 2007 were divided for coding by joumal among the research team members. In addition, each team member provided the number of research articles and the number of total articles for each joumal issue.

Coding Variables

Several of the following coding variables for the present study were subjectively identified by raters (i.e., type of research, research population, sample description, research focus, setting in which the research was conducted, subpopulation, and a process variable). These seven coding variables were analyzed for interrater agreement and subjected to a generaliz- ability analysis, and they are referred to in the generalizability analysis as different methods by which the raters rated the articles. Scott's P. coefficient is provided for each of the seven identified coding variables.

Number of authors. The number of authors was listed as an open-ended continuous variable.

Ethnicity of sample. Sample populations were coded for reported ethnicity for each study. Because of the specificity and variation of ethnicities reported (e.g., immigrant Kore- ans in the United States), researchers concluded that broad

categories of ethnicity would not reveal an accurate level of diversity in studies. Thus, we chose to summarize data by coding whether ethnicity was reported or not as a measure of standard of practice for research publications.

Other characteristics of sample. Other characteristics of samples were coded when reported, including income, age, education level, marital status, sexual orientation, and profes- sional status (e.g., working or unemployed).

Randomization of the sample. Each study was coded for its use of randomization of the sample with the following categories: randomly assigned, not randomly assigned, or not applicable for research design.

Type of research. Each article was coded as quantitative, qualitative, or mixed methods {P^ = 0.85). Because the focus of this research was on exploration of quantitative research and because mixed-methods research (n = 171) resulted in a small number of studies, mixed methods was collapsed with the quantitative category to accurately represent quantitative studies. Thus, it should be noted that the total number of qualitative research articles is underreported.

Research population. We identified eight categories of coun- seling and assigned each study to an area of counseling that the research most likely benefits {P. = 1.00). For example, if a research study explored the attitudes of mathematics majors in college, we categorized the study as benefiting adult counseling. Categories included counselors in training, adult counseling, child counseling, adolescent counseling, family counseling, couples counseling, counselor educators, and counselors in practice. During discussion of population categories, it was noted that college populations would be categorized as adults because being over age 18 years is the demarcation of adult- hood legally in the United States. In addition, the research team concluded that information regarding college research would be better coded under the research setting variable.

Subpopulation. In coding the content of research studies, there seemed to be a preference among counseling research- ers to study specific types of populations beyond diagnoses or clinical issues. These populations appeared to be indepen- dent variables that were of particular interest in counseling research. The groups identified in the literature were typically considered multicultural populations. Articles were coded {P.^ = 1.00) if the researchers specified a particular group as an independent variable or focus of the study, such as attitudes of Native Americans. Because we found that authors some- times specified two independent population variables, such as experiences of African American women, we decided that this category could be multicoded. Categories included females, males, Afiican American, Asian American, White/Caucasian, Hispanic, Native American, multiethnic, intemational popu- lations, religious, GLBT (gay, lesbian, bisexual, and trans- gender), socioeconomic status, disabilities, and older adults.

Clinical versus nonclinical sample population. Each article was coded as clinical or nonclinical for its sample population (P. = 0.77). Clinical samples were defined as participants

Journal ofCounselingôC Development • Summer 2011 • Volume 89 351

Ray et al.

who had been referred for a clinical problem or were already receiving clinical services. Nonclinical samples were defined as participants who had been recruited for the study.

Research setting. Studies were coded for the setting in which the research was conducted {P.^ = 0.76). Categori- cal variables were listed including college, school (pre- K-12), clinical (including counseling clinics, hospitals, etc.), and other.

Intention of research related to counseling field. The purpose of this coding variable was to identify what area of counseling this research describes and to what area does it make a contdbution. There were 11 categories in this area; to provide more meaningful data, the coders were forced to choose one that was most representative of the researchers' intentions (P. = 0.85). The following is a list of descriptions for each area. Effectiveness of counseling intervention was coded if the research study was a design intended to investigate the effect of a specific counseling technique or modality; this was typically represented by experimental, quasi-experimental, or single-group pretest-posttest design. The counselor effective- ness category included studies that addressed the effectiveness of the counselor, including building counselor competencies such as multicultural competencies. The counselor variables category was idendfied if the study focused on descdptors of counselors, such as developmental level of counselors. The client variables category included studies that used mea- surements to descdbe clients, such as client personality or behavioral characteristics. The diagnosis variables category descdbed a diagnosis or population with a specific diagnosis. The process of counseling category included research that explored dynamics dudng counseling sessions. The counselor educator vadables category included research that described counselor educators, such as comparing attitudes of non- tenured with tenured faculty. The measurement/instrument category contained any studies in which the primary objec- tive was to investigate the use of a particular instrument. The population vadables category included research studies that descdbed a group of people who were not clinical, such as self-esteem of intemadonal college students. The supervision category included any research on the process or outcome of supervision of counseling students or counselors in practice. And the "other" category was provided to categodze any study that did not meet cdteda for any of the previously mentioned categorizations.

Dependent variable. Studies were coded to identify content areas of expected outcome {P^ = 0.81). We identified 22 areas of content (see Table 6, p. 355) that incorporated areas of focus for counseling researchers and then coded each study for one of these content areas. In this process, we attempted to define the single most important dependent vadable assessed by the researcher, resulting in a forced-choice coding vadable (i.e., no multicategodcal coding).

Data analyses. Studies were coded for use of data analysis methods, such as t tests, correlation, regression analysis, and

so on. Because the coding team did not categodze analyses into broader categodes but listed each type of data analysis separately, interrater reliability was not established for this vadable. Hence, results should be interpreted with caution. In addition, individual studies were multicoded because of the use of multiple types of analyses in a single article.

Reliability of Coding Procedures

Scott's Pj coefficient (Scott, 1955; analogous to Cohen's kappa coefficient; Cohen, 1960) was used to examine inter- rater reliability, whereby each rater was crossed with every other rater on a subset of 10 randomly selected articles and each article was rated on seven separate subjective methods by all six raters. No coefficients were less than 0.75. However, when measurement procedures, such as the one used in the present study, introduce multiple sources of measurement error, generalizability theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Brennan, 2001; for an overview, see Shavelson & Webb, 1991) uses classical analysis of vadance (ANOVA) to estimate meastirement errors and G coefficients for the entire measurement procedure. We identified three facets, which can be thought of as independent vadables or factors in ANOVA, that constitute the qualitative conditions under which the data were obtained: methods (M), articles (A), and raters (R). The observed ratings on one article with one method of measurement is not fully dependable; that is, it is unlikely that we would observe exactly the same rating frequency on a different article or for a different method of measurement. Implicit in this idea is that the ratings result from a steady state within the person providing the radng, and we assume that any differences in ratings from one article to the next or between different measurement scales (methods) are not attdbutable to systematic changes in the individual providing the rating.

The value in applying generalizability theory is that mul- tiple sources of measurement error can be estimated separately in a single analysis. This allows the observation of relative error vadance associated with each facet in the study and at the same time provides a single coefficient refiecting the level of dependability, analogous to a reliability coefficient (such as Cronbach's alpha). In the present study, all facets (methods, articles, and raters) were ftiUy crossed, thus there are no con- founded or nested sources of measurement error. The aim of the ANOVA, then, is to assess the relative importance of the idendfied sources of rating vadation. This is accomplished through the esdmadon of vadance components in a simple three-facet crossed design MAR, in which methods (M) were independently applied to articles (A) by different raters (R). The score decomposidon equation would be

Y=a + M.+ A.+ RA- MA.+ MR A- AR.+ MAR.^e... ijk f^ I J k y I* jk ijk' ¡jk

where 7..̂ is the rating frequency obtained on Method / of Article^' for Rater k, \i is the overall mean score (the average

352 Journal of Counseling & Development • Summer 2011 • Volume 89

Research in Counseling

of all ratings), M. is the effect of Method i (the deviation of the ith measurement scale's average score fVom the grand mean), Aj is the effect of Article 7 (the deviadon of theyth article's average score from the grand mean), and Ä̂ is the effect of Rater k (the deviation of the kth rater's average score from the grand mean). MA.., MR.^, and AR.^^ represent Method x Article, Method X Rater, and Article x Rater interaction effects (e.g., for AR.1^, this is the degree to which Rater / deviates from his or her normal severity/leniency from article to article). MAR..^^ represents completely confounded variance among methods, articles, and raters and thus is synonymous with an error term (e..^. Because effects are assumed to be independent, the total score variance ( a p represents the sum of the variance components: methods (CJ^), articles (a^), raters ( o p , and the interactions between Methods x Articles (o^^). Methods x Raters (o^^). Articles x Raters (CJ^^), and Methods x Articles X Raters (o^^^), which includes the residual variance:

The G study was performed using EduG software (Car- dinet, Johnson, & Pini, 2010), which is designed to carry out generalizability analyses. The seven different methods for examining articles were treated as a fixed facet, because these are the only attributes of interest. Each facet or facet interacdon is shown in the ANOVA source in Table 2. Sums of squares, degrees of freedom, and mean squares are pro- vided in the first three columns, in addition to three sets of estimated variance components. Colutnns 4 through 6 contain the estimated variance components for an entirely random effects model, a mixed model, and corrected model using Whimby, Vaughan, and Tatsuoka's (1967) correction factor. Negative variances are not possible in theory, but the raters variance and Methods x Raters variance contain sampling fluctuations around small component values that resulted in negligible negative component esdmates. While these nega- tive values appear in the ANOVA table, they were replaced

by zeros in the computadon of generalizability parameters. The most important column of this table (column 4) shows the percentage of variance associated with each facet or facet interaction. Note that the different methods contribute the largest amount of variance to the total, combined with the interaction of the different methods with the different articles used for the reliability analysis. Both of these high variance components should have high contributions to any variance observed in this measurement procedure. Moreover, the contribution of rater effects to variance is very low, as are the rater interaction components.

In Table 3, the different error variances are displayed along with the various facets of the study, and their interactions are separated into differentiation facets (the first two columns) and instrumentation facets. These are based on the data in the ANOVA results from Table 2 and are calculated using the estimation algorithms appropriate to the sampling status of the different facets. Methods are treated as a differentiation facet, which is the one single measurement component that produces an effective difference between measured entities. Two separate G study analyses are represented; the first rela- tive study includes an a priori measurement design in which the interactions with the differentiation source are examined, and the second analysis, which is more important for our purposes, is the absolute error variance showing the contri- budon of all components of variance with the exception of the differendadon variance (the component we expected to produce the most variance). The differendation variance is more than eight dmes that of the absolute error variance; all other things being equal, this higher differendation variance indicates that the sources of measurement error (articles and raters) are highly consistent (reliable).

A ratio of the estimated differendation variance to estimat- ed total variance of observed scores yields a G coefficient. The absolute G coefficient is an indication of the global reliability of the measurement procedure when the methods differentia- tion variance (the different ways the articles were examined)

TABLE 2

ANOVA Source Table Demonstrating Percentage Contribution of Each Source to Totai Variance

Source

Methods Articles Raters Methods and

articles Methods and

raters Articles and

raters Methods,

articles, and raters

ss 500.67 74.00

2.48

434.33

11.99

29.14

174.72

6 9 5

270

83.44 8.22 0.50

8.04

0.40

0.65

Random

1.26 0.00 0.00

1.23

-0.02

0.00

0.65

Mixed

1.26 0.18 0.00

1.23

-0.02

0.09

0.65

Variable Components

Corrected

1.08 0.18 0.00

1.06

-0.02

0.09

0.55

36.5 6.1 0.0

35.6

0.0

3.1

18.7

0.70 0.09 0.01

0.25

0.01

0.02

0.06

Note. ANOVA = analysis of variance.

Journal ofCounseling& Development • Summer 2011 • Volume 89 353

Ray et al.

TABLE 3

G Study Table (Measurement Design: Methods, Articles, and Raters)

Source of Variance

Methods Articles Raters Methods and articles Methods and raters Articles and raters Methods, articles, and

raters

Sum of variances SD

1.08

1.08 1.04

REV

0.11 0.00

0.01

0.12

% Rei.

92.0 0.0

8.0

100.0

AEV

0.02 0.00 0.11 0.00 0.00

0.01

0.13

% Abs.

13.4 0.0

78.6 0.0 1.1

6.9

100.0

Note. Coefficient G relative = 0.90; Coefficient G absolute = 0.89. Grand mean for levels used = 2.67. Variance error of the mean for levels used = 0.02. Standard error of the grand mean = 0.14. Relative standard error = 0.34. Absolute standard error = 0.37. DV = differen- tiation variance; REV = relative error variance; % Rel. = percentage relative; AEV = absolute error variance; % Abs. = percentage absolute.

is partitioned out of the model and can be interpreted similar to an alpha coefficient in the case of a simple fully crossed design as shown here. The G coefficient certainly suggests that measurement precision is satisfactory, as .89 represents a good degree of measurement precision. Reliability coefficients (relative and absolute) for each of the 10 articles and the six raters are provided in Table 4.

•Results Research Question 1 : How Much Research Is Being Published in Counseling-Related Journals?

We first attempted to answer this question by listing the num- ber of articles divided by year of publication. The following is a breakdown of number of studies published for each year across all 15 journals: 1998 (n = 128), 1999 (n = 117), 2000

TABLE 4

G-Facets Analysis Facet and Levei Coefficient G Reiative Coefficient G Absoiute Articies

1 2 3 4 5 6 7 8 9

10 Raters

1 2 3 4 5 6

0.88 0.87 0.88 0.89 0.91 0.88 0.87 0.89 0.93 0.92

0.90 0.90 0.90 0.91 0.90 0.90

0.86 0.85 0.86 0.87 0.90 0.86 0.85 0.88 0.91 0.90

0.89 0.89 0.89 0.89 0.89 0.89

in = 123), 2001 (« = 146), 2002 (n = 129), 2003 (n = 138), 2004 (n = 147), 2005 (n = 149), 2006 (n = 154), and 2007 (n = 151). From 2004 to 2007, publication of research seemingly stabilized for the final 4 years reviewed.

Table 1 provides a ñill listing of journals as well as the number of quantitative/mixed-methods articles and number of qualitative articles in relationship to total publication. Measurement and Evaluation in Counseling and Development published the highest percentage of research to total articles with 84% of its articles dedicated to research, and the next highest percentage was represented by Journal of Employ- ment Counseling (55%). Journal for Specialists in Group Work published the lowest percentage with 9% of its articles reporting research. The percentage of the total number of research articles in comparison to total number of published articles is 31% (n - 1,382). If only quantitative and mixed- methods articles are compared with total publication articles, the percentage drops to 26% (« = 1,139).

Research Question 2: In Relation to the Field of Counseling, What Were the Intentions of the Research?

Table 5 lists the intentional focus of research in relation- ship to the field of counseling. As can be seen in Table 5, the highest number of counseling research articles among the categories listed was focused on population variables (36.8%), which is an exploration of nonclinical groups for nonclinical problems. Examples of this type of research are characteristics of first-generation college students or attitudes of human resource employees. In a distant second, measurement/instrument research accounts for approxi- mately 15% of counseling research. The fewest number of research studies were conducted on issues related to specific diagnoses and groups of people with specific diagnoses (diagnosis variables = 0.7%). Also, it is interesting to note that the study of counseling interventions was ranked 6th of 10 categories with 6% of research focused on this intention.

TABLE 5

Intention of Research Related to Counseling Field

Research Focus Area

Population variables Measurement/instrument Counselor variables Client variables Counselor effectiveness Effectiveness of counseiing interventions Counselor educator variables Process of counseling Supervision Diagnosis variables Other

No. of Articies in Focus Area

509 210 143 128 114 83 48 42 34 10 61

% Focus toTotai

Research

36.8 15.2 10.3 9.3 8.2 6.0 3.5 3.0 2.5 0.7 4.4

Note.N= 1,382.

354 Journal ofCounseling& Development • Summer 2011 • Volume 89

Research in Counseling

Research Question 3: On What Type of Population, if Any, Did Counseling Research Focus?

Results addressing this research question were derived from three coding variables: subpopulation, clinical versus nonclini- cal sample population, and research setting. Approximately 70% (« = 959) of research publications did not identify a specific subpopulation. Thus, the vast majority of research articles used study samples with general characteristics, ostensibly to allow generalization of findings to the general population. Of those studies that focused on a particular sub- group ofthe population (n = 423), females were studied the most, in 109 studies, which represent approximately 8% of all research articles. African Americans were the second most studied population with 63 articles (4.6%). Asian Americans were third (n = 59; 4.3%), intemational populations outside ofthe United States were fourth (n = 55; 4%), and males were fiflh (n = 50; 3.6%). White/Caucasian (3%), Hispanic (3%), religious (2%), GLBT (1.9%), socioeconomic status (1.7%), disabilities (1.3%), older adults (1.2%), multiethnic (1.1%), and Native American ( 1 %) were the subgroups of least focus.

In another attempt to address this research question regard- ing what populations are being studied, we coded two other pertinent variables; clinical versus nonclinical populations and settings in which the research was conducted. On the question of clinical and nonclinical population, 134 stud- ies (10%) reported investigating a clinical sample, whereas the overwhelming majority, 1,248 studies (90%), reported investigating nonclinical study participants. We also coded studies according to the setting in which the research was conducted. Ofthe 1,382 research studies, 609 (44%) were conducted in colleges/universities, 283 (21 %) in schools, 130 (9.4%) in clinics or clinical settings, and 360 (26%) in other settings, which were typically workplaces such as studies in the employment literature.

Research Question 4; What Were the Primary Dependent Variables?

Table 6 lists each of the categories of dependent variables and how many studies emphasized that variable. Not sur- prisingly, processes related to counseling were identified as the dependent variable represented in the largest ntimber of studies (n = 400; 28.9%). These could include a coimselor's skills or dynamics in the counseling process. Second, career- related variables were highly represented with 250 publica- tions (18.1%). Development, which included Wellness and counselor development, was the third most likely dependent variable to be studied (n = 141; 10.2%). Grief and loss were the least likely topics to be studied among the counseling literature (n = 4; 0.3%).

Research Question 5: What Areas of Counseling Will Benefit From This Research?

The final research question attempted to use the population coding variable to identify specific areas of counseling to

TABLE 6

Dependent Variables Identified in Research Publications

Dependent Variabie (DV)

Counseling Career Development Relationships Multiculturalism Academic Substance use/abuse/addiction Crisis Self-esteem Spirituality/religion Depression Attachment Disruptive behaviors Body image/eating disorders Ethics/legal Stress Anxiety Attention-deficit/hyperactivity disorder Medical Other Diagnostic and Statistical Manual

disorders Grief/loss Other

No. of Studies

400 250 141 gg 73 47 43 3g 38 33 ig i g 18 15 14 14 g 8 6

5 4

% DV to Total

Research

28.9 18.1 10.2 7.2 5.3 3.4 3.1 2.8 2.7 2.4 1.4 1.4 1.3 1.1 1.0 1.0 0.7 0.6 0.4

0.4 0.3 6.4

Note. N = 1,382.

which research made contributions. The following is a break- down of counseling categories and their matched studies: adult counseling (« = 642; 46.5%), counselors in practice (n = 244; 17.7%), counselors in training (n = 152; 11%), adolescent counseling (n = 134; 9.7%), child counseling (n = 79; 5.7%), counselor educators (n = 69; 5%), couples counseling (« = 36; 2.6%), and family counseling (« = 26; 1.9%). In comparison with other categories, the highest numbers of research studies were being conducted to benefit adult counseling.

Supplemental Results Data

Other results related to data collection were summarized as potential points of interest to the counseling community. With regard to number of authors, authorship ranged from 1 to 13 authors for one article, with approximately 16% of articles published by single authors. Most articles were published by 2 to 4 authors, representing approximately 76% of all re- search publications. When research studies involved human participants, most authors reported ethnicity of sample (n = 936; 67.7%); however, a large number of publications did not report ethnicity (n = 446; 32.3%). Authors reported other sample characteristics at various rates, including income (« = 126; 9.1%), age (« = 670; 48.5%), education level (n = 693; 50.1%), marital status (n = 173; 12.5%), sexual orientation (n = 43; 3.1%), and professional status (n = 198; 14.3%). Randomization of the sample was rare, with frequency at 130 studies (9.4%). The most utilized statistical analysis was correlation (n = 327; 23.7%), with ANOVA (« = 297; 21.5%),

Journal ofCounseling & Development • Summer 2011 • Volume 89 355

Ray et al.

regression procedures (n = 282; 20.4%), multivariate analysis of variance (« = 211; 15.3%), and t tests (« = 175; 12.7%) rounding out the top five most fi-equently used analyses. No- tably, the nonparametric chi-square analysis was conducted in 160 studies, representing 11.6% of all research studies.

•Discussion Over a decade ago. Mate and Kelly ( 1997) stated, "Publication of original research is a key criterion for judging the identity development of professions" (p. 286). Results of the present study indicate that the counseling profession is producing research at a consistent rate. The rise of research publications ti-om 2004 to 2007 demonstrates that the field of counseling is attempting to produce research that progresses counseling to an adequate level ofprofessional status. Interpretation of the 31% of research articles to total articles found in this research is challenging. Previous publications have reviewed the research publication in individual journals (Mate & Kelly, 1997; Nilsson et al., 2007), but there has never been a comprehensive exploration of all counseling-relatedjoumals. Although all ACA-affiliated joumals publish original research, some concentrate on research at the expense of practice ar- ticles, whereas others concentrate on practice at the expense of research, leading to a wide variation of research publica- tions in comparison to total publications across joumals. JCD attempted to address this balance by adding a Best Practices section to its joumal in 2003. The purpose of the Best Practices section was to encourage counselors to "apply the research evidence that undergirds the practice of counseling" (Marotta & Watts, 2007, p. 491). However, this statement inspires the question, "What research base do counselors have for the practice of counseling?"

Evidence-Based Counseling: The Need for More Intervention Research

Although evidence-based psychological interventions can be defined through strict adherence to rigorous experimental intervention designs and identified through concrete criteria set forth by professional organizations such as the American Psychological Association (APA; see www.psychology.sun- ysb.edu/eklonsky-/divisionl2/faq.html), counseling literature appears to link effective practice with a broader definition of evidence. Evidence for counseling practices includes the coconstruction of practice between counselor and client, clinical practicalities, consideration of diverse populations, and theoretical belief systems, along with empirical support for practice (Marotta & Watts, 2007). The debate within the mental health field regarding evidence-based practice has grown to new heights over the last two decades and is often a point of contention among professionals (Nathan, 2004; Rubin, 2008; Schloss & Haaga, 2005). Some mental health professionals believe a strict definition of evidence-based practice is reductionistic and mechanical and propose more

humanistic methods of evaluation (see Cain & Seeman, 2001); altematively, other mental health professionals view as unethical counselors who do not exclusively use identified evidence-based practices (see Nathan, 2004). And then there are the majority of counselors who fall somewhere in the middle of the debate.

However, the debate over evidence-based practice does not appear to serve as a driving force in counseling literature when one considers the results of the present study. Perhaps the most conceming tinding of this study is that only 6% of all counseling research articles are focused on exploring the effectiveness of a counseling intervention. There are few answers in the counseling literature to the most basic ques- tions of the profession, "What works with clients?" and "Are counselors doing what works?" The absence of counseling literature on intervention motivates further questions regard- ing counseling practice. Are counselors looking to other fields such as psychology or social work for direction in practice? Are counselors operating without empirical evidence regard- ing practice? Do counselors believe that empirical evidence is less relevant to practice? Do counselors rely on theory and experience rather than empirical research to guide practice? The low publication rate of intervention-based research leads to more questions than answers. Future research might explore the counseling practitioner's use of published research. In ad- dition, a retrospective review of joumals to interpret the 6% counseling intervention research publication rate is warranted.

Although counseling joumals experienced an increase in research publications over the 10-year span, this may not be specific to intervention research publications. Findings ftom other disciplines indicate that intervention research may be on the decline. Hsieh et al. (2005) reported that intervention re- search in selected educational psychology joumals decreased ti-om 40% to 26% over a 21-year period from 1983 to 2004. In addition, intervention research published in the American Educational Research Journal dropped trom 33% to 4% in the same time span. Robinson, Levin, Thomas, Pituch, and Vaughn (2007) further concluded that intervention studies in teaching and leaming joumals declined trom 45% to 33% over a 10-year period ti-om 1994 to 2004. Henson, Hull, and Williams (2010) suggested that a lack of quality research publications may be infiuenced by a lack of knowledge and skills in doctoral-level education. In each of these studies, authors emphasized the need for intervention research to support practice.

There has been speculation that although counselors are interested in research, they are often tied to practice or un- skilled at conducting or understanding research (Bangert & Baumberger, 2005; Lundervold & Belwood, 2000). If this speculation is true, then it is unclear whether a lack of inter- vention studies is due to counselors not conducting research on intervention or joumal venues not publishing intervention research. Although there are no verifiable answers to the many questions regarding intervention publications, it does appear

356 Journal of Counseling & Development • Summer 2011 • Volume 89

Research in Counseling

that the field of counseling would benefit from a greater concentration on increasing intervention research. Even if counselors are not limited to pracdcing only those techniques with empidcal evidence, most professionals would agree that evidence for practice is helpful in several ways. Most important, empidcal exploration of interventions provides information descdbing under what conditions and with what populations certain interventions are most effective.

Operating from empidcal evidence provides protection from legal downsides of practice, including lending support for current practices. And finally, empidcal evidence for in- tervendon helps to gain funding from government and pdvate sources and is helpful for marketing services to clients.

Utility of Current Counseling Research to the Practitioner

Results of this review indicate that the typical research study published in counseling joumals seemed to be a research exploration descdbing aspects of general or specified popula- dons that were not considered clinical and often using a col- lege sample. This statement is not meant to be an indictment of counseling research, only an explanation. There is certainly value to the counseling profession to discover personality and behavioral aspects of nonclinical populations. After all, cotinselors concenfrate on the healthy development of people; hence, the study of general developmental aspects of people seems warranted. However, counselors also work with people who have significant mental health problems. Of the research articles examined, few focused on common clinical issues such as depression, anxiety, and gdef and loss (2.4%). Given reports that major depressive disorder is the leading cause of disability in the United States for individuals ages 15-44 years and that anxiety disorders affect approximately 40 million adults each year (Kessler, Chiu, Demler, & Walters, 2005), the need for research on depression and anxiety ap- pears significant. In addidon, gdef and loss are considered to be universal human expedences, as well as dsk factors for the development of depression and anxiety (Servaty-Seib, 2004). Mental health counselors work with clients who are labeled with diagnoses that are threatening to everyday funcdoning. School counselors work with children who are dismptive to their own leaming as well as others. Human resource counsel- ors work with employees who may lose their career because of mental health issues. These types of services call for an increase in cotmseling research related to clients and client problems. This study indicated that of 1,382 research studies, only 134 explored a clinical population. In other words, less than 10% of counseling research is dedicated to clients most in need of counseling services.

One argument against increasing research related to client and client issues is that as a cotmseling profession, counselors are focused on development and Wellness. Yet, this appears to be the very reason for increasing research in the positivistic area of development to identify effective approaches and

detennine what positive approaches work with clients. The medical model adopted by other mental health professions concentrates on finding what is wrong with clients and then fixing the problem. A counseling approach to client research is based on looking at the whole person and how impaired functioning is related to the whole, not just the problem. This holisdc perspective contdbutes to finding new research designs and ways of working with clients, as well as new im- portance attdbuted to clinical significance of interventions. An emphasis on conducting research from a holistic perspective with real clients in real life may provide infinite possibilities for the counseling practitioner.

Counseling Areas as a Focus

As previously descdbed, much of the counseling research re- viewed for this study was committed to adding information to the pracdce of adult counseling, which for the purposes of this study included college coimseling. Informadon is provided about the attdbutes or charactedsdcs of adult populadons or possibly measurements that are designed for adult popula- tions. Almost 47% of all research publicadons benefited adult coimseling. Counselors in pracdce and counselors in training were the second and third most frequent areas that benefited from the research literature, combining for close to 29% of all research publications. Research to inform child counsel- ing made up only 5.7% and adolescent counseling made up only 9.7% of the publications. These particular percentages are disappointing considedng that Mellin (2009) referred to children's mental health as a cdsis and summoned counsel- ors to increase fraining and research in response to current needs of children. Also, we were surpdsed to see the scarcity of research related to couples and family counseling, which made up fewer than 5% of all research publicadons. Certainly, counseling pracddoners are seeing couples and families on a consistent basis, and research in counseling should reñect such practice. In faimess, it is noted that conducting research on children, couples, and families is challenging and includes many more barders than conducdng research with individual adult populations. Barders may include legal challenges in- volved in child research, informed consent and assent with participants, and pracdcal considerations of organizing inter- vendons for multiple participants simultaneously.

Limitations

Limitations of this study include the subjective nature of cat- egodzation among independent raters. Although substantive interrater reliability with dgorous statisdcal generalizability analysis was established pdor to coding, each study was coded by an individual rater who is subject to systemadc bias. It is possible that another research team might come to different conclusions regarding the categodzation process of the vad- ous content areas. In addition, each rater attempted to access hard copies of all issues of each joumal. However, in some cases, raters relied on electronic databases, which may pos-

Journal of Counseling & Development • Summer 2011 • Volume 89 357

Ray et al.

sibly result in missing individual articles within journal issues. To address this issue, we cross-referenced each electronic article with the electronic table of contents for each journal issue. Regarding the immense scope of this research project, it is doubtful that the review of 4,457 counseling articles with meticulous reading of 1,382 identified research articles resulted in error-fVee data.

In the context of extensive data collection, decisions regard- ing presentation were made to aggregate data into categorical findings. The purpose of aggregadng the data is to provide some level of order for the reader's understanding. However, aggregadon of data may also lead to more globalized presenta- don, which may provide less meaning and possibly devalue small qualitative differences.

Implications for the Counseling Profession

The good news is that the counseling profession is producing original research that is intended to have an impact on the pracdce of counseling. Close to one third of all counseling publicadons over a 10-year period were research based. The counseling profession produces research related to practitioners, counselors in training, counselor educators, clients, diagnoses, and the general population. However, the benefits of current research to counseling practice are questionable. There seems to be a strong emphasis on conducting research with nonclinical populations to define characteristics and attributes, but there is little focus on conducting research regarding coimseling inter- ventions that will likely benefit a clinical population.

Conducting intervendon research is challenging at best. Counseling researchers face hurdles from the beginning with the inidal approval of human participants, where committees are often hesitant to approve the use of specific techniques for persons with mental health problems. Recruiting sample participants who meet criteria and are willing to participate is often where many researchers lose modvadon. When there is an attempt to randomize the sample, the researcher may encounter problems related to setting (Hsieh et al, 2005). In addidon, it is often difficult to find measurements to appropriately isolate dependent variables and then administer those measurements on schedule. There are many reasons not to conduct intervendon research. However, as demonstrated through this study, there is a great need for intervention research.

Challenges related to intervention research can be met through different options available to the counseling com- munity. First, research teams are the best defense against developing a sense of hopelessness and for developing creative solutions to research problems. A research team involves several people of similar professional backgrounds working toward the same goal. Hence, when one member meets a research challenge, there are others to provide inspiration for condnuance. Findings from this study indicate that research articles are more likely to be published by two to four authors (76%) than by a single author (16%). This result is consistent with that of Jones et al. (2010), who reported a mean of 2.6

authors per article in the field of educational psychology for the 2008 publicadon year. Second, acquisition of funding is particularly helpful for assurance of research completion, specifically to provide incentives to participants to complete experimental or control conditions and/or complete mea- surement administrations. Funding for interventions to work with common societal problems such as depression, atixiety, or disruptive behavioral problems is possible through both govemment and private sectors. Setting reasonable expecta- dons for research participation is another option to encour- age intervention research. Often, counselors believe that experimental research must be conducted with htmdreds of participants to be considered of substance. However, rigorous experimental designs can be conducted with a small number of participants. For example, 20 participants might be randomly assigned to each of two conditions, experimental and control, thereby only requiring 40 participants in all and allowing for statistical analysis. Although limited in power, small rigorous designs lend credibility to specific interventions and can later be used in meta-analyses and systematic reviews to contribute to evidence-based practice.

Notably, research studies must include details regarding sample characterisdcs, methodology, and data analysis to be used for the broader purpose of meta-analyses and systematic reviews. Results from the present study indicate that sample characteristics such as ethnicity, gender, or other relevant descriptors, as well as sampling procedures, frequently may not be reported in published research. Joumal editors are in a position to require reporting procedures as criteria for publication.

Finally, cotmselor education programs can address the need for more clinical research by initiadng the research process at the student level. As described in Ray, Barrio Minton, Schot- telkorb, and Brown (2010), students in counseling programs can participate in research projects early on, from entering data to midlevel contribudon of delivering intervendon to later leadership in organization and administradon of an interven- tion research design. Once cotmselors in training experience the success of research completion, it is logical that they will be more willing to conduct research as a practitioner or counselor educator.

•References American Counseling Associadon. (2005). ACA code of ethics.

Alexandria, VA: Author. Bangert, A. W., & Baumberger, J. P. (2005). Research and stadsdcal

techniques used in the Journal of Counseling & Development: 1990-2001. Journal of Counseling & Development, 83,480-487.

Barrio Minton, C. A., Fernando, D. M., & Ray, D. C. (2008). 10 years of peer-reviewed publications in counselor education: Where, what, who? Counselor Education and Supervision, 48, 133-143.

Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer.

358 Journal ofCounselingSc Development • Summer 2011 • Volume 89

Research in Counseling

Cain, D., & Seeman, J. (Eds.). (2001). Humanisticpsychotherapies: Handbook of research and practice. Washington, DC: American Psychological Association.

Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. In G. Marcoulides (Series Ed.), Quantitative methodology series. New York, NY: Routledge.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37—46. doi: 10.1177/001316446002000104

Cronbach, L. J., Gleser, G. C , Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of gener- alizability for scores and profiles. New York, NY: Wiley.

Fouad, N., & Byars-Winston, A. (2005). Cultural context of career choice: Meta-analysis of raee/ethnicity differences. The Career Development Quarterly, 53, 223-233.

Henson, R., Hull, D., & Williams, C. (2010). Methodology in our education research eulture: Toward a stronger collective quantitative proficiency. Educational Researcher, 39, 229-240. doi:10.3102/0013189X10365102

Herr, E. L., & Fahian, E. S. (1993). The Journal of Counseling & Development: Its legacy and its aspirations. Journal of Counseling & Development, 72, 3-4.

Hsieh, P., Acee, T, Chung, W, Hsieh, Y, Kim, H., Thomas, G., . . . Robinson, D. (2005). Is educational intervention research on the decline? Journal of Educational Psychology, 97, 523-529. doi: 10.1037/0022-0663.97.4.523

Jones, S., Fong, C , Torres, L., Yoo, J., Decker, M., & Robinson, D. (2010). Productivity in educational psychology journals from 2003 to 2008. Contemporary Educational Psychology, 35, 11-16. doi:10.1016/j.cedpsych.2009.08.001

Kessler, R., Chiu, W, Demler, O., & Walters, E. (2005). Prevalence, severity, and comorbidity of 12-month DÄW-/F disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617-627. doi:10.1001/archpsyc.62.6.617

Lapan, R. (2005). An editor's top ten wish list. Professional School Counseling, 8, ii-iv.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.

Lundervold, D. A., & Belwood, M. F (2000). The hest kept secret in counseling: Single case (Af= 1 ) experimental designs. Jowrna/o/ Counseling & Development, 78, 92-102.

Marotta, S. A., & Watts, R. E. (2007). An introduction to the Best Practices section in the Journal of Counseling & Development. Journal of Counseling & Development, 85, 491-503.

Mate, R. L., & Kelly, K. R. ( 1997). Research publication in the Jour- nal of Mental Health Counseling as an indicator of professional identity development. Journal of Mental Health Counseling, 19, 286-294.

McGowan, A. S. (2003). New and practical sections in the Journal of Counseling & Development: Information for the prospective author and the readership. Journal of Counseling & Develop- ment, 81, 387-388.

Mellin, E. A. (2009). Responding to the crisis in children's mental health: Potential roles for the counseling profession. Journal of Counseling & Development, 87, 501-506.

Nathan, P. (2004). The evidence base for evidence-hased mental health treatments: Four continuing controversies. BriefTreatment and Cri- sis Intervention, 4,243-254. doi:10.1093/brief-treatment/mhh021

Nilsson, J. E., Love, K. M., Taylor, K. J., & Slusher, A. L. (2007). A content and sample analysis of quantitative articles published in the Journal of Counseling & Development between 1991 and 2000. Journal of Counseling & Development, 85, 357-363.

Ray, D., Barrio Minton, C , Schottelkorb, A., & Brown, A. (2010). Single-case design in child counseling research: Implications for counselor education. Counselor Education and Supervision, 49, 193-208.

Robinson, D., Levin, J.,Thomas, G., Pitueh, K., & Vaughn, S. (2007). The incidence of "causal" statements in teaching and learning research journals. American Educational Research Journal, 44, 400-413. doi:10.3102/0002831207302174

Ruhin, A. (2008). Practitioner's guide to using research for evidence- based practice. Hohoken, NJ: Wiley.

Schloss, H., & Haaga, D. (2005). Nature of the evidence base for evidenced-based mental health care. Journal of Cognitive and Behavioral Psychotherapies, 5, 207-216.

Scott, W. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 17, 321-325. doi:10.1086/266577

Servaty-Seib, H. (2004). Introduction: Perspectives on counseling the bereaved. Journal of Mental Health Counseling, 26, 95-97.

Sexton, T. L. (1996). The relevance ofcounseling outcome research: Current trends and practical implications. Journal of Counseling & Development, 74, 590-600.

Shavelson, R., & Webb, N. ( 1991 ). Generalizability theory: A primer. Newhury Park, CA: Sage.

Skowron, E., & Reinemann, D. (2005). Effeetiveness of psycho- logical interventions for child maltreatment: A meta-analysis. Psychotherapy: Theory, Research, Practice, Training, 42, 52-71. doi: 10.1037/0033-3204.42.1.52

Whimby, A., Vaughan, G. M., & Tatsuoka, M. M. (1967). Fixed ef- fects vs random effects: Estimating variance components from mean squares. Perceptual and Motor Skills, 25, 668.

Wood, N., Crane, D., Sehaalje, B., & Law, D. (2005). What works for whom: A meta-analytic review of marital and couples therapy in reference to marital distress. American Journal of Pamily Therapy, 33, 273-287. doi:10.1080/01926180590962147

Journal ofCounseling& Development • Summer 2011 • Volume 89 359

content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's

express written permission. However, users may print, download, or email articles for individual use.