Summary

Revista de Asisten]\ Social\, anul X, nr. 1/2011, 25-33 25

Measuring Effectiveness in Direct Social Work Practice

Bradford W. Sheafor*

Abstract. In many parts of the world social workers are increasingly expected to provide documentation of the effectiveness of their services. One useful approach to such documentation is to measure the amount of change clients experience relative to the issues in their lives being addressed with the social worker. This is one expression of the popular demand for evidence-based practice: evidence-based evaluation. While it is not possible to prove that a social worker�s intervention caused the change, empirical documentation of change can be shown to be associated with the intervention and the work of the social worker. This trend is somewhat controversial in social work and, indeed, there are advantages and disadvantages to efforts to quantify client change. In this article a process is described for conducting an evidence-based evalu- ation of client change when working in a direct service capacity, i.e., face-to-face intervention with individuals, families, and groups. In addition to the usual process followed in assessing and intervening to help change the client situation, additional steps in the process are to: 1) generate researchable questions that will inform the social worker�s actions with this client (formative research) or provide summary infor- mation about the practice outcomes (summative research) to inform future practice activities; 2) quantitatively measure change in the important variables related to the issue(s) being addressed; 3) organize the resulting data in a format that helps to interpret the client outcomes.

Keywords: direct practice evaluation, evidence-based practice, measurement, single- -subject designs, empirical practice evaluation

Introduction

As social work has evolved, at least in industrialized nations, simply asserting that we are doing good when serving our clients is increasingly viewed with suspicion. When called upon to prove that our interventions make a difference for clients, social workers are often hard-pressed to uphold their claims of success-or defend against others� claims of our failures.

How can we accurately determine if we are truly helping our clients? One approach is to ask the opinions of the clients who clearly have an important perspective on our work. However, there are serious limitations to client assessments of the social worker�s perfor- mance. Clients may not have an accurate basis of comparison to other service providers,

* School of Social Work, Colorado State University, 119 Education Building, Fort Collins, CO 80523, USA, Tel.: (970) 4915654, E-mail: [email protected].

B.W. Sheafor / Measuring Effectiveness in Direct Social Work Practice26

may base their judgments or disliking the social worker as opposed to assessing his or her competence in addressing the issues, and the client�s assessment may be subject to manipu- lation as social workers often are in a position to reward or punish clients.

Another approach to worker assessment is to depend on supervisors or managers in our employing organization who may regularly evaluate our work and base employment reten- tion, salary increases, and promotions on the outcome. Yet, if these assessments are to be objective the organization must develop protocols for the evaluation that may minimize creativity and be more focused on efficiency (lowest cost for a unit of service) than effectiveness (client improvement). Social work practice, then, becomes shaped by the protocol and the activities rewarded by the agency, yet may have limited relevance to the clients being served.

Also, social workers should be responsible for assessing their own work through their informed professional judgments. A downside of self-assessment is that for most social workers the motivation for entering a helping profession is a passion for contributing to the improve- ment of the quality of life for the people served. With that strong bias, the tendency is for social workers to overestimate the improvement of clients and/or their social functioning.

How, then, can social workers be responsive to the demands to produce objective evaluations of their practice effectiveness and still practice with the flexibility of using professional judgment to best serve clients? One approach is to focus the measurement on client change, rather than on direct judgments about the worker, at least as one important indicator of practice effectiveness.

Evidence-Based Practice

The growing demand for documentation of practice effectiveness is part of a current movement of social workers and other professionals. This involves basing practice deci- sions on systematic reviews of practice effectiveness and is traced by an English physician, Thomas Beddoes, who in 1808 argued for the systematic indexing of medical facts and increasing the publication of scientific research (Goodman, 2003). The popular term for this movement, �evidence-based practice�, was introduced in the medical profession in Great Britain in the 1990s (Guyatt et al., 1992) and then spread to other helping profes- sions. Sometimes expressed under different terms (e.g., empirically supported treatment, research-based practice), the basic intent of evidence-based practice is that the art of social work practice should be combined with the science of carefully researched outcomes. Gambrill (1999) has helpfully distinguished the difference in these approaches. She depicts evidence-based practice as interventions where the social worker initially informs his or her practice decisions from the professional literature versus authority-based practice in which the authority of the individual social worker�s knowledge and judgment is the primary source of practice decisions.

In the United States, for example, evidence-based practice has become institutionalized in the Educational Policy Statement associated with accreditation standards of the Council on Social Work Education (CSWE, 2008). These standards require schools of social work to demonstrate that they prepare their graduates with the competency to �engage in research- -informed practice and practice-informed research� (CSWE Competency 2.1.6). Further, the National Association of Social Workers (NASW) maintains a website as part of its Social Work Policy Institute that keeps an updated comprehensive list of evidence-based practice registries and databases (http://www.socialworkpolicy.org/research/evidence-based-practice-2. html#EVP). Internationally, the highly respected work of the Campbell Collaboration

Revista de Asisten]\ Social\, nr. 1/2011 27

(http://www.campbellcollaboration.org/frontend.aspx) provides evidence of effective in- tervention approaches in specific practice situations that have undergone evaluation through rigorous protocols. The United Kingdom�s Social Care Institute for Excellence also pro- vides a very useful database at http://www.scie.org.uk.

For many social workers the term evidence-based practice has become a concept used in so many contexts that its meaning has been diminished. It is useful to understand the focus of evidence-based practice in three different phases of the social change process: assessment, intervention, and evaluation.

Evidence-Based Assessment: At the root of effective work with clients is careful assess- ment of the client and the client situation that is the purpose of the social work practice.

We have considerable well-researched evidence about human growth and development, social interaction, family functioning, and the cultural uniqueness of different population groups to aid in understanding the clients with whom we work. We are also having considerable information about many of the social issues our clients� experience (e.g., poverty, child protection, discrimination, mental illness, physical disability, aging). This evidence has been generated primarily through traditional forms of research based on application of the scientific method. Social workers clearly must be skilled in evaluating the literature reported in this research and able to make judgments about its applicability to the specific clients we are serving.

Evidence-Based Intervention: We are now beginning to develop evidence about the relative effectiveness of different intervention strategies or approaches in work with specific client groups or related to specific issues these clients are experiencing. This �best practices� research is largely based on meta-analysis research and suggests to social workers the services that are likely to be most effective is specific practice situations. While not offering conclusive evidence that any intervention approach(es) will be successful, this evidence clearly helps the social worker to give serious consideration to using the documented best practices.

Evidence-Based Evaluation: Once the intervention has begun an entirely different form of evidence-based practice can be used to monitor the client�s progress (or lack thereof) and to draw final conclusions about the change experienced by the client when the work is terminated. As opposed to research based on the scientific method that is based on carefully selected control and experimental groups and the application of (sometimes) sophisticated statistical analysis, direct practice evaluation is largely focused on tracking change during the time of the intervention in the important conditions affecting the specific individual, family, or small group being served by the social worker. This approach is in its �infancy�, or perhaps �early childhood�, but shows promise for the social worker documenting for clients, supervisors, and especially for himself or herself the effectiveness of services delivered to clients.

Pros and Cons of Evidence-Based Direct Service Evaluation

The emergence of evidence-based evaluation for direct practice has not been without controversy. On one side of the argument are those social workers who consider the work of this profession to be primarily an art form based on the worker�s natural abilities for building helping relationships and drawing on intuition and practice experience or practice wisdom. At the other extreme are those who view social work as requiring much more science at its base and contend that the direct practice evaluation tools are too primitive to provide useful data. Both views are partially valid, yet neither perspective helps to answer the larger concerns about the need to accurately assess our practice outcomes.

B.W. Sheafor / Measuring Effectiveness in Direct Social Work Practice28

More specifically, the ongoing arguments about the merits and demerits of empirical direct practice evaluation centred around the following issues.

� Depersonalization of the client. When specific client conditions are measured (e.g., depression, self-esteem, family relations) the focus of practice moves to those factors being measured and the client as a �whole person� is neglected and the social worker�s artistic expression in practice is minimized. Countering that view is the perspective that clients are involved with social workers to address specific issues and practice should be focused on addressing those issues and the best evidence (science) available should be brought to bear on those issues.

� Empirical evaluation is time consuming. One position argues that social workers invari- ably have heavy caseloads and time spent in activities other than face-to-face interaction with clients that deprives clients of a needed resource. Especially when beginning to use these evaluation tools it takes time to construct the study designs and collect data, although that time commitment decreases with practice. However, if a social worker learns what works and doesn�t work with his or her clients, in time he or she becomes both more efficient and effective thus making the up-front investment of time payoff.

� The practitioner/scientist dilemma. Some question the utility of expecting the social worker to simultaneously be both a practitioner and a scientist. Indeed, the thought processes required for temporarily entering the lives of others to develop empathy, inspire hopefulness that change can occur, and achieve other characteristics of effective helping relationships is different than the need for the researcher to stand back and objectively observe the change that is occurring. Others contend that developing empiri- cal evaluations of practice does not require �rocket science� and that professionally educated social workers are surely capable of being both caring with clients and being objective in their evaluations. In fact, social workers are expected to be both practition- ers and scientists � whether depending on their traditional impressionistic assessments or increasing the objectivity through empirical assessments.

� Limitations of the evaluation tools. Some social scientists contend that direct practice evaluation is at such a preliminary stage of development that it risks basing important practice decisions on faulty evidence that is not powerful enough to establish a cause and effect relationship between the intervention and the client�s situation. Few would argue that measuring change in one client or client group can yield information generalizable to all other clients, or that it is possible to control for enough external variables in the client�s life, or that the measurements obtained are completely accurate representations of the factors being addressed. We can, however, identify an association between the initiation of our intervention and change in the client�s condition as partial evidence of the effectiveness of practice. When assessing client outcomes we can provide a layer protection to the process if empirical evaluations are considered only one vantage point for judging our clients� change. A useful concept for this check and balance, triangula- tion, is borrowed from the field of surveying and simply suggests bringing at least three different perspectives to the judgment. In addition to the empirical evidence, for exam- ple, the perspectives of the client, the client�s family members, the client�s associates such as teachers or employment colleagues, the social worker and/or the worker�s supervisor, and so on should be considered. When these perspectives are in alignment, the empirical data can be more trusted and provide more specific information on degrees of change.

Revista de Asisten]\ Social\, nr. 1/2011 29

Steps in Conducting Empirical Direct Practice Evaluation

Many of the actions taken in direct practice evaluation are identical to what one does in practice when not undergoing this evaluation. The primary differences are in formalizing questions for which the answers will inform one�s practice, using numerical measures of change in the relevant client conditions, organizing the measurements in a format that will facilitate interpretation of the data, and adding the empirical evidence to other perspectives (i.e., triangulation) of the client�s situation. The following steps capture the process.

Step 1: Analyze the practice situation. Practice begins with understanding the client and the client�s situation. As indicated above, the particular client must be understood in the context of his or her culture, gender, age, sexual orientation, family constellation, and other factors relevant to the practice situation. The practice situation to be addressed, too, must be fully understood and related to evidence about that condition as found in the evidence-based literature. These activities should occur regardless of the form of evaluation to be used.

Step 2: Generate Research Question(s). Sound empirical evaluation forces the social worker to be clear about the practice questions to be answered through the empirical research and the purpose for asking those questions. Usually this is to either monitor what is transpiring in the practice to inform what the worker does with the client being served (i.e., formative research) or to provide a summary at the point of termination to report the changes the client has made and to inform work with future clients (i.e., summative research).

Some questions relate to the client(s) being served. For example, �To what extent does Mrs. M�s level of depression change when the services of a Hospice worker are provided?� Or, �To what extent do the Johnson�s family relations change when Jack (a foster child) is placed in the home?� Or possibly, �To what extent do members of a group perceive that the outcomes they considered important were met through the group experience?�

Other questions might be framed in a way to make judgments about the effectiveness of different intervention approaches or combinations of approaches. For example, �To what extent is the cognitive behavioural approach effective in working with Mary in relation to strengthening her self-esteem?� �To what extent is combining the cognitive behavioural approach with an assertiveness training group effective in strengthening Mary�s self-es- teem?� Or, �To what extent were the goals that Steven, his teacher, and the social worker established for the intervention attained?�

Finally, questions may relate to identifying the social worker�s areas of strength and weakness. A worker might ask �To what extent do my clients perceive that they have achieved their goals for each of the ten practice outcomes expected to be addressed in this unit of the hospital?� Or, a supervisor might ask �As viewed by clients, to what extent is each of the social workers on my unit successful in helping clients achieve each of the ten outcomes the workers are assigned to address?�

Step 3: Select tools for measuring client change. Perhaps the most difficult part of empirical practice evaluation is measurement. Some factors social workers address are tangible and already in a numerical format. For many other client conditions, however, we must have people rate the degree of pain, emotion, problem, or other pertinent factors and those ratings must reflect at least ordinal-level (ranking) data. These measurements are found in three distinct formats.

� Frequency counts. Often numerical data are already maintained in agency records such as a school�s count of the number of times a child is tardy for his/her classes, or police

B.W. Sheafor / Measuring Effectiveness in Direct Social Work Practice30

reports of the number of domestic violence calls to a household. If data are not already available, we may keep track of indicators of the issue being addressed or we may help clients keep such records. Keeping track of events such as arguments among siblings or bedwetting episodes for an older adult in a journal or on a calendar can provide useful information of patterns or trends. So long as the data are truthful and consistently collected, frequency counts can be accurate indicators of the intensity or duration of a problem.

� Individualized scales. With a little practice, it is not difficult for a social worker to construct scales that measure an important factor in the practice activity. Some factors are best expressed in the client�s own words so that when the degree of emotion or frequency of feelings is measured over time, the client�s reference point is revisited and the measure- ments are consistent (Nugent, Sieppert and Hudson, 2001; Bloom, Fischer and Orem, 2009). This �face validity� strengthens the usefulness of the measurements. The terms that anchor the numerical scores on each scale can also be indicated by some language that is commonly accepted in a culture. The essential characteristic of these anchor points is that they provide at least ordinal data (i.e., ranking) where each term captures a progression of the amount of the factor being rated. The anchor points may also be symbols (e.g., for children or people who are very ill � smiley/frowny � faces with a progression of expres- sions), thermometers showing amounts of the factor, or numbers-although numbers (e.g., �Rate your degree of anger between 1 and 10�) without anchoring terms or symbols tend not to be very accurate.

Typically three to seven anchor points on any scale are identified. Two factors should be considered when establishing the number of anchor points. First, how many degrees of difference in the factor can the respondent accurately differentiate? For example, children, older adults, and people with a mental disability may have difficulty in discriminating between more than three or four points on a scale, while others may have greater ability to accurately identify differences. In general, more valid points on a scale yield a greater amount of data to help identify change. Second, should there be an odd or even number of points on the scale? Odd number scales are used most frequently, yet they have the limitation of the respondent selecting the midpoint and not really considering if he or she leans one direction or the other on that factor. An even number scale forces a choice. When creating a scale, the number of anchor points should depend on nature of the factor(s) being assessed.

� Standardized scales. At the most sophisticated level, a surprisingly large set of carefully developed self-rating scales has been developed in relation to many factors that social workers and their clients address (Corcoran and Fischer, 2009; Hudson, 1997). Through rigorous psychometric testing the factor to be measured is isolated from other factors, usually 20 to 25 simple questions are selected to represent dimensions expressions of the concept, and an appropriate rating scale with anchor points created. Through testing with various population groups the reliability and validity of the scale is established, the standard error or measurement to be used in interpreting the amount of instrument error when analyzing results determined, and, in some cases, cutting scores are established indicating when clinical intervention is likely to be needed and when the respondent is in severe crisis. These are useful assessment tools and, in addition, repeated application can yield scores that track changes in the factor or condition the client experiences.

Step 4: Select an appropriate tool for organizing the measurements. The question(s) asked in Step 2 will, to some degree, determine the format for data organization. The social worker equipped to select any of the following formats should be able to compile measure- ments for meaningful monitoring of the change during the intervention or summing up the

Revista de Asisten]\ Social\, nr. 1/2011 31

change at the point of termination. (Note: A more complete description of these four tools with examples of their application may be found in recent editions of Sheafor)

� Service Planning Outcome Checklist (SPOC). The SPOC involves the use of a check- list at the initiation of service on which clients are asked to identify items on a menu of possible goals or outcomes that might be addressed. The menu is created as a list of the services an agency or a social worker typically offers. Clients are asked to mark the items they would like to address with the social worker and then to select the two or three highest priority items. While this format can be used with a single client, it is most useful when applied to groups of clients. With client groups, the percent marking each item he or she wants to address can be identified and the percent selecting each item as a priority item can also be determined. By adding together the two percentages an Importance Index is constructed and the items on the menu can be organized in order of their importance to the group.

When items are ordered by importance, the Importance Index becomes a useful tool for determining what to emphasize in the group sessions or identifying what clients view as the most important issues for them to address. At the point of termination the clients are again given the menu and asked to rate (usually on a five or seven point scale) the degree to which they believe they achieved each menu item. A mean score for achievement, when compared to the Importance Index, readily identifies areas where the clients believe they were or were not successful. If given to all of a social worker�s clients for a period of time, this information can lead the worker to evaluate where he or she needs to strengthen his or her practice competence.

� Task Achievement Scaling (TAS). Particularly when adopting the task-centred ap- proach (Epstein and Brown, 2002), but also when using other practice approaches, social workers and clients may agree on tasks or specific activities to be completed between sessions. By creating five-step scales of increasing success in completing each agreed upon task, it is possible to measure success in achieving each task. For example, if a client reaches step three (of five) in the task of obtaining rent-subsidized housing, he or she has achieved 60 percent of the task. When the client actually moves into the housing (if that was the final task on the scale), he or she would have achieved 100 percent of the task. Assuming that more than one task is being addressed at any one time, a percentage of overall task accomplishment can be computed as an indicator of the client�s overall task achievement.

� Goal Attainment Scaling (GAS). As compared to the short-term nature of tasks, social work practice might be focused on achieving long-term goals. A measure of practice success, then, is the degree to which the client attains the identified goals. Typically a practice situation involves an effort to address three to five goals. For example, working with a child experiencing problems at school might reflect a goal of enhancing self-esteem, reducing anger outbursts with peers, and decreasing the frequency of discipline referrals. The child�s self-esteem might be measured by a standardized or individualized scale, while the other two goals could be reflected in numerical counts of anger outbursts and discipline referrals recorded by the client and/or teachers. In Goal Attainment Scaling these factors are measured early in the helping process and again when service is terminated with five-point scales reflecting change for each goal (Bloom, Fischer and Orem, 2009; Kiresuk, Smith and Cardillo, 1994).

Recognizing that all goals do not make an equal contribution to client success, Goal Attainment Scaling incorporates weighting the importance of each goal to overall success. Thus if a student�s self-esteem is viewed as critical for improving his or her relations with other students or avoiding discipline issues, the weights might be 50 for change in

B.W. Sheafor / Measuring Effectiveness in Direct Social Work Practice32

self-esteem and 25 for each of the other two goals. Then, when the overall goal attainment score is computed, the actual weighted change can be compared to the possible weighted change and an overall percent of change is completed reflecting the differential importance of the goals.

� Single-Subject Designs (SSD). The most well-known and most versatile of the tools for organizing measurement data are the single-subject designs (also know as time-series designs, single-case designs, or single-system designs). This format allows the social worker to visually track periodic measurements of one or more factors being addressed in a practice situation and, in some cases, to perform statistical analysis of changes (Bloom, Fischer and Orem, 2009). If anticipated change is not occurring, the social worker and client can use this information to determine if the service should be continued or if a different intervention approach should be used. Or, if the change is progressing in the desired direction, to reinforce what is being done and encourage the client to continue with the intervention.

Typically social work practice begins with the identification of factors in the client�s life that the intervention is intended to help change. By developing baseline measurements of those factors, change can be traced by conducting measurements of those variables on a periodic basis. This baseline score is charted on the y-axis of a grid and across the bottom (x-axis) the dates the measurement was taken are noted. Thus by connecting the points of measurement, a simple line-graph of the variable(s) is created.

The usual procedure is to designate the baseline score(s) as the A phase of the change process, then labelling the measurements taken while each intervention approach is being used as the B, C, D, and so on phases. If more than one intervention is used at a time, the combination can also be identified. For example, if a cognitive-behavioural approach (B intervention) is the initial intervention and the client later also enters an anger manage- ment group (C intervention), while the two interventions are simultaneous this would be graphed as the B/C phase. Strength in this form of evaluation is that the design can be adapted to follow what occurs in practice. The goals are to develop an empirical record of what change is occurring in important client variables and to determine if there is an association between desired change and a particular intervention approach of combination of approaches.

Conclusion

Social work and other human services professions are increasingly under pressure to demonstrate that what they do truly makes a difference for their clients. Our �professional judgment� assertions that we positively affect client outcomes may be correct, but clients, taxpayers, agency boards of directors, and insurance companies that contribute to paying the cost of our services rightfully demand more objective evidence. The call for �evidence-based evaluation� is often resisted by social workers and it implies inserting a scientific approach into our practice activities for measurement to occur when the social worker�s art of helping is critical to the helping process. Our challenge is to provide empirical evidence without compromising the quality of services we deliver to our clients.

Clearly, social work is in the early stages of developing tools for measurement of client change that are not terribly invasive or demanding of the client. In fact, these evaluation tools are often resisted more by the worker than the client. Just as most patients of physicians do not resist having their blood pressure measured on a routine basis and even want to know the results, so social workers� clients are also interested in the outcome of our empirical measures. Also, relatively simple tools now exist for organizing these measurements to

Revista de Asisten]\ Social\, nr. 1/2011 33

increase the ability to analyze the patterns of change. It is quite possible for the social worker to find a middle-ground between being a scientist and a practitioner in order to both be flexible in servicing clients and, at the same time, measuring client change.

References

Bloom, M., Fischer, J. and Orme, J.G. (2009) Evaluating practice: Guidelines for the account- able professional. 6th ed. Boston: Pearson/Allyn and Bacon.

Corcoran, K. and Fischer, J. (2009) Measures for clinical practice: A sourcebook. 4th ed. New York: Oxford University Press.

Council on Social Work Education (2008) Educational policy and accreditation standards. Alexandria, VA: CSWE.

Epstein, L. and Brown, L. (2002) Brief treatment and a new look at the task-centered approach. Boston: Allyn and Bacon.

Gambrill, E. (1999) Evidence-based practice: An alternative to authority-based practice. The Journal of Contemporary Human Services, 80, 341-350.

Goodman, K.W. (2003) Ethics and evidence-based medicine: Fallibility and responsibility in clinical science. New York: Cambridge University Press.

Guyatt, G., Cairns, J., Churchill, D. et al. (1992) Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, 268, 2420-2425.

Hudson, W.W. (1997) WALMYR assessment scales scoring manual. Tallahassee, FL: WALMYR Publishing. Also available at www.walmyr.com.

Kiresuk, T.J., Smith, A. and Cardillo, J.E. (1994) Goal attainment scaling: Applications, theory, and measurement. Hillsdale, NY: Erlbaum.

Nugent, W.R., Sieppert, J.D. and Hudson, W.W. (2001) Practice evaluation for the Twenty-First Century. Belmont, CA: Brooks/Cole.

Sheafor, B.W. and Horejsi, C.R. (2008) Techniques and guidelines for social work practice. 8th ed. Boston: Pearson/Allyn and Bacon.

Copyright of Social Work Review / Revista de Asistenta Sociala is the property of University of Bucharest,

Faculty of Sociology & Social Work with Polirom Publishing House and its content may not be copied or

emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.

However, users may print, download, or email articles for individual use.