Evaluation Plan for Youthworks ML2

USCB HMSV 2016 Program Evaluation Plan Page 1 of 6

DEPARTMENT OF SOCIAL SCIENCES Human Services Program

PROGRAM EVALUATION PLAN

TABLE OF CONTENTS

I. DESCRIPTION & PURPOSE

Background

Purpose

Ethical Considerations

Evaluation Type

Stakeholders

II. EVALUATION DESIGN

Key Evaluation Questions

Data Collection & Analysis Methods

Limitations of Evaluation Design

Design Matrix

III. MANAGEMENT & MONITORING

Plan Implementation Timeline

USCB HMSV 2016 Program Evaluation Plan Page 2 of 6

IV. DESCRIPTION & PURPOSE

Background The Human Services (HMSV) program at the University of South Carolina Beaufort (USCB) is an interdisciplinary liberal arts degree that draws heavily from the disciplines of psychology and sociology. Graduates of the HMSV program gain the attitudes, values, knowledge, and skills for employment or advanced study in the art and science of care. Program graduates will also be able to more effectively manage non-profit organizations, deliver direct care as a human service professional and have the skills to assist in building informal networks of care. The HMSV program at USCB is guided by the principles of the national Council on Standards for Human Services Education (CSHSE). A review of the standards for CSHSE accreditation revealed the need to improve the evaluation process for USCB’s HMSV program. The Department of Social Sciences identified comprehensive evaluation of the HMSV program as a departmental goal in 2012, and made steady progress toward achieving that goal in the subsequent two-year period. In 2014, the Department organized the HMSV Program Evaluation Taskforce to oversee the design of a CSHSE-compliant evaluation plan. The ongoing mission of the Taskforce is the execution, review and modification of the HMSV evaluation plan, and development and implementation of program improvement strategies. Purpose The primary purpose for the 2016 evaluation of the HMSV Program is accountability. USCB intends to submit an application for accreditation of its Human Services program and the findings from this evaluation will help demonstrate the extent to which the program is meeting its previously established goals and objectives. The outcomes from this evaluation will also allow USCB to demonstrate accountability to various stakeholders in the HMSV program; these individuals and entities are discussed in greater detail herein. The secondary purpose for the 2016 evaluation is program enhancement. The HMSV program has been in operation since fall 2004 and a number of positive outcomes have been achieved in the 11 years of the program’s existence. The program has seen significant increases in student enrollment rates, the creation of an online degree completion component, additional faculty, revamping of the Advisory Council, and expansion of the student-led Human Services Student Organization. Moreover, in 2015, USCB welcomed a new Chancellor. Now is an ideal time to take a step back and carefully consider the program in the context of changes happening within the larger USCB institution, as well as changes occurring within the regional and national Human Services sector. Ethical Considerations Ethical issues will be considered at each step in the HMSV program evaluation process. This will result in an evaluation that benefits students and other stakeholders in USCB’s HMSV program, and builds the knowledge base of the Human Services profession. The

USCB HMSV 2016 Program Evaluation Plan Page 3 of 6

need for ethical considerations have been identified and will be included during implementation for the following areas of this evaluation: design, research questions, data collection and reporting. In terms of the evaluation design and key research questions, selected variables are clearly focused on the evaluation questions, measurement processes are appropriate and culturally sensitive, and participant selection methods will ensure diversity and inclusiveness throughout the evaluation. Participants will be informed of the voluntary nature of the evaluation, and appropriate steps will be followed to ensure privacy and maintain confidentiality of evaluation participant information. Ethical practices will be employed during the data collection and reporting for this evaluation. Data analysis will follow appropriate methods to ensure results of the evaluation are fairly represented. The evaluation findings will be presented in accurate and comprehensible reports; reporting formats will be selected in consideration of the relevant stakeholder groups. All stakeholder groups noted in this plan will be provided with results from the HMSV evaluation in order to generate informed discussion. Negative and problematic findings will be reported with sensitivity and tact. The aforementioned ethical considerations will also incorporate cultural competence practices. A culturally competent evaluation acknowledges that culture has an effect on communication, perceptions, and processing of information for stakeholders. Moreover, the PET shall abide by the code of ethics of the CSHSE as adopted in 1996, as well as the American Evaluation Association’s Guiding Principles for Evaluators revised in 2004. Evaluation Type The Department will conduct a comprehensive outcome evaluation for the HMSV program in 2016. An outcome evaluation considers the extent to which a program’s objectives are achieved. This type of evaluation is ideal for serving the purposes of providing feedback for decision making, improving program services to customers, and generating knowledge for the profession. The PET selected this evaluation type over the other options (needs assessment, process evaluation, or efficiency evaluation) due to the intended purpose of the evaluation at this stage in the HMSV program’s existence.

Stakeholders Several stakeholder groups will be essential to the successful implementation of this HMSV evaluation plan. The PET recognized the importance and value of engaging stakeholders early in the planning process. The following groups have provided input and will continue to be incorporated during the execution of this evaluation plan: HMSV students and alumni, faculty members, including adjunct faculty members, the HMSV Advisory Council, the Beaufort County Human Services Alliance and representatives of the regional HMSV sector, USCB’s Office of Institutional Effectiveness and Research, as well as the CSHSE and National Organization of Human Services.

USCB HMSV 2016 Program Evaluation Plan Page 4 of 6

V. EVALUATION DESIGN

Key Evaluation Questions The following is a list of primary research questions that will guide this HMSV evaluation plan:

1. How well is the HMSV program meeting its stated goals and objectives? 2. How well does the HMSV program prepare its graduates to work in the Human

Services sector? 3. How well is the HMSV program preparing its graduates to be Servant Leaders in

the Human Services sector? 4. How does the HMSV program’s online learning experience compare with the

program’s traditional classroom learning experience? 5. What are the most viable concentrations or minors for the HMSV program? 6. What new course offerings are needed for the HMSV program? 7. What is the potential benefit of the HS-BCP credential for HMSV graduates and

the local/regional Human Services sector?

Data Collection & Analysis Methods A mix of qualitative and quantitative data will be collected in order to answer the key evaluation questions. Secondary data sources include internal reports from 2012 through 2015, HMSV internship evaluation surveys, and graduate exit surveys. Two surveys will be deployed to collect additional primary data from all HMSV students and alumni, as well as a sample of organizations in the region that employ graduates of the HMSV program. This data will be supplemented with follow-up interviews and focus groups to further contextualize findings. Data will be analyzed using the Statistical Package for Social Sciences (SPSS). Descriptive statistics will be used to describe and understand the basic demographics as well as the education and employment outcomes of program graduates. Bivariate t- tests will be used to assess systematic differences among survey respondents and to explore possible correlates between research question variables. Interview and focus group data will be transcribed and analyzed using code sheets or NVIVO coding software, depending on the volume of qualitative data collected. Limitations of Evaluation Design This evaluation plan was designed in consideration of the Department’s and University’s finite resources. Reasonable efforts will be made to ensure a large enough population size is included in the evaluation. However, response rates may vary, and this will impact generalizability of evaluation findings. Also, self-selection bias may further limit the evaluation, particularly with regard to findings related to data collected from the student and employer surveys, focus groups and interviews. The design matrix on page 5 illustrates key evaluation questions, data sources, sampling and collection methods for the HMSV program evaluation plan.

USCB HMSV 2016 Program Evaluation Plan Page 5 of 6

U S C B H U M A N S E R V I C E S P R O G R A M E V A L U A T I O N D E S I G N M A T R I X

Key Evaluation Questions Data Type of Data Collection Experimental Design*

Source & Sampling Survey Individual

Interview

Focus

Group

Direct

Observation

Internal

Records

Case

Study

Pre-

Post-test

Time Series

1. Is the HMSV program meeting stated

goals and objectives?

HMSV IE-OA reports from 2012-13

thru 2014-15 X

2. How well does the HMSV program

prepare graduates to work in the

Human Services sector?

Internship evals. - HMSV graduates

2012-13 thru 2014-15; all HMSV

graduate exit survey responses;

sample of HMSV employers in the

region; sample of CSHSE-

accredited schools

X X X

X X

3. Is the HMSV program preparing

graduates to be Servant Leaders in the

Human Services sector?

Sample of HMSV employers in the

region; 2012-13 thru 2014-15

grads

X X

X

4. How does the HMSV program’s

online learning experience compare

with the program’s traditional

classroom learning experience?

All current HMSV majors;

X

X

5. What are the most viable

concentrations or minors for the HMSV

program?

All current HMSV majors; 2012-13

thru 2014-15 grads X

6. What new course offerings are

needed for the HMSV program?

All current HMSV majors; sample

of HMSV employers in the region X

7. What is the potential benefit of the

HS-BCP credential?

All current HMSV majors; all HMSV

graduates 2011-12 thru 2014-15;

sample of HMSV employers in the

region

X X

Draft Survey Links: HMSV Organizations/Employers (https://www.surveymonkey.com/r/USCBHMSV); HMSV Students (https://www.surveymonkey.com/r/HMSVstudent)

USCB HMSV 2016 Program Evaluation Plan Page 6 of 6

VI. MANAGEMENT & MONITORING

Plan Implementation Timeline The PET has primary responsibility for management and monitoring of plan execution. This plan will be implemented beginning January 2016, with final results and recommendations expected by December 2016, pending Advisory Council review and approval by the USCB Office of Institutional Effectiveness and Research. The following table outlines specific goals, objectives, tasks, resources and timeframes for plan implementation. Goals Objectives / Tasks Required Resources Month Status Involve stakeholders in the evaluation process

Host at least 2 stakeholder input sessions during evaluation plan development

HMSV Advisory Council; HMSV B302 Students

Aug. – Nov. 2015

Complete

Use stakeholder feedback in development of key evaluation questions and survey protocols; pilot test surveys

Program Evaluation Taskforce (PET); HMSV Student Organization (HSSO)

Aug. – Nov. 2015

Complete

Adhere to ethical standards for program evaluation

Submit study to USC Institutional Review Board (IRB)

PET; USC IRB online portal

Dec. 2015 – Jan. 2016

Complete

Address any IRB concerns; document final approval

PET; USC IRB online portal

Jan. 2016 Complete

Conduct effective data collection to address key questions

Deploy HMSV organization / employer survey

PET, Advisory Council, HMSV Alliance

Feb. 2016

Deploy HMSV student / alumni survey

PET; Institutional Effectiveness

Feb. 2016

Conduct appropriate data analysis to answer key questions

Analyze internal records and reports

PET; Institutional Effectiveness

Feb. – Mar. 2016

Analyze employer and student survey responses

PET; Institutional Effectiveness

Feb. – Mar. 2016

Triangulate findings with qualitative data

Conduct follow-up interviews & focus groups

PET Mar. – Apr. 2016

Analyze interview & focus group data

PET; Institutional Effectiveness

May 2016

Discussion initial findings PET May 2016 Accountability and program enhancement

Review findings; develop recommendations

PET; Advisory Council June – Aug. 2016

Prepare full evaluation report

PET Sept. 2016

Effectively disseminate findings to all stakeholders

Posting on USCB HMSV website; notices via email

PET; Administrative Assistant

Oct. 2016

Publish slide presentations of key evaluation findings

Program Evaluation Taskforce;

Oct. 2016

Conduct at least 2 stakeholder discussion sessions

HMSV Advisory Council; HMSV B302 Students; HSSO

Nov. – Dec. 2016

Enhance the HMSV Program

Develop plan for implementing recommendations

Program Evaluation Taskforce;

Nov. – Dec. 2016