2D1-09 - What are the hallmarks of a reliable and valid evaluation that shows change over time? How do you measure program effectiveness?

Program Evaluation Life Cycle

Evaluation is the systematic acquisition and assessment of information to provide useful feedback about a career development center's programs or services. The general goal of evaluations in career development is to provide "useful feedback" to a variety of audiences including center staff, internal and external interested parties, students, job seekers, adults in transition, community members, state and local leaders, federal entities, and other relevant constituencies that address accountability. Most often, feedback is perceived as "useful" if it aids in decision-making and making the programs and services more effective. Presently these strategies are used in a number ways to address efficacy, identify cost effectiveness, and move to adequately address the changing nature of work and careers.

Two Types of Evaluation

Formative Evaluation in Career Centers Summative Evaluation in Career Centers

Needs Assessment Determines who needs the career center programs and services, how great the need is for these, and what might work to meet the need at this time and place.

Outcome Evaluations Investigates whether the particular career program or web based career resource caused demonstrable effects on specifically defined and targeted program outcomes.

Evaluability Assessment Determines whether an evaluation is feasible and how the advisory committee of internal and external members can help shape its usefulness.

Impact Evaluation Broadly assesses the overall or net effects—intended or unintended—of the career program or web based career resources as a whole.

Structured Conceptualization Uses the advisory committee of internal and external members to define the career center programs or web based career resources, the target populations, and the possible outcomes of the programs and services.

Cost-effectiveness and Cost-benefit Analysis Addresses questions of efficiency by standardizing career center outcomes in terms of their dollar costs and values.

Implementation Evaluation Monitors the extent of which the career program aligns with the initial vision and goals.

Secondary Analysis Reexamines existing data on the career center to address new questions or use methods not previously employed by the career center leadership or staff.

Process Evaluation Considers the process of delivering the career center program or web based career information through technology, including alternative delivery (i.e. social networking, blogging, twitter).

Meta-analysis Integrates the outcome estimates from multiple studies to arrive at an overall or summary judgment on an evaluation question about the career center.

Phases of Evaluation

Phase One: Planning Phase

Formulation Defining the major objectives, goals, and hypotheses of the career center program or web based center.

Conceptualization Operationalizing the major components of the evaluation to be done of the career center – the career programs, participants, settings these are used in, and measures to be used to see if these are effective.

Detailing How the components of the evaluation will be coordinated. For example, what's the timeline for evaluation? Will all programs be queried or just some? Who will conduct the evaluations from the career center, or will there be an outside source of help? Is training needed for career center staff or administrative leadership in preparation for the evaluation?

Evaluation Assessment of the alternatives and the selection of the best one; and the implementation of the selected alternative.

Phase Two: Evaluation Phase

Formulation Create major objectives, goals, and hypotheses of the career center program or web based career resources/technology.

Two Types of Evaluation

Conceptualization Operationalization of the major components of the evaluation—the career programs, participants, settings, and measures (what you are trying to capture).

Design Who will design the evaluation? How will you validate it? What questions do you want to ask? What questions would external entities need to know?

Analysis Qualitative and quantitative. How will you write the implications? Conclusions? Areas of future study? Next steps in the career center's movement once this information is available?

Utilization How will you use the evaluation results? What are the next steps in the career center's movement having this data and information? Who will it be shared with internally and externally?

L i c e n s e d u n d e r a C r e a t i v e C o m m o n s A t t r i b u t i o n 3 . 0 L i c e n s e .