8084 MD3 Dis

MEASURING WHAT MATTERS: EXERCISES IN DATA MANAGEMENT

EXERCISE 2: COLLECT

Revised

This document was originally developed with funds from Grant #90HC0003 and modified with funds from Grant #90HC0014 for the U.S. Department of Health and Human Services, Administration for Children and Families,

Office of Head Start, and Office of Child Care, by the National Center on Parent, Family, and Community Engagement. This resource may be duplicated for noncommercial uses without permission.

For more information about this resource, please contact us: [email protected] | 1-866-763-6481

Acknowledgments The National Center on Parent, Family, and Community Engagement would like to acknowledge

the leadership of the Harvard Family Research Project, with support from the Brazelton Touchpoints Center, in developing this resource. These organizations represent diverse roles, expertise, and

perspectives; their input and feedback were essential in creating this resource. We recognize and value the role of parents and programs in making a difference for children, families, and communities.

Suggested citation: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Head Start, National Center on Parent, Family,

and Community Engagement. (2020, Revised). Measuring What Matters: Exercises in Data Management—Exercise 2: Collect.

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 1

Measuring What Matters Exercise 2: Collect

The exercises in this series are organized to follow the Four Data Activities. Each of these exercises focuses on a specific activity:

• Prepare: Get ready for data collection by thinking about the program goals, objectives, services, and expected outcomes that you need to show the reach and impact of your work.

• Collect: Identify how to gather data that are useful and easy to interpret.

• Aggregate and Analyze: Learn ways to look at data to examine progress for families and your program.

• Use and Share: Understand the importance of sharing accurate data in appealing and accessible ways and how data can inform various aspects of programming.

Begin with the Prepare exercise and follow with Collect, Aggregate and Analyze, and Use and Share. There may be times when it is useful to revisit one of the Four Data Activities as you learn more about your program’s data and progress.

CollectPrepare

Use and

Share

Aggregate and

Analyze

The Four Data Activities to Support Family Progress Toward Positive Family Outcomes

Exercise 2 is about collecting data. Data collection requires thoughtful planning. This exercise highlights how programs can prepare for data collection, choose data-collection methods, and develop systems to gather data effectively.

In this exercise, you will follow a scenario about a fictional Head Start program that collects data related to the Family Well-being Outcome of the Head Start Parent, Family, and Community Engagement (PFCE) Framework.

You can use this exercise to:

• Understand the importance of data collection.

• Create a data-collection plan that applies to the baseline and continuing grant application process.

• Guide data collection to measure both the efforts and effects of your program’s services and activities.

How to Use Exercise 2:

On Your Own • Read the scenario, Collecting Family Outcomes Data. • Reflect on similarities with your program’s PFCE work. • Review the instructions for completing Tables 5, 6, and 7. • Complete Tables 5, 6, and 7 using information from your program.

With a Group • Share your answers to the prompts in the tables. • Work together to create a plan for applying the data concepts to

your program.

1

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect2

Introduction

The Head Start PFCE Framework is an organizational guide for collaboration among families and Head Start and Early Head Start programs, staff, and community service providers to promote positive, enduring outcomes for children and families. The Framework identifies equity, inclusiveness, cultural and linguistic responsiveness, and positive goal-oriented relationships as important drivers for these outcomes.

The PFCE Framework shows how family engagement strategies can be systemic, integrated, and comprehensive across services and systems in line with the Head Start Program Performance Standards.

PROGRAM FOUNDATIONS

PROGRAM IMPACT AREAS

CHILD OUTCOMES

Program Leadership

Professional Development

Continuous Learning and Quality Improvement

Program Environment

Family Partnerships

Teaching and Learning

Community Partnerships

Access and Continuity

Family Well-being

Positive Parent-Child Relationships

Families as Lifelong Educators

Families as Learners

Family Engagement in Transitions

Family Connections to Peers and Community

Families as Advocates and Leaders

Children are:

Safe

Healthy and well

Learning and developing

Engaged in positive relationships with family members, caregivers, and other children

Ready for school

Successful in school and life

FAMILY OUTCOMES

Equity, Inclusiveness, Cultural and Linguistic Responsiveness

Positive & Goal-Oriented Relationships

Parent and Family

In this resource, “parent” and

“family” refer to all adults who

interact with early childhood

systems in support of their child,

including biological, adoptive

and foster parents, pregnant

women and expectant families,

grandparents, legal and informal

guardians, and adult siblings.

Head Start Parent, Family, and Community Engagement Framework

You can use data to engage families and support progress toward one or more of the seven Family Outcomes of the PFCE Framework. You can also use data to track progress as your program sets goals and develops and implements plans within the five-year project period.

2

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 3

Learning Objectives

• Identify measures of effort and measures of effect to track progress related to expected Family Outcomes.

• Help your program staff align their goals, objectives, and services with related Family Outcomes.

• Identify methods to collect data about progress toward expected Family Outcomes.

Garden Street Head Start Program Scenario: Collecting Family Outcomes Data Preparing for Data Collection: Creating Goals and Objectives for the Continuing Grant Application

Garden Street Head Start is preparing its continuing grant application for the next five-year project period. Amelia Posada, the Family Services Manager, and the program planning team are working together to collect data and develop program goals and objectives.

First, they conducted a community needs assessment and a program-wide self-assessment. They learned the following: • Garden Street’s county ranked last in their state on quality-of-life measures. Eighty-three percent of the

child and adult population in the program’s census track were in poor physical health. • Nearly 65 percent of the children at Garden Street Head Start had a body mass index (BMI) above the

recommended range, indicating an increase in child obesity. • Families wanted more opportunities to connect with each other.

Results from a short health survey completed by families during the family partnership process revealed other worrisome patterns (see Figures 1, 2, and 3). • More than 70 percent of families reported not having access to healthy food choices. • Eighty-one percent of families reported not having time to prepare healthy meals. • Ninety percent of families reported getting less than 5 minutes of exercise a day.

Figure 1. Family Access to Healthy Food Choices

Do you feel that you and your family have access to healthy food choices?

3

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect4

Figure 2. Family Time to Prepare Healthy Meals

Figure 3. Family Exercise

Do you have time to prepare healthy meals for you and your family?

How much exercise do you get each day?

4

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 5

Based on this information, Garden Street Head Start decided to focus on Family Well-being, specifically family health, as a major program goal for the next 5 years. Amelia Posada and other family services managers knew that they would need to collaborate with community partners and families to work toward this goal.

Garden Street entered into a partnership with Healthy Clinic, one of the local community health centers. Several staff from the clinic joined the Garden Street’s health advisory committee. They decided to:

• Include members of the Latino and Spanish-speaking community to ensure that efforts are helpful and services are accessible to families.

• Create the Growing Healthy Together Team involving staff and families from both programs.

• Develop a Memorandum of Understanding (MOU) that outlined both agencies’ roles and responsibilities.

• Provide a variety of health-related services, including a workshop series for Garden Street families.

Table 1 shows the goal, objective, services (actions), and expected Family Outcomes that the planning team proposed in its application. The table identifies the measures of effort and measures of effect that align with the program’s expected outcomes related to Family Well-being. The team outlines these measures to guide the types of data to collect.

Measures of Effort and

Measures of Effect

Measures of effort and effect can be used to track progress.

Measures of effort count what and how much family programming is offered. They describe whether and to what extent activities were carried out as planned. For example, a measure of effort is a count of the number of parent education workshops or the number of families who participated in them. These measures show what was done, but do not tell you about the results of your services and activities.

Measures of effect record changes in knowledge or behavior as a result of the activity. They track whether your activities have made a difference. For example, a measure of effect shows how reading routines change in the home after (or as a result of) a family literacy intervention.

Both types of measures are important to help you understand whether your program is making progress partnering with families and reaching program goals or objectives that relate to Family Outcomes.

5

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect6

Table 1. Garden Street Head Start’s Baseline Application Information for the Growing Healthy Together Initiative

Goal Objective Services (Actions) Expected Outcome Measures

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What actions are we going to take?

Which expected outcomes relate to our goals, objectives, and services (actions)?

Which PFCE Family Outcome does this represent?

Measures of Effort: How much programming are we offering? Are we carrying out services (actions) as planned? Which expected outcomes relate to our goals, objectives, and services (actions)?

Measures of Effect: What difference is our program making? What are the changes in knowledge and behavior?

Garden Street Head Start, in collaboration with Healthy Clinic, will enhance the health and well-being of children and families.

Increase families’ knowledge and skills to promote health and well-being for the entire family by encouraging family participation in at least 6 of the 12 activities led by Healthy Clinic from Year 2 through Year 5.

Recruit and encourage families to participate in Healthy Clinic events and activities, and support Healthy Clinic in planning and leading events.

Families have the information, skills, and knowledge to promote their children’s and their own health and well-being.

(Family Well-being)

Effort: Number of Healthy Clinic events offered and the number of parents who attend the events.

Effect: Parents report having access to healthier foods, cooking healthier meals, and getting more exercise.

6

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 7

Garden Valley Data Collection: Year 1 Phase 1 Action Items (September through November) Focusing on goals, objectives, services (actions), expected outcomes, and measures

Garden Street Head Start chose the first half of Year 1 in the five-year project period to plan for its focus on family health and well-being. The Growing Healthy Together Team needed time to plan the events and activities that would support their goal and objective. As part of the planning process, the team reached out to local promotoras (Hispanic and Latino community members who specialize in health education) who could consult on relevant health issues and identify available resources in the community.

After several weekly meetings, the Growing Healthy Together Team developed a 12-week workshop series based on their goal and objective. The series would be offered over 6 months each program year. It would include 1) presentations by experts, 2) cooking courses, 3) family dance classes, and 4) group trips to local grocers. Activities informed by the traditions of participating families would be delivered in Spanish and English.

Based on feedback from parents on the planning team, the team decided to add an extra component to the program. Any family member who attended at least eight events, took an additional training course, and passed a test could become a certified community health advocate. This additional train-the-trainer component would help sustain the program after the project with Healthy Clinic ended. These community health advocates could then play a variety of roles, including:

1. Leading workshops in the community to complement the events conducted by Healthy Clinic

2. Consulting with small groups of families on health topics during support group meetings

3. Connecting with local restaurants and shops to encourage more healthy food choices

Amelia Posada was excited about the community health advocate certification. This model would help Garden Street Head Start address other Family Outcomes in addition to Family Well-being. (See Tips for Thinking about Family Outcomes to the right.) The Growing Healthy Together Team added two new objectives related to the new health advocate component. (See Objectives 2 and 3 in Table 2.)

Tips for Thinking About Family

Outcomes

Often you will find that your objective(s) will lead to more than one expected Family Outcome. For example, by becoming community health advocates and participating in health support groups and workshops, families at Green Street Head Start were able to:

• Improve their own health (Family Well-being)

• Further their education and skills (Families as Learners)

• Develop the competencies needed to be leaders (Families as Advocates and Leaders)

• Broaden their social networks and develop friendships with other families in the community (Family Connections to Peers and Community)

7

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect8

Table 2. Revised Goals, Objectives, Services, Expected Outcomes, and Measures for Garden Street Head Start

Goal Objectives Services (Actions) Expected Outcomes Measures

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What actions are we going to take? Which expected outcomes relate to our goals, objectives, and services (actions)?

(Which Family Outcome does this represent?)

Measures of Effort: How much programming are we offering? Are we carrying out services (actions) as planned?

Measures of Effect: What difference is our program making? What are the changes in knowledge and behavior?

Garden Street Head Start, in collaboration with Healthy Clinic, will enhance the health and well- being of children and families.

1. Increase families’ knowledge and skills to promote health and well-being for the entire family by encouraging family participation in at least 6 of the 12 activities led by Healthy Clinic from Year 2 through Year 5.

Recruit and encourage families to participate in Healthy Clinic events and activities, and support Healthy Clinic in planning and leading events.

Families have the information, skills, and knowledge to promote their children’s and their own health and well- being.

(Family Well-being)

Effort: Number of Healthy Clinic events offered and the number of parents who attend the events.

Effect: Parents report having access to healthier foods, cooking healthier meals, and getting more exercise.

2. Assist families to become certified community health advocates by supporting them in completing a 5-hour training course and passing a certification exam, with a percentage of families becoming certified each year.

Recruit parents who have participated in at least 8 of the above sessions to participate in a 5-hour training course led by Healthy Clinic and specifically designed to give parents additional skills to lead health workshops and support groups for other families.

Families become certified community health advocates.

(Family Well-being and Families as Learners)

Effort: Number of parents becoming certified trainers through a train-the-trainer model.

Effect: Parents who participate in the training course report increased health awareness and confidence in their abilities.

3. Support health advocates in leading at least 3 workshops in the community each year and in leading at least 5 health support groups each year in Years 3, 4, and 5.

Encourage parents to participate in 1) community-based workshops led by health advocates, and 2) support groups led by health advocates.

Support health advocates by providing resources, supervision, and encouragement as needed.

Families who participate in health advocate workshops and support groups build support systems to meet their goals and build connections to others in the community.

(Family Connection to Peers and Community)

Health advocates improve their parent leadership skills.

(Families as Advocates and Leaders)

Effort: Number of health advocate workshops offered and the number of parents who attend parent-led workshops and support groups.

Effect: Parents report an increase in the number of social contacts they have as a result of attending health advocacy workshops and support groups.

Effect: Health advocates demonstrate enhanced leadership skills by promoting collaboration among families during respectful and productive workshops and support groups.

8

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 9

Phase 2 Action Items (December through March) Choosing Data-Collection Methods

Next, the Growing Healthy Together Team focused on ways to collect data to show its progress toward Family Outcomes for the five-year project period. The program team needed to link data-collection instruments, such as surveys and observations, with their measures of effort and effect. Amelia took the lead in this process. She guided the group by focusing on four main points:

1. There are many different ways to collect data, and there are benefits and limitations to each.

2. We already collect a lot of data, and it is helpful to link new data collection to existing efforts.

3. It is useful to be creative and try different approaches.

4. Informal observations can yield valuable information.

1. There are many different ways to collect data, and there are benefits and limitations to each. Amelia shared a handout she developed about different ways to collect data. She grouped data-collection methods into four categories: 1) surveys and questionnaires, 2) interviews and focus groups, 3) observations, and 4) tests and assessments.

The group discussed the advantages and disadvantages of each method. For example, one of the biggest advantages of surveys is getting answers from large groups of people. Surveys also provide a general sense of how these groups perceive or feel about an issue. One disadvantage is that the information tends to lack detail.

Surveys and questionnaires also miss the reasons why people answered questions in the way they did. In contrast, interviews and focus groups help explain people’s experiences in detail. However, they require more time to conduct and analyze.

The group discussed the value of reviewing documents the program already has (e.g., counting the number of certificates completed). They also considered the benefits and drawbacks of collecting different health indicators (e.g., cholesterol levels, weight, blood pressure, etc.). Amelia also talked more generally about the Four R Approach to support family progress. This approach includes using data responsibly and respectfully and making sure that data are relevant and relationship-based.

Measures of Effort and

Measures of Effect

Measures of effort and effect can be used to track progress.

Measures of effort count what and how much family programming is offered. They describe whether and to what extent activities were carried out as planned. For example, a measure of effort is a count of the number of parent education workshops or the number of families who participate in them. These measures show what was done, but do not tell you about the results of your services and activities.

Measures of effect record changes in knowledge or behavior as a result of the activity. They track whether your activities have made a difference. For example, a measure of effect shows how reading routines change in the home after (or as a result of) a family literacy intervention.

Both types of measures are important to help you understand whether your program is making progress partnering with families and reaching program goals or objectives that relate to Family Outcomes.

9

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect10

No matter what data-collection method the team adopted, it would be important to use this approach to guide their work. For Amelia and her team, this meant translating all data-collection tools into Spanish, the predominant home language of families in the program. The team wanted to make sure that questions were culturally appropriate and kept the same meaning during the translation process. (For more ideas on how to engage families in the Four R Approach, see the sidebar on the left: How can you engage families in the data-planning and collection process?)

2. We already collect a lot of data, and it is helpful to link new data collection to existing efforts. Garden Street already collects a great deal of information about the children and families in the program. The team might need to develop some new data-collection tools, but they should be able to connect most of their new efforts to existing data-collection activities.

Using Table 3, the team documented what data was currently being collected. For each objective, the team counted the number of events, workshops, or support groups offered and how many families attended them (measures of effort).

The team members noted that they had several resources to build on. The team could adapt generic sign- in sheets from other family workshops offered throughout the year (e.g., family literacy workshops, parent meetings, etc.) and use them for new data collection. The program’s database system already had a field to track individual family attendance at events. Amelia could work with the software developer to customize fields specifically for the new events.

How can you engage families in

the data-planning and collection

process?

• Give families opportunities to be part of the data-collection processes.

• Ask families to write down their child’s interactions with books, toys, and materials in the home or classroom; and share these observations with teachers. Provide families with reflection journals, checklists, or guiding questions to capture this information.

• Include families in family, program, and community-wide data-collection activities whenever possible. Consider inviting families to lead parent focus groups, or train families in using classroom observation measures.

• When deciding what types of data to collect, encourage families to share what they would like to know about children, families, staff, relationships, classroom and home visitation, and community partners.

• Invite families to share what they like and don’t like about different data-collection methods.

• Explore new ways to collect data with families (e.g., storytelling, scrapbooking, photographing, videotaping) that will complement surveys and Program Information Reports (PIRs).

• Ask a small group of parents to help develop and translate surveys, interviews,

and focus group questions.

10

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 11

Table 3. Aligning Current and Additional Data-Collection Methods to Goals, Objectives, and Measures

Goal Objectives Measure Current Data-Collection

Methods

Additional Data-Collection

Methods Needed

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

Measures of Effort: How much programming are we offering? Are we carrying out services (actions) as planned?

Measures of Effect: What difference is our program making? What are the changes in knowledge and behavior?

What data does our program currently collect related to the goal and objectives? How do we collect the data?

What additional data- collection methods can our program use?

Garden Street Head Start, in collaboration with Healthy Clinic, will enhance the health and well- being of children and families.

1. Increase families’ knowledge and skills to promote health and well-being for the entire family by encouraging family participation in at least 6 of the 12 activities led by Healthy Clinic from Year 2 through Year 5.

Effort: Number of Healthy Clinic events offered and the number of parents who attend the events.

Sign-in sheets (entered into family workshop portal in the database management system).

N/A

Effect: Parents report having access to healthier foods, cooking healthier meals, and getting more exercise.

Family Partnership Agreement survey items related to accessing healthy foods, cooking healthy meals, and making time to exercise. Data is entered into a data management system.

Parent focus groups during self- assessment.

Family food and exercise diaries

2. Assist families to become certified community health advocates by supporting them in completing a 5-hour training course and passing a certification exam, with a percentage of families becoming certified each year.

Effort: Number of parents becoming certified trainers through a train-the-trainer model.

Sign-in sheets (entered into family workshop portal in the database management system).

Certificates earned (Document review)

Effect: Parents who participate in the training course report increased health awareness and confidence in their abilities.

N/A Post-session evaluation survey with items related to confidence

3. Support health advocates in leading at least 3 workshops in the community each year and in leading at least 5 health support groups each year in Years 3, 4, and 5.

Effort: Number of health advocate workshops offered and the number of parents who attend parent-led workshops and support groups.

Sign-in sheets (entered into family workshop portal in the database management system).

N/A

Effect: Parents report an increase in the number of social contacts they have as a result of attending health advocacy workshops and support groups.

Effect: Health advocates demonstrate enhanced leadership skills by promoting collaboration among families during respectful and productive workshops and support groups.

List of social contacts generated during community mapping exercise on initial home visit.

Observation tool to assess group dynamics

11

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect12

Turning Informal Observations

Into Structured Data Collection

The observations that Head Start and Early Head Start staff make every day can be a source of data if collected systematically. For example, a director whose office was near the center’s entrance noticed that, over the year, the discussions during arrival times among families increased. She felt this was evidence of the effectiveness of some of the program’s strategies.

To help make your daily observations more systematic:

• Begin by observing families. What patterns do you notice about how they talk, act, or behave?

• Plan to record your observations regularly (a few minutes each day at the same time and place, for example).

• Review your recorded observations regularly (e.g., weekly, monthly).

• Reflect on your observations and summarize them.

• Use the summaries in your program reports and in planning for strengthening your work with families.

3. It is useful to be creative and try different approaches. Using multiple data-collection methods would help paint a complete picture of Garden Street’s progress and areas for improvement. Amelia also stressed that by using more than one data-collection method the team could be more confident about their results. For example, families already responded to a series of questions about their access to healthy food and their exercise routines. The team could use that information as baseline data to track whether families were increasing their access to healthier foods, cooking healthier meals, and getting more exercise. (See Tables 1, 2, and 3.)

Amelia encouraged the team to think about additional data to help show progress. The group felt that it would be important to understand how families were changing their eating and exercise habits. They decided to ask families to keep a food and exercise journal at the beginning and end of the project. In return, the families would receive a basket of healthy food from a local grocery chain. The team members thought focus groups at the end of the series would help give them insight into the ways the activities and events changed families’ health routines.

4. Informal observations can yield valuable information. Sometimes even informal observations, if put together in an orderly way, can yield valuable information. For example, the group thought that families who participated in health advocate workshops and support groups would increase their social networks.

One parent suggested they use the data collected at home visits about families’ social networks. At the start of the year, during the initial home visit, classroom teachers completed a short community-mapping exercise with the family. After spending time learning more about families’ contacts and connections to the community, teachers asked families whom they rely on in times of an emergency (e.g., help with carpooling, needing a babysitter, etc.).

The same parent suggested that the teachers could reframe the question slightly by asking for a list of the names of people they would be comfortable calling in case of an emergency. This question would be repeated a year later. In this very informal way, they would be able to determine whether the number of contacts increased based on participation in a support group.

12

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 13

Phase 3 Action Items (April through June) Developing a Data-Collection Plan

After choosing data-collection methods, it was time for the Growing Healthy Together Team to start organizing the data-collection effort. They created a Data-Collection Plan (see Table 4) to address key data management issues, such as:

1. Responsibilities: Everyone had a role in data collection. Families and staff had different responsibilities related to collecting and entering data explained in the plan.

2. Schedule: It was important that everyone involved in the project knew when data would be collected. The plan included when and how often data would be collected and who would do it.

3. Training: Data collectors needed training. The team outlined when data collectors would be trained and noted who would do the training. Training involved guidance about how to collect information and how to enter data into the data-management system. Training also included a reminder on keeping information confidential and following with the program’s privacy guidelines.

4. Monitoring: Amelia would supervise the data-collection process. The team added a column to identify how and when she would monitor the data-collection efforts, including record keeping and reporting.

13

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect14

Goal Objectives Data-Collection Methods Staff Responsible Data-Collection Schedule Training Monitoring

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What data-collection tools will we use for the project?

How and where will data be kept?

Who will collect and enter the data?

When and how often will the data be collected?

When will they be trained and by whom?

How often will the Family Services Manager monitor the data-collection effort?

Garden Street Head Start, in collaboration with Healthy Clinic, will enhance the health and well-being of children and families.

Increase families’ knowledge and skills to promote health and well- being for the entire family by encouraging family participation in at least 6 of the 12 activities led by Healthy Clinic from Year 2 through Year 5.

Sign-in sheets (hard copy saved in training binder and data entered into a data management system)

Family Services Worker

October through April (after each event)

August by Family Services Manager

Bi-weekly during the 12-week series

Family Partnership Agreement survey items related to accessing healthy foods, cooking healthy meals, and making time to exercise (data entered into data management system)

Family Services Worker

September/October (during family partnership process) April/May (as a short follow-up on identified health items)

August and March by Family Services Manager

Weekly during data collection

Parent focus groups during self- assessment (taped and transcribed; transcription saved on the program’s server)

Parent Policy Council Leaders (data entered by an intern)

April (during self-assessment) March by Garden Street Head Start Team

May

Family food and exercise journals (data entered into an Excel worksheet on the program’s server)

Healthy Clinic Trainers (data entered by an intern)

October (1 week) and March (1 week)

September by Garden Street Head Start Team

Daily during diary collection

Assist families to become certified community health advocates by supporting them in completing a 5-hour training course and passing a certification exam, with a percentage of families becoming certified each year.

Sign-in sheets (hard copy saved in training binder and data entered into a data management system)

Family Services Worker

May (after training course) August by Family Services Man- ager

After training

Certificates (collected, counted, and entered into a data management system)

Family Services Manager

May/June (after tests complete) April by Family Services Manager

After training

Post-session evaluation survey with items related to confidence (entered into Excel and saved on the program’s server)

Healthy Clinic Trainer May (after training course) April by Family Services Manager

After training

Support health advocates in leading at least 3 work- shops in the community each year and in leading at least 5 health support groups each year in Years 3, 4, and 5.

Sign-in sheets (hard copy saved in health advocate binder)

Health Advocate After each session Record keeping is part of a training course

Quarterly during review of records

Count of social connections (kept with home visit folder in a child’s portfolio; entered into a data management system)

Teachers August (during initial home visits) June by Garden Street Head Start Team

During the file audit after home visits

Observation tool to assess group dynamics (entered into an Excel worksheet)

Family Services Manager (data entered by an intern)

Twice each year for each health advocate

Family Services Manager will attend training

Quarterly (by Director)

Table 4. Year 2 Data-Collection Plan

14

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 15

Goal Objectives Data-Collection Methods Staff Responsible Data-Collection Schedule Training Monitoring

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What data-collection tools will we use for the project?

How and where will data be kept?

Who will collect and enter the data?

When and how often will the data be collected?

When will they be trained and by whom?

How often will the Family Services Manager monitor the data-collection effort?

Garden Street Head Start, in collaboration with Healthy Clinic, will enhance the health and well-being of children and families.

Increase families’ knowledge and skills to promote health and well- being for the entire family by encouraging family participation in at least 6 of the 12 activities led by Healthy Clinic from Year 2 through Year 5.

Sign-in sheets (hard copy saved in training binder and data entered into a data management system)

Family Services Worker

October through April (after each event)

August by Family Services Manager

Bi-weekly during the 12-week series

Family Partnership Agreement survey items related to accessing healthy foods, cooking healthy meals, and making time to exercise (data entered into data management system)

Family Services Worker

September/October (during family partnership process) April/May (as a short follow-up on identified health items)

August and March by Family Services Manager

Weekly during data collection

Parent focus groups during self- assessment (taped and transcribed; transcription saved on the program’s server)

Parent Policy Council Leaders (data entered by an intern)

April (during self-assessment) March by Garden Street Head Start Team

May

Family food and exercise journals (data entered into an Excel worksheet on the program’s server)

Healthy Clinic Trainers (data entered by an intern)

October (1 week) and March (1 week)

September by Garden Street Head Start Team

Daily during diary collection

Assist families to become certified community health advocates by supporting them in completing a 5-hour training course and passing a certification exam, with a percentage of families becoming certified each year.

Sign-in sheets (hard copy saved in training binder and data entered into a data management system)

Family Services Worker

May (after training course) August by Family Services Man- ager

After training

Certificates (collected, counted, and entered into a data management system)

Family Services Manager

May/June (after tests complete) April by Family Services Manager

After training

Post-session evaluation survey with items related to confidence (entered into Excel and saved on the program’s server)

Healthy Clinic Trainer May (after training course) April by Family Services Manager

After training

Support health advocates in leading at least 3 work- shops in the community each year and in leading at least 5 health support groups each year in Years 3, 4, and 5.

Sign-in sheets (hard copy saved in health advocate binder)

Health Advocate After each session Record keeping is part of a training course

Quarterly during review of records

Count of social connections (kept with home visit folder in a child’s portfolio; entered into a data management system)

Teachers August (during initial home visits) June by Garden Street Head Start Team

During the file audit after home visits

Observation tool to assess group dynamics (entered into an Excel worksheet)

Family Services Manager (data entered by an intern)

Twice each year for each health advocate

Family Services Manager will attend training

Quarterly (by Director)

Implementing Data-Collection Plan: Year 2 After six months of planning, the group was ready to begin the data-collection process. The Healthy Clinic events started in October and were met with great enthusiasm and high participation rates. As data were collected, Amelia focused on monitoring the data-collection efforts and entering data when necessary.

1. Monitoring data collection. Amelia tried out a few different ways to monitor the data until she found a strategy that worked best for her. First, she hung a large calendar on the wall of her office. The calendar had the months of the project written across the top and all of the data-collection tasks in columns on the side. She color-coded the different data-collection methods so that she would be able to see them easily every day (e.g., the family partnership survey items were coded in red, the focus group in blue, etc. See Figure 4).

Amelia entered important deadlines into her email calendar. She set reminders when data-collection or monitoring deadlines were approaching. She worked with her program’s database developer to create a screen that would allow her to quickly see how many families had participated in events.

Data Collection For Year 2

September October November December January February March April May

W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4

Objective 1

Sign-in sheets from events X X X X X X X X X X X X

Family Partnership Assessment survey items X X X X X X X X X X

Journals completed X X X X

Focus groups X

Objective 2

Sign-in sheets X

Certificates collected X X

Health advocate evaluation sheets X

Figure 4. Data-Collection Timeline

15

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect16

2. Entering data. Most of the data were entered into the database management system by those who collected the information. But there were a few items that needed to be coded and entered into Excel spreadsheets. A graduate intern from Healthy Clinic entered data from 1) hand-written surveys that came in from the health advocacy training, and 2) information from families’ food and exercise journals.

Amelia reminded the intern about the program’s confidentiality policies. The intern entered the data into an Excel spreadsheet. Amelia double-checked what was on the spreadsheet with the paper journals. The intern also transcribed the recording of the parent focus group at the end of the year.

Amelia and the Growing Healthy Together Team felt proud of the work they had done to collect data. They reflected on their process and the following strategies:

• Using the Four R’s as a guide for honoring all families in data-collection, data use, and data storage

• Linking new data-collection to existing data-collection activities

• Choosing a mix of data-collection methods to paint a more complete picture of progress

• Creating a detailed data-collection plan

The team would spend the remainder of Year 2 aggregating and analyzing data. During that process, they would consider whether they needed to make adjustments to the program and whether additional data was needed to inform their decisions about ways to change.

16

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 17

Suggestions for Developing

Program Goals and Objectives

Program goals for the provision of services for families and children describe what a program intends to accomplish in its work with and in support of families.

Program goals should be BROAD: Bold, Responsive, Organization-wide, Aspirational, and Dynamic.

An objective expands on a goal. It describes what the program is intending to do to reach the goal.

Objectives should be SMART: Specific, Measurable, Attainable, Realistic, and Timely.

See the revised Foundations for Excellence: A Guide for Five- Year Planning and Continuous Improvement, 2nd Edition on the Head Start Early Childhood Learning and Knowledge Center (ECLKC) website for more information about goals, objectives, and expected outcomes.

Your Turn Now that you have read about Garden Street’s data-collection efforts, you can take the first steps in thinking about collecting data for your program.

Table 5 gives you an opportunity to develop goals, objectives, services (actions), expected outcomes, and measures of effort and effect for your work. 1. Review Table 5.

2. Identify one or more BROAD goals and SMART objectives related to expected program outcomes for families and children. Use the PFCE Framework to guide you.

3. Identify the services and strategies that align with or can be effective in reaching your objectives.

4. Relate your goals, objectives, and services (actions) to the expected outcomes.

5. Write ways that you will measure your program’s efforts and your program’s effects. Refer to the Garden Street Head Start’s Table 1 and Table 2 for inspiration.

Table 6 provides an opportunity for you to brainstorm your program’s current data-collection methods related to goals and objectives. It also includes space for you to write about additional data-collection methods you might adopt. 1. Review Table 6.

2. Copy your goals, objectives, and measures of effort and measures of effect over to Table 6.

3. Think about what current data-collection methods you have in place related to goals, objectives, expected outcomes, and measures. How are they collected? Refer to Garden Street Head Start’s Table 3 for guidance.

4. Reflect on your chart. Think about what additional data-collection methods you might need.

5. Consider how you will ensure quality in the data you collect. (See the text box titled Quality Control on Page 21).

Table 7 is the outline of a data-collection plan. 1. Review Table 7.

2. Copy your goals and objectives from Table 6 into Table 7, and write your chosen data-collection methods into the data-collection column.

3. Write the names of the staff who will be responsible for each point of data collection.

4. Think about your timeline and schedule for data collection. How often will the data be collected?

5. Consider how and when responsible staff will be trained. Who will train them?

6. Finally, decide how the effort will be monitored. How often will data-collection efforts be supervised, and how?

17

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect18

Table 5. Revised Goals, Objectives, Services, Expected Outcomes, and Measures

Goal Objective(s) Services (Actions) Expected Outcome(s) Measures What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What actions are we going to take?

Which expected outcomes relate to our goals, objectives, and services (actions)?

Which Family Outcome does this represent?

Measures of Effort: How much programming are we offering? Are we carrying out services (actions) as planned?

Measures of Effect: What difference is our program making? What are the changes in knowledge and behavior?

18

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 19

Table 6. Aligning Current and Additional Data-Collection Methods to Goals, Objectives, and Measures

Goal Objectives Measures Current Data-Collection Methods Additional Data-Collection

Methods Needed What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

Measures of Effort: How much programming are we offering? Are we carrying out services (actions) as planned?

Measures of Effect: What difference is our program making? What are the changes in knowledge and behavior?

What data does our program currently collect related to the goal and objectives? How do we collect the data?

What additional data-collection methods can our program use?

19

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect20

Goal Objectives Data-Collection Methods Staff Responsible Data-Collection Schedule Training Monitoring

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What data-collection tools will we use for the project? How and where will data be kept?

Who will collect and enter the data?

When and how often will the data be collected?

When will staff be trained in data collection and by whom?

How often will the data-collection efforts be monitored?

Table 7. Year 2 Data-Collection Plan

20

Measuring What Matters: Exercises in Data Management—Exercise 2: Collect 21

Goal Objectives Data-Collection Methods Staff Responsible Data-Collection Schedule Training Monitoring

What program goal does our program want to accomplish?

What are we planning to do to reach our program goal?

What data-collection tools will we use for the project? How and where will data be kept?

Who will collect and enter the data?

When and how often will the data be collected?

When will staff be trained in data collection and by whom?

How often will the data-collection efforts be monitored?

QUALITY CONTROL: GOOD DATA COLLECTION MEANS FINDINGS YOU CAN TRUST

Here are some tips to consider to collect high-quality data:

1. Think of the group you want to learn about (your sample). Depending on your goal(s), you might want to narrow or expand your group. For example, you can decide to direct a survey to caregivers in general (parents, grandparents, and other relatives who provide care), or you can survey parents only. Consider these key questions: What information do you want to get? Who do you want to get the information from? What group do you want to learn about?

2. The more the merrier. Regardless of the method you decide to use, keep in mind that you are aiming for as many responses as possible. This is particularly true for surveys and questionnaires. For example, if you are surveying parents, count the total number of parents, and then decide how many surveys you need to collect. A general rule of thumb is that a response rate below 30 percent may limit your ability to trust your findings. If using interviews or observations, this percentage is more flexible, but make sure that your sample represents the diversity of families in your program. A representative sample will allow you to understand how different groups are experiencing the issue you are exploring.

3. Incentives and follow-up. The higher your response rate, the better your results. You might want to think about how to reward the people who take the time to fill out the survey or attend the interview or focus group session. Small prizes, public recognition, or any other way you can recognize their effort is helpful. Also, follow-up calls or notes and regular reminders are effective ways to increase your number of responses. Follow-up efforts can increase your response rates significantly.

4. Think of reliability. Reliability means that the method you use measures the same thing, every time, with every group you want to study. It is essential that what you are trying to learn about, or measure, is captured consistently. In real life, this is often difficult. The variety of settings or data collectors can alter the consistency of the method or instrument used. You want to collect your data as consistently as possible. If surveying, make sure that surveys look the same for everyone and that they are all administered in the same setting (e.g., house, classroom, office). Make sure that the instructions are identical, as well. Similarly, if conducting interviews or focus groups, develop a protocol of questions ahead of time so the interviewer can stick to them. Training your data collectors is one of the best ways to ensure reliability. This will help avoid biases and distortions in your findings.

5. Consider how data-collection measures fit with the languages spoken by families in your program. You can only collect high-quality data if all families can participate fully.

21

Measuring What Matters: Exercises in Data Management—Exercise 2: CollectMeasuring What Matters: Exercises in Data Management—Exercise 2: Collect22

Closing Thoughts

The second activity in data management is data collection. Data collection includes:

• Getting clear about the goals, objectives, and expected outcomes of your work.

• Developing measures of effort and measures of effect to track information and know what data to collect.

• Choosing different data-collection methods and tools.

Data collection means developing a plan for who will collect and enter data, when it will be collected, when data collectors will be trained, and how the effort will be monitored. Above all, data collection is about gathering information in respectful and responsible ways.

Now that you have completed the steps in this exercise you can proceed to Exercise 3, Aggregate and Analyze. In this exercise, you will learn how to combine (aggregate) and separate (disaggregate) data across sites, services, and expected Family Outcomes related to the PFCE Framework. You will also explore how to examine data across time.

Are You Interested in Learning More About Using Data to Support Family Progress?

Explore other Measuring What Matters Resources on the Head Start Early Childhood Learning and Knowledge Center (ECLKC) website:

• Overview

• Exercises in Data Management Series

• Resource Guide

Related Resources

Foundations for Excellence: A Guide for Five-Year Planning and Continuous Improvement, 2nd Edition

Integrating Strategies for Program Progress

Strategies for Implementing the Head Start Parent, Family, and Community Engagement Framework

22

1. table5_goal_1:
2. obj_1:
3. service_action_1:
4. expected_outcome_1:
5. measures_1:
6. obj_2:
7. service_action_2:
8. expected_outcome_2:
9. measures_2:
10. obj_3:
11. service_action_3:
12. expected_outcome_3:
13. measures_3:
14. measures_4:
15. measures_5:
16. measures_6:
17. table6_goal_1:
18. objectives_1:
19. table6_measures_1:
20. current_methods_1:
21. additional_methods_1:
22. objectives_2:
23. table6_measures_2:
24. current_methods_2:
25. additional_methods_2:
26. objectives_3:
27. table6_measures_3:
28. current_methods_3:
29. additional_methods_3:
30. table6_measures_4:
31. current_methods_4:
32. additional_methods_4:
33. table6_measures_5:
34. current_methods_5:
35. additional_methods_5:
36. table6_measures_6:
37. current_methods_6:
38. additional_methods_6:
39. table7_goal_1:
40. table7_obj_1:
41. collection_method_1:
42. staff_responsible_1:
43. collection_schedule_1:
44. training_1:
45. monitoring_1:
46. table7_obj_2:
47. table7_obj_3:
48. collection_method_2:
49. staff_responsible_2:
50. collection_schedule_2:
51. training_2:
52. monitoring_2:
53. collection_method_3:
54. staff_responsible_3:
55. collection_schedule_3:
56. training_3:
57. monitoring_3:
58. collection_method_4:
59. staff_responsible_4:
60. collection_schedule_4:
61. training_4:
62. monitoring_4:
63. collection_method_5:
64. staff_responsible_5:
65. collection_schedule_5:
66. training_5:
67. monitoring_5:
68. collection_method_6:
69. staff_responsible_6:
70. collection_schedule_6:
71. training_6:
72. monitoring_6:
73. collection_method_7:
74. staff_responsible_7:
75. collection_schedule_7:
76. training_7:
77. monitoring_7:
78. collection_method_8:
79. staff_responsible_8:
80. collection_schedule_8:
81. training_8:
82. monitoring_8:
83. collection_method_9:
84. staff_responsible_9:
85. collection_schedule_9:
86. training_9:
87. monitoring_9:
88. collection_method_10:
89. staff_responsible_10:
90. collection_schedule_10:
91. training_10:
92. monitoring_10: