Fieldstone Alliance Logo
Fieldstone Alliance: Tools You Can Use e-newsletter
Tools You Can Use

Getting the Most from Evaluation

Resource
Information Gold Mine: Innovative Uses of Evaluation

Contents
How One Organization Used Its Findings
    Service improvements
    Policy and marketing uses
The Numbers Added Up to Real Change

 

From Vince Hyman, former Publishing Director, Fieldstone Alliance:

MOST NONPROFITS—especially those in the social services—undergo evaluation at some point in their history. Traditional use of evaluation—for improving service quality—is well known. However, nonprofits have only scratched the surface when it comes to using evaluation findings as a tool for public relations, educating consumers, influencing policy, and boosting staff morale.

Paul Mattessich, PhD, is director of Wilder Research, one of the nation’s premier evaluation organizations. He is lead author on three of our books. Actually, make that four, as of June 2007. His just-released book is titled Information Gold Mine: Innovative Uses of Evaluation.

Paul is a long proponent of “mining” evaluation data for a host of uses beyond measuring service effectiveness. In true evaluator fashion, Paul and his staff interviewed 40 organizations to uncover ways that they were using evaluative information. They identified three large categories of evaluation use:

  1. To improve service (perhaps the most typical use)
  2. To influence policy and legislation
  3. To market programs

Of course, there are many varieties of use within each of these broad categories. To help show how one organization leveraged its evaluation findings in a number of ways, this issue of Tools You Can Use profiles, YouthZone a youth-serving organization in Colorado. The issue is adapted from chapter three of Information Gold Mine, pages 39-48. That chapter contains a more detailed profile of the organization.

How One Organization Used Its Findings
YouthZone, located in western Colorado, serves youth ages six to eighteen and their families. It delivers counseling, mentoring, life-skills education, court services, and other forms of prevention and intervention services. Started in 1976, the organization has the overall mission to promote the development of young people as responsible and contributing citizens.

Intervention Services, a project of the YouthZone organization, has goals to reduce delinquency, reduce substance abuse and use, improve self-concept, increase decision-making skills, and increase pro-social development. Intervention Services focuses on youth who have exhibited drug-use behaviors. As part of its effort to increase pro-social development, the program strives to keep youth connected to schools, families, and communities.

Debbie Wilde, executive director of YouthZone, is enthusiastic about what evaluation has done for her organization: “As an executive director, I am so glad that I had a backup for my stories about what our program does. Lots of programs have touchy-feely stories. We have those too, and we do good work, just like other programs. But I can tell you what we do and what impacts it has in the scientific language of this business. This ability has given me more confidence in being bold about talking with people who have financial resources, in doing public relations, and in being assertive in asking for referrals. I feel like I really have solid ground to stand on. Not just me, but my board as well. It’s really given us confidence in our organization.”

Most of YouthZone’s innovative uses of evaluation were in the categories of service improvement and marketing, with some use to influence policy improved funding of programs. Here are some of the many ways YouthZone used its evaluation findings.

Service improvements
Wilde notes that YouthZone’s first evaluation included a list of recommendations. “We took that list, methodically reviewed the service recommendations, and…followed through with some changes.”

These changes included:
Improving initial assessment. YouthZone used the data to begin identifying the “high-risk” kids. This enabled YouthZone  to more quickly focus on potential issues.

Staffing improvements. YouthZone saw that they needed some staff changes to work more effectively with certain kids, and that it needed to change some staff requirements. The agency also learned to give better attention to the high-risk kids, and doubled the number of supervisors looking at the high-risk kids.

Changing its program model. YouthZone’s evaluation revealed that with regards to its drug and alcohol programming, education was most effective. Another part of the model, public service, was not correlated with improvements. So YouthZone changed the “youth public service” part of its program. The resulting improvements led to other changes to its model related to self-concept and pro-social development; YouthZone began focusing more on getting kids better connected to family, school, and community. On its second evaluation, YouthZone saw significant improvement.

Improving post-service outcomes. YouthZone noticed that some of the students showed little or no improvement after going through the program. In Wilde’s words, “Part of what we identified through the evaluation is that kids aren’t done when they finish our program.” So, they instituted an exit interview that involved the parents, explaining to the parents the ongoing concerns that the parents would need to attend to. (Or, when a youth has done well, explaining the successes to the parents.)

Program infrastructure improvements. The evaluation also helped YouthZone recognize areas it was spending money that weren’t necessary. Recognizing that some parts of the model weren’t as effective as others helped them cease investing energy in ineffective actions.

Policy and marketing uses
YouthZone developed talking points for board and staff based on the evaluation. It used these both to influence policy and to market the program.  Wilde notes, “We were able to say ‘Look at this: kids aren’t re-offending again. Here’s the significant impact that we’ve made.” They used this message in meetings with local government, private donors, and foundations.

Policy use. Government funding of programs is a direct expression of policy. The ability to numerically demonstrate impact gives YouthZone an edge in seeking funding. Per Wilde, evaluation enabled YouthZone to say, “You should fund us. See the impact. Help us keep serving kids.” She continues,  “When the rubber meets the road, it’s the dollar. When things are tight people want to know about results. What is my dollar getting me? That’s where we really use evaluation heavily.”

Public relations use. YouthZone uses the evaluation to tell about the agency and to show that the agency is effective. Their access to good data has helped them move into circles that were otherwise unavailable. Wilde says, “For example, the Rotary Club—it’s really hard to get in to do a presentation. But we had another angle... we could report on our evaluation. We could do a ‘profile of youth’ in the valley. Because the sample is so large—with such a statistically valid instrument—we had a great analysis. We could apply it to a million things there. It really gave me an avenue to talk about what we’re doing at YouthZone, but I can also discuss what the kids look like in the area. I can give a multi-service profile of youth. This also gave me a whole different angle for several newspaper and radio stories as well.”

Morale and recruitment.While evaluation sometimes makes staff wary, YouthZone has found that the strong evaluation results have improved staff morale. It has also helped the agency attract good board members and volunteers. Wilde points out that the reputation brought by the evaluation makes people want to be part of the program—as staff or as volunteers.

Increasing program use. One result of the positive publicity and reputation is an increase in referrals. YouthZone used information from the evaluation to target different referral sources: criminal justice agencies, youth-serving organizations, county government, towns, and so on. YouthZone’s evaluation enabled them to document what was happening with kids, point to the barriers and ways to overcome them, and show that the success of kids in schools was having an impact on other areas of their lives.

Expanding alliances. YouthZone used the evaluation to find more partners, enabling them to work with more kids and have a much better impact.

The Numbers Added Up to Real Change
Of her evaluation, Wilde states, “Having the numbers, showing that things are effective, getting more kids involved, seeing the positive outcomes—this provides accountability, legitimacy. It let us have a sense that important things are happening—that there are outcomes, and that there is impact.”

“We did the first evaluation, and we used the recommendations. Then, we saw increased improvement in the second evaluation. We saw significant differences later.” YouthZone used the evaluation to push for excellent programming based on sound data. They made real changes—dropping parts of their model and changing others—to get genuine results.”

If your organization is already evaluating its programs, or if you are planning on it, don’t do so without thinking through the many ways you can employ the results. Involve as many staff as you can, and find out the ways that a cross-section of the organization might employ the results, not just the program being evaluated. As Wilde says, “Spend money on evaluation, so you can use it.” That kind of thinking is at the heart of creating what Mattessich calls culture of evaluation—about which, more in our next issue.

Sincerely,

Vince Hyman
Publishing Director
Fieldstone Alliance

June 6, 2007

 

Copyright Fieldstone Alliance. For reprint permission, click here.