Evaluation: How do we know we are doing what we say we are doing?

Posted February 2000

Program evaluation has never been more important than it is today in a competitive fundraising market. As we scramble for grants and donations, we better be able to show that our programs are effective and that they are doing what we say they do.

This is as it should be. Nonprofits can't measure success at the end the year by the value of their stock or by adding up the profits. Evaluation of program (and organizational) effectiveness is the answer.

Achieving results is the key to establishing long term self-sufficiency for any Traffic Safety Program. Everything depends on results. Agreeing on desired results helps focus resources, reduces conflict between competing goals, and provides the basis for feedback and evaluation. Measuring results makes possible the celebration of achievement.

The purpose of evaluation is to determine how effectively and efficiently a program or project achieves the objectives it was designed to achieve.

Effective means having the expected or intended effect.

Efficient means acting with minimum waste or unnecessary effort.

Evaluation can:

· Document and measure programmatic effort
· Measure outcomes at numerous levels of accomplishment
· Identify strengths and weaknesses
· Indicate needs for correcting activities
· Enable the modification of inappropriate objectives
· Improve management policies and procedures
· Provide the basis for reporting to stakeholders

A manager's concerns about evaluation include the following:

· Building in feedback systems from the beginning
· Identifying the questions that need to be answered
· Assuring objectivity, but not overlooking subjective observations
· Obtaining understandable, useful reports describing both effectiveness and efficiency

Types of evaluation include:

Administrative--Measures the scope and degree of implementation achieved.

· Critical for ascertaining whether planned activities were carried out according to schedule
· Imperative in order to adjust plans when conditions change
· Necessary for acknowledging the work of people in completing their tasks
· A vital link for assessing overall effectiveness, since interpreting final results requires knowing the degree of implementation that was achieved

Impact--Measures the extent to which activities have contributed to achievement of objectives related to crashes.

· Often termed "scientific evaluation" since it involves formal experimental design
· Measures "the bottom line": crashes, deaths, injuries, property damage, behavior change, and changes in public awareness and attitudes
· Shows relationship of costs to benefit from the activities
· Conducted only on major projects when there are:

Anecdotal--Contains short narrative descriptions of activities, events, or circumstances that explain or clarify project results.

· Often used in reports to stakeholders: Congress, legislative bodies, funding sources
· Often involves a "case study"
· Frequently helps to explain findings or exemplify success

Process--Uses monitoring, interviewing, observation to indicate what is "really going on."

· Depends on qualitative rather than quantitative data, i.e., samples are usually not random
· Interviewing and observation are essential tools
· Indicates the capacity for success, or the difficulties being encountered
· Provides immediate feedback for corrective action or replanning

Steps in evaluation

1. Define project objectives and chain of action.
2. Develop evaluation questions for each objective.
3. Define evaluation measures and data system requirements.
4. Write an evaluation plan.
5. Assign responsibilities and negotiate agreement.

The program manager's responsibilities

· Ensure that the evaluation planning is considered from the start.
· Help identify appropriate evaluation methods.
· Set guidelines to ensure information arrives on time.
· Review information for decision making.
· Assure that evaluation results are used.

All people possess skills to work with evaluation concepts, as in buying a new house or car. It is not crucial that managers perform the evaluation. However, they need to be able to identify people who can help, communicate with them, and participate in devising the questions they want answered.

Realities of evaluation

· Limited funds available to develop experimental designs, collect data, and conduct analysis
· Unrealistic expectations of stakeholders from small investment in the program
· Lack of data
· Contamination of data
· Difficulty in teasing out effects of particular activities when results are due to many variables

Resources for evaluation

· Local universities--marketing, public health, business departments
· NHTSA--Regions and Headquarters, numerous manuals
· Private sector firms

The better we evaluate, the better we can do our jobs!

Printer-friendly version | discuss this article in the forums