Goal-Oriented Evaluation

It is conventional in most discussions of program evaluation to begin with the idea that evaluation can best be done when goals and objectives are clearly stated, preferably in quantitative terms. However, the quantitative specification of program goals remains one of the most difficult aspects of evaluation. Goal-oriented evaluation is a term that refers to a set of techniques by which the effectiveness and efficiency of programs in meeting their goals and objectives may be measured. These are procedures by which quantitative values may be assigned to relative degrees of performance of a program over a defined period of time. Comparison of these quantitative estimates of effectiveness and efficiency for two or more programs (or parts of a single program) can lend support to decisions about the redirection of resources to better achieve important goals and objectives.

Fitting Goal-Oriented Evaluation into the SRAP Structure

Each project has been asked to develop a "project logic" tracing actions to outcomes. Those project logics will be used to construct a series of measurable objectives for each project. Given the variety of interventions that are being developed, the measures will be different for the various projects. However, all projects will be subject to the same logic-building rules and the need to identify achievable and measurable objectives that fit within the overall goals of improving access for rural populations

Useful Terminology

Because terms that have a variety of common meanings are used in the following discussion, it is desirable to indicate their special (and more limited) meanings in the present context. Ten terms will have specific meanings in our discussion:

Evaluation: The process by which programs are studied to ascertain their effectiveness or efficiency in the fulfillment of goals. In this case, the evaluation of the SRAP is intended to assess the progress of the overall program identifying elements that work or do not work in improving access. Within each state, evaluation of their individual progress in meeting objectives serves more as a management tool and feedback loop

Activity: A positive action to be taken by the program being evaluated during a fixed time period, usually one year. An activity is that which answers the question: "What will be done?"

Target: A numerical quantity (in appropriate units) that specifies that minimum desirable level of achievement for a particular activity. The target answers the question: "How much will be done, by whom, by when?"

Objectives: The activity being evaluated, along with its corresponding target and resources.

Goal: A general statement indicating the general direction to be followed by the program, in the longer term, usually at least three years.

Standard or Indicator: A numerical quantity in appropriate units that specifies the minimum desirable level of achievement for each goal.

Short-Term Achievement Index: The degree to which program objectives are fulfilled in any particular year.

Long-Term Achievement Index: The degree to which program goals are fulfilled each year.

Efficiency: The amount of output (in terms of short-term achievement) per unit of input (in terms of resources expended).

General Methods for the Measurement of Goal Fulfillment

This approach to the measurement of goal fulfillment was designed as a flexible tool for program management, planning and evaluation. It builds on work used in earlier evaluations of projects similar to the SRAP. Its validity is derived from its interpretation and use within a given project context. Its external validity is limited; that is, the approach was not developed for the purpose of comparing one project with another. This approach gives a project the framework within which it can organize its goals and objectives and a quantifiable way to measure progress (in degree and pace) in meeting these goals and objectives. The methodology is flexible, allowing projects to selectively implement parts of the approach, depending on time and resources available.

Projects that attempt to influence a goal which has many elements, such as access, usually begin with a set of goals which are expected to remain constant over a period of time, say four-to-six years.

These "long-range goals," frequently derived from a general statement of the intent of a program, are usually stated in narrative or non-quantitative terms. Here are some examples of long-range goals for programs which have undertaken to improve access:

• Decrease the proportion of people who do not have a regular doctor.
• Increase the proportion of rural and minority applicants to state-supported medical schools.
• Increase the percentage of the target population who are screened for common chronic disease.
• Decrease waiting times for appointments in clinics.
• Reduce inappropriate emergency room utilization.

A conceptual model of the approach to the measurement of goal fulfillment is presented in Figure 1. Once the long-term (five-year) goals of an access-enhancing project are stated, standards for each long-term goal should be specified. These standards are the criteria by which one may judge that a given goal has been reached. Table 1 provides some examples of standards for the first long-term goal listed above (i.e., Reduce travel time to primary care). Ideally, these standards should be measurable and realistically attainable, and they should be monitored on a yearly basis.

Once the standards (or criteria) are clearly stated, it is possible to set up a data collection system to capture the indicators of progress.

It is important that there be objectives addressing the structure, process, and outcome indicators of program effectiveness. During the first years of the project, most objectives will tend to deal with structure and process indicators, while toward the end of the project the objectives related to outcome become more important and measurable.

Figure 1. Conceptual Model for the Measurement of Goal Fulfillment

Three Components of a Program Objective

Each objective is composed of (1) an activity, which tells what will be done; (2) a target, which tells how much will be done by whom by when; and, if possible, (3) the needed resources, indicating what kind and amount will be required to achieve the target. Resources required for the attainment of a given objective are usually defined in terms of dollars, personnel, equipment, or facilities.

An example of a process objective related to the effort to improve the leadership capacity for rural primary care in one state:

Implementation of the Community Health Team Leadership Initiative

Curriculum for Community Health Team Leadership Initiative will be developed by June, 2000; Five Community Health Teams will be identified to participate by September, 2000; Each Community Health Team will have conducted a team project by November, 2000.

Approval, publication, and dissemination of curriculum. Rosters of Community Health Teams and minutes/reports of meetings. Documentation of project including target community, activity, internal feedback or evaluation.

When program long-term goals and short-term objectives have been stated as we have illustrated above, three important indicators of goal fulfillment can be measured at the end of the first year of this project. Similar goals are set for the subsequent years initially but modified as the overall project moves forward. These specific activities and targets are derived from the project logic structure which takes the general form:

Measuring Short-Term Achievement

Short-term achievement addresses how well a program has progressed toward meeting its stated yearly objectives. Provided that multiple, preferably up to ten, objectives over five years, are stated for a given goal, measurement of this index can be as simple as the ratio of objectives "completed as specified" to the total number of objectives stated for that goal. Measuring the proportion of each objective (target) completed by the end of the year allows for monitoring the amount of over- or under-achievement for a particular objective. Achieving the target level would be indicated by a score of "1.00," while achievement at a level less than the target for an objective would be indicated by a score of less than "1.00." The actual measurement of achievement would be dependent on how the target is stated. The following are a few examples of how measures can be stated:

1. Target Met Versus Target Not Met: This is the simplest type of measure which all programs should be capable of using. It scores each objective "1" if the target was met (or exceeded) and "0" if the target was not met. The level of achievement with respect to any objective can be measured in this way.
2. Target is a Date: Number each month in the year for which objectives are being stated, assigning "12" to the first month and "1" to the last month. The measure would then become the number assigned to the month the objective was completed divided by the number assigned to the month in the target date. This process is illustrated in Figure 2.
3. Target is a Proportion or a Number: Many times objectives are stated in terms of proportions as absolute numbers. An example would be "There is an increase of 25 percent in the number of person-visits using van services at the clinic." The measure would become the proportion achieved by the end of the year (outcome) divided by the proportion stated in the target. An example is given in Figure 3.

Weighting the objectives according to their importance and the accuracy with which they can be measured, as was recommended for the standards, can further increase the validity of the index. Table 2 gives three examples of how short-term achievement would be calculated for our hypothetical access-enhancing leadership. There are no firm rules for interpreting these results, but intuitively one might consider a short-term achievement score less than 50 percent as undesirable and a score of 70 percent or more as an indicator of substantial progress.

Measuring Long-Term Achievement

The first indicator of goal fulfillment is long-term achievement, the degree to which the program has progressed toward meeting its goals. This is an interim index by which a program can assess the relative pace with which forward movement is (or is not) taking place with respect to particular goals.

The calculation of the index of long-term achievement begins with some kind of measurement of the level of completion for each standard relevant to a particular goal. Measures of completion can be derived in a variety of ways, but quantitative "estimates" of this variable are essential. The goals are those that are anticipated for the end of the program and they should not vary substantially from the onset of the program. There are, however, instances where the goals may change, even dramatically. These occur when there are major environmental changes: a reorganization of the state Medicaid Program, the passage of major national legislation, the closing of a medical school in the state, for example.

Once the level of completion for each standard has been derived, this value is then divided by the proposed level of accomplishment stated in the original standard in order to yield the proportion of each standard met. Long-term achievement with respect to the larger goal of the program can then be calculated by averaging the proportions for each standard pertaining to this goal.

The long-term achievement scoring is similar to that in Table 2 but the assignment of weights is more important. The weights should represent the relative degree of importancee associated with each standard vis-a-vis the objective in question, and (if possible) the relative accuracy with which each standard can be measured. The weighting of standards adds to the validity of the index.

Each standard can be assigned a weight between 1 (least important and/or accurate) and 5 (most important and/or accurate), indicating its relative importance in meeting the standards specified for that goal and the accuracy with which it can be measured.

These weights can be derived from a group process (like a Nominal Groups or Delphi process) that allows consensus judgments to be reached among a number of knowledgeable persons involved in or informed about a given program (e.g., staff, clients, etc.). The measurement of long-term achievement for each goal would be the weighted average of the proportions of each standard met with respect to that goal. The development of there weights in the SRAP evaluation will be done through discussions between the evaluation team, the NPO and the leadership of the projects.

Goal-Oriented Evaluation and Program Management

The approach suggested here is intended to be a pragmatic one of value to program directors who wish to assess levels of accomplishment across a number of areas of overall program activity as well as allow for the comparable evaluation of projects with varied approaches. Goal-oriented evaluation relies on a simple method of assigning quantitative values to program activities. The index values derived are to be used only as indicators of relative accomplishment, not as absolute numbers reflecting other qualitative or quantitative properties of the programs or activities being evaluated.

This method of evaluation feeds very nicely into two common techniques of program management. Measurement of short-term achievement is really a technique for management by objectives (MBO). Although both of these techniques have received mixed reviews, the concepts on which they are based are extremely useful when used in a "reasonable" way to make management decisions. Programs are usually expected to have at least a one-year plan and to monitor their progress relative to that plan. Further resource allocations can thus be made based on data concerning the strengths and weaknesses of the programs. This does not necessarily mean that "problem" areas (objectives with short-term achievement and/or efficiency scores less than 1.00) would be defunded. If the objective is considered important to the long-term outcomes of the program, resources might be reallocated to it from less problematic areas (objectives with short-term achievement scores greater than 1.00).

A Helpful Reference Illustrating the Application of this Approach

Guild, P. A. (1990). Goal-oriented evaluation as a program management tool. American Journal of Health Promotion, 4, 296-301.

Siegel, E., Gillings, D., Guild, P. Nugent, R. (1977). Planning and evaluation of regionalized perinatal care: A rural example. Seminars in Perinatology, 1, 283-301.