|
November 30, 1999 The evaluation will include three primary strategies: (1) the development of a system for tracking primary care professionals in the rural portions of the 7 states and eastern Texas, (2) a system of individualized, goal-oriented evaluation based on the project logics of the 8 grantees, and (3) the use of the US Centers for Disease Control and Prevention's Behavioral Risk Factor Surveillance System (BRFSS) surveys or their equivalent to provide information about population based measures of access in the states. The Goal-Oriented Evaluation approach has been used to assess the progress of complex multi-site programs and a more detailed description of that approach is appended. Within the individualized goal-oriented design we will be focusing on the development of networks and the use of the loan funds. Although the goal-oriented approach is designed to create customized evaluation structures within each state, there may also be common activities across a subset of states such as the use of AHEC systems or and emphasis on mentoring, this will allow the use of the same indicators and scales across some states. For example, in states that have heavy emphasis on pipeline development, the numbers of rural students applying to medical school can be tracked so that the measures and indicators will be as comparable across states as possible for comparison purposes. The specifics of each goal-oriented evaluation component, including the identification of specific indicators will be finalized over the period December, 1999 through June, 2000 through contacts and site visits to the state programs. 1. Tracking Locations and Practices of Primary Care Professionals The data collection standard for health professions is based on the Evaluation Planning Memo Number 2, "Physicians and Primary Care Professionals: Essential Data Elements for the Study of Access, Supply, and Needs." That document is also attached. The evaluation will request from each state data on the supply and distribution of primary care professionals in the target regions and, if possible, for all rural areas effective as of January, 2000 and annually until January 1, 2004. The data will identify the actual practice locations and activity of primary care physicians, nurse practitioners, physician assistants and nurse midwives. The evaluation intends to create specific milestones and objectives for the collection of the data in each state by the end of May 2000. The evaluation team will provide assistance to the states in the development of inventory and tracking systems either as part or the licensing process within the states or as a recurring mechanism to assess access. The degree to which all three types of professionals may be tracked within each state may vary; however, the overall goal is congruity across all states for all professionals. We anticipate that some states will be able to implement annual inventories immediately and others will have to either create systems or do special data gathering. The process of developing data systems will also be considered as a part of the goal oriented evaluation approach. Where states have set objectives of developing data and tracking systems as part of the SRAP, then their progress will be assessed against their stated goals and objectives. 2. State-specific Goal-oriented Evaluation Using Project Logics The second component will have each state participate in a "goal-oriented evaluation" approach based upon measurable outcomes identified in their project logics. In the course of the site visits in 1999 we have found that, as of October 1, 1999, not all states were able to create firm logic structures that identified their activities and their associated objectives, goals, and effects on access (see table 1). Projects are making progress in developing these plans and we believe we will be able to assist the states in refining the project logics over the period December 1999 through March 2000 to produce measurable objectives. We have created a standardized structure for the project logics that includes the following elements: Table 1. Project Logic Structure
The evaluation will further develop standards for the level of information we will use to monitor specific program progress. The primary activities of the projects, including network development, leadership and pipeline development and the loan funds will be examined in the goal-oriented structure. For example, for a leadership program in a specific state, the goal-oriented evaluation will identify measurable activities related to that focus. These may include indicators such as (1) final approval of curriculum by a target date and implementation of the curriculum by a target date, (2) the number of students participating in classes compared to expected enrollment, and (3) students' evaluations of the program scaled against expected evaluation scores. The creation of specific project logics to be used in the evaluation will be iterative and continuous with agreement on the individual items worked out between the state's project leadership, the UNC team and the National Program Office. The latter will review the logic structures and provide comments to the evaluation team. The project logics, along with the specific indictors, will then be included in a goal-oriented evaluation structure that assigns numerical values to objectives and calculates proportional progress. The numerical assignment process is not meant to be absolutely precise or to drive the evaluation process, rather, it is intended to allow for comparisons of relative progress within and across the projects. The details of this process are described in the Evaluation Planning Document Number 3, "Goal-Oriented Evaluation as a Program Management and Assessment Tool which is attached. The information necessary to monitor the projects' progress will be drawn from individualized contacts with the projects and their recurring reports to the National Program Office. The overall goal will be to make use of existing monitoring and reporting, and avoid undue reporting burdens on the projects. We anticipate at least one site visit per project per year through the evaluation for the purpose of collecting these data and assisting us in interpreting the data. The development of goal-oriented evaluation structures for network activity will potentially produce the greatest challenges. The state-of-the-art in the assessment of network activity is largely descriptive and performance measures are not apparent but there are a series of potential measures of outcomes and impacts that have been proposed by Moscovice and colleagues . Indicators of structural network activity including the existence of formal agreements for referrals, cost-sharing, and joint governance mechanisms can be used to track the implementation of networks. Moscovice, Christianson and others saw important indicators of (1) network formalization, (2) organizational complexity, (3) scale of operations, and (4) commitment of members as workable and measurable indices of implementation of networks . Long terms measures of health status and access change require long term assessment of populations, and, where possible, the BRFSS surveying can potentially provide outcome measures. However, the goal-oriented portion of the evaluation as it focuses on networks, will use structural measures. 3. Measuring Access Changes Using the Behavioral Risk Factor Surveillance Survey and Comparable Surveys of Access Indicators. Objective, secondary measures of the impact of the program on access are difficult to develop within a restricted budget. We have explored alternative indicators including Medicaid program utilization changes, participation in Medicaid by professionals in the target areas, indicators of avoidable hospitalizations, and utilization statistics from public providers. Each of these measures can be shown to reflect an aspect overall access to primary care, however, the capacity to collect the data to produce these measures varies widely among states and the specificity of each measure may not match the target population. The one general population survey that potentially can produce comparable population based data on access and changes in access over time is the annual Centers for Disease Control and Prevention (CDC) Behavioral Risk Factor Surveillance System (BRFSS) survey access component and its optional health care utilization module. The BRFSS is administered yearly in all 50 states with the cooperation of states. The CDC is responsible for the overall survey design, question development, and general guidance for the sampling. The actual samples are drawn and the surveys administered by state agencies or contractors. The surveys currently are designed to produce statewide estimates of health risk behavior and access to health care. In order to create reliable estimates of access for the targeted or statewide rural areas, the team will negotiate with the CDC and state liaison agencies to expand the samples in the project target areas. The core questions focusing on access and the optional health care coverage and health care utilization modules of the BRFSS are attached to this proposal as Appendix 3. During the first 6 months of 2000 we will work with the states to develop the samples and any additional access questions beyond those included in the standard core access portion or the optional modules 4 and 5 for coverage and utilization. The BRFSS is administered throughout the year and the effective date of any targeted sample surveying or the addition of special access questions will not take place until later in 2000. The statewide data from each prior year is also available but those samples are not focused enough to evaluate fairly the targeted rural programs. The BRFSS, if the samples can be created, will provide a baseline of population access indicators that are comparable across the states. The evaluation team has contacted the CDC coordinator for the surveys, David E. Nelson, and will be working with state contacts during November, 1999 through June 2000. The BRFSS system allows the participating users access to the edited data with far fewer restrictions than are placed upon surveys such as the Medical Expenditure Panel Survey or the National Health Interview Survey. Citations Guild, P. A. (1990). Goal-oriented evaluation as a program management tool. American Journal of Health Promotion, 4, 296-301. Moscovice, I., Christianson, J. B. Wellever, A. (1995a). Measuring and evaluating the performance of vertically integrated rural health networks. The Journal of Rural Health, 11, 9-21. Moscovice, I., Wellever, A., Christianson, J., Kralewski, J. Manning, W. (1995b) Building Rural Hospital Networks, Health Administration Press, Ann Arbor, MI. Siegel, E., Gillings, D., Guild, P. Nugent, R. (1977). Planning and evaluation of regionalized perinatal care: A rural example. Seminars in Perinatology, 1, 283-301. Proposed Evaluation Schedule
|