v1n6 Context Evaluation for the Academic Support Programs

Management Notes Online
June & July 1997

Alan L. Joplin, Publisher

Context Evaluation for the Academic Support Programs


The purpose of context evaluation is to define the environment, its unmet needs, and the opportunity for change, and to diagnose the problem to be solved it needs are to be met or opportunities used.

The design for context evaluation is described through a series of steps involving:

1. Delineation of evaluation requirements
2. Collection of Information
3. Organization and storage of information
4. The analysis and reporting of information

The first step in designing context evaluation is delineating the evaluation. This is to:

1. Identify the program to be evaluated.

Identifying the program to be evaluated is a very complex problem.

Decision makers might define for example, curriculum in many ways, and their verbal statements may deviate greatly from the actual practice.

2. Identification of decision situation and levels to be served.

This item is essential in the focusing of the evaluation.

This information aids in determining:

A. The purpose the information is to serve, quality control development or acquisitions.

B. A timetable for providing the information

C. The level of decision making for which the report is prepared.

NOTE: Well designed Context Evaluation will provide data for all relevant levels.
3. Describing the current program.

This will require gathering information relating to the instructional and behavioral variables for each program component.

1. Demographic data on the population

2. Data on program activities

a. Instructional

b. Counseling, etc.

4. Identifying mission, goals and objectives.

The task of the evaluator is to state the mission, goals and objectives so that they facilitate communications between the evaluator and the designers and implementers of the program. The approach to the problem of stating objectives must be in terms of meeting the needs of the component and the decision level to be served.

One the mission, goals and objectives have been identified the final step of the evaluation process in the delineating stage is to identify the questions to be answered by the evaluation.

1. The forces which act upon program design to meet the needs of students.
2. Forces: Instructional, behavior and Population Variables

This combination of variables provides the evaluator with a system for identifying questions concerning those forces that influence the achievement of program objectives.

Delineating the evaluation as described provides the evaluator with a complete picture of the program to be evaluated and the data needed for decision making. The generation of questions may be as complex as the need demands. The decision maker must be responsible for the decision relative to the selection of objectives. If the information concerning content, student and cognitive behavior is all that is considered necessary, then the evaluator's task is much less complicated in terms of instrumentation, data gathering, and storage. The evaluator has at least called attention to possible categories of questions that might be relevant to the need for decision making.

The delineation process of the context evaluation systems provides the basis for a well organized data collection system.

NOTE: Descriptive data will be collected and stored as soon as each step of the delineating state is completed.

Instrumentation should be designed to fit the information storage and retrieval system developed by the evaluator. In most cases, forms will be designed with information relevant to the coding systems used for the storage and retrieval of information.

Once the questions to be answered by context evaluation have been developed, the source of information, sampling procedures and instruments must be identified.

Whenever possible, administering too many instruments to the same person should be avoided. Many questions can be answered by a single instrument. Where a number of instruments are involved, sampling might be employed.

The final component of the data collection system is the master schedule for collecting information. A well organized plan involving deadlines for the creation and administration of instruments must be developed.

The third link in the strategy of context evaluation is an information processing and storage system. The monitoring of programs through context evaluation should include a system that provides for

1. The processing of data.

2. A central information and storage system, including data bank for research, innovative practices, demographic data, test instruments and items and program descriptions.

Properly designed information systems possessing adequate facilities make it possible for the evaluator to provide up-to-date and immediate access to information vital to decision making. Program planning and implementation also requires information relevant to program objectives, facilities, personnel and program alternatives, etc. Each of these complex tasks require valid and reliable information. Inadequate information and processing systems lead to poor decision making and ultimately to serious weaknesses in the educational process.

The fourth and final component of the strategy for context evaluation is the analysis and reporting of information, a complex task requiring a cooperative effort on the part of many individuals.

The first step is to develop a report prepared by the evaluator directly responsible for the context evaluation. Data will be packaged in a report, with interpretations made from the data analysis and resource. The objective of the first step is to provide a baseline for the evaluator in considering all the facilities available through the information storage and processing system.

With the evaluator's report involving the analysis and synthesis of data the second step is to identify needs and opportunities. Even though the program is meeting current objectives, the evaluator and decision makers together must consider future needs and keep up to date on innovations in programming. At this point recommendations are made concerning possible changes or recycling of the program. Changes may be recommended because the program is not meeting its objectives, current objectives are not sufficient to meet demands, or innovative practices indicate significant opportunities for improvement.

Depending on the recommendations in steps two, the final report is prepared to meet the needs of the various audiences it is to serve.

To provide information on the forces that influence a student's achievement, the evaluator needs to work within a framework that offers a wide range to potentially relevant variables reflecting theory and practice of teaching and learning in a wide range of institutional settings.

A framework designed to meet these specifications was developed at the EPIC Evaluation Center in Tucson, Arizona. The structure is composed of three sets of variables:

1. INSTRUCTION

2. POPULATION

3. BEHAVIOR, and it has been most useful as a heuristic device to reveal combinations of variables leading to more complete description and analysis of the program.

In considering the dimensions, analysis of variables is limited only by the nature and scope of the program and the desire for simple and complex analysis. A description of these forces affecting program results is produced through the interaction of variables on each of the dimensions.

In the set of variables identified with instruction, content is that structure or body of knowledge which is identified with the subject matter of the discipline and controls its inquiries. Generally, content is described through a structure of topics to be covered within and across grade levels in the institution.

The scope and sequences of the content of the program is divided into topics to be covered for freshmen through the senior year. Each topic is further broken down according to content to be covered under a topic at a given instructional level..

Such a breakdown provides the evaluator with the necessary content of instruments for evaluation purposes. Additional breakdown within topics provides specifications for item development and analysis.
methodology

The second variable is defined as "that process designed to facilitate learning"

Three components are:

1. Teaching activities, such as lecture, discussion, drill, inquiry, and debate.

2. Types of interaction between students, instructors.

3. Learning principles pertaining to motivation, practice and reinforcement.

The degree of detail involved in the evaluation process will be determined directly by the ability of the instructor to describe what is taking place.

Organization

A third variable of instruction, concerns the way students and instructors are brought together so that instructions can take place.

Classifications of organization:

1. Vertical organization, serving to classify students and move them upward from admission to departure.

2. Horizontal organization, dividing students among instructors.

3. Time organization, involving both duration and sequence of segments of instructions.

Since many organizational patterns may be operating within a given institution, the evaluator must describe and store data for each program.

Facilities

Are the space, special equipment and expendable needed for the educational program.

A second set of variables to be considered in program description are those regarding the population directly or indirectly involved in the program. Each acts as a force affecting program outcomes. One classification includes student, teacher, administrator, educational specialist, family, and community. Demographic data are required for each category. Examples include (1) general identification data for students such as age, grade level, sex, achievement, and mental health; (2) identification data for teachers, administrators, and educational specialists, such as educational background, work experience, and environmental factors (professional salary and professional and non-professional affiliations); (3) general characteristics of families, such as size, age distribution, income, residence, education, and mobility; (4) characteristics of the community, such as geographic setting, historical development, population, economic and social structures, government, power structure, socioeconomic stratification, and commercial and financial status.

In the past evaluators have gathered demographic data on an ad hoc basis, so that new information had to be gathered each time a particular decision situation occurred. storage systems must be developed and kept up to date, and this requires the development of instruments and coding systems for storage. Much time must be spent on this during the early states of evaluation.

Behavior is a third set of variables to be considered the reactions of the persons who are in the instructional settings. One approach to classification is that of Bloom, Krathwohl, and others involving cognitive, affective, and psychomotor behavior. Taxonomies have been generated for the cognitive and affective domains and attempts have been made to do so for the psychomotor domain. Information about these is readily available to the evaluator. The primary task of the evaluator is to identify the levels and categories of behavior on which information is needed.

_____________________________________________________________________________________________

Management Notes is published six times a year a year by Voices of the Tribe. Editorial offices are located at 933 Washington - Davenport, Iowa 52804/USA. The Purpose of this publication is to provide readers with down-to-earth management information, ideas and techniques they can put into action to motivate employees and spur productivity.