ETEC 561 Section 3: Evaluating, Implementing and Managing Instructional Programs and Projects

All too often instruction is developed with little thought as to how evaluation of learning or the effectiveness of the instruction will take place. When evaluation is considered on the front end of the instructional design process, it is often limited to evaluating whether the instructional design is more effective than traditional methods.

For this week's reflection activities, I would like for you to concentrate on the following:

Chapter 10 discusses evaluation in instructional design and provides you with two evaluation models, the CIPP and Kirkpatrick models for evaluation. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction.

Standards Based and Responsive Evaluation Models

(Robert E. Stake)

After reading many different sources online I found a presentation that Stake gave at the IPIEN conference on Development and Evaluation in 2006. In his presentation he discusses two different model for evaluating a group or organization.  These two models are the Standards based and the Responsive Evaluation Models.

To summarize the Standards Based Model, Stake says it is “comparing measurements”. This model is more formal and has set variables and limits. This model leans itself more to a formative assessment. Being standards based it is looking for tested results to show for growth. Often the model will evaluate one or a limited set of standards and look to these to gage a measurement of performance. This has become big in modern education with the standardized test being the chief form of data gaining tool. This makes for what I think is a very narrow and in some ways faulty evaluation model. I don’t think one can gage overall performance based on a limited sample. I use Standards Based Evaluation several times a year usually at the end of a unit, to gage the knowledge mastery of the learner over particular information. I believe this model is also used evaluate classes, the instructor, or segments of instruction. In theory the better the instruction or the more engaging the lesson the better the standards should be met by the leaner.

I have also summarized Stake’s Responsive Evaluation Model. This model is designed so that the evaluator can be responsive to the learner’s needs and concerns. This model more leans itself to formative assessments. The stakeholder is at the center of this model and allows the evaluator to be flexible. Responsive Evaluations have to lean on the evaluators experience and judgments based on their personal experience to evaluate. Stake calls the Responsive Evaluation an “experiential understanding. This differs from the more traditional models in that it is a much more informal model. This model also can take into account the experiences of other experts. I think we as teachers probably use this methods often. I also believe this model has short comings as it relies too heavily on others experience in order to make a judgment. I use this model often in my instruction and the evaluation of my learners. I make judgments everyday about whether my class is “getting it” or not. I think that this model allows for quick informal evaluation to gage success of instruction. I have experienced and relied on this model, by having other experts observe my instruction and supply me with feedback just like I do for my leaners. Further information on these models can be found HERE.

Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. What else would be useful to know?

When looking at different options for evaluation an instructional designer should ask themselves a few questions before deciding on one or more evaluation models. The following are a few examples I came up with that might be useful to know:

a.    What is the “value” (Scriven) to the learner?

b.    What type of data is being gathered?

c.    How will the data gathered be used?

d.    What type of environment will the evaluation by done in?

e.    What type of organization will be evaluated?

f.     What is the focus of the evaluation?

g.   How long will the evaluation last?

h.    Who is being evaluated?

i.     What are the desired results of the evaluation?

Brinkerhoff and Patton also had lists associated with their models of evaluation.

All of these questions have the commonality of a focus of determining how an evaluation will be given and used and what the “picture of success” looks like. I think that we must think about these things before we develop instruction. If we design with the end in mind then we should be more successful in the long term with our results and the performance of our learners.

Chapter's 12 & 13 focus on project management and how to manage projects when resources are scarce. You have been assigned to develop a series of professional development sessions focusing on technology use in the classroom for teachers during a time of economic decline. How will you use Situational Leadership to facilitate this project and manage scarce resources?

If I consider this from the perspective of my district, I would consider any project like this to be done with scarce resources. Now, I what is scarce here is the question. Is it time to get the sessions ready, lack of personnel to prepare and present this training, or it is it that more funding is needed to make this happen. Again I will use what I know of my own district again. Time would probably not be an issue in preparing the sessions. However, the time to deliver it effectively to all staff might be an issue. There are ways this could be accomplished. It could be done in inservice training. It could also be down after school for factuality meetings. Another option would be to have teachers come to training during the instructional day. The final way could be to develop this as an online course that teachers could do as they have time with in a given week. These all have strengths and weaknesses. The best solution might be to do a mixture of these. Now when doing them during the instructional day cost must be considered. Cost must also be considered in the development and implementation of the training if done online. Inservice would be a time that would not cost the district extra money, but it might rob teachers of valuable time that is needed for other activities. After school meetings are never the best time to do train at least as far as most teachers are concerned. What I would suggest to the district would be to do a day face to face. Then do a couple of online sessions with assessments and end it with an after school question and answer session. There might also be a project or demonstration that could be produced like a portfolio for the teachers to show they understand the concepts.

As far a developing these lessons there are always limited personal at my district. In most cases one person would be responsible for most of the aspects of design and another would be responsible to put together the class and register participants. Either of these people might instruct the sessions. If I was the director of Instructional Technology and was tasked with this project, I would first discuss the focus and expected outcomes with the Superintendent and the head of Academics. After understanding the expectations, I would get my best instructional planner, or in the case of my district and its technology department it’s only planner. After discussing with instructional planner the limitation and expectations I would work with that person to start the process of designing the sessions. As I felt the planner was getting the focus of these sessions I would start to step back, but still check in often the help with questions and redirect misunderstandings. When the questions and misunderstands have been resolved I would step back from the development of the informational or instructional side to focus on the end product being ready to present and making schedules and online notices for teachers of the upcoming face to face session. This would mean working with campus administrators heavily. I would while focusing on this still check in with the planner and thank and reward when possible and/or necessary. At the stage where my focus must switch to finalizing the details of presenting the first session I would be letting the planner finish almost totally on their own. I would still review the session presentation before it is deemed finished. I would repeat this process for the first online session also. The second one might not need much involvement from me until it was finished. However, that does not mean I would not be involved with stating focus and expected outcomes and then checking in from time to time. I would handle the question and answer session, but it would be nice to have planner there to see how their work was received and understood. To me this scenario is probably more the norm at most districts instead of the exception.