Wednesday, May 16, 2007

Evaluating Teaching Evaluations

Teaching evaluations are best viewed in a general, cosmic, slightly unfocused way as an indicator of some aspects of a teacher's methods and effectiveness. I just got my spring evaluations yesterday, and I am pleased with the results, but that won't stop me from musing about the bad (and good) aspects of evaluations.

I think that if you look at someone's teaching evaluations for a range of classes (small and large, majors and non-majors, graduate and undergraduate) over 3-5 years, you will get a sense for whether that person is a 'good' teacher or not. Ideally, 'good' means interesting, caring, challenging, accessible, organized, and fair, and doesn't correlate with a propensity to give A's for mediocre work and effort.

At my previous university, I had 2 teaching mentors who sat in on my classes at random times during the semester and provided me with feedback. They were supposed to tell me in advance when they would be sitting in, but they didn't. This was fine with me because it meant they saw the class more as it was on a routine day, rather than on a day when I had prepared in a different way knowing I'd be monitored. I didn't really get any useful feedback from these teaching mentors except about some technical issues on how to more effectively use the A/V equipment in the classroom. Even so, I liked the fact that there was an additional evaluation method of my teaching other than just the student evaluations.

At my present university, our only feedback comes from the teaching evaluations that students fill out at the end of each semester, so these evaluations are the sole evidence for whether we are good teachers or not. That's why it's important that the evaluation process be improved.

Problems with the current process (written from the perspective of someone teaching at a large university):

1. Some of the questions are strange, irrelevant, offensive: I have written about this before, but some of the primary questions that are the basis for the quantitative aspect of the evaluations are inappropriate, e.g.,

- A question about the physical environment of the classroom asks students to rate me on something over which I have no control.

- A question about how the students would rate the professor's knowledge of the course material is offensive as written. That question could be worded to ask how effective the professor is at conveying their knowledge to students, rather than asking the student how much they think the professor knows.

- A question about how much a student learned in the course is not only rating the effectiveness of the teacher, but also how much effort the student put into the course. Sometimes the two are related (students may put more effort into an interesting and well-taught class), but sometimes they are not. A student who does not attend class regularly and doesn't do assignments etc. will not learn much in the course, yet he/she gets to rate how much they learned in the course as part of the evaluation of the professor's teaching. In theory, these students are a small minority and their low ratings will be outliers in an overall positive evaluation, but their effect on the evaluation depends on class size.

The questions I think are good and useful include:

- A very general question on the professor's overall teaching ability.

- A question or questions about the professor's level of respect, concern, accessibility for students.

- A question about whether graded work is returned in a timely way.

Even for those questions, a student who is not doing well in a course will in some cases give a professor a low rating, but again, if the class size is sufficiently large, these will be outliers.

And then there are the detailed comments. Reading these can be alternately heart-warming and enraging. I am fortunate that this semester I got the heart-warming kind. More typically there are at least a few that will say "I wish you had done X", where X is something unreasonable (e.g., "I wish you could have given us all the answers to the quiz questions in advance so that we could have studied them.").

And of course there is no chance for rebuttal/response, and there is no way to fix any problems for that specific class. That can be frustrating.

In summary: it's a flawed process that sort of works in a general way.


Anonymous said...

I have answered these kind of surveys/evalutions of teachers- and often one has to respond to how much reading per week, and own evaluation of one`s efforts. These kind of questions should definitely be included, because they can explain outliers- as you point out.

right now I`m having a headache trying to sort out how to solve a given "exercise"-- to me it seems like the teacher asks us to write something a bit top of the hill.. She is in fact advisor to phd- students- so I think she should know better... Or perhaps that is exactly her problem?? BUt- she will know my point of views at the end of the semester- as asked for by herself... Of course in a nice and polite manner!

I would also like to recommend something we here call for a "referencegroup". If there is any problems or questions DURING the semester some appointed spokespersons on behalf of the class can be contacted, and find a reasonable solution together with the teacher. Then it will be possible to adjust the teaching during the course- and one would correct the errors also for the time- being class! Just a suggestion for you!


Female Science Professor said...

I like to take care of mid-semester issues with an optional informal evaluation that I give out to request feedback on things that I can easily fix before the end of class.

Anonymous said...

The unreasonable feedback, and the untruthful feedback (e.g., "not available" when I reply to email within 12hrs and am willing to meet with students whenever works for them) are annoying, but nothing compared to the hurtful comments on, for example, my race or reproductive status.

I don't read my feedback anymore. I ask someone to read it for me and give me the gist while filtering the abusive comments. I also use a midterm form which is semi-open (I discuss the feedback in class with the whole group) and I get much more useful (and more positive) comments in that format. I think there is something about the anonymity and lack of accountability that makes end-of-semester evaluations a vehicle of "revenge" for a few.

Female Science Professor said...

You've brought up an important point. Some students use the anonymous reviews to write really hateful comments about a professor's appearance, personal life etc. I got some of these comments on evaluations when I was younger, but I haven't gotten one in a while (though of course now I get them as comments on my blog). It would be nice if there were an official mechanism to discard the truly obscene and personal-attack comments in teaching evaluations.

Anonymous said...

yeah, I don't get people who are negative and insulting on blogs. At least my students with nasty comments seemed to think they had been wronged in some way!

lost clown said...

The prof I TA for has his own evaluation form that he gives a point for completing. It is tailored to the class and less vague then the university forms. I also like it because there's a section on the TAs, so I get to know how to better in the labs.

Female Science Professor said...

There should always be TA evaluations along with the faculty evaluations. Labs are essential parts of many classes, and we need input on them as well as the lecture part of the class. Also, as you mentioned, TA's need feedback. TA evaluations are important in my department also because there is an award for 'best' TA, and it involves a financial reward as well as the names engraved on on a wall. The award is decided in part from evaluations and in part from nomination letters from faculty and students.

Irie said...

I think the question regarding the knowledge of the professor is valid. I have suffered through classes where the professor was frequently off topic, gave information that conflicted with the text that they assigned, and/or other behaviors that made me question their ability to teach.

Female Science Professor said...

Yes, but does that mean they don't know their subject or they just don't know how to teach it? I think the question would be better worded if it related to teaching rather than knowledge.

pluto said...

< I don't read my feedback anymore. I ask someone to read it for me and give me the gist while filtering the abusive comments.

What a great solution. Nice thinking!

pluto said...

About evaluation questions: at least they don't get to rate you on your personal hygiene like they do at my university.

Jenny F. Scientist said...

In our TA evaluations there are several questions that ask about things the TA has no control over- what the content of the sections is, our refusal to hand out test questions in advance, how we go over the material that will be on the quiz (because the instructor specifically mandated it). I think standardized evaluations tend towards the useless questions, unfortunately.

Jessica said...

Glad you got some good evaluations! It's not a perfect system but it's the only one we've got. I think evals show a great deal over a long period of time. Good teachers get consistently good evaluations, with a rogue student stuck in there every now and then.

As to the classroom, I totally agree---that should not be a part of the instructor evaluation! And if I were teaching I'd find it offensive to be evaluated on my knowledge of the subject. I mean damn if I'm not knowledgeable why am I there? However how I convey that knowledge is a different thing and ultimately very important. About how much the student learned--I agree about it depending partly on how much the student put into it. I think that question could be re-worded on those evaluations.

One of the problems I see in evaluations is students who don't put forth much effort who tend to blame it on the professor. Students who enter a non-thesis graduate program believing it's merely an extension of their undergrad program are FLOORED when they find out they have to do graduate level work. Amazing isn't it? And they get upset and reflect that in the evaluations.

Anyway I do agree with your post, just wanted to let you know!

Carrie said...

One other problem with student evaluations is that often times the students really don't care. In that they have no idea that their evaluations are actually going to be read and that they DO have an impact on the professor's job/salary/status. They just have no clue. I know when I was an undergrad I did not put the effort into the evaluation that I should have, now that I know what can be riding on them.

lost clown said...

I wish we had a 'best TA' award. My stidents love me, even though I am a harsh grader. I try to give them feedback at the beginning of the next lab for what come up as common problems of understanding in each lab. Fortunately as the labs progress they ask more questions and are more involved with the process. It's great to see that.

TW Andrews said...

A question about how the students would rate the professor's knowledge of the course material is offensive as written. That question could be worded to ask how effective the professor is at conveying their knowledge to students, rather than asking the student how much they think the professor knows.

I think this is a legitimate question for students to answer. They may not be able to differentiate levels of knowledge past a given point, but it's pretty easy to tell when the professor doesn't know a damn thing about the subject matter at hand. I'd guess this is much less an issue in science engineering departments than it is in the humanities, where a professor with no background in a author/time-period/genre gets assigned a class covering that material.