Thursday, January 14, 2010

Tell Me Something I Don't Already Know

Like many of you who teach, I have recently been poring over my teaching evaluations from last term.

Brief attempt to forestall a few of the usual cynical comments: Yes, I care about teaching even though I am a full professor and my evaluations don't affect whether I get to keep my job. Yes, I read my teaching evaluations even though I know evaluations are flawed (except when they are really positive..).

Let's say that you feel pretty sure that your class(es) had overall gone well, and the teaching evaluations confirm that. So, if the evaluations just confirm what you already knew (or thought you knew), are they any use at all to someone like me who doesn't 'need' them?

Yes, in some ways. For example, in one of my classes last term I tried some new things: I tried a slightly new format and I introduced some new reading materials. Also, in this class the students do a lot of writing. I wanted to know what they thought about all of these things. I asked them in class and got some opinions that way, but not everyone expressed an opinion at the time. I had 100% participation in the teaching evaluations, so I got a lot more information that way. It was interesting to read that many of the students liked the writing assignments and the feedback they got on them.

And when I teach a new course -- either one that is new to me or one that I create -- it is good to get some feedback on the content and organization of the course.

Of course there are the usual vague comments: "The order of topics didn't seem quite right." End of comment. Which topics seemed out of order? What would be a better order? This information is not provided, but even vague comments like that can be semi-useful because they make you think about the organization and whether anything can be improved for next time.

After reading my teaching evaluations, I started thinking about the correspondence between professorial perception of whether a course went well and student evaluation of how the course went. Possibilities:
  • Professor thought course went well : teaching evaluations agree
  • Professor thought course did not go well : teaching evaluations are negative and hostile
  • Professor thought course went well : teaching evaluations are negative and hostile
  • Professor thought course did not go well : teaching evaluations are very positive
My hypothesis is that the last two are more rare than the first two.

I think we generally know when students are happy or unhappy with a course. We may be surprised at the magnitude of the general assessment -- e.g., you might be fairly sure that a course went well and then be pleasantly surprised at how positive the evaluations are; or you might be fairly sure that a course had some rough spots but be unpleasantly surprised and distressed at the level of hostility in the comments -- but I think we generally know the result to the first order.

Agree? Disagree?

19 comments:

a physicist said...

Agree. I generally know the result to first order.

What often surprises me is that I may think that the biggest flaw with the course was X, but the students don't mention X at all and instead mention Y. (Where both X and Y are valid critiques.) This is why I'm always interested to read my evaluations. But still, the overall reaction to my teaching is usually what I expect, even if I'm incorrect about the details.

Most surprising critique from a disgruntled student last semester in my intro physics class: "Professor spent too much time on a side topic and not enough time on the main topics." This was a criticism of a side topic which consumed 1 class period out of 40 and which many students commented positively about.

Anonymous said...

I taught one semester when I decided to change how I was teaching completely, using a new textbook, completely new lecture notes, a bunch of untested ideas. The whole reason is that the subject is quite difficult and I was trying to improve how students learn it. On top of that, this was a large (> 150 students) course, for non-majors, many of whom have no interest in the subject and are not used to problem-solving courses of that nature.

I thought it went poorly, and the public hostility of many students was severe. I did get the worst teaching evaluations I ever had that semester, but interestingly, they were still above average. The number of students saying I was "excellent" or "good" was greater than those saying "neutral", "bad", or "really bad". Many of them in particular liked the very aspects of the course that angered the other students. This doesn't mean, of course, that I did everything right and those negative students are idiots. I do believe that there is no way to teach such a course so that 1) everyone likes it and 2) they all learn what they need to learn. But I probably could have taught it so that more students liked it, without compromising the learning.

Anonymous said...

Maybe this reflects my insecurity about teaching, but "thought it didn't go well"/"positive evals" is actually the most common outcome for me.

amy said...

I only predict my evals well after I've been at a school for a couple of semesters. When I moved from the west coast to the east coast for my first teaching gig as a professor, I had a class with lots of discussion -- students were challenging every idea I presented. I thought it was great; I love lively debate in class. But I got creamed on evals: they thought I wasn't authoritative enough and that I didn't know what I was talking about. My openness to discussion, which had worked so well when I was in California, was interpreted by them as weakness. Then when I moved to the Midwest for my current job, I thought my first classes were horrible -- the students just sat there, didn't look interested or excited at all, didn't want to talk. But my evals were great -- they said they really enjoyed the class and found it very interesting. After a while I realized Midwesterners just have bland expressions on their faces a lot of the time; they don't display much affect.

Dan said...

Another interesting evaluation matrix which I've often fantasized about having is student performance vs overall evaluation.

I'm really interested in negative reviews from good students, mildly interested in positive reviews from bad students, possibly interested in positive reviews from good students, and definitely not interested in negative reviews from bad students.

Unfortunately, all the info I usually get it the (anonymous) review. My previous university had the students predict their own grade in the course on the evaluation - I never realized at the time how useful that could be.

John V said...

Um, the only thing that matters is whether the students think the course went well and whether they feel more educated. They or their parents paid the tuition, they put in many more hours taking the course than the prof did preparing the course.

Maybe a few students, years later, lamented their sophomoric judgment in preferring unchallenging courses and entertaining profs. Most stick with their initial opinions.

I'm personally appalled at the profs who proclaim their courses were good for the students despite the students' complaints of overwork, inscrutable presented material, and cruel profs.

Particularly aggrieved at the teachers of courses where my undergrad and grad students tell me of the tedious hours listening to painful lectures and busy work, as the profs are simultaneously telling me how the students are benefiting from their old-testament teaching philosophy.

One needs to factor out the bias towards students liking to get a good grade or having little work (or being mislead into thinking their grades will all be good long enough to conduct the evaluation), but otherwise, student opinions rule.

Lorac said...

I must be one of the few students (I take continuing education courses in my field) who believes that evaluations should be specific and offer suggestions, not just complaints. I also typically sign evaluations.

Sadly, I increasingly find myself in a minority group of people who value helpful criticisms and are more interested in service/helping/training/promoting others than in my own glory.

Anonymous said...

Thought you would enjoy watching this video.
http://www.youtube.com/watch?v=WwslBPj8GgI

Kevin said...

John V. claimed "Um, the only thing that matters is whether the students think the course went well and whether they feel more educated. They or their parents paid the tuition, they put in many more hours taking the course than the prof did preparing the course."

Not true, John. In engineering, at least, it matters whether the students actually learned the material, not whether they felt more educated. Also, in courses for non-majors it is not at all unusual for the faculty to put in more effort than the average student. Our evaluation forms include student estimates of how much time they put into a course (in hours/week). The low level of effort from some of the students is astonishing---and that's only including the students who actually showed up for class on the day evaluations were done, so the numbers are biased by including more of the diligent students.

a physicist said...

John V, I agree with 95% of what you're saying here, but I disagree with an important 5% of it.

A big problem, as you say, is professors who claim their teaching methods are awesome and yet students hate them. Students are the customers and customer satisfaction is crucial, which is the point of your post. But another point is that the profs who claim their teaching methods are awesome are, often, clueless themselves about the ineffectiveness of their teaching.

More and more, I'm seeing a lot of talk about assessment of teaching effectiveness. Schools may start being required to assess whether or not students are learning the material. And perhaps even making the results publicly available... although that may not happen for obvious reasons. At our school, there is a big push from the administration to begin assessing learning outcomes.

So, those evil profs who claim their teaching methods are good for the students... we can all hope that some day they'll be exposed as not only unpopular, but also as ineffective! And then shortly thereafter we'll all get magic sparkle ponies and live happily ever after.

John V said...

Kevin,

It's a bit tough to prove that courses crammed with material serve students better than less feared classes. I suspect it is more productive to engage students with tractable problems and engaging lectures than drilling them with long painful assignments, but that's just my opinion.

Also, I meant the students COLLECTIVELY have invested orders of magnitude more effort in the class than the prof, even though each one usually invests less (but not always).

John V said...

A. Physicist,

I think we both mostly agree, in an ideal world, teaching effectiveness is measured and appropriately rewarded or punished. Perhaps advances in measuring learning will weed out the most useless teaching strategies eventually.

I don't see progress yet, but am an optimist. Unfortunately, we're probably headed for standardized curricula, as individual teachers would be hard-pressed to assess their own idiosyncratic methods.

I don't know how good profs are at self-assessment, aside from reading their evaluations, but I'm dubious.

Kevin said...

"It's a bit tough to prove that courses crammed with material serve students better than less feared classes."

True. It is hard to prove anything about education. I did not make any claims though about the relative value of different styles of courses, though. Only about satisfaction being less important than what was learned (to use the grossly over-used business analogies: students are our products, rather than our customers).

"I suspect it is more productive to engage students with tractable problems and engaging lectures than drilling them with long painful assignments, but that's just my opinion."

I agree. What makes you think I was claiming anything else? Perhaps you were stereotyping engineering classes without having experienced them??

Unbalanced Reaction said...

Personally, I agree. I wonder why so many of my colleagues are ALWAYS surprised by terrible teaching reviews. I never understand why some people don't realize that they aren't doing a good job.

John V said...

Kevin,

Where did engineering come from? I wasn't even thinking of engineering. And I have taken both undergrad and grad classes in engineering, for what that is worth, albeit in the eighties and only a handful of each.

re students are our products, rather than our customers - the business model of the schools for which I've taught is that the students select the schools, and further, select their majors at the schools. They select their classes, too.

Classes, majors, and schools with no students fold. Give the students some credit for intelligence to wisely choose classes.

adam said...

I have found that my undergraduate student evaluations (based on comments) are more positive than my own judgements of how the course went.

Unlike one previous commenter, I don't think that I'm insecure. about my teaching. Rather, I think that:

(a) the rest of the teachers in my department suck even more badly than I do;

(b) the students tend to be nice on their evaluations if they think you care, which I do (and they apparently believe);

(c) The students who really hate the course attend class less regularly and hence tend to miss the (unannounced) course evaluations.

Just to pre-empt a few criticisms: my course, as I teach it, is one of the hardest ones in my department, so I don't get any student brownie points for being easy. I do, however, do things like learn students' names.

So my comment is really about two questions for FSP and others:

- Do the factors above positively influence teaching evaluations at your institutions?

- Are teaching evaluations in your department made public, or even visible to other faculty? [They are closely guarded secrets at my (large state) university, and I haven't been able to find anything out about how I am doing relative to my colleagues other than vague assurances that my evaluation scores are solid.]

Pagan Topologist said...

"Professor thought course went well : teaching evaluations are negative and hostile"

I find this to be too common in large lecture courses for non-math-majors..

Lola said...

I fit into the category of "thought it didn't go well/positive evals." However, my belief that it didn't go well had less to do with the actual course and more to do with the fact that I'm a grad student and it was my first time teaching my own class.

I expected to receive some of the negative comments that were mentioned in your earlier blog post (ex. "I can't believe that I'm spending $40,000 per year in tuition to take a class from a GRAD STUDENT") but I found that my students were actually very supportive.

I think that teaching evals are extremely useful to those of us in the earlier stages of our careers. The suggestions that I received for areas of improvement were generally well-founded. Moreover, it was a huge confidence boost to realize that I did a better-than-average job (compared with all of the faculty in my department). I'll definitely go into the next course that I teach with higher expectations for myself.

Madscientistgirl said...

When I was a grad student in physics at an ivy league school, the options for ranking various qualities of a class on the student evaluation were something like: poor, acceptable, good, very good, superior, excellent. This from professors who complain about grade inflation... When my teaching evaluations had mostly the highest rank, I found these evaluations unreliable. And how do you even interpret "very good" verses "excellent"?

I think good teachers usually know how they did before evals (and good teachers can have a bad semester,) but I don't think bad ones do. Sure, student evaluations are imperfect, but biased data are better than no data.

I also think that the students who do poorly in the class have a very important perspective - clearly these are the students the teacher did not reach.

Students are evaluated by us constantly, and grades have a huge impact on students' lives. If we dish it out, we should be able to take it. I have seen many professors tell a student in his/her class that grades don't matter, but then later profess the brilliance of a student because he or she got all A's. Not all grades are fair, and they can be somewhat arbitrary - I only understood how much the first time I had to assign grades. There is generally a correlation between what a student learned and the grade a student gets, but it is not perfect. The same is true of teaching evaluations.