Some of my professor colleagues and I were recently chatting about end-of-semester type issues, including the excellent exams we were creating to provide our students with challenging and intellectually stimulating experiences that would perfectly and fairly test their understanding of the course material. And that would be easy to grade but that would not involve going over to the dark side of multiple choice exams, except in classes with > 200 students.
We found ourselves focusing on one particular perplexing issue that we had all recently encountered -- that students seem to be having more trouble understanding what a test question is asking for. Let me restate that, in case what I wrote is ambiguous and/or makes no sense: Some students are confused about what some sentences, phrases, or words mean. They may understand the concept the question is trying to test, but they don't know what the question is asking.
It is entirely possible that some of us write poorly worded test questions, but, for the sake of discussion, let's ignore that possibility.
Example: One professor who was teaching a geometrically oriented topic said that some students in a class couldn't deal with the phrase "not all of the angles [of a particular geometric object that was shown in an image on the quiz] are at 90 degrees". They had no idea if this meant that none of the angles were at 90° or if possibly some or all of the angles were at 90° but didn't have to be or if the statement required that some be at 90° and some not be at 90°. In fact, no students asked a question about the ambiguous phrase ("not all of") during the test, which is ideally when such issues are resolved, but that's another issue.
Students in another class had trouble with the concept of at least. In fact, I noticed that many (but not all) of the examples we were discussing involved phrases such as some of, all of, or at least. My hypothesis, which I proposed to my colleagues, is that long experience with multiple-choice type tests leads some students to try to psych out answers that involve quantities or time (never, always, sometimes, often) and to look for the 'trick' of a question rather than taking the question at face value. Even when a test is not multiple-choice, but instead requires the writing of words or sentences, some students may still treat a question as if there is some trick to it.
Every once in a while, a student will ask me "Is this a trick question?" about something on a test; or, retroactively "Was that a trick question?". I do not ask trick questions on tests.
Many multiple choice tests, however, do involve tricklike questions. By definition, you have to provide several wrong answers along with the correct answer, and one way to make this challenging is to make some of the answers similar, or to ask the question in a 'tricky' way (e.g., using those ambiguous time/quantity words or phrases).
The first time my daughter took a multiple choice test at her elementary school, she came home incredulous. She told me that she had just taken an extremely stupid test. She said "They gave us the answer right there in a list and we just had to show them which one it was."
This isn't a rant against multiple choice tests. Sometimes they are the only practical choice in large classes, but I think the multiple choice test culture might be creating a generation of skeptical and suspicious students who are always (or often) looking for the trick in a question. Or maybe there are legions of professors out there who do ask trick questions. I suppose that is possible, but at least I know that not all of them are me.
14 years ago
22 comments:
As a university student, I know that I often look for the 'trick' in a question, especially if it seems 'too easy.' Some of this might be because of the reasons you mentioned, but I think part of the reason is also that my high school specifically trained students to parse questions and determine exactly what was being asked because, they told us, that was what college professors would expect. Unfortunately, in a world where college professors (quite logically) expect students to read questions just as they would read any other bit of text presented to them, this can make a straight-forward exam nearly impossible to decipher.
It might also help to remember that, as children, many of us were exposed to a culture where wording and careful language were paramount - political corrections were going on left and right as words we had just learned (such as 'stewardess') became taboo and were swapped for new, better words ('flight attendant'), and we watched a president debate the meaning of the word 'is' on national television. Maybe some of this seeped into our expectations of the world at large, including exams.
Interesting. All these problem cases seem to involve scalar implicatures (e.g., even though "some" is logically compatible with "all", people generally assume that "some, but not all" is meant). Your students seem to perceive the difference between the literal reading and the conventional reading, and wonder whether they are being tricked.
I suspect the typical undergrad is an expert at multiple-guess tests. But their secondary teachers may very have included trick questions, and used language incorrectly (such as "not all" meaning "none").
The students I TA/tutor for are always asking me if the professor is going to be asking trick questions on the exams. I've asked them what makes a question tricky, and their answer has been (as far as I understand it--they usually give examples and discuss what was tricky about them) making it so that they have to notice words in the question and know/think about their definitions.
I never really know how to respond to that.
I'm proctoring an exam in a large class right now (in truth there are 4 students left). I *almost* laughed out loud when reading about your daughter's response to multiple choice exams. I had a student last fall from Indonesia who did the same thing at my first multiple choice exam.
I notice similar phenomena with lengthy multiple choice questions in my own courses; i have actually wondered about some types of generational declines in reading skills, as my students have problems not in the context of quantity-phrases that might point to 'trick' questions, but simply in parsing sentences with more than one phrase. I guess i should note that I teach at a large, research-1 state institution; so my experiences may be peculiar to my state's woefully inadequate k-12 system.
One of my main struggles as a freshman undergrad was the crazy multiple choice exams. For my first year biology lecture, the multiple choice answers could be: A, B, C, B and C, A and C, A and B, none of the above, all of the above.
Adding to this mess, I used to joke that this professor had his own "lingo" in terms of the way he phrases his sentences. As FSP had mentioned, the placement of "at least," or "some of" in a question could dramatically alter the perception of what the questiion is really asking.
To further complicate matters, this professor relies heavily on assigning reading materials in the textbook, so that in a way we were learning the material in the "textbook language," but writing the exams in the "professor language."
I took it upon myself to really pay attention in class, and to go to his office hours, not only to learn the material, but to get a feel for his lingo. While trying this didn't propel me to exactly ace his class (didn't end up too badly), getting into the routine of learning the professor's lingo to understand the exam questions definitely helped me later on in the upper year courses. I mean, we all have our idiosyncracies in the way we speak and phrase our sentences, so why not professors!
*Good* multiple choice tests are the hardest to write; it wouldn't surprise me at all if most of our tests are poorly written. I respectfully submit that using distractors that are similar to the correct answer in language, but differ in minor details are in fact tricky questions. (*Unless* your purpose is to teach careful reading, in which case students should know that from the beginning and have had practice quizzes.)
The discipline of physics, much to my chagrin since I'm not a physicist but am married to one and I hate having to admit that he's right *again*, has done the absolute best job of researching undergraduate teaching in the sciences. Through very, very laborious work, they've developed good MC tests: the distractors deal with common misconceptions. Eric Mazur and Ed Prather are two people to look up if you're interested in seeing some really reliable and well-written tests. Fwiw.
I love your daughter's reaction to her first MC test. :-)
In math we find that students struggle with the quantifiers "for all" and "there exists". For example, the definition of the limit in calculus starts "for all epsilon there exists a delta ...", and students typically confuse this with the other three possibilities. "For all delta there exists an epsilon" seems to be the most common confusion. Is it possible that this is related to the problem you are taking about?
Us mathies have the advantage of a standard phrasing (eg. "not all angles ..." could be changed to "there exists an angle which is not 90 degrees"), so variations in phrasing don't seem to explain the problem. Also these problems come up even before the first quiz, so at least in our case I don't think it is the test environment that is to blame. I suspect that it is actually a lack of experience with formal logic. It seems like quantifiers are not used precisely in general conversation, with the meaning usually being picked up from the context.
By the way, great blog. I discovered it a few weeks ago, and have been reading my way through the archive with great enjoyment.
I find that my students struggle with language in general. I had a light-bulb moment with a discrete math student six weeks into the course when I discovered that he didn't know what the word "suppose" meant. They have trouble with "at least" or "at most" or even "all". In my case it does not seem to be an inclination to parse things too closely or look for tricks. It seems that they simply aren't used to using exact language for anything. This shows in the appalling grammar and word usage of their writing as well.
A colleague had to explain to his calculus students that they should search for the imperative sentences: "prove..." means to do this, but a problem that begins, "A function f is continuous..." means to assume this is the case, not to prove it!
I have been repeatedly accused of writing trick questions in a multiple choice test. After further probing, I often find that when a student says "That was a trick question" they actually mean "Some of the other choices contained words that you have used in class and thus sounded like something you might say."
Other times, they use "trick" to describe [wrong] answers that contain 'always,' when if the same answer contained 'sometimes' instead, it would be correct. I don't know why that's a trick. Isn't it legit to see if a student knows the difference between when something is always the case and when it isn't?
Squeaky wipers, that sounds like the evil multiple choice questions we got in high school chemistry. We got a list of statements and had to choose between:
A) 1, 2 and 4 are correct
B) 1, 2 and 5 are correct
C) 1, 4 and 5 are correct
C) 3 and 4 are correct
D) 3 and 5 are correct
Nasty for students, but probably a better test of understanding than a straight multiple choice!
The MC questions I found the most challenging were the ones a particular undergraduate biology professor liked to write where he asked you to "circle all correct answers" from a list that was 6-10 things long. Any number could be correct, so you really had to know the concept being tested to get it 100% correct. This was the same professor who usually included in his long lists of potential answers something that was so out of left field that it was blatently wrong and frequently chuckle producing. In retrospect, I wonder if he did that intentionally to monitor students' progress...
I deliberately avoided like the plague fields where multiple choice is the norm.
That puts me in fields where we had to show our work or write out extended answers.
I tended to assume they couldn't be asking something so stupid, so I would always overthink the question. I don't think that had anything to do with taking too many multiple choice exams (but I could be wrong about that).
I would argue that many times the exam-writer has a specific number of examples in mind (for example, they want 4 types of X that fit the category they're asking about) but they don't say in the question that the grading will be based on how many of these 4 things you're able to come up with.
They did this a lot at my grad school on the assumption, I guess, that we just memorized the lecture notes verbatim.
I remember being penalized because I picked 4 OTHER examples and although they were in the same category and were strictly correct, scientifically speaking I was given a zero on the question because I didn't pick THE 4 THEY WERE LOOKING FOR.
Message received: Only regurgitation will be rewarded! No actual knowledge, application of knowledge or thinking will be allowed!
Little did I know, this should have given me more warning about how scientists deal with new information. These aren't the droids you're looking for...
I will take a moment to thank the lord now, because I am so very, very glad I'm done being a student.
I will never, ever visit those kinds of ambiguities or mind games on my students!
I am in math and also find that many students have trouble understanding simple sentences. Many students have incredible difficulty understanding that a statement and its converse mean different things. For example, they cannot grasp that the sentences "If a shape is a square, then it has four sides" and "If a shape has four sides, then it is a square" have different meanings.
I have come from physics and math, where multiple choice was not a problem, to grad school in a biological science. I have no problems with "for every epsilon there exists a delta," and the other standard qualifiers in physics/math problems. The bio classes where I have seen multiple choice problems tend to fixate on 'scalar implicatures' to determine an answer, instead of on conceptual knowledge. In other words, the question must be thoroughly read and re-read to assure you parse the "some" "all" "most" "few" "many" words correctly, as opposed to thinking about the phenomenon discussed.
Maybe it is just my lack of experience in the field.
The first thermodynamics course that I took was in the Mech E department. This professor was determined that no engineer would graduate from his department by continually earning partial credit or by just earning all C's. Thus he would always give 5-6 question tests, two of which had numerical multiple choice answers. Everyone always complained bitterly about how missing one of those questions would instantly dock you two letter grades. The multiple choice questions weren't particularly different in difficulty level and I remember that the answers were almost always plausible. It was a pretty effective roadblock, just like the professor wanted.
An excellent resource for writing multiple choice questions on a very high cognitive level can be found at http://web.uct.ac.za/projects/cbe/mcqman/mcqman01.html
My only teaching experience is from High School science, but one of the things I tried to do was point out places where I was using specific meaning words. Things like "positive" means "not negative" but also "not zero".
I remember a particular example from my graduate courses; it took me several weeks to figure out exactly what the professor meant by "finite". I understood the "not infinite" part, but was missing "not zero".
Great post! Late to the conversation, but you may get a kick out of this. I point out, in class, at the beginning of the semester, that what people generally mean when they call something a "trick question" is that they completely missed a pivotal piece of information. This has really cut down the number of accusations I've had to deal with.
I try hard not to include trick questions, but some questions do (must!) require a fairly nuanced understanding of vocabulary and concepts. If a question is at risk of being "trick," I try to put the pivotal word or phrase in bold or offer a hint. Some students will still give a wrong answer, especially if (and this is an idea some students seem to be able to grasp) they don't really know the material very well. Yes, I'm in science. And yes, I allow students to discuss questions with me after a test, and I have often given partial or full credit back on a problem if a student could make a compelling point for another answer. I think that's pedagogically defensible, since the student must make the effort to put forth a valid argument, but I have wondered if I'm rewarding squeaky wheels while shyer or less-confident students are left out.
When learning the sciences, or higher mathematics, you start with a lower level of knowledge, and generalize or even ignore advanced concepts in order to get the information across.
The problem is, if you do have higher levels of knowledge due to your personal diligence, you can accidentally outwit a question.
Let's look at Bio 101.
"What are two things all life depends on?
a. water an sunlight
b. sunlight and air
c. plants and oxygen
d. carbon and oxygen"
The answer is supposed to be "a," but "a "is not correct. Members of Domain Archaea simple metabolize compounds in rocks. To make things more confusing, if you are familiar with chemistry, all life is made of carbon and oxygen.
Maaaaan I hate multiple choice, when I make multiple choice questions I make them blatantly different or I ask so many questions that it outweighs bias introduced by more knowledge or confusing wording.
Post a Comment