Justify the distractors when writing multiple choice items

Before I get into the meat of this post, a caveat:  Don't actively WRITE multiple-choice items for your classes unless you have no other option.  If you're teaching at the AP or Regents level -- even if you're not technically teaching a course labeled "AP" or "Regents" -- multiple-choice questions are more plentiful than Tribbles in an unpredated colony.  Use them.  I still, after 17 years of teaching, don't ever use multiple-choice questions of my own design on tests in these classes.  That said, good multiple-choice for conceptual physics, and (currently) for the new AP Physics 1 course, are more comparable to American Bison.  They're out there, the population is slowly increasing, but they're tough to find outside of Yellowstone National Park.

Multiple-choice questions must be carefully constructed if they are to test physics knowledge rather than general test-performance instincts.  Just as we don't want a free-response question to be game-able by anticipating the rubric, a multiple-choice item should not be game-able by eliminating choices for reasons unrelated to physics.  If a distractor* in an item is so silly as to be instantly eliminated without physics reasoning, the question is poor.  For example, a calculational question in which only one choice could be obtained by any possible algebraic manipulation of the given values; a question with three short, poorly-phrased phrases and one long, clearly written sentence; or a verbal question in which one or more choices doesn't even refer to physics principles.  

* "Distractor" in ETS-speak means "wrong yet plausible choice"

Questions for the new AP Physics 1 and 2 exams will have only four, not five, choices.  Why?  At the meeting for AP consultants last April, we were told that in their research, ETS could find no statistical difference in performance between four-choice and five-choice questions.  Furthermore, it was contended that the fifth choice increases the reading comprehension burden on the student, to the extent that students in studies were often not reading all the way through the fifth choice.*  And finally, the committee's perception was that, all too often, test authors grasp at straws in coming up with a fifth plausible distractor.  

* This should surprise no one who sends class emails beyond one sentence in length.  Try burying an offer of extra credit in the middle of an email's second paragraph, and you'll see.

In order to make the question authors focus more on the reasonability of the distractors, ETS now asks authors of each item to justify each distractor.  We of course mark the correct answer, perhaps with the reasoning that we hope will lead the student to choose that answer.  But then for each of the three incorrect options, we fill in a box to say why we think a student might plausibly, if incorrectly, choose that wrong answer.

The process of justifying distractors makes writing questions more time consuming, in that the expected incorrect reasoning must be articulated in words.  But the questions become of such higher quality.  Plenty of times I've improved the language in a distractor, or changed it entirely, because of a disconnect between what the problem stated and my justification for a student's thought process.  The item review process can be more efficient, not only because the justifications for each choice are spelled out, but because fewer poor-quality items are submitted to begin with.  (In previous years I've reviewed questions in which the author seemed to just make up random crap for a couple distractors.  With the requirement for written justification, many of these questions will be weeded out before the review stage.  It's not likely that even a less-skilled test author will submit a justification that says "No student would ever pick this choice unless he's a total dipstick.  I'm out of ideas.")

And writing justification for each potential wrong answer forces the writer to consider common misconceptions.  One of my favorite sources of inspiration in writing multiple-choice is to think about open response problems that I've given to my class, or that I've graded at the AP exam reading.  On such problems I generally have an intimate familiarity of the myriad ways in which students can be wrong.  So I rephrase the problem as a multiple-choice item, sometimes with distractors nearly verbatim from actual incorrect student answers.

If you have written multiple choice items for your own test, look at some with an eye to justifying the distractors.  Why would an intelligent student with an incomplete or incorrect knowledge of physics choose each one?  If you can't say, then eliminate the choice.  (If you're not teaching AP or Regents, who says every multiple-choice item has to have the same number of choices?  A three-choice question, with answers like "greater than, less than, or equal to" can be just fine.)  Mercilessly cull any question that  can easily be answered without physics knowledge.  Have a trusted and knowledgeable colleague review the items you've written.  The quality of your tests will skyrocket; and your students' understanding of physics will improve as well.