I’m just returning from what is becoming one of my favorite conferences, the SABER meeting on biology education research. As a company that is driven by research on how to help students learn better, having a weekend surrounded by others who are doing rigorous studies on undergraduate biology education is really stimulating. There were lots of ideas to think about long-term, but a couple of thoughts on writing test items stuck out for me right away.
One was a set of rules of thumb for writing multiple choice questions, enumerated in the introduction to a talk by Teresa McElhinny and Julie Libarkin from Michigan State University. They included:
- Use the simplest language possible in both question stems and response options.
- Use plausible response options.
- Keep response options homogenous—similar language, similar length, etc., so students cannot guess the correct answer because it is, for instance, longer than the others.
- Never use parenthetical statements in questions—students don’t read them.
- Try not to introduce new concepts in response options.
- Avoid questions with answers that are combinations of other answers such as “Both A and B are true”.
Though these rules make a lot of sense, and we already try to follow many of them, they do have some problems if you are trying to draw response options out of students own words (a great way to make research—quality questions). For instance, if you ask students about a process in biology, they will often spontaneously talk about other processes. Using their own words would then violate the second to last rule above. More fundamentally, Ross Nehm’s group at Ohio State presented evidence that students often have mixed-up mental models of biological processes. Some of their thoughts about the process are correct, while others are incorrect. If you use a multiple choice question, you miss that complexity—one reason why essay questions are so much more powerful for finding an accurate picture of students understanding.
Nehm’s colleagues, along with a group at Colorado University including Brian Couch, Bill Wood, and Jenny Knight, are trying a different idea which I thought was promising for avoiding some of the problems with multiple choice. They set up a story or problem with a question stem, just like multiple choice, but then ask a set of true/false questions around that story. Each true/false addresses one aspect of the process that the student might understand separately from the rest, or one common confusion that students have. While we’ve all seen series of true/false questions before, the way these groups explicitly write them to test different aspects of understanding around one concept seems promising as an advance over multiple choice.
Questions are key for the virtual labs and interactive chapters we write as well as in our research around those materials, so I’m looking forward to trying multiple true/false questions to see if we can better hone in on what students truly understand.
