Using open-ended situations for exam review

I'm in the process of writing 5 Steps to a 5: AP Physics 1.  The success of this book in its mission of preparing students for the new AP exam will hinge on how well I can communicate the changed expectations in the new exam:  calculation will still exist, but will take a backseat to verbal explanations, graphical analysis, semi-quantitative analysis, and other ways to describe physics that don't involve a calculator.  

My instinct is that students are willing to work through example problems in a text or prep book.  But once they start, they're extraordinarily answer-focused.  Once the answer is right, the reader moves on; the occasionally follow up questions like "what if the initial speed were increased?" remain wanting for attention, because the student got the right answer, dagburnit, and so he understands just fine and is moving on.*

*Or, the follow-up questions are ignored because the student didn't get exactly the right answer, and so it's far more urgent to him to find the math error than to keep reading.

Thing is, that's no way to learn physics.  A deep understanding of physics, like that required for the new AP exams, requires probing beyond the algorithm that obtains an answer.  And even when straight-up calculation is tested, the ability to explain the methods behind the calculation will be critical.  

In my classes, I've used class demonstrations as the starting point for teaching how to go beyond the calculation.  Sure, I focus my demonstrations on a solvable example problem, such as “predict the reading in this spring scale.”  But the actual presentation of the demonstration includes all sorts of questions and ideas beyond the final calculation.  I’m finding the correct answer, sure, but I’m also modeling the general approach to physics problem solving: determine what physics concepts are relevant and why, and then figure out how those concepts apply in this particular situation. 

An alternative approach would be to eliminate the calculational focus of a physics problem altogether.  In her blog, Kelly O’Shea describes her overarching theme of the “goal-less” physics problem: by the end of the course, she aims to present a physical situation by itself, then to make the students decide for themselves what specifics can be calculated and described.  Ability to deal with the goal-less problem is, ironically, the ultimate goal of physics teaching.  It certainly takes time and practice to bring students to the point at which they can handle a goal-less problem, as Kelly points out; but as an end result, or in a broad review of a course, the goal-less problem is where we aim.

The structure that I've adopted for my book modifies the traditional "solve a bunch of typical example problems" approach in a way that provokes the reader to consider multiple representations of a physics problem, just like my in-class demonstrations, just like Kelly’s goal-less problems.  In each chapter, I’m posing several “examples,” which include only the stem of an AP-style problem.  Then I’m citing the specific facts of physics that are relevant to each example, followed by a narrative explanation of how these facts can be used in conjunction with the example situations to make calculations, graphs, or other predictions. 

I’d encourage you to try a goal-less approach in your own class, at least in conjunction with a topic review.  After you’ve taught kinematics, offer up a situation, such as “a cannonball is shot off of a 150 m high cliff at a 30^o angle above the horizontal.”  Ask what facts of physics apply, what can be determined about the situation, what can be calculated, what could be calculated with further information, what graphs can be constructed, how all these predictions would change by changing the height or the angle, and so on.  When the class can’t contain themselves for all their ideas, that’s when you know they’re ready for AP Physics 1.

Don't Game the Test -- each year's AP rubrics are different!

I've always been annoyed by Big Test Prep, and not just because of the competition for my AP prep book.  The entire industry promotes the attitude that standardized tests -- and by extension, all tests -- are games to be won via trickery, rather than authentic evaluations of knowledge and skill.  It's all BOOST YOUR SAT BY 100 POINTS USING THIS ONE WEIRD TRICK THAT THE COLLEGE BOARD DOESN'T WANT YOU TO KNOW!!

Folks, the AP physics exams are excellent exams, well-constructed by experts in physics, experts in physics teaching, and experts in test design.  The free response portions are graded to rubrics by savvy, intelligent physicists.  There's only one approach that will lead to success on the exam: know physics, and know how to communicate that physics knowledge.  Nevertheless, as an AP reader and consultant, I see teachers and students trying to game the exam.  

The biggest misconception about the AP exam is that there are hard and fast year-to-year rules behind assignments of points on rubrics.  

Folks, the rubric for any particular exam question originates with the original author of the item draft.  The development committee revises the question and its rubric.  After the exam is administered, table leaders are assigned to each question.  These table leaders show up a few days early to the Reading so that they can page through hundreds of sample exams.  They adjust this rubric again so that it can be applied consistently and usefully for the range of actual observed responses to the question.  

And there's more!  The table leaders have to present and defend their rubric to the entire group of exam leadership.  Suggestions and advice flows from problem to problem.  Now with the perspective gained from seeing all the rubrics from all the problems, each set of table leaders revises and finalizes their rubric.

The upshot of all this rubric design is that a "trick" that would have worked on a question on this year's exam is unlikely to work on a future exam.  

For example:  Take a look at the rubric for problem 3 on the 2011 AP Physics B exam.  The first part of this experimental question asks the student to check from a list the items he plans to use in the experiment.  The rubric awards two points for checking boxes: one point for checking a ruler or meterstick, the other for checking the light source.  Then part (b) asks for a diagram of an experiment; as long as every item that was checked in part (a) shows up somewhere, this part earned the full two points.

The unfortunate instinct of so many teachers and students is to exclaim, "Oh, if I just check every box and draw a labeled picture of each item, I get four points."  Well, that was true in this case.  No one, not even the development committee, could have really known that ahead of time in order to make use of that information, but sure, it's true.

Then the even more unfortunate logic says, "So, whenever you see an experimental problem, check every box and draw all the equipment, even if you're not sure you need it."  Bzzzz.  We who grade AP exams are not stupid, despite the hopes of Big Test Prep.  In other experimental questions, credit has been awarded only for a diagram that would work; or for only one specific aspect of the diagram; or credit may have been lost for silly checkmarks.  In future years, who knows what such a question's rubric might look like.  I talked with a group of readers who agreed with me that one good approach might be not to award credit for checkboxes at all!  

It's NOT true that you always get points for putting units on the answers.  Incorrect arrows on energy level diagrams or on free-body diagrams often (but not always) lose credit. 

So teach your students how not to game a test.

In your own class, use authentic and therefore unpredictable AP rubrics for authentic AP questions.  If you give your own questions on quizzes, don't set up a game-able rubric.  Don't allow students to ask questions on a test or quiz.  Don't allow the student to give two answers, hoping that one is right!  AP readers are instructed that if they see two answers* to grade the one that earns fewer points.  

* For example, "the acceleration could be to the right because of the net force, or the acceleration could be to the left because of the speeding up."

Above all, please don't even imply that the exam can be gamed.  Just encourage students to communicate how well they know physics... which is pretty darned well, because they have you to teach them, right?  There's no need for tricks!

Structure of a class: AP Physics 1 [Formerly AP Physics B]

At Woodberry, the first-year AP class is generally taught to bright seniors.  Now, this includes a wide swath of folks, from the valedictorian all the way to the dedicated student with a 550 SAT math score.  Although everyone in the class is by definition a serious, motivated student, some are more motivated than others.  Even in AP it is still necessary to grade homework daily, hold students accountable for their preparation via quizzes and such, and provide entertainment enough to keep attention as the students trudge through a seven-period academic day.

A typical class starts with a quiz, which begins at the bell.  Daily quizzes in AP are sometimes 3-5 question multiple choice; sometimes a question about the previous night's homework; sometimes pure recall; sometimes even an authentic AP problem.  The purpose is not only to review, but to provoke a good discussion.

I collect the problems from each student's desk during the quiz.  

We take as long as necessary to discuss the quiz.  Then I'll ask the class pointed questions about the previous night's homework, possibly provoking a good discussion.*  Once questions about previous topics have petered out -- or once I decide we've talked long enough -- we move on to the day's lesson.

* Though I never just do the problem on the board for the class.  The dialogue here is more like "So, how did you figure out the mass of the cart, since it wasn't explicitly given?"

The "lesson" is generally an example problem that I pose and work through for the class.  Equipment is set up on the demonstration table so that the answer to the problem can be verified experimentally.  This is the "quantitative demonstration" -- we develop the intellectual habit of placing every problem in a laboratory context.  

This "lesson" is as much performance art as it is classical lecture.  I'm modeling the habits of good problem solving through the way I structure the board work; I'm interrupting frequently to ask "check your neighbor" questions; I'm engaging the class at every opportunity, and using every trick at my disposal, in order to maintain focus.  A typical demo takes anywhere from 10 to 30 minutes to complete.  

I try to time the end of the lecture with some sort of cliffhanger -- "Now we've solved for the cart's speed, will the measurement match the prediction?"  or "So tomorrow, we'll double the mass of the cart, and decide how the cart's speed changes."  

Since AP Physics 1 is less broad than the B course, I frequently have time to end class with 5 minutes of individual problem solving on the night's homework.  Students are expected to get something accomplished in those five minutes: the "ticket out the door" is to show me the written work they've done.  (I don't care whether said work is right or wrong... the only way anyone gets in trouble is if he shows me a blank paper and says "I don't know what to do.")

What about lab days?

The above class structure is used about three days each week.  One double-period is used for laboratory work.  I discuss the specific structure of a lab day in this post.  

And tests?

In AP, I only test once a month or so.  Those tests are entire period tests, usually using the 90-minute lab class.  Sometimes I'll use a class day soon after to do corrections.  

Review for the AP exam in April requires a different structure, too -- but by then I've established the class norms enough that I can use the 9th grade class structure with music and independent work.  

Structure of a class: 9th Grade Conceptual Physics

I believe in establishing a predictable routine for any class that I teach.  At the lower ages, structure is even more critical.  Because my students know generally what to expect from class each day, they are less vulnerable to distraction.  After a couple of months, the class is almost able to carry out the daily functions without me even saying a word; thus, we can focus on learning physics rather than on what specifically to do.

Now, your routine will almost certainly differ from mine, depending on your personality, your class time, class size, age of student, level, all sorts of things.  But I'm asked often enough about what actually happens in class on a day-to-day basis that I think it useful to go through the routine.

In 9th grade, on a typical class day:

We start with a 3 to 4 minute quiz, during which I collect the problem set that was due.  (Since I collect the problems from each student's desk personally, it's nearly impossible for a slacker to skate by without me noticing incomplete homework.)  The students trade and grade the quizzes.

I take just a few minutes to answer questions and to show the class any information necessary for the day's activity.  If we have truly new material, I've already printed and handed out a fact sheet for their reference.

I ask who had the highest quiz score; this student gets to choose the Pandora station that I play during the rest of class.*

* Music is perhaps the most interesting innovation from this past year of teaching freshmen.  Since the last half of class almost always involves students working independently as opposed to me talking at the front of the room, there's no reason NOT to put some music on in the background.  The quiz is taken that much more seriously, knowing that music selection is the reward for performance.  It's amazing to me just how important this reward is to the class.  And, woe to the class when someone doesn't complete his problem set.  We don't listen to music unless everyone turned in the homework.  Peer pressure can be useful...

Students are released to work at their own pace on a set of problems and experiments.  Generally, the students solve a problem or a series of problems, as they would on a nightly set.  As they finish each problem, they show me their work.  If they're wrong, I send them back to their seat to do it right.  If they're right, they proceed to the next step.

Usually the "next step" after solving a problem is to head to the back of the room, where the problem is set up as an experiment.  The students are asked to perform the experiment to measure whatever they predicted on the in-class problem.  For example, the in-class problem might include questions about the motion represented by a velocity-time graph; the experiment would then be to produce the graph with a cart on a track and a motion detector.  They show me a printout or a picture of the experimental results... and then they get a new problem to do.

We end class in time to straighten up quickly, and for me to hand out the next day's problem set.

This format has variations... sometimes we do a more traditional laboratory exercise in groups instead of the individually-focused problems and experiments.  Sometimes we do test corrections, or problem set review.  But the typical quiz - brief talk - independent work model is almost always used.

No "Mercy" -- train students to treat exams dispassionately

Saturday we finished grading all 100,000+ AP physics exams.  I was assigned throughout the week to two problems – B7, the one about atomic energy levels; and B4, the one about two blocks connected over a pulley that create projectile motion.  I feel like I’ve seen every available correct approach to these problems, as well as every possible misconception that our students have.

As you can probably imagine, a non-negligible number of exams were taken by students who were woefully unprepared.  We saw blank papers, and not just on question 7.  We read clever doodles, poems, messages, “Mr. Lipshutz should be fired,” “Thank you for grading my exam, hope you have a great day,” and, of course, the classic “Kick Mr. Kirby* in the butt for me.”

* Observed multiple times over the years at the reading:  “Hi, you’re Martin Kirby?”  “Yes, I’m Martin, nice to meet you.”  “Hey, could you turn around?  I have a message to deliver that one of your students wrote on his exam.”

I don’t mind all these.  If a student didn’t study appropriately all year, his punishment – or at least the natural consequence – is to sit still for three hours taking an exam he has no chance of passing.  If he wants to spend his time entertaining me, more power to him.

One of the things students write that most bothers me, though, is the serious or semi-serious plea for mercy.  “Please, I’ve worked hard all year, my teacher is new, I really want credit in college, have some mercy!  Give me some points, please!  We didn’t do a problem just like this in class, but I’m answering the best I can, be nice and give me a 3.”

What makes the plea for “mercy” bothersome, while I laugh respectfully at the animal drawings? 

Perhaps the principal battle that physics teachers must fight involves those students who don't adjust well to the unique nature of a physics course.  Students who have earned As throughout their school career because they can memorize and they have a big vocabulary often become frustrated by creative problem solving.  As they see they might not earn an A in physics, as they (think they) see their visions of being valedictorian of their Harvard Medical School class going up in smoke, they -- and their parents -- fight.  Such a student tends to use "compassion" and "mercy" as a weapon.  They attempt to portray the physics teacher as a mean person, hoping to pressure him or her to back off the course expectations.

This sort of smear campaign would be comical if it weren't so effective.  All it takes is one or two physics-ignorant teachers to champion such a student's cause; then even if the teacher stands her ground, a subset of students feels validated enough to persist in their hostile attitude, spreading their incompetence and despondence throughout the class.  And if the principal doesn't give an emphatic smack-down to the first whiny parent, the teacher is up a creek.  

I want to change the conversation everywhere, not just in physics.  An exam should be viewed as a dispassionate, objective evaluation of a student's skills.  Teachers do not "give" grades, students "earn" grades.  A score, good or bad, on a test doesn't reflect on the character of the teacher: a teacher is not "kind" if the class does well, a teacher is not "mean" if the class does poorly.  

Poor performance on an exam should be viewed like a loss at an athletic contest: it doesn't necessarily reflect on the character of the test taker, it's just an evaluation that "your team was not as good as the other team today."  A loss should not be attributed to the fact that the coach is mean; a win doesn't mean the coach is compassionate.

So when I see students pleading for mercy on their AP exams, I despair.  Such pleas are too often learned behavior.  They see whining, begging, smearing as effective in their own school, so they try it on the AP exam itself.  Now, I don't know what makes anyone think that an AP reader, who is generally a consummate professional bound to grade precisely to a rubric, has any ability or desire to raise or lower a particular student's score.  That's as crazy as suggesting that NFL officials are out to get Seattle, or Cleveland, or whoever.  

Strike down the language of "mercy" early in the year, so you can focus on learning physics.  If you ever need help on that account, I am more than happy for you to show this post to parents, colleagues, or administrators; or I'll even add an amicus curiae on your behalf.  None of us anywhere can teach properly if our students are gaming schools' social structure rather than practicing their problem solving.

Trick to writing paragraph-length comments: Sort the roster first.

Every marking period -- that is, six times each year -- teachers at my school are asked to write a paragraph-length comment on each of our students.

Now, most schools don't require their teachers to write narratives about students.  But those teachers who have a similar requirement, even if it's only twice a year, are asked to take on a daunting writing task.  That's four to five thousand words each marking period.  How is it possible to streamline the process?

A colleague of mine, Matthew Keating, shared the key secret with me several years ago.  See, I used to run my grades, print out the spreadsheet, and then go in order:  "Okay, Adams.  What can I say about him... now Baker.  Now Cabrera.  Now Davis...."  While alphabetical order is the way my spreadsheet and my school's computer system sort the students, it's not a *meaningful* order.  You'll be jumping around mentally, writing about dissimilar students in quick succession.

Matthew pointed out the one easy trick:  Sort the class from lowest grade to highest.  Then write.  This sorting places similar students in quick succession.  Phrases and descriptions can be reused or adapted in the very next comment, not from the comments you wrote half an hour ago.  

More importantly, such a sorting improves the quality of the comments as well as the speed of writing.  Comment writing is a long, intellectually draining process.  Sometimes I'm running on fumes by the end; sometimes I'm pushing up against a deadline by the end.  

So why not get the tough comments written first?  The comments for students with the lowest grades are the ones that will be most scrutinized.  These are the comments most likely to get a teacher into trouble if they are not carefully phrased with politically correct but nevertheless clear and direct language.  These are NOT the comments I want to bang out hurriedly in a desperate attempt to just be done.

This way, when the brain is beginning to fade, what's left to do?  The A students, whose parents are more likely to skim over your praise as just a small part of a hagiographic report card.  For these folks, a two-sentence note with phrases like "I very much appreciate his strong efforts this marking period" will be sufficient.  I can write those kinds of comments in my sleep, which is sometimes essentially what I have to do.