Teaching versus Grading or Learning versus Assessment

Some weeks ago, I mentioned the SCHEV course redesign conference I attended in November. A theme of the conference was how to use the power of computers to teach more efficiently and effectively. (This is the premise of Aplia, which probably does a better job than I witnessed at the course redesign conference.)

I went to the social sciences session at the conference, which was presented by a psychology instructor. He explained how he had redesigned his Introduction to Psychology course to fit the above mentioned theme. What was interesting to me was the central role that online quizzes played in his course, not merely for assessment, but for teaching and learning. He had constructed (or otherwise obtained) a large set of multiple choice questions, two or three thousand of them. For each course topic, students were assigned the online quizzes. Not only did the students take the quizzes, but they were encouraged to take them as many times as they liked, since only the highest grade was recorded. The instructor intimated that the students didn’t necessarily read the book (or even come to class); rather the quizzes were the course. I’m overstating here, but not by much. Students learned by doing the quizzes, rather than by reading the text or listening to lectures.

This concept is quite interesting and it certainly made me think, being a step beyond the idea I’ve explored of regular, low-stakes assessment. The problem that I’ve encountered with such assessment is that poor questions limit the usefulness of the assessment. That is why this past semester I told students that success in the online quizzes would assure them (only) a C+ in the course. In my experience, publishers don’t put near as much care and effort into test banks as they put into text books. (I certainly didn’t when my publisher asked me to add problems to my book.) While I use test bank questions, over the years I’ve modified them to better fit my assessment needs.

At the Course Redesign conference, as the psychology instructor demonstrated his quizzing software as well as some of the questions, I found myself seeing potential if not actual problems. What were the students actually learning? The quiz questions seemed to emphasize recall rather than any higher order skills. I asked about that and got two responses. First, since each time a student takes a quiz they get a different draw from the pool of questions, they’re not necessarily getting exactly the same questions, but rather similar questions on the same topics. But even if students weren’t getting the same questions, perhaps they were learning more about how to be successful on the quizzes, how to suss out the correct answer from the form of the questions, than what the correct answer was. The response to this was, “It’s only a 100-level course, so if they’re just memorizing concepts, I’m okay with that.”

This made me start thinking about the differences between learning and assessment, between how we teach students to learn, and how we test that learning. Returning to the previous context, if the assessment makes students go through the process of solving the problem, then using the assessment process to learn is fine. But if the assessment doesn’t require students to go thru the process of solving the problem, and I suspect most multiple choice assessments do not, then students can “learn” how to answer the questions without learning the underlying content. In this case, using assessment to “learn” is not fine.

Often, I think I test differently than I teach. I don’t think this problem is unique to me—it may reflect my disciplinary approach. The fact that students find economics (exams) “hard” may be a symptom of this. The basic problem is transfer, a form of external validity. We teach in one context, but ideally we want to see how students do in another. If we test on the same examples we use to teach, we can’t necessarily assess how deep the learning is, since we’re not assessing transfer ability. As such, the test is subject to the same criticisms I raised above about the psychology class. As teachers, what we want students to be able to do is use the same content correctly in a different context. That’s transfer. And that’s what I tend to ask for on exams. Not exclusively, of course. But the trick is to find the right balance between assessing basic content and transfer. (A further wrinkle is that I like to make exams a learning experience as well as an assessment. That is, I try to teach students something about the topic via the exam, too.)

My challenging student this past semester made me reflect more than I normally do about the way I assess student learning in my intro courses. After the second exam, he asserted that the appropriateness of my test questions should be based on their ability to measure achievement of the course goals. I agree. After reflecting, I decided I’m not sure the students and I are on the same page about what those goals are. Of course I have goals stated on the syllabus, but I’m not sure that my students have internalized them. Principles of Macroeconomics isn’t primarily an economics course—it is, or should be a general education course designed to introduce students to the social sciences. I think my course does a pretty good job of that. But I’m not sure the course exams have the balance right between the general education goals (which emphasize lower levels of cognition) and the economics goals (which emphasize higher levels). Or more precisely, I’m not sure the students correctly understand that balance. There are complicating factors here: Most of my students are first semester freshmen. The early material in the course focuses on the general education goals and is very easy to master. By the time students discover they’re not mastering the more complex material using the same (high school) study habits, it’s late in the semester and hard to change.

I think my efforts over the last few years to teach students metacognition were an attempt to fix this disconnect. By giving students more practice doing the kinds of problems they would encounter on exams, I hoped to teach them better as well. The problem that I ran into was that the online quizzes which I used for this purpose, coming from text book test banks, mostly assessed lower level learning. Thus, they weren’t teaching students all that I wanted them to learn.

I began to reflect on my intermediate macro course, where I think I do a better job of assessment. The students find the course difficult, and the tests challenging, but they’re not surprised by the types of questions as they seem to be in the intro course. The reason is that we practice those types of questions. That type of analysis is, in fact, exactly what economists do. My reflection made me wonder if I shouldn’t give students more practice of this sort. I’ve tended to limit the amount of practice because, more so than in the intro course, there is only a limited number of questions I can ask. Economic theory is fundamentally about using theories (or models) to make predictions about how events affect the economy. In the intermediate course, we only learn a handful of models, each of which has only a handful of parameters, so given that I want to ask students to use a model to assess the effects of changing one or more parameters, there aren’t really a large number of possibilities. I tell the students not to “try to memorize the different cases,” but to learn how to work thru whatever scenario is presented. If we work through all the options in practice, then how do I assess transfer?

But this raises another set of questions: Should every student be able to learn to do this? I want to say yes–if not, why not? Should every student be able to earn an A? What exactly should a grade mean? Are we confusing grading with teaching? We shouldn’t teach less well so that we can more easily discriminate between different levels of learning—Are we teaching students to analyze, which is what we say, or are we asking students to infer how to do it, something only the best students can do well? What does it even mean to say that a student is “one of the best?” In practice, it means they scored well on exams and other assessments. But does that mean they brought more to the course in terms of intellect or experience? Does it mean they learned more than most students in a value-added sense? Does it mean they learned to do economic analysis at a higher level than the one they’re in? Say at the level of a senior major for a freshman taking the intro course, or at the level of a graduate student for a senior major?

While I do a good job of balancing these issues in the intermediate course, I think I’ve erred on the side of protecting the sanctity of the grade rather than insuring that students learn. I’m going to make a concerted effort to change that this coming semester by giving more problem sets based on the types of analysis I want students to learn well enough to transfer. If I’m right, it should raise the mean exam scores (since more students are learning how to do macro well) while compressing the distribution of scores. Hopefully, I’ll still be able to discriminate between different levels of learning. It probably won’t affect the mean grade though since I’ve always ended up curving those.

This brings me back to the intro course. If the problem is getting students to practice more of the types of analysis I want them to learn, the solution is to come up with more good questions for them to practice on. That’s my next goal. One possibility would be to adopt the Aplia package, even though I don’t plan to use all the bells & whistles, merely the better quality quiz questions. Another possibility would be to apply for a course redesign grant to buy some time to create a larger personal testbank. I have only revised and expanded my test questions incrementally before, because devising good quality questions will take a great deal of time and it hasn’t seemed to be worth it or valued by the institution or the discipline. But perhaps it’s time anyway.