Pedablogy: Musings on the Art & Craft of Teaching

Some weeks ago, I mentioned the SCHEV course redesign conference I attended in November. A theme of the conference was how to use the power of computers to teach more efficiently and effectively. (This is the premise of Aplia, which probably does a better job than I witnessed at the course redesign conference.)

I went to the social sciences session at the conference, which was presented by a psychology instructor. He explained how he had redesigned his Introduction to Psychology course to fit the above mentioned theme. What was interesting to me was the central role that online quizzes played in his course, not merely for assessment, but for teaching and learning. He had constructed (or otherwise obtained) a large set of multiple choice questions, two or three thousand of them. For each course topic, students were assigned the online quizzes. Not only did the students take the quizzes, but they were encouraged to take them as many times as they liked, since only the highest grade was recorded. The instructor intimated that the students didn’t necessarily read the book (or even come to class); rather the quizzes were the course. I’m overstating here, but not by much. Students learned by doing the quizzes, rather than by reading the text or listening to lectures.

This concept is quite interesting and it certainly made me think, being a step beyond the idea I’ve explored of regular, low-stakes assessment. The problem that I’ve encountered with such assessment is that poor questions limit the usefulness of the assessment. That is why this past semester I told students that success in the online quizzes would assure them (only) a C+ in the course. In my experience, publishers don’t put near as much care and effort into test banks as they put into text books. (I certainly didn’t when my publisher asked me to add problems to my book.) While I use test bank questions, over the years I’ve modified them to better fit my assessment needs.

At the Course Redesign conference, as the psychology instructor demonstrated his quizzing software as well as some of the questions, I found myself seeing potential if not actual problems. What were the students actually learning? The quiz questions seemed to emphasize recall rather than any higher order skills. I asked about that and got two responses. First, since each time a student takes a quiz they get a different draw from the pool of questions, they’re not necessarily getting exactly the same questions, but rather similar questions on the same topics. But even if students weren’t getting the same questions, perhaps they were learning more about how to be successful on the quizzes, how to suss out the correct answer from the form of the questions, than what the correct answer was. The response to this was, “It’s only a 100-level course, so if they’re just memorizing concepts, I’m okay with that.”

This made me start thinking about the differences between learning and assessment, between how we teach students to learn, and how we test that learning. Returning to the previous context, if the assessment makes students go through the process of solving the problem, then using the assessment process to learn is fine. But if the assessment doesn’t require students to go thru the process of solving the problem, and I suspect most multiple choice assessments do not, then students can “learn” how to answer the questions without learning the underlying content. In this case, using assessment to “learn” is not fine.

Often, I think I test differently than I teach. I don’t think this problem is unique to me—it may reflect my disciplinary approach. The fact that students find economics (exams) “hard” may be a symptom of this. The basic problem is transfer, a form of external validity. We teach in one context, but ideally we want to see how students do in another. If we test on the same examples we use to teach, we can’t necessarily assess how deep the learning is, since we’re not assessing transfer ability. As such, the test is subject to the same criticisms I raised above about the psychology class. As teachers, what we want students to be able to do is use the same content correctly in a different context. That’s transfer. And that’s what I tend to ask for on exams. Not exclusively, of course. But the trick is to find the right balance between assessing basic content and transfer. (A further wrinkle is that I like to make exams a learning experience as well as an assessment. That is, I try to teach students something about the topic via the exam, too.)

My challenging student this past semester made me reflect more than I normally do about the way I assess student learning in my intro courses. After the second exam, he asserted that the appropriateness of my test questions should be based on their ability to measure achievement of the course goals. I agree. After reflecting, I decided I’m not sure the students and I are on the same page about what those goals are. Of course I have goals stated on the syllabus, but I’m not sure that my students have internalized them. Principles of Macroeconomics isn’t primarily an economics course—it is, or should be a general education course designed to introduce students to the social sciences. I think my course does a pretty good job of that. But I’m not sure the course exams have the balance right between the general education goals (which emphasize lower levels of cognition) and the economics goals (which emphasize higher levels). Or more precisely, I’m not sure the students correctly understand that balance. There are complicating factors here: Most of my students are first semester freshmen. The early material in the course focuses on the general education goals and is very easy to master. By the time students discover they’re not mastering the more complex material using the same (high school) study habits, it’s late in the semester and hard to change.

I think my efforts over the last few years to teach students metacognition were an attempt to fix this disconnect. By giving students more practice doing the kinds of problems they would encounter on exams, I hoped to teach them better as well. The problem that I ran into was that the online quizzes which I used for this purpose, coming from text book test banks, mostly assessed lower level learning. Thus, they weren’t teaching students all that I wanted them to learn.

I began to reflect on my intermediate macro course, where I think I do a better job of assessment. The students find the course difficult, and the tests challenging, but they’re not surprised by the types of questions as they seem to be in the intro course. The reason is that we practice those types of questions. That type of analysis is, in fact, exactly what economists do. My reflection made me wonder if I shouldn’t give students more practice of this sort. I’ve tended to limit the amount of practice because, more so than in the intro course, there is only a limited number of questions I can ask. Economic theory is fundamentally about using theories (or models) to make predictions about how events affect the economy. In the intermediate course, we only learn a handful of models, each of which has only a handful of parameters, so given that I want to ask students to use a model to assess the effects of changing one or more parameters, there aren’t really a large number of possibilities. I tell the students not to “try to memorize the different cases,” but to learn how to work thru whatever scenario is presented. If we work through all the options in practice, then how do I assess transfer?

But this raises another set of questions: Should every student be able to learn to do this? I want to say yes–if not, why not? Should every student be able to earn an A? What exactly should a grade mean? Are we confusing grading with teaching? We shouldn’t teach less well so that we can more easily discriminate between different levels of learning—Are we teaching students to analyze, which is what we say, or are we asking students to infer how to do it, something only the best students can do well? What does it even mean to say that a student is “one of the best?” In practice, it means they scored well on exams and other assessments. But does that mean they brought more to the course in terms of intellect or experience? Does it mean they learned more than most students in a value-added sense? Does it mean they learned to do economic analysis at a higher level than the one they’re in? Say at the level of a senior major for a freshman taking the intro course, or at the level of a graduate student for a senior major?

While I do a good job of balancing these issues in the intermediate course, I think I’ve erred on the side of protecting the sanctity of the grade rather than insuring that students learn. I’m going to make a concerted effort to change that this coming semester by giving more problem sets based on the types of analysis I want students to learn well enough to transfer. If I’m right, it should raise the mean exam scores (since more students are learning how to do macro well) while compressing the distribution of scores. Hopefully, I’ll still be able to discriminate between different levels of learning. It probably won’t affect the mean grade though since I’ve always ended up curving those.

This brings me back to the intro course. If the problem is getting students to practice more of the types of analysis I want them to learn, the solution is to come up with more good questions for them to practice on. That’s my next goal. One possibility would be to adopt the Aplia package, even though I don’t plan to use all the bells & whistles, merely the better quality quiz questions. Another possibility would be to apply for a course redesign grant to buy some time to create a larger personal testbank. I have only revised and expanded my test questions incrementally before, because devising good quality questions will take a great deal of time and it hasn’t seemed to be worth it or valued by the institution or the discipline. But perhaps it’s time anyway.

This post is a continuation of my thinking on the Product of Learning. I was discussing with a colleague the way I grade process over product in my research methodology course. I respect this colleague a great deal, and these remarks should not be construed as a criticism of his views, but rather a narrative about how his views enhanced my own thinking. I told him that 60% of the grade in the course was based on competently completely a series of assignments which cumulate into a final research paper. I told him that if students did a competent job, they got full credit on the assignment, but if they didn’t, they were allowed to redo the assignment to get it right. My colleague argued that he thought it better to urge his students towards excellence rather than mere competence. I agree that excellence should be our goal, but how do we get there?

One way to teach for excellence is to be a very tough grader. This is what I used to do. I would tell students, “Here is the standard for an A. You should all strive for this. Now go for it.” Students would turn in an assignment. I would either make careful suggestions for improvement (if it was a draft), or explain where it fell short of ‘A’ work (if it was a final version). One problem with such an approach is that it assumes students fully understand the standard for an A, and that if they fall short, it’s because they are not putting in appropriate effort to achieve that standard. Of course, the hidden assumption is that not all students are capable of excellent work. I question that first assumption, and wonder if more students would be capable of excellence if we explained to them more clearly what it was.

I have another colleague who used to write exam questions which many students found ambiguous. When asked about that, he responded that an important part of the course was learning how to correctly interpret the questions. Perhaps, but I found myself wondering, if students don’t know what you’re asking, is it any wonder they don’t give you what you want?

I think there may be a parallel here between my second colleague’s approach to exam questions and my first colleague’s approach to student essays. In my research class, the purpose of the research assignments and their competency-based grades is to get students to engage in the course work more fully than they otherwise would. It may be that the best students would do this anyway, but this approach aims to capture they rest. And as I demonstrated in the previous post, it works.

In class, I explain (for example) why researchers do a literature survey or search for appropriate data or ask an interesting and significant research question. I model how to do the task. I ask students to practice in class what I’ve modeled. And then I ask them to do it on their own in the form of an assignment. For years, I’ve told them they don’t have to do these things perfectly to excel; they just have to do them all. I now think I was right about this though I didn’t fully understand why until the other day. There’s something synergistic about the process which achieves the goal of the course for those students who participate in all (or nearly all) of the assignments. If they don’t complete most of them, they don’t end up getting it. [ I described what ‘it’ is in the last post.]

Students see the research assignments as “easy points,” but that’s not their purpose. The fundamental purpose of the course is not to write an excellent research paper, nor is it to get students to go through the motions of the process. The purpose is to teach students how to genuinely engage with a subject, which I would argue is more important than “the product” per se. Or rather, it’s not the product (e.g. research paper) that’s the product—it’s the process that’s the product. It’s what a liberal education is supposed to be about.

P.S. to my first colleague: Students still have an incentive in the course to write excellent papers. They get to take those to present at a professional conference each year. Isn’t that a better incentive than a grade?

I was consulting with Martha on my TLT Fellows project #1, which involves teaching a seminar in advanced macroeconomics. The seminar is divided into eight or ten topics, each of which has a list of scholarly readings: books, journal articles, etc. As the seminar progresses, we study the various readings, discuss them in class, and try to sort out our understanding of them.

This time, I’d like to build the seminar around a project to create a digital document, summarizing our findings and addressed to economics majors who have not taken the course. Ideally, we should be able to use this document as a resource for subsequent offerings of the intermediate macro course.

When I have attempted less ambitious projects along these lines, the resulting analysis has lacked depth. Martha suggested that what I may have been lacking is an effective model of collaborative learning that promotes deep analytical thinking.

While I have some ideas, I thought I would appeal to readers with more experience in this area than I have. Can you suggest one or more appropriate models?

I read Martha’s post “Bring Us Your Fear” and emailed a comment to Martha. She asked me to blog it so here it is:

I wanted to comment on your post but didn’t because I couldn’t think of any way to say what I want to say without sounding self-serving, so I figured I’d just email you and you can do with this what you will.

I am intrigued by this fear of change/ fear of using technology thing. I know it’s there. I’ve seen it in the eyes of colleagues, but I don’t get it. For me, a new technology is either interesting or it’s not. If it’s not interesting, I ignore it. (Tenured faculty can generally do that.) If it’s interesting, I can think of a way to use it in teaching, learning or thinking. And so I give it a try. Sometimes I’m wrong. But even when it doesn’t work out the way I anticipated, I usually see something unexpected that still makes it interesting. This change or new technology for me is what makes my job continually interesting. It’s as if my job changes over time, which is for me, fun. I think that if I was stuck doing the same thing all the time, teaching the same course in the same way each semester, I’d get bored.

I also like the idea of being part of the team of folks on the frontier of learning about technology. I never felt this way about my economics research. But I don’t think this can explain my apparent lack of fear entirely, because I was interested in technology before DTLT even existed.

My semester is coming together well. I’m particularly pleased with the research methodology class. The group seems to be very tight, and they seem to be getting it. In the past when I’ve felt this way, it’s because the class was completing an unusually high number of excellent papers. I realized this week that that wasn’t the case here. Most of the papers are decent, but probably fewer than normal are excellent. So why do I think they’re getting it. Am I merely settling for adequacy instead of excellence. I don’t think so. Rather I realized that the product of the course is not primarily the final paper. The product is hard to articulate, but it involves an understanding of the different facets of a research project (literature survey, data collection, data analysis, drawing conclusions), an appreciation of the problems posed by each of these facets, an appreciation for ill-structured problem solving, in short an understanding of what real research is (dare I say, real life?). I want to suggest that these are learnable regardless of how bright one is (within the domain of our students) or how well one’s research project turns out. Indeed, it may be that those that learn the best are those that ran into a real world problem that resulted in a less than excellent final paper. (A great deal of the outcome is based on chance; for example, the ability to find the right data set.) Here are some quotations from blog posts in the class last week that to me, at least, demonstrate that the authors got it. (You can find the complete posts on the course website.):

When professors used to say that you can usually see a difference between the approach freshman take on a class, to that the upperclassman take on a class, I could see where they were coming from, however, it was not until this class that I fully understood what they meant. As a freshman, I would learn the material and do the assignments as specified. There were very few assignments like this one that made me spend more time thinking then writing. I am by no means saying I had easy classes or bad teachers; I’ve learned a lot over the past two and a half years. It was just a different approach I had taken to the classes. However, it was not until this class, where it was up to me to meet the deadlines, and up to me to plan out the project with an unspecified topic, that I got a glimpse of the real world. No one is going to be out there handing you the problem and how to find the solution. In this project in particular, I saw that most of the trouble lies in finding out what the problem is and then determining what it all means. I had no problem putting data into Eviews, running regressions and tests, and telling what the tests meant, but when it actually came down to interpreting results for myself, without a book telling me what to do, it came down to thinking ‘outside the box.’ This cliche used all the time, was put into full practice in this assignment. Thanks!

So I’ve just finished my powerpoint for my presentation Monday. In going through it, I found holes in my research, and trying to explain it to myself in a way that I could explain it to others has helped my understand my topic more fully. I apologize in advance for the complexity of my topic. If I’d have only known…

Now that I’ve finished my Spanish presentation, I’m going to sit here and think about what causes my variables [i.e. statistical results] to be bad. I figure if I do nothing but think I might get somewhere. I might even draw some graphs on the board, I figure if Nash did it on hard topics, maybe I can figure out some easy ones.

here is my “Carl Post” for what I’ve learned in this class this semester:
1. Research, Research, Research! I was lazy when it came to really researching my topic early in the semester. This came back to bite me when I had to write my literature review and prepare my presentation. It’s hard to introduce a topic when you have very few facts other than your basic knowledge of “it does exist.” I was also too narrow when it came to my research. I read articles only about the wage difference in self-employment, when I should have read articles that were also about the wage-difference in general. This would have helped introducing my topic during my presentation.
2. It’s a good idea to have a firm idea of what you’re studying before you study it. Through out this enitre semester, I kept learning about my topic as I was doing it. Further, my research topic, hypothesis, research questions, results, etc. only became absolutely clear at the end of the semester when I had everything done. Basically, I really understood what I was doing after I completed my regressions. If I had figured out everything before, it would’ve saved me some stress and frantic conversations with some fellow classmates.
3. Instead of stressing out about something, talk to Dr. Greenlaw. Through out the semester I learned that the best way to dodge sleepless nights and stressed out days was to just ask for help.
4. Your classmates are your biggest supporters. I honestly felt so much better though out the semester knowing that everybody in the class was in the same boat as me at times. From “I have no idea what my topic is and I have to turn in my first assignment next class” to “I can’t find data” to “I spent so many hours entering data into e-views I can’t see straight anymore” to finally “I’m officially sick of my topic” it made me feel better hearing that all of us were going through the same struggles.

What I’ve learned This Semester:
1. Think, think, think, then read, read, read. Think about what you want to research, go over it, think about if there would be available data on the subject and then check previous studies. Read as much as you can, it can only improve your research and help you to understand your limits and goals. The more I read the more I wanted to alter my topic slightly and the more I read the more I learned on the subject.
2. Find data, and then find it again and again and again. First make sure you can find adequate data for you topic early on. Second once you’ve found data don’t just say okay looks good I’ll start regressions in a month or so. You may need to switch up your variables a few times and you don’t want to be doing this last minute. Find data think about it, run a regression, and change what needs to be changed.
3. Don’t be discouraged by not so great regression results. Your classmates are in the same boat. Just try your best to improve them, and in the end if you can’t do your best to explain why they suck.
4. It feels good to see the end. I saw Professor Greenlaw today and he asked me how I was feeling, I didn’t get it at first (I can be slow at times) but he was referring to the fact that my research project is almost over. The end of the tunnel is near, maybe I’m a pessimist or I’ve just been so focused on finishing my draft and working on other classes’ assignments that I haven’t sat back and realized how much we’ve all accomplished this semester. We should all feel proud and most of all relieved that the end is near and we’ve made it!

For about the last month, I’ve found myself thinking deeply about questions I pondered when I first started blogging: What is learning? What should the objectives of college courses be? What should course grades be based on? This recent train of thought has been prompted by a number of things: the Course Redesign Conference sponsored by our State Council on Higher Education, a thought-provoking challenge from my challenging student, discussion in my First Year Seminar, the TIP Workshop on Student-Centered Learning, and finally, lunch with Martha. I plan to publish several blog posts on these questions shortly, but first let me provide some preliminary thoughts.

I’ve struggled to sort out the importance of content vs. skills, and product vs. process. All courses are some combination of content vs. skills. C Programming is probably more skills. Western Civilization is probably more content. Neither is more legitimate than the other as far as college-level learning goes. They are both relevant to education. This distinction has turned out to be a digression. I think the distinction between product and process is going to be more critical. I’ve been assuming that product is more important, more valid to education than process. What content and skills do we know at the end of the course vs. what have we done during the course? I’m not sure that’s correct anymore. How do we assess those options and how much should each be weighted in determining grades?

This is a follow-up to the challenging student post, though it’s not primarily about the student. We met and discussed what he hoped to get of the course. I offered to give him a more advanced textbook, one designed for intermediate macroeconomics. He accepted saying that was a good idea. We agreed that he would take the course exams, but his homework and class participation would be based on his reading and our discussion of the alternative text.

While I think this may be a good solution for the current situation, I’m not so sure it’s something I could do if more students requested this option. I’m already supervising five independent studies. That would be my limit, but for the fact that all five are excellent, highly motivated students. I would be hard pressed to supervise more well, given that I’m also teaching a full load—four courses. I feel somewhat hypocritical here. I’ve criticized the industrial model of higher education and I believe that every student should have an individualized course of study. Yet at the same time, scale is a genuine concern. I haven’t figured out completely what that alternative higher education would look like, but I’m pretty sure it involves greater responsibility on the part of the student to manage his or her curriculum. It’s not simply replacing a class meeting with 30 one-on-one meetings.

But there’s another issue here. The challenging student asked me if there was any point in his continuing to attend class. Given that he had been coming but spending the entire period reading a paper back book, I said, “probably not.” I’m having second thoughts about this. My class sessions are not merely lectures, nor are they all defining of terms and framing the context for the course as was common in the first weeks. If he doesn’t come to class he will miss a number of hands-on exercises as well as discussions of specific real world applications of economics. In other words, he’ll be missing a substantial part of the course experience. Some of that material will show up on the exams, which presumably he will have a harder time getting right, especially since much of this material isn’t in the text. But it’s not his potential poor grade that bothers me, but rather his lost learning. A teacher can’t make a student learn, but am I enabling what may be poor choices on his part? What obligation does an instructor have to provide a customized but equivalent learning experience for a student who doesn’t like the standard option?