Archive for the 'The Experiment' Category

More on metacognition and assessment

Published on July 24, 2008 in The Experiment. 2 Comments

For a couple of years now, I’ve tried to build low stakes, formative assessment into my intro courses. The intent was to help students become more responsible for their own learning and promote more mid-course adjustments to study strategies while there was still plenty of time for that. (Contrast this approach with the mid-term, final exam model.) The way I implemented this metacognitive effort was to encourage students to take the online quizzes provided by the text book publisher at the end of each chapter. As long as students took the quizzes, I gave them extra credit regardless of how well they did. And, to judge by their scores, nearly everyone did well. When individuals did very poorly, which was rare, I spoke to them publicly about whether they had taken the quiz seriously or just for the credit. The message was received and the students almost always did substantially better next time.

Unfortunately, the plan didn’t work as well as I had hoped. In past years I found that the assessment quizzes, based on textbook test banks, didn’t correspond well to my exam questions. The quizzes were not very challenging and thus not adequate preparation for my exams. Students could do well on the quizzes and then not so well on the exams. I felt like the lack of correlation between quiz and exam questions defeated the purpose of the exercise and encouraged students to think of the former as busy work, unrelated to assessing their mastery of the material.

This past semester, to address this concern, I adopted the Aplia product, customized for the text I was using. I made this selection principally because of the higher quality of their quiz (problem set) questions compared with textbook test banks. The result was disconcerting. Student scores on the quizzes, were low, and as the semester went on they got worse. After about the fourth week, the class mean never reached 70%. I got the feeling that students were not taking the quizzes seriously, that they were just going through the motions. Economics suggests that people behave rationally, predicting that students would merely make an effort to get the credit, while not taking the assessment seriously. And yet this seemed to conflict with my experience in past semesters when the students did consistently well on the quizzes.

To examine this paradox, I added a question to the final exam, which I told students I wouldn’t look at until after grades were in. The question asked:

The average score on the Aplia problem sets was pretty low. How did you approach them?
a. I tried to do my best, but the problems were difficult; when I did badly on the problem sets, I took that at as a sign that I hadn’t mastered the material. (11%)
b. I tried to do my best, but the problems were unlike the ones we did in class. (32%)
c. I tried to get the right answer but it didn’t bother me when I got the questions wrong. (41%)
d. I didn’t take the problems too seriously; I was just trying to get the points for trying. (16%)

I intended ‘a’ to be the right answer, the answer which indicated that the exercise was achieving its goal. Answer ‘b’ was included to reflect the fact that mid-way through the semester, many quizzes began to include numerical questions, which usually were based on methods of solution that I didn’t emphasize in the course. The advantage of numerical questions is that they result in “exact” answers. The problem is that often numerical questions are driven by the need to keep the math simple, which requires the economics to be trivial or an unusual case.

The frequency distribution of the responses is given in parentheses. Thinking about the likely bias in the responses this suggests to me that probably 60% of the students didn’t take the quizzes seriously, which is a problem.

What I’ve decided to do is raise the bar a bit. For this coming semester, I will give credit for the quizzes as long as the score is 70% or more. We’ll see if that makes it more of a useful assessment while still keeping the stakes low.

More on the ‘Do over’

Published on April 12, 2008 in Teaching and Learning, The Experiment and Uncategorized. 1 Comment

Some weeks ago, I blogged about a new way of curving exam grades which I’m trying in my principles course. Monday I handed back the second exam, which is the most difficult of the year, primarily because it covers the most analytical material. The average grade was 63%, which is normal for this exam. This week the students have been coming by my office for the ‘do-over.’ Their charge is to explain orally why the correct answer on the problems they got wrong is correct. This shouldn’t be that difficult, since they have several days to re-view the problems and re-think the answer. For each problem which they correctly explain, I give them half credit.

I’ve found this to be an interesting (and different) sort of assessment. Not only do students have a chance to explain their reasoning (which they don’t on the multiple-choice exam), but I give them more than one chance by prompting for more information. When students still don’t understand the reasoning behind the correct answer, it is very clear. Whether that is due to lack of preparation, or a true lack of understanding I’m not sure, but I have no trouble denying them credit.

I think I like this was of curving exam grades because it reinforces that exams should be opportunity for learning as well as assessment. Additionally, I think it has the incentives right, unlike my past practice of using a traditional curve. Here, they get more credit if they get the problem right the first time, so there’s a disincentive to just blowing off the in-class exam. After all, 62.5% (assuming 25% randomly correct) is not a very good grade for someone able to get all the questions right orally.

It’s not just semantics

Published on October 12, 2007 in Teaching and Learning and The Experiment. 6 Comments

I returned the first exam in my intro course the other day. The average grade was a bit lower than normal. I wonder if that reflects the somewhat higher than normal number of first year students in it.

I did the return a bit differently this year. Usually, I hand back the exam and spend the class going over the questions. The mood is somber and the attention questionable. This year, I gave back the exams at the end of a class lecture, and asked students to look over theirs and try to figure out what they did wrong. (The exams are multiple choice; the correct answers are provided on the corrected answer sheets.) My thinking was that rather than my doing the processing of results for them, I would ask them to do it. As a final step, I asked them to email me the numbers of any questions they couldn’t figure out on their own (or otherwise wanted me to go over). I collated those choices and we will go over them in class today. We won’t waste class time on questions everyone answered correctly or on those where students made simple mistakes. Additionally, the students should be no longer in the trauma of just receiving a poor grade.

For the first years, this was the first time many had ever received a ‘poor grade’ on an exam. In some cases, this poor grade was a C. I’m not making fun of this reaction–these students have performed well in high school and see themselves as good students. But as I noted previously, the rules are different in university. Because of the larger than normal number of first years this time, I’ve thought more carefully about how they react to this.

I think there are faculty who see the rules of higher education as not much different than in high school, just with higher standards. Students who were successful in (e.g.) science in high school, don’t do as well in college, and this simply represents a winnowing of the field when the bar is higher. Survival of the fittest. Those students likely will find another discipline which better matches their talents and abilities.

There is no doubt some truth to this view, but over time I’ve come to subscribe to a different one. The rules of higher education are substantially different and not merely, or perhaps even primarily, the higher standards of competence. Learning is far more than grades. Grades are at best a rough indicator of learning. Over the years, most of my best students were not the students who got the best grades, at least in the lower level courses. Rather, they were the ones who had to struggle with the material. Strangely we tend not to test on the most important parts of learning. In part that’s because it’s difficult. Assessing factual knowledge is much easier than assessing critical thinking ability. Over time, I’ve tried to re-align my assessment to match my assessment with the more important parts of teaching and learning, but it remains imperfect. It seems to work best in less structured formats where I can ask students to ‘show me what you’ve learned,’ rather than asking every student to answer the same questions.

What I try to teach my students is an understanding of how to do economics, as well as an appreciation for the power of the economic way of thinking. Writing the previous sentence seems almost to trivialize the two goals. Both goals are richer than the words suggest. In high school, I think most teaching is about a discipline. In university, we should be teaching how disciplinary practitioners practice their arts and sciences. I see a fundamental difference there. When students see an application of economics in the real world, I want them to be able to analyze the situation the way an economist would, even without the complexity or deep understanding that a professional has. When I talk about the appreciation for the power of economic thinking, I mean something like what an art or literary critic brings to the table though at a lower level.

These are achievable goals for my undergraduates and they don’t correspond necessarily to the normal structure of grades. Indeed, in some ways, the grade structure is orthagonal to what I’m trying to accomplish. In 25 years of teaching, I’ve only had two or three students who I concluded were unable to learn what I was trying to teach. (That doesn’t mean only a few students didn’t learn, just that only a few were unable to.)

When I discuss the exam today, these are some of the things I will say. Unlike the view of higher education laid out in the fourth paragraph above, I believe that all students can achieve a level of competency in what I’m teaching. Getting a failing grade on an exam doesn’t mean one is unable to learn the subject. It just means one hasn’t figured out how to learn it yet. This is not just semantics. There are different ways of learning, and different faculties to tap into. This lesson is particularly important for first year students who may not have study skills or habits amenable to higher education. A student who appears attentive in class and yet did less well on the exam than I expected said to me yesterday:

You mentioned that staying up all night before the exam to study wasn’t good practice. Were you serious? That’s the way I’ve always studied.

I am confident she can learn a different way, one which relies less on absolute talent, and more on more thoughtful uses of the talent one has, one in which she can learn economics effectively.

UMWBlogs, Baby; They’re taking over!

Published on September 23, 2007 in The Experiment. 2 Comments

I was reading the first metacognitive activity submitted by students in my intro course, prior to discussing it tomorrow. (I’ve decided to move that discussion outside of class time to avoid the potential problems I ran into last year. I’m meeting with any student who turned in the meta after my last class, the meta being the entry fee.)

So I opened the next student’s email containing, I supposed, their meta. I was pleasantly surprised to read:

Hey Dr. Greenlaw,

I have my first Meta complete now. It’s in the form of a UMWblog, and all the other Meta assignments will be posted on the blog.

http://dmoore201b.umwblogs.org/

Enjoy,

I opened the link in my browser and discovered an incredibly creative and thoughtful submission, which I commend to your review.

What was particularly interesting about this was that I haven’t said a thing about blogs this semester, other than to build the course website on a WordPress platform. So the students aren’t supposed to be blogging for me. This student just decided to do it anyway. Fascinating!

Quick note on teaching my intro course

Published on September 15, 2007 in The Experiment. 2 Comments

One of the ideas I explored this summer was the finding that students bring misconceptions into a course that need to be explicitly refuted if learning is to be transferable. It turns out that there is not that much known about what these misconceptions are in economics. What I plan to do this term is start each new topic with a question that illustrates a possible misconception. For example, for the topic we began yesterday (The Theories of Supply and Demand), I posed the following question:

An increase in the price of a product leads to a decrease in the amount people buy (what economists call ‘the quantity demanded’). But an increase in demand for a product leads to an increase in the price.

Is this a paradox?

Before the end of the topic, I’ll make an explicit effort to show students how that question should be answered.

I also plan to pay careful attention to what the students are learning and having trouble learning this semester so that I’ll have a better idea next time what the misconceptions are.

More on Student Metacognition and Motivation

Published on August 2, 2007 in Teaching and Learning and The Experiment. 1 Comment

My latest favorite blogger is Lanny Arvan. (Thanks Martha!) Aside from being a fallen economist turned instructional technologist, he’s a fascinating and deep blogger. One of his (relatively) recent posts struck a chord as it related to my interest in student metacognition and motivation, both extrinsic and intrinsic.

Beginning in the 16th paragraph of the post, he discusses the use of (on-line) quizzes as a way of helping students get regular, low-stakes assessment of their performance.

Lanny argues that quizzing forces students to engage with the course and put in the effort they otherwise wouldn’t, especially in non-major courses, and as such is “a triumph of extrinsic assessment.” But therein lies the problem (recall Harry Truman’s desire for a one-armed economist):

But many educators, among them the psychologist Jerome Bruner and Ken Bain, author of What the Best College Teachers Do, emphasize an appeal to students via intrinsic motivation, to wit curiosity, a desire to understand the puzzles that real life circumstances pose from the perspective of disciplinary expertise.

Lanny goes on to suggest that intrinsic motivation enters into several aspects of instruction: what students are asked to read, what topics are discussed in class, and “clever assignments.” He describes one of the latter as follows:

The first assignment I gave to those honors kids was for each of them to identify Principles of Economics textbooks that are in the top 10 by market share, with each student receiving 10 points of credit per book if they were the sole provider of the title and no credit at all if the title was offered up by another classmate as well. The assignment worked like a charm the first time I did this, when I had 15 students. The outcome was that they identified all books in the top 10 and then some, one student earned 10 points but otherwise all the titles that were submitted came in duplicates, and then they had to puzzle over why they put in effort but (except for that one student) got no credit for their travails. This assignment was my introduction to the core idea that economics is about incentives. It was a great introduction. I had them hooked for the rest of course.

How cool is that? This is an order of magnitude better than the way I introduce this topic, because it takes what students would otherwise perceive as a fact or finding, boring in the abstract, and turns it into a mystery to be explored.

How can I built exercises like this into my courses this semester? Stay tuned. I’ll have more to say about this.

Postmortem on my experiments with teaching this year.

Published on June 4, 2007 in BigWiki, Teaching and Learning and The Experiment. 1 Comment

At the end of the school year, about a month ago, I was pretty unhappy with my teaching experiments this year.

In my intro course, the metacognition experiment didn’t seem to go very well this year. Part of it was me–the FSEM was taking so much of my time and energy that I wasn’t able to devote as much to the intro course. But it also seemed as if the students weren’t taking the meta assignments seriously. It made me wonder if trying to teach freshmen and sophomores metacognition was perceived as teaching it out of context. While I explained carefully what I was trying to teach them–that experts learn differently than novices, and that here was a framework they could use to learn economics the way economists see it, they seemed to perceive that explanation as abstract and unrelated to the content of the course. It seemed beyond their ken, at least for most students. Clearly teaching novices to learn like experts is a challenge.

Another part of my discomfort was my insistence on not curving the test grades this year to provide a clear incentive for using the metacognitive framework. The students ended up doing poorly in terms of grades. Unfortunately, when students did badly on the exams, it didn’t seem to make them want to do the meta activities any more. A contributing problem is that the first exam tends to be fairly easy since the material includes a great deal of common knowledge. This provided a false signal for the second exam which was much more difficult as was the final. With only two midterms and a final, students apparently weren’t getting enough timely feedback on their learning.

What I did learn was that providing regular formative feedback, in the form of automated quizzes is a good thing. The challenge is finding quiz questions that adequately represent the ones I test with. Perhaps the answer is to write my own questions, if I can then find a testing platform to put them on.

The whole experience made me think carefully about the best way to teach introductory economics–I ended up wondering if the writing intensive method I used for years was more effective than any of my recent experimental approaches. The reason for that conclusion is the writing assignments I gave required students to do economics, which learning strictly from the text and lectures doesn’t.

As I discussed previously, my experimental approach to intermediate macro was largely a failure in that students didn’t learn as much as they did during my previous approach. At the same time, I learned a great deal that I will be able to apply to courses in the future.

I also learned a lot about social software and learning in the international finance seminar. I’m going to apply some of that to the FSEM next fall–e.g. group research presentations instead of a research paper.

At one level, my understanding of the best ways to teach my courses seems to be right back where I started several years ago. But then it occurred to me that I may have experienced what Jerome Bruner calls “the spiral of learning.” I may be back to the approaches I originally used, but my understanding of what makes them effective is much greater than it was when I started. Additionally, some of the details are quite different.

I guess that’s progress.

Thoughts on Meta 2

Published on October 17, 2006 in The Experiment. 0 Comments

As I read over my principles students’ second metacognitive activity this weekend, I had a couple thoughts. First, the administrative one: Fewer students submitted this meta: 54% versus 70% for the first meta. Still, it’s more submissions than meta 2 last year. There may be a time constraint at work here–we are beginning midterms, indeed, our first exam will be this coming Friday. [Note: This post was started a week ago.] We have one more meta to do before then. My plan is to discuss Meta 2 on Monday, then do the same for Meta 3 on Wednesday before the exam.

The way I’m doing the metas this semester seems to have helped with the timing problems I experienced last year, when I ended up having to review several metas the class before the exam. This year I discuss the meta before I’ve read the student responses. Then after I’ve reviewed the responses I make any short follow-up observations that seem necessary.

Now, the more interesting thought: I was struck again that reviewing the students’ meta submissions isn’t grading. Grading isn’t the point here. When I’m “grading”, the emphasis, especially when I’m tight for time, is giving the students appropriate credit and writing comments to justify the grade. What I’m doing here is more a conversation with the students. Do I understand what they think and how they are thinking? That’s what I comment on. It’s a very different mindset and dynamic for me at least.

Rethinking Grades and Learning

Published on September 27, 2006 in First Year Seminar, Teaching and Learning, The Experiment, University 2.0 and What is Education?. 4 Comments

Is learning fundamentally a flow or a stock? Is learning about improving your skills and knowledge or about being skilled and knowledgeable? I ask these questions because often it appears to me that we’re grading what students know more than what they’ve learned. Bright students get good grades. Students who write well, complete A papers. Grading is an exercise in sorting based on what students know, which may have little correspondence with what students have learned on an assignment or in a course. I’ve encountered many students who, at least in lower level courses, didn’t have to learn anything to get good grades. What a shame, or is it a sham from the perspective of education?

This creates a contradiction for me or at least a discomfort: Our objective as teachers is to help students learn, to help them improve their thinking and understanding. But we grade not on their learning in the sense of improvement, but rather on what they know.

Okay, so be it. Let grades be an indication, an estimate, of what a student knows or is able to do at a point in time. But let’s not pretend that grades measure learning.

So where does that take us? Actually, quite a distance. It reminds me that teaching should focus on learning and not grades, that teaching is a craft while grading is a mere administrative function, and that grading provides only a poor incentive for learning. What does this mean in terms of one’s teaching? I will explain.

I asserted that grading is a poor incentive for learning. What do I mean by that? After all, economists argue that people respond in predictable ways to incentives. Rewarding students with a good grade for learning, or punishing them with a poor grade for not learning should encourage students to do what is necessary to learn more. But what is necessary? Grading is a blunt object for teaching. The signal to noise ratio is quite low. Grades less than A tell students that what they did wasn’t right, or wasn’t completely right. But grading, the way it is often done, provides little direct information about how to perform better. (A grade of A is almost as bad: telling students they are “completely right”, as if that is possible in the real world.) By contrast, I suspect that carefully thought-out, formative assessment can be a strong tool for enhancing learning. At least that’s my hypothesis. As Christopher Miller, President of St. Johns College, said today in a panel presentation, “Assessment should be an integral part of learning itself.”

Over the last ten days, I’ve reviewed two sets of student papers. Both were non-traditional assignments: one was the first meta assignment in my intro course, which I blogged about the other day, where students reflect on what was important in the previous topic, but more importantly, why it was important. The second assignment was the first major essay in my first year seminar on globalization. For this assignment, I asked students to identify, explain and justify what they see as the most important questions that need to be answered about globalization. I asked why the questions identified were important, and who they were important to.

For both of these assignments, I read and responded to their papers, but I didn’t grade them. I didn’t sort the papers (students?) and label them A, B or C. Rather, my responses were guided by the Inquiry Method, which says not to provide answers to students, but only to respond with additional questions, that is questions that lead students to think more deeply.

I must say I found this approach difficult. I have been training for more 25 years to sort, categorize,and label assignments (students?). It’s harder and it takes a bit longer, but it is opening my eyes to a new way of viewing teaching. Instead of reading a student’s paper and looking for what it lacks and what I can deduct points for, I start from “zero” and look for what I can suggest to improve the paper. What is the limit in the deductive approach? You can knock a student’s grade down to zero. What is the limit to this new approach? There is none. Students can improve without limit. In the first year seminar, I’m ~~allowing~~ encouraging students to revise their work as many times as they wish until the end of the semester. With each revision, I’ll give them additional feedback. When is an assignment done? When the student is convinced they have nothing more to add or when I have nothing more to suggest. I haven’t gotten rid of grading (which comes at the end); I’ve just put it into the back seat in favor of regular formative assessment.

This approach implies a very different view of teaching and learning, one where an instructor treats each student as engaged in a personal and in some ways unique journey towards education. Putting it another way, one must accept students’ abilities where they are and strive to help them improve. Each of them. Contrast this with the one size fits all method of the industrial model of education.

Can this approach be scaled? At one level that’s a question from the industrial model, though I admit it can be legitimate from the perspective of an institution or a school system. It may be possible to do something approximating this approach with the tools of Web2.0, but that remains to be seen. The question for me is whether I do it with a class size of 35, since that is the general limit in my courses.

I think I can, said the Little Engine that Could.

Status Report on the Metas

Published on September 26, 2006 in The Experiment. 0 Comments

Last week I reported on the first meta discussion in my intro course this term.Â Over the weekend I read over their responses and drew some conclusions.Â First, the majority of students completed the assignment, 70% as compared to 35% for the same assignment last year.Â What was the difference?Â I’m not sure.Â This time, I didn’t emphasize that this assignment was optional, but I did indicate that it was essentially extra credit.

Last year as I read over the metas I asked myself, what are they choosing for their answers?Â This year I found myself asking, what are they thinking?Â What is their reasoning for selecting what they did?Â For students who explained this well, I had a strong sense that they understood the material.Â By encouraging the students to spell out the metacognition, I think I’ll be able to make these assignments more effective learning tools.