Course Evaluations for my Waymaker Course Pilot

LumenWaymaker-400x80This past term, as part of a nation-wide pilot including more than 9000 students, I pilot-tested Lumen Learning’s Waymaker platform in the online version of my ECON 202, Principles of Microeconomics course.  I hoped that Waymaker would provide a useful structure for my online course.  At the end of the term, I asked the students to complete a course evaluation that I constructed. The responses were anonymous, though I gave each student who submitted one extra credit towards their final grade. (I ask them to email their responses as anonymous attachments to my colleague, Bob Rycroft, who sends me the list of names, but holds the evaluations until final grades are turned in.) Out of 36 students in the course, I received 28 submissions so the response rate was decent.

The evaluation included questions on the online course (since I’m still trying to improve my online teaching) as well as on the Waymaker platform. The questions asked for student perceptions and included both quantitative and open-ended responses. My statistical analysis of learning outcomes for this course, as measured by student performance, will come later.

This past week I reviewed the student submissions and coded them into Excel. I prefer to do this myself, rather than asking a student aide to do so, because in reviewing and coding I find myself doing some processing of the results. More precisely, because I’m working one submission at a time I get a sense for what and how each student is thinking; I get a sense of consistency or inconsistency in responses to different questions, and I get a sense of the relationships between responses on one question and responses on another.   For example, and this is just a casual feeling, it seemed like non-traditional age students were less likely to buy into the Waymaker approach, which attempts to facilitate deep learning. This manifested in things like not watching the video content, not doing the simulations, and asking for study guides that would explain exactly what they were expected to learn. Don’t get me wrong; I am not anti non-traditional age students. Some of my best students fit that demographic.

So here are my initial impressions.

ECON 202, Principles of Microeconomics is an introductory-level course. The course is required for the business, environmental science and international affairs programs, as well as for certifying to teach K-12. While the intent is for students to take this course early in one’s academic career, some defer it until later. The majority of my respondents were in their first (10 students) and second year (11 students). The remaining seven students were third (5 students) and fourth years (2 students). Most of the students (18) live on campus. Three live close by; the remaining seven live as far as 50 miles away.

Interpreting a survey like this is challenging. Some students seemed to try to say what they thought I wanted to hear. Some gave answers to related questions that were inconsistent with one another. Some students didn’t interpret the questions as I intended or didn’t think carefully about them, dashing off answers that don’t provide much useful information. One student wrote,

“The hardest material to learn was anything really extensive and difficult.”

An additional complication was that this was an online course, which is relatively rare at my institution, so some of the criticisms expressed by students were criticisms of online courses, rather than of the Waymaker platform. Similarly, economics is a challenging subject, since unlike English or History, many students haven’t studied it before. So some of the criticisms were really about the discipline rather than Waymaker.

Waymaker is a next generation courseware environment, combining text, video, and animation/simulation, all embedded in a mastery learning framework with formative and summative assessments interleaved through each content module (analogous to a book chapter).

Three quarters of respondents were generally favorable towards the Waymaker modules, and only 18% said they preferred a traditional textbook. 42% of respondents said Waymaker was easier to understand than a traditional textbook, while 32% said a commercial textbook was easier. As one student said,

“The combination of videos, self-checks [formative quizzes], and worked examples allowed for small checks for understanding that a textbook does not provide.”

Students indicated that the hardest material to learn was specific economic theories, understanding graphs or understanding the math. Anyone who has taught principles of economics would find this pretty normal, but I wonder if Waymaker could do better than text, given its multimedia approach?

Let’s look at some of the specific features of Waymaker.


None Some All  
1 20 7 How many videos did you watch?
3 13 12 Did you watch them in their entirety?
No Yes
12 16 Did you ever watch the same video more than once?
Not very useful 1 2 3 4 5 Very useful
0 6 6 8 4 On a scale of 1 to 5, how useful were the
25% 25% 33% 17% videos for learning concepts or models?

Only 25% of students (7) indicated that they watched all the videos. Is this a problem? Slightly less than half the students watched the videos in their entirety. More than 60% (16 students) watched the same video more than once, while 75% found them somewhat to very useful.  25% indicated the videos were not very useful.


None Some All  
1 18 9 How many animations/simulations did you play?
2 15 11 Did you play them in their entirety?
No Yes
10 18 Did you ever play the same animation/simulation more than once?
Not very useful 1 2 3 4 5 Very useful
1 6 10 5 4 On a scale of 1 to 5, how useful were the
4% 23% 38% 19% 15% animations/simulations for learning concepts or models?

About a third of the students played all the animations/simulations, and the same number found them somewhat or very useful.  27% found them not very useful.   My impression is that students who like them, liked them a lot.  I didn’t get the same impression with the videos.

Study Groups:

Less Same More
3 10 15 Did you learn more from working with your study group than you would have learned on your own?
Own Group When you did group assignments, did you mostly work on your own & share, 
7 21 or did you work together as a group and go over each others’ contributions? 

One of the features of the online course was study groups, which I used to counter possible feelings of isolation for students working online. Slightly more than half the students (54%) said they learned more from working with their study group than they would have learned on their own. About 11% said they learned less. Three quarters indicated that they worked together as a group on assignments, while one quarter divided up the assignments, worked individually, and then shared answers. Some study groups were effective, because the student drew on each other’s strengths. Others were less effective, either because one or more students didn’t participate easily (or were hard to reach), or because “no one in the group knew the answers.” I need to do more research on group work to see if I can improve these results.


No Scores Yes  
1 8 17 Did you use the self-checks to assess your learning? i.e. Did you use the self-checks to think about what you knew or didn’t know, or did you just focus on the scores.
Never Some Often  
4 11 13 Did you use the module quizzes to study for exams, aside from doing them for the grade? 
Less Same More  
7 10 11 Compared to using a traditional textbook, did you think about how well you were learning as you went through the modules?

Nearly all students used the self-checks (formative quizzes); only 28% (8) focused on the scores, while a majority (17 students – 68%) used them metacognitively to think about which topics they had mastered and which they had not. Nearly half (13 students – 46%) used the summative end-of-module quizzes often to study for exams, while another 39% (11 students) used them sometimes. 39% (11 students) indicated that thought more about how well they were learning using Waymaker compared to a traditional textbook. 25% (7 students) said they thought less about how well they were learning using Waymaker.

Less Same More  
5 17 6 How much time did you put into this course compared to others you have taken at UMW? 
Easier Same Harder  
0 7 16 How did this course compare with the others you have taken at UMW?

A majority of students (57%) found the course harder than others at UMW, but only 26% put more time into the course, which seems odd to me.

What tentative conclusions have I drawn from these survey responses?

Some students liked Waymaker; some students didn’t. What was the difference? Was it merely a question of taste, or something more substantive that could be improved?

Weaker students seemed to not like Waymaker, but was this cause or effect?

Some students didn’t seem to take the Waymaker features very seriously or they didn’t give them a serious try. For example, a number of students didn’t do much with the videos or the animations/simulations. Look back at the Video & Animation/Simulation results above. Students who responded None to the video or animation/simulation questions obviously didn’t benefit from them, but I suspect that many of those who responded Some/Some/No were also disengaged.

At the same time, 50% of the respondents said that the videos were useful or very useful for learning concepts or models, while only 25% said they were not useful. Similarly, 35% said the animations/simulations were useful or very useful, while 27% said they were not.

My impression is that some students, especially first semester first year, or transfer students, don’t seem to understand learning at the university level. Some don’t know how to study deeply and effectively. They have adopted the “school is about grades” motivation, and their approach to studying is to skim the readings looking for key terms to memorize. I know not all FY students or transfers are like this, but it’s a pattern I saw in the survey.

Waymaker was designed for deep learning; it was designed to mimic Socratic questioning of the student. It’s not perfect yet, but those were some of the goals.

Two common themes in the student comments were time management/self-discipline and the greater time required compared to a face-to-face course. The first theme is common to online courses. As one student explained it, “Being responsible and getting assignments done without having the professor in class reminding you.“

The second theme was more interesting. A number of students objected to the time required to work through the modules. But compared to what? The time spent reading a traditional text? But that’s not the right comparison. Waymaker tries to provide the structure students get in a face-to-face course. So the proper comparison should be time in the classroom plus time spent reading the text. I don’t think most students understood that.

Several students mentioned that the course required them to teach themselves! But what do they mean by that, and how is it different in a face-to-face course? If they simply mean the lack of lectures meant they had to filter the content and discern meaning for themselves, that’s almost tautological. And some research suggests that might actually enhance learning, for those that do it seriously.

But what if part of the pushback was students’ lack to experience with studying in depth? Then if they had bought into the program they could be more successful, but if they didn’t buy in they could be frustrated and unable to see the superficiality of their study approach, possibly leading to giving up on Waymaker. We might be able to test this if we have time-on-task data or some other measure of engagement.

I’m ending up with at least as many questions as answers here. Much to ponder. Next fall, when I teach both face-to-face and online sections using Waymaker in both, perhaps I’ll be able to tease out some more answers.


Posted in Uncategorized | Leave a comment

Response to Robin DeRosa’s “My Open Textbook Pedagogy & Practice”

This is an amazing and thought-provoking post. There’s so much here that I just want to put my initial thoughts out; otherwise, I may put it aside and not respond at all.

I’ve done this sort of thing before but never as ambitious a project as a complete text—see and more recently —so I can speak to how much work and energy on your part is involved. I can also affirm that students who participate engage with the project far more deeply than with most academic assignments.

First, the naysayer: Did you review and/or edit the students’ commentary? That could move it from an excellent student project to a more serious professional resource, albeit at the cost of taking something away from the students, and more of your time. For my financialcrisis project, I ended up spending a huge amount of my summer reviewing, editing and other production work. Do you have a better way?

In my view, there are a couple of takeaways here:

  • Don’t let the perfect be the enemy of the good. One shouldn’t compare an open text like this one to a finished, glossy commercial text. A better comparison is with a first draft manuscript—the one the author submits. The content is pretty much all there, though the finish may not be. But that’s okay.
  • It’s hard to understate the learning experience of the students. This learning experience is so much more profound than what a student normally gets from reading a book, because the process of curation requires students to examine assumptions, consider alternatives and evaluate what they conclude—all things that we at least subconsciously hope students will do when they read, but which realistically very few do—in part because we haven’t taught them to read that way, and in part because it’s hard work and takes a lot of time.

I look forward to reading the next chapter in your exploration of open texts.

PS: As I was thinking about responding to Robin’s post it occurred to me that the way I respond to a blog post is a little different from the way I respond to a colleague’s paper. Perhaps it’s the nature of blog posts, which are less formal than papers. Whatever the reason, I find I respond affectively to blog posts as well as cognitively.  It’s not that a post can make me feel good or bad, but rather my response is coming from a deeper level than when I respond to a formal paper.


Posted in Uncategorized | 1 Comment

Is the Purpose of Teaching Student Learning or Student Grading?

Brief Remarks for the UMW “Faculty Conversation,” May 11, 2016 

Is the purpose of teaching student learning or student grading? I don’t believe that you can have it both ways. That is, grading, at least the way it’s commonly done, inhibits student learning. This is not merely my idea. David Brooks wrote a piece in the NYT about this, saying

“We all know why it exists, but the grade-point average is one of the more destructive elements in American education.

In life we want independent thinking and risk-taking, but the  GPA system encourages students to be deferential and risk averse, giving their teachers what they want.“

Think about that.  Does anyone who’s taught for any length of time disagree on this point?

The “F” grade, in particular, is an artifact of the current focus on teacher as grader, rather than teacher as learning-facilitator. It’s also an artifact of the system where students don’t want to be in the class. Even the “good” students more often than not see (most) classes as things they have to take, rather than things they want to take, or things they have a real reason to take.

Contrast this with the real world: In the real world (or for adult learners), there is a real purpose in learning something. If the learner doesn’t achieve mastery in some period of time, they keep at it until they do. What about those who are time constrained? They either need to work smarter or take longer. If they really need to learn something, they will find a way.

Only in school do we give students fake assignments where the answer is already known so the task is to identify the one specific answer. If you get something else, you are WRONG. But I digress.

Let’s take a lesson from writing teachers. No one writes a perfect paper on the first draft! Rather, good writing teachers understand that writing well is a process; that it takes multiple attempts to improve, to get it right.

Suppose you read a student paper, and put an A on it. How will the student react? How will they probably feel? They will feel great, successful. What’s the likelihood that the student will want to improve the paper?   Answer: zero.

Suppose you read another student paper, and put an F on it (with no comments). How will the student react/feel? This student will likely be upset, possibly angry, possibly bewildered. How will the F help the student become a better writer? By itself, it won’t.

Suppose you put an F on that paper AND you write comments to explain the grade. If you allow the student to revise the paper, he or she will likely try to fix the problems you identified. If they do, will you give them an A? Probably not, because a good writing teacher won’t identify everything that’s wrong with an F paper, they won’t cover the paper in red ink, rather, they will just focus on the highest level flaws.

Notice what’s going on here: The student is trying to satisfy the teacher, rather than themselves.

Suppose you read the paper, write the comments and omit the grade. How will the student react: Not upset, not angry; hopefully just interested in improving.

More and more, I am attracted to Mastery Learning. Don’t worry! I’m not using it in all of my courses, just some of them. Mastery Learning is a binary system: Either you’ve learned the content sufficiently to use it. Or you haven’t yet.

In other words, Mastery Learning is something like a pass/fail system, except that failure doesn’t mean the student is a failure (though I think we often treat it that way).  Rather, the alternative to Mastery is “Not Yet,” as in “the student hasn’t mastered it yet.” This is not just semantics!

Think about the artificiality of term lengths. What is sacrosanct about a 14 week term? Why don’t we recognize/accept that students learn at different speeds? A senior colleague once told me that speed and intelligence are highly correlated, so the student who completes the test first tends to score the highest. As a slow learner, someone who likes to chew while thinking, I don’t agree and I’d like to see the literature on it. Why not extend the length of time for students that need it?

Imagine, if instead of the current system, students were trained to see learning as a process, taking work, but ultimately leading to success.

Imagine that students were trained to expect to make mistakes along the way, that mistakes were not signs of failure, of hopelessness, but rather just steps towards learning.

Imagine if we taught students to evaluate their own work, rather than deferring to the authorities to do it for them.

Just imagine.


Posted in Uncategorized | 2 Comments

Helping FY Students Take Ownership of their Writing

Remarks for the UMW FSEM Swap Session, May 5, 2016

This year is the 10th anniversary of our First Year Seminar program (FSEM).  Since the beginning, introducing FY students to college writing has been an element of FSEM. Through the years, we’ve had a number of workshops on how to teach writing to first year students and what writing problems first years bring to college.  These remarks are intended to be a small continuation of that conversation.

One of the goals of FSEM is to change students’ perception of writing assignments from something they do for us, to something they do for themselves.  I tell students I’m not looking for them to identify the right answer on their papers, but rather that I genuinely want to know what they think.  Some students seem sceptical, as if (to quote Gardner Campbell), “[they] assume there is a rule they don’t k about that teachers will hold against them.”

Some years ago, I noticed that grading student papers seemed to be an adversarial process. I would read a student paper and then start thinking about what was wrong with it, how I could deduct points to determine a grade.  Yes, I knew that I was supposed to include positive as well as negative comments, but when one is grading a lot of papers, it’s much easier to focus on what’s wrong with a paper, rather than what’s right.  After all, students are supposed to be able to write, so why should I credit them with what they should be doing already?

One day, I had an epiphany:  The way I read/comment on my colleagues’ papers is very different from how I grade students’ papers.

Why is that?

One reason is that I’m not reading colleague’s papers for a grade.  Rather, I’m trying to help them improve their work.  But isn’t that, or shouldn’t that be, the same goal for reviewing student writing?  That depends on whether reviewing student papers is perceived as formative (part of the learning process), or summative (a final judgment).

As I developed my first FSEM, I made the choice to change the way I read student papers:

  1. I give students a hard deadline for the first draft.  If they meet that deadline, they qualify for the following.
  2. I review the draft like I was reviewing a colleague’s:  I identify parts of the paper I don’t understand and ask for clarification or elaboration.  I identify parts of the paper that piqued my curiosity, and/or that I’d like to hear more about.  I identify places where I lose the argument.  I make suggestions for improvement.
  3. I tell students that these are my comments, but that the paper is theirs to revise as they see fit. I hope they will take my comments seriously, but any changes are up to them.
  4. I don’t say anything about a grade. In fact, while I’m reading I’m not thinking about a grade. Rather, I’m just thinking exclusively about what suggestions for improvement I can make.  (By contrast, when I grade papers, the comments I write are very different than the ones I put on drafts.)  This was difficult when I started the practice, but it comes without thinking now.
  5. I tell students if they revise their paper I will review it again, as many times as they like for the next 4 weeks until the final paper is due.  Only when they decide their paper is “done” will I put my grading hat on.

I want the students themselves to evaluate their writing, to decide when it’s done.  I think we’ve trained students to expect the teacher to tell them how good their work is and when it is done.  I think when we do this for them, they miss out on an important part of the learning process.  Suppose on the first draft I told a student s/he had an A paper. What more work would they do?  Is an A on an FY paper as good is it could be?  Suppose a student writes a weak paper.  Asking s/he to reflect about how good it is might help them take their writing more seriously, especially if you draw out their thinking.  (What makes you think it’s strong/weak?)

This probably sounds like a lot of work, and it is. But it’s not as much as you think. FSEM’s are small classes capped at 15 students.  These are short papers (3-5 pages).  Only a few students write more than three drafts.  Except for the first one, the drafts come in in a staggered fashion, allowing me to read and comment on them one at a time.  All students write fewer drafts as the semester progresses.

As a bonus, I feel much better about how I teach writing in the FSEM.

The next step in my evolution as a writing instructor, which I will try next fall, is to lead my FSEM students to develop a rubric for me to use in evaluating their papers. I’m thinking of a list of what makes a good paper (and what makes a poor one), rather than a rubric listing requirements for formal grades. HT to Robin DeRosa for inspiring this.  Now that I’m planning this, it seems like an absolute no brainer.

Posted in Uncategorized | 11 Comments

Friendly Amendment to “OER: Some Questions and Answers”

A couple weeks ago, David Wiley responded to a clever Pearson op-ed, which attempted to damn [OER] with faint praise. Among other things, the op-ed states:

“OER often shine in their variety and ability to deepen resources for niche topics.“ (emphasis added)

I have a couple of thoughts on the op-ed and David’s response.

The op-ed asserts that while OER could be high quality, few “instructional design-minded instructor[s]” would be willing to put in the required effort without “fair compensation”. The author’s implication is that OER is extremely unlikely to replace commercial textbooks.

To this point, David responds that the public good nature of digital OER refutes this:

“The nonrivalrous nature of digital resources (the technical ability to share copies of resources at almost no cost) combined with open licenses (the legal permission to share copies of resources at no cost) means that only a handful of people need to be actively involved in producing or making substantive improvements to OER in order for the public to have free and open access to resources whose effectiveness is on par with those created by commercial publishers.”

David goes on to state that the classic theory of public goods,

“lacks an account of why people volunteer for or donate their time, money, and effort to a range of charitable and other causes, including the creation, improvement, and maintenance of open source software and open educational resources.“

I respectfully suggest that the theory suggests otherwise.

A positive externality occurs when the benefits of some action accrue to people other than the consumer who pays the cost. A good example is flu shots. The rational individual will get a flu shot if the benefit he or she receives exceeds the cost they pay. But note that if you get a flu shot, I benefit also since I won’t catch the flu from you. In other words, there is an external benefit to your getting a flu shot (or to put it into economic jargon: the marginal social benefit exceeds the marginal benefit to you). But you don’t take the benefit to me into your calculations. For some people, the cost will exceed their perceived benefits. Those people won’t get the flu shot even when the social benefit (to them and others) is greater than the cost. This is the argument that private markets will under-produce goods with positive externalities.

A public good is a little different. With a good like a flu shot, the main benefits go to the consumer, while the external benefits are secondary. Or to put it more generally, the private benefits make up a substantial part of the total social benefits.   With public goods it is the opposite.

Public goods are defined as those with a high upfront cost, but a zero (or low) marginal cost for others to benefit once the public good is provided. With a pure public good, no one finds the benefit to them worth the substantial upfront cost. Thus, no public goods are provided, even though collectively the social benefits exceed the social costs.   Provision of public goods requires a mechanism for the collective decision to consume and pay for them.

“Non-rival” means that your consumption doesn’t detract in any way from my consumption. An apple is a rival good. If I eat it, you can’t. A printed text is largely rival, since if you are reading it, I can’t read it too, at least not easily at the same time. If more people want copies, they have to pay for them. This is why students generally obtain their own copy of a text, rather than sharing with someone else.

A digital text is non-rival. The cost of creating the text is substantial, but an infinite number of people can download and use a digital text at no additional charge (except for maintaining the server). Thus, a digital text is a public good, while a printed text is a private (rival) good. What about commercial digital texts? Commercial digital texts are technologically-speaking, public goods, but economically-speaking they are not. Property rights to the producer make them artificially private goods. Note that it is (near) impossible to prevent one student from benefiting from another student’s flu shot, but it is entirely possible through DRM to prevent one student from benefiting from a commercial digital text unless they pay the price. This is true even though the actual cost of an additional download is zero.

The classical public good argument is not an argument for why public goods won’t be produced by people who don’t see any personal financial benefit. A flu shot doesn’t convey any (direct) financial benefit to the recipient. There are benefits, but they are not financial. What the argument says is that the personal benefits do not exceed the personal costs. This raises a couple of questions. If this condition characterizes a major work of OER, like a complete textbook, one that could replace a commercial text, then such OER is, indeed, a public good, and by the theory requires public support (e.g. a subsidy) for the OER to be provided. Thus, the grant funding behind a lot of OER makes sense in the context of public good theory.

But there’s another dimension to OER, and this is where I think David is mistaken. When David claims that the theory of public goods can’t explain why people devote time, money and effort to creating OER, I disagree. The theory suggests that people will do this when the benefit to them exceeds the cost. I think that the benefits perceived by these individuals are the benefits their actions provide to other people. Altruism provides a benefit even though the benefit is not financial. I think that explains that why people are often willing to contribute to improvements to OER, once the OER exists. And this makes no sense to anyone who limits their understanding to the realm of commerce. To phrase it differently, new major OER is a public good, but improvements to the same are positive externality goods.

[ By the way, I’m not wowed by the Wikipedia quote on public goods that was cited in the post. Free riding does not cause private costs to exceed private benefits. Rather, it strengthens the case for collaborative action because private action is less likely to occur.]

Beyond this point, my reading of the op-ed suggests that Pearson is setting up a strawman to represent OER. The author implicitly compares comprehensive commercial text books to the single piece of OER created by a random faculty member to teach Topic X, or even an open source text written by a faculty member on his/her own with no editorial or publisher support and no extra quality control. This is not the type of major OER that David and other OER publishers like OpenStax are about. Of course, it’s in Pearson’s interest to promote this myth.

It also seems to me that the commercial publishers are trying to design products for instructors who want to minimize the amount of work they must do to integrate the text into their course–time-constrained adjuncts and lazy tenured or tenure-track faculty come to mind. An obvious example is the powerpoint slides provided by publishers to mirror the book’s content. The op-ed alludes to this when it brings up the costs to faculty of adopting OER, and the implication that these costs are less for commercial texts.

I have to admit that earlier in my career, I adopted textbooks that I had never read. I suspect that’s pretty common across academia. I simply assumed that if they were commercially published, they must be good, right?  David’s post exposes the iie in that assumption.

Posted in Uncategorized | 1 Comment

2016 OpenEd Presentation

[ to be posted here ]

Posted in Uncategorized | Leave a comment

Online Shouldn’t Mean a Robo-Course!

This is the time in the semester when the workload starts to increase and the ability to think metacognitively decreases in proportion.  Many folks, both students and faculty, are just trying to keep their heads above the water.  Which means this is a bad time to enter into the most analytically complex material in principles of microeconomics: namely, the theory of the firm.

I sense that my students are struggling.  It’s not just intuition–Waymaker is sending me students’ scores on module quizzes and the scores have dropped significantly compared to earlier in the term.  Students who are ahead of the pack, who had been earning 80+ percent on their first quiz attempts are getting 60% plus or minus.  Ideally, these students will review the material carefully and deeply before going on to their final quiz attempts. These are good students, but I worry about the rest, not just the bottom cohort who tends to have trouble finding time to do the work, but the middle group who can generally be successful with enough effort.

This morning I was reminded of an experiment I tried in the mid-1980s:  I attempted to teach without formal textbooks, instead using books for popular audiences (e.g. Freakonomics, The Undercover Economist, and the like.)  My hope was that students would find those books more appealing and thus get more out of them.  I learned a great deal from that experiment, including that while students can learn most of the content of principles of economics from popular books, they can’t learn theories & models as well.  They need to have a textbook-like treatment that they can review multiple times if necessary.  As a result, I switched to more basic, less encyclopedic textbooks, which seemed to work fine in my face-to-face courses.  I was able to clear up any problems or misconceptions my students got from the text.

Now that I’m teaching online, I don’t have the same opportunity to either read students’ body language to see when they are having problems, or to explain difficult material.  How can I provide the same support in an online course?

One option is to change nothing.  The Waymaker content is pretty good, though to be fair this is the first time I’ve taught with it, so there may be shortcomings in places.  That’s why they call this a pilot!  It may be that the Waymaker approach will work.  Students will struggle on the quizzes, they will return to the content, study carefully and pass on the second quiz attempt.  Perhaps, but since this approach didn’t work with textbooks, I wonder if there’s something I can’t do to insure better outcomes.

One thing comes to mind, and yes, I know this is a “duh” moment. But like Nobel Prizes, it’s only a duh moment after you’ve had it.  The first time I taught online my students struggled to grasp supply & demand analysis.  So what I did was create a small group assignment that required groups to do several supply & demand problems.  Working thru multiple problems and working in groups seemed to do the trick.  Since I’ve already got study groups in my online course, I’m going to give them some “theory of the firm” problems to work thru and we’ll see if that helps.

Image “Dog Paddling” courtesy of Lorenia (via flickr)

Posted in Uncategorized | Leave a comment

When less is more

I have a relatively light teaching load this semester. I’m teaching a senior seminar, our introductory research methodology course, and an online principles of microeconomics course (technically, two smaller than average size sections taught as a single course). And at least so far, my scholarly projects have been on hold.

The light load has allowed me to put a lot of effort into the online course, making it highly interactive. The course is pilot testing (for a second semester) the Lumen Waymaker platform. Waymaker is designed for mastery learning: Students work until they achieve mastery, that is, when they fail to reach the mastery level on an assessment, they keep working through the material until they do.

I’ve found that students find the mastery approach very countercultural. When students fail to achieve mastery, they seem to feel like failures, as if they took their shot and missed. But in my experience that doesn’t characterize much of the real world. Instead, one is rarely given a pass if you don’t succeed on the first try. Rather, you are required to fix whatever went wrong, that is, you keep trying until you get it right.

An integral part of the Waymaker philosophy is for instructors to reach out to students experiencing difficulties. Waymaker informs instructors when a student fails to achieve mastery on any end-of-module quiz (e.g. chapter test). When I have reached out to students this term, I’ve found the students to be very apologetic or even guilty. The students’ natural tendency seems to be to think I am scolding them. But this is the opposite of what I’m actually trying to do.  Rather, I’m trying to build a coach/mentor/tutorial-like relationship to help each student negotiate the learning process and to encourage students to see that learning is achievable, that there is no failure unless you quit. Again, this seems to be a very different approach than most of my students are used to.

The Waymaker platform has facilitated this approach in my principles course, but it’s really the availability of time that allows me to do this, in part the time Waymaker frees up, and in part my lighter than normal teaching load this term.

The load has also enabled me to use the same approach in my other courses. I’ve been able to keep up with the near weekly research assignments in the methodology class. I’ve been able to respond with more thoughtful feedback than I have typically been able to do. While I don’t know for sure yet, I feel this is enabling earlier intervention into prospective problems than in the past—and students always encounter problems in the research course since problems are inherent in doing research.

This all feels good, but I also feel some guilt that I must not be working hard enough, that I’m not supposed to be on top of things like this. I know this feeling is wrong, but it shows how deeply engrained the traditional grading & overworking system of academia is and, in my opinion, how pathological the system is. All of this has made me think about how we grade, and how difficult it is for both students and instructors to disentangle summative assessment from the learning process.

Something to ponder.

Posted in Uncategorized | Leave a comment

Journey into OER: My Quandary

VivienRolfe: I fear we will build the education system that we can measure – not the one that we need. #opened15 #analytics

This is the second in the series.  The first was here. I need to admit upfront that this project is causing some serious cognitive dissonance for me, because it is forcing me to reconcile two conflicting views I have about higher education.  You can see these views in my posts over the years. These views are not unique to me: They were on display at and around the OpenEd 2015 conference, where there was significant pushback by thoughtful commentators who argued that open education is more than open text books. Well, yes, but one has to start somewhere. (I think of open text books as the “gateway drug” to open ed more generally.)

I believe that college has an experiential dimension that goes beyond the content, and to a certain extent, skills learned. The seminars that I teach are oriented around this perspective, so that the most important thing is the degree to which students have participated in, or more precisely engaged with, the process. It’s not that there isn’t content to interact with and skills to learn, but I give students a great deal of discretion about what content to explore, and about which skills they hone. My introductory courses, by contrast, are very much content and skills-oriented. If students haven’t learned the content and skills, they haven’t been successful in the course. My intermediate courses are somewhere in between.  It seems clear to me (though I don’t recall anyone actually saying this) that the importance of what I’m calling content/skills vs. experience varies by discipline.

I believe that learning is fostered when student actively engage with the material. The traditional lecture course, where the instructor lectures on the content, students (ideally) read the text, and then (hastily) study for the exams, is not a good example of an active learning approach. Students need to engage with the material, which suggests the need for regular, low stakes assignments with feedback so students will be able to monitor their learning. Again, the form of this engagement & assessment probably varies by discipline. Whether these assignments are essays, problems, or quizzes, an active learning approach of this type adds to the workload of instructors. Instructors have only 24 hours in the day, like everyone else, so we have had to make choices about how many assignments to give and of what type. There are simply not enough hours in the day or days in the week for me to give and grade as many assignments as would be ideal. Enter computer-based learning and data analytics. A computer is ideal for giving unlimited assignments with almost instantaneous grading. Of course, some assignments work better than others, and computers can only grade what they can measure. As a social scientist, I am trained to construct and use data to draw conclusions. If data analytics can help students monitor their learning, and help me help students be more successful with their learning, then I am all for it.

Which leads to my quandary: If I were to incorporate computer-based learning with data analytics, it could only assess things that can be measured. This would leave out anything that can’t (easily) be measured, like the experiential aspects of my seminars.   I believe both these things, so what to do?

Digression:  The shortcoming of courses that emphasize experience over content/skills is that grades are more subjective. But the reality is that all grades are subjective, and the less subjective they are, the more they emphasize only those aspects of learning that can be measured.  I am willing to accept this in my intro courses, but not in my seminars where I want to give students the freedom to learn what’s important to them.  Perhaps ironically, that meets the learning objectives of the course.

Another reality is that students typically don’t do that well on my introductory exams, and this is pretty common in the discipline. One response is to claim that economics is just difficult, that some/most people are just not suited to learning it. I have not yet given into that cynical view, but I don’t have a lot of evidence to support my hopes.  I want to believe that all students at the introductory level can learn the material, and that if they don’t, it’s the fault of the instructor or the system.  Don’t get me started on the pathology of final exam week

The solution I’ve found for my quandary comes from being a pragmatist and an empiricist. This academic year, I am pilot testing the Lumen Waymaker courseware I mentioned in my previous post (Principles of Macroeconomic last semester; Principles of Microeconomics this semester) to see how well it works.  I wouldn’t use courseware like this for a senior seminar, but it may make sense for an intro course.

(To be continued.)

Image Credit: John Fowler (via flickr) Balanced Rock

Posted in Uncategorized | 2 Comments

Journey into OER

 This is the first in a series of posts about my journey into OER.  If you are not familiar with Open Educational Resources (OER), you should be.  OER is course materials: syllabi, assignments, lecture notes, and even texts, which are available for adaptation and use for free.  There is almost certainly some OER in your field. If you are like the majority of faculty that I know, you care about your students’ learning and you want them to be successful. One of the major impediments to college graduation is the cost, which has grown faster over the last few decades than nearly any other component of inflation.  Textbook costs are a major part of the cost of higher education.

The strength and the weakness of OER is that it is free. One of the first principles of economics is that there is no such thing as a free lunch—everything has an opportunity cost. If the user doesn’t pay it, someone else has to or else the good or service will not get provided. Much of existing OER is craft work, interesting and useful things created as more or less one-off projects. This is not a bad thing—indeed, it describes pretty much everything I’ve done in my professional life. Until recently.

Two years ago, I helped create a free, open source principles of economics text for OpenStaxCollege. OpenStax has introductory text books in a variety of fields. You should check to see if yours is represented. The books are free in a variety of formats, but OpenStax has been challenged with the question of how to keep their textbooks current and available. Someone has to pay the bills to run the servers. Someone has to pay for updates and revisions. Or these things won’t happen. OpenStax’s strategy has been three pronged: One approach is to ask for donations. I don’t know how successful that has been, but I suspect donations alone will not be enough to pay the bills. Another approach is to work with commercial firms to provide after-market products that are not free, though they may be effective at enhancing learning and they are not very expensive compared to traditional commercial textbooks and their accoutrements. In return, the commercial firms provide a modest revenue stream to OpenStax, enough perhaps to maintain the existing collection of books. The third approach is to solicit grant funding for major projects, either creating new books or making major revisions of existing ones. Granting agencies support innovative new ideas, but they tend to be less supportive of maintenance operations, including revisions. The Gates Foundation has been this way.

One of the goals of the Gates Foundation’s Next Generation Courseware Challenge has been to develop a business model that allows for operation at scale, but that doesn’t require continued grant funding. Enter Lumen Learning. Full disclosure: I am working for Lumen on the project I’m about to describe.   This means I am not an objective observer, which means I am going to tell you what I really think: Go figure!

Lumen is a for-profit company. How can a for-profit company provide OER? I had a hard time wrapping my head around this, but I may have figured it out now.  Lumen is giving away its content for free.  After all, that’s the way OER is supposed to be.  Here is the principles of macroeconomics text.  Here the the principles of micro text.  Then Lumen used this OER to create digital courseware that will replace the textbooks students use in (at present) four of their courses. Lumen will sell this product to colleges and universities at a significantly lower price than students would pay for a text book (approximately 10%), something comparable to what OER after market companies are charging for their products. It will be up to schools to determine how they are going to pay for the Lumen product, perhaps by charging a course fee, but once the school opts in, the students will have immediate access to the text from day 1. Research has shown that when students have access to the textbook on the first day of class, they are far more likely to finish and succeed in the course. In short, the product is not free, but neither is it very expensive. The purpose is to allow the product to be used by a large number of schools and students, but in a way that is economically sustainable and won’t require constant searches for money to support.

(To be continued.)


Image Credit: oer_logo_EN_1 by Breno Trautwein via twitter.





Posted in Uncategorized | Leave a comment