Course Evaluations for my Waymaker Course Pilot

LumenWaymaker-400x80This past term, as part of a nation-wide pilot including more than 9000 students, I pilot-tested Lumen Learning’s Waymaker platform in the online version of my ECON 202, Principles of Microeconomics course.  I hoped that Waymaker would provide a useful structure for my online course.  At the end of the term, I asked the students to complete a course evaluation that I constructed. The responses were anonymous, though I gave each student who submitted one extra credit towards their final grade. (I ask them to email their responses as anonymous attachments to my colleague, Bob Rycroft, who sends me the list of names, but holds the evaluations until final grades are turned in.) Out of 36 students in the course, I received 28 submissions so the response rate was decent.

The evaluation included questions on the online course (since I’m still trying to improve my online teaching) as well as on the Waymaker platform. The questions asked for student perceptions and included both quantitative and open-ended responses. My statistical analysis of learning outcomes for this course, as measured by student performance, will come later.

This past week I reviewed the student submissions and coded them into Excel. I prefer to do this myself, rather than asking a student aide to do so, because in reviewing and coding I find myself doing some processing of the results. More precisely, because I’m working one submission at a time I get a sense for what and how each student is thinking; I get a sense of consistency or inconsistency in responses to different questions, and I get a sense of the relationships between responses on one question and responses on another.   For example, and this is just a casual feeling, it seemed like non-traditional age students were less likely to buy into the Waymaker approach, which attempts to facilitate deep learning. This manifested in things like not watching the video content, not doing the simulations, and asking for study guides that would explain exactly what they were expected to learn. Don’t get me wrong; I am not anti non-traditional age students. Some of my best students fit that demographic.

So here are my initial impressions.

ECON 202, Principles of Microeconomics is an introductory-level course. The course is required for the business, environmental science and international affairs programs, as well as for certifying to teach K-12. While the intent is for students to take this course early in one’s academic career, some defer it until later. The majority of my respondents were in their first (10 students) and second year (11 students). The remaining seven students were third (5 students) and fourth years (2 students). Most of the students (18) live on campus. Three live close by; the remaining seven live as far as 50 miles away.

Interpreting a survey like this is challenging. Some students seemed to try to say what they thought I wanted to hear. Some gave answers to related questions that were inconsistent with one another. Some students didn’t interpret the questions as I intended or didn’t think carefully about them, dashing off answers that don’t provide much useful information. One student wrote,

“The hardest material to learn was anything really extensive and difficult.”

An additional complication was that this was an online course, which is relatively rare at my institution, so some of the criticisms expressed by students were criticisms of online courses, rather than of the Waymaker platform. Similarly, economics is a challenging subject, since unlike English or History, many students haven’t studied it before. So some of the criticisms were really about the discipline rather than Waymaker.

Waymaker is a next generation courseware environment, combining text, video, and animation/simulation, all embedded in a mastery learning framework with formative and summative assessments interleaved through each content module (analogous to a book chapter).

Three quarters of respondents were generally favorable towards the Waymaker modules, and only 18% said they preferred a traditional textbook. 42% of respondents said Waymaker was easier to understand than a traditional textbook, while 32% said a commercial textbook was easier. As one student said,

“The combination of videos, self-checks [formative quizzes], and worked examples allowed for small checks for understanding that a textbook does not provide.”

Students indicated that the hardest material to learn was specific economic theories, understanding graphs or understanding the math. Anyone who has taught principles of economics would find this pretty normal, but I wonder if Waymaker could do better than text, given its multimedia approach?

Let’s look at some of the specific features of Waymaker.


None Some All  
1 20 7 How many videos did you watch?
3 13 12 Did you watch them in their entirety?
No Yes
12 16 Did you ever watch the same video more than once?
Not very useful 1 2 3 4 5 Very useful
0 6 6 8 4 On a scale of 1 to 5, how useful were the
25% 25% 33% 17% videos for learning concepts or models?

Only 25% of students (7) indicated that they watched all the videos. Is this a problem? Slightly less than half the students watched the videos in their entirety. More than 60% (16 students) watched the same video more than once, while 75% found them somewhat to very useful.  25% indicated the videos were not very useful.


None Some All  
1 18 9 How many animations/simulations did you play?
2 15 11 Did you play them in their entirety?
No Yes
10 18 Did you ever play the same animation/simulation more than once?
Not very useful 1 2 3 4 5 Very useful
1 6 10 5 4 On a scale of 1 to 5, how useful were the
4% 23% 38% 19% 15% animations/simulations for learning concepts or models?

About a third of the students played all the animations/simulations, and the same number found them somewhat or very useful.  27% found them not very useful.   My impression is that students who like them, liked them a lot.  I didn’t get the same impression with the videos.

Study Groups:

Less Same More
3 10 15 Did you learn more from working with your study group than you would have learned on your own?
Own Group When you did group assignments, did you mostly work on your own & share, 
7 21 or did you work together as a group and go over each others’ contributions? 

One of the features of the online course was study groups, which I used to counter possible feelings of isolation for students working online. Slightly more than half the students (54%) said they learned more from working with their study group than they would have learned on their own. About 11% said they learned less. Three quarters indicated that they worked together as a group on assignments, while one quarter divided up the assignments, worked individually, and then shared answers. Some study groups were effective, because the student drew on each other’s strengths. Others were less effective, either because one or more students didn’t participate easily (or were hard to reach), or because “no one in the group knew the answers.” I need to do more research on group work to see if I can improve these results.


No Scores Yes  
1 8 17 Did you use the self-checks to assess your learning? i.e. Did you use the self-checks to think about what you knew or didn’t know, or did you just focus on the scores.
Never Some Often  
4 11 13 Did you use the module quizzes to study for exams, aside from doing them for the grade? 
Less Same More  
7 10 11 Compared to using a traditional textbook, did you think about how well you were learning as you went through the modules?

Nearly all students used the self-checks (formative quizzes); only 28% (8) focused on the scores, while a majority (17 students – 68%) used them metacognitively to think about which topics they had mastered and which they had not. Nearly half (13 students – 46%) used the summative end-of-module quizzes often to study for exams, while another 39% (11 students) used them sometimes. 39% (11 students) indicated that thought more about how well they were learning using Waymaker compared to a traditional textbook. 25% (7 students) said they thought less about how well they were learning using Waymaker.

Less Same More  
5 17 6 How much time did you put into this course compared to others you have taken at UMW? 
Easier Same Harder  
0 7 16 How did this course compare with the others you have taken at UMW?

A majority of students (57%) found the course harder than others at UMW, but only 26% put more time into the course, which seems odd to me.

What tentative conclusions have I drawn from these survey responses?

Some students liked Waymaker; some students didn’t. What was the difference? Was it merely a question of taste, or something more substantive that could be improved?

Weaker students seemed to not like Waymaker, but was this cause or effect?

Some students didn’t seem to take the Waymaker features very seriously or they didn’t give them a serious try. For example, a number of students didn’t do much with the videos or the animations/simulations. Look back at the Video & Animation/Simulation results above. Students who responded None to the video or animation/simulation questions obviously didn’t benefit from them, but I suspect that many of those who responded Some/Some/No were also disengaged.

At the same time, 50% of the respondents said that the videos were useful or very useful for learning concepts or models, while only 25% said they were not useful. Similarly, 35% said the animations/simulations were useful or very useful, while 27% said they were not.

My impression is that some students, especially first semester first year, or transfer students, don’t seem to understand learning at the university level. Some don’t know how to study deeply and effectively. They have adopted the “school is about grades” motivation, and their approach to studying is to skim the readings looking for key terms to memorize. I know not all FY students or transfers are like this, but it’s a pattern I saw in the survey.

Waymaker was designed for deep learning; it was designed to mimic Socratic questioning of the student. It’s not perfect yet, but those were some of the goals.

Two common themes in the student comments were time management/self-discipline and the greater time required compared to a face-to-face course. The first theme is common to online courses. As one student explained it, “Being responsible and getting assignments done without having the professor in class reminding you.“

The second theme was more interesting. A number of students objected to the time required to work through the modules. But compared to what? The time spent reading a traditional text? But that’s not the right comparison. Waymaker tries to provide the structure students get in a face-to-face course. So the proper comparison should be time in the classroom plus time spent reading the text. I don’t think most students understood that.

Several students mentioned that the course required them to teach themselves! But what do they mean by that, and how is it different in a face-to-face course? If they simply mean the lack of lectures meant they had to filter the content and discern meaning for themselves, that’s almost tautological. And some research suggests that might actually enhance learning, for those that do it seriously.

But what if part of the pushback was students’ lack to experience with studying in depth? Then if they had bought into the program they could be more successful, but if they didn’t buy in they could be frustrated and unable to see the superficiality of their study approach, possibly leading to giving up on Waymaker. We might be able to test this if we have time-on-task data or some other measure of engagement.

I’m ending up with at least as many questions as answers here. Much to ponder. Next fall, when I teach both face-to-face and online sections using Waymaker in both, perhaps I’ll be able to tease out some more answers.


This entry was posted in Uncategorized. Bookmark the permalink.

4 Responses to Course Evaluations for my Waymaker Course Pilot

  1. Robin DeRosa says:

    This is so interesting! I am really trying to get my head around Waymaker, which honestly I haven’t had much time to look into. From reading your post, it seems like an LMS-type set-up that integrates OER content with multimedia learning activities? How much of what was in Waymaker was your own stuff, how much was OER from around the web, and how much (if any) was Lumen’s stuff? And what are the costs ($) of using Waymaker? I couldn’t easily find that info on the Lumen site (I didn’t click “Request a demo”). I am trying to figure out how “open” a course can feel inside Waymaker. Are there ways that is is less “locked down” than an LMS? Do students produce work inside it or does it focus more on assessments, and outputs for students to watch? I think if I spend a bit more time poking around in Waymaker, which I really have planned to do so thank you for reminding me, this post is going to become really useful to me. As of right now, I see the benefits for competency-based approaches, but wonder if there is anything in Waymaker that might work with more inquiry-based approaches… Thank you for reminding me to learn more about this!

  2. Steven, what a useful step in your experiment.

    Several thoughts.
    1) Great point about the difference between economics and other disciplines, which undergrads are likely to have already experienced.
    2) Were you able to draw connections between attitudes towards Wayfarer and students’ majors?
    3) I’m surprised at engagement with videos, which is lower than I would have thought.

  3. Hi Steve,
    Your post came about at a time when I was at a huge teacher conference in Texas, so apologies for the time it has taken for me to respond. You are such a champion of our work at Lumen, and we’re all so thankful for your support. Suzanne shared the raw data with me, and I’ll admit that I think better with a story like this than I do with numbers on a spreadsheet.
    First of all, I’m quite charmed to see Bryan call Waymaker–Wayfinder. It’s got a nautical ring to it that Kim might like since it sounds like the name of a sailboat. I was just talking to folks who made the same mistake and it could seen as a branding issue, but I see it another way. At the heart of what we’re trying to do is to help students make and find their way towards what they already know and how it connects to what they need to learn. In the most simplistic terms, anyway, that’s how I understand our courseware’s potential. As Audrey Watters reminds us, it’s nothing new–it’s just good teaching to take into consideration what people know and how it connects to their future learning.

    I was struck by the student’s lack of engagement with the videos and the simulations as well. Bryan beat me to it–as well as having the talent to grow an awesome beard (lol). Here’s my theory as an instructional designer–I don’t think we have enough instructions to contextualize what students should do with the simulations and the videos. It’s amazing what we’ve been able to pull off with the early days of Waymaker. I think that some students may not know to click on them or that they aren’t “extra.” This is something we addressed in our Marketing course, and an idea that I shared with our brilliant course editors. Attaching assessments, discussion posts, and the like to these features may also increase their use. We’ll get there. As we work on continuous improvements of our courses, I think embedded instructions are necessary for FY students.
    Recently I got to see two Business teachers, Linda Williams (Tidewater CC) and Charlene Barker (Spokane Falls CC) talk about their use of Waymaker demonstrating two different LMSs (Canvas and Blackboard). It was incredible to see how they themselves personalized Waymaker using their LMS. Their additional assignments showed their individual pedagogy and personality. Both commented how easy it was to connect to the students that were doing well in the course along with those that were struggling. The messaging system (faculty tools) was the key to that satisfaction, and I loved hearing teachers talk about it.

    I’m also very interested in your second theme that “a number of students objected to the time required to work through the modules. But compared to what? The time spent reading a traditional text? But that’s not the right comparison. Waymaker tries to provide the structure students get in a face-to-face course. So the proper comparison should be time in the classroom plus time spent reading the text.”

    So now let me hop on my soapbox as an administrator/leader–this is where we fail students at the institutional level to prepare them for online courses. This is where we fail preparing teachers to teach using online materials. This is where students fail–they struggle to see that the time engaging with the material is the time they would have been spending listening to a lecture or time in class. When I taught, I got killed in my evals for “requiring too much work” or that I “easier in f2f classes” or my personal favorite, “I took too long to grade.” When I became an administrator, I “certified” new online teachers and I started having experienced online teachers ask me if they could take the course for professional development. I was elated! They were always miffed by “instructor presence” and that was usually their best insight to about what I preached. You have to engage people’s ideas not just their attendance or their quizzes or the gradebook. For people like you and I, Steve, this is really easy. Tweeps and OL pals are some of my strongest enduring relationships these days. It’s not always the case for our students and our teachers. Institutions fail teachers and students by not supporting professional development o f teachers to learn this skill. [end ranty rant]

    Perhaps this something we can work with the student module where we teach students how to use Waymaker. I was pretty new to Lumen when I worked on it, and you’ve given me good insight on how to improve it. As always, Steve, I really appreciate your commitment to open education. Much to ponder indeed.

  4. Excellent thoughts, Alyson.
    I think you really hit the nail on the head re: instructions.
    That gives me more to think about re: NGDLE.

Leave a Reply

Your email address will not be published. Required fields are marked *