Course Evaluations for my Waymaker Course Pilot

This past term, as part of a nation-wide pilot including more than 9000 students, I pilot-tested Lumen Learning’s Waymaker platform in the online version of my ECON 202, Principles of Microeconomics course.  I hoped that Waymaker would provide a useful structure for my online course.  At the end of the term, I asked the students to complete a course evaluation that I constructed. The responses were anonymous, though I gave each student who submitted one extra credit towards their final grade. (I ask them to email their responses as anonymous attachments to my colleague, Bob Rycroft, who sends me the list of names, but holds the evaluations until final grades are turned in.) Out of 36 students in the course, I received 28 submissions so the response rate was decent.

The evaluation included questions on the online course (since I’m still trying to improve my online teaching) as well as on the Waymaker platform. The questions asked for student perceptions and included both quantitative and open-ended responses. My statistical analysis of learning outcomes for this course, as measured by student performance, will come later.

This past week I reviewed the student submissions and coded them into Excel. I prefer to do this myself, rather than asking a student aide to do so, because in reviewing and coding I find myself doing some processing of the results. More precisely, because I’m working one submission at a time I get a sense for what and how each student is thinking; I get a sense of consistency or inconsistency in responses to different questions, and I get a sense of the relationships between responses on one question and responses on another.   For example, and this is just a casual feeling, it seemed like non-traditional age students were less likely to buy into the Waymaker approach, which attempts to facilitate deep learning. This manifested in things like not watching the video content, not doing the simulations, and asking for study guides that would explain exactly what they were expected to learn. Don’t get me wrong; I am not anti non-traditional age students. Some of my best students fit that demographic.

So here are my initial impressions.

ECON 202, Principles of Microeconomics is an introductory-level course. The course is required for the business, environmental science and international affairs programs, as well as for certifying to teach K-12. While the intent is for students to take this course early in one’s academic career, some defer it until later. The majority of my respondents were in their first (10 students) and second year (11 students). The remaining seven students were third (5 students) and fourth years (2 students). Most of the students (18) live on campus. Three live close by; the remaining seven live as far as 50 miles away.

Interpreting a survey like this is challenging. Some students seemed to try to say what they thought I wanted to hear. Some gave answers to related questions that were inconsistent with one another. Some students didn’t interpret the questions as I intended or didn’t think carefully about them, dashing off answers that don’t provide much useful information. One student wrote,

“The hardest material to learn was anything really extensive and difficult.”

An additional complication was that this was an online course, which is relatively rare at my institution, so some of the criticisms expressed by students were criticisms of online courses, rather than of the Waymaker platform. Similarly, economics is a challenging subject, since unlike English or History, many students haven’t studied it before. So some of the criticisms were really about the discipline rather than Waymaker.

Waymaker is a next generation courseware environment, combining text, video, and animation/simulation, all embedded in a mastery learning framework with formative and summative assessments interleaved through each content module (analogous to a book chapter).

Three quarters of respondents were generally favorable towards the Waymaker modules, and only 18% said they preferred a traditional textbook. 42% of respondents said Waymaker was easier to understand than a traditional textbook, while 32% said a commercial textbook was easier. As one student said,

“The combination of videos, self-checks [formative quizzes], and worked examples allowed for small checks for understanding that a textbook does not provide.”

Students indicated that the hardest material to learn was specific economic theories, understanding graphs or understanding the math. Anyone who has taught principles of economics would find this pretty normal, but I wonder if Waymaker could do better than text, given its multimedia approach?

Let’s look at some of the specific features of Waymaker.

 Videos:

None	Some	All
1	20	7	How many videos did you watch?
3	13	12	Did you watch them in their entirety?

No	Yes
12	16	Did you ever watch the same video more than once?

Not very useful	1	2	3	4	5	Very useful
	0	6	6	8	4	On a scale of 1 to 5, how useful were the
		25%	25%	33%	17%	videos for learning concepts or models?

Only 25% of students (7) indicated that they watched all the videos. Is this a problem? Slightly less than half the students watched the videos in their entirety. More than 60% (16 students) watched the same video more than once, while 75% found them somewhat to very useful.  25% indicated the videos were not very useful.

Animations/Simulations

None	Some	All
1	18	9	How many animations/simulations did you play?
2	15	11	Did you play them in their entirety?

No	Yes
10	18	Did you ever play the same animation/simulation more than once?

Not very useful	1	2	3	4	5	Very useful
	1	6	10	5	4	On a scale of 1 to 5, how useful were the
	4%	23%	38%	19%	15%	animations/simulations for learning concepts or models?

About a third of the students played all the animations/simulations, and the same number found them somewhat or very useful.  27% found them not very useful.   My impression is that students who like them, liked them a lot.  I didn’t get the same impression with the videos.

Study Groups:

Less	Same	More
3	10	15	Did you learn more from working with your study group than you would have learned on your own?

Own	Group	When you did group assignments, did you mostly work on your own & share,
7	21	or did you work together as a group and go over each others’ contributions?

One of the features of the online course was study groups, which I used to counter possible feelings of isolation for students working online. Slightly more than half the students (54%) said they learned more from working with their study group than they would have learned on their own. About 11% said they learned less. Three quarters indicated that they worked together as a group on assignments, while one quarter divided up the assignments, worked individually, and then shared answers. Some study groups were effective, because the student drew on each other’s strengths. Others were less effective, either because one or more students didn’t participate easily (or were hard to reach), or because “no one in the group knew the answers.” I need to do more research on group work to see if I can improve these results.

Metacognition

No	Scores	Yes
1	8	17	Did you use the self-checks to assess your learning? i.e. Did you use the self-checks to think about what you knew or didn’t know, or did you just focus on the scores.

Never	Some	Often
4	11	13	Did you use the module quizzes to study for exams, aside from doing them for the grade?

Less	Same	More
7	10	11	Compared to using a traditional textbook, did you think about how well you were learning as you went through the modules?

Nearly all students used the self-checks (formative quizzes); only 28% (8) focused on the scores, while a majority (17 students – 68%) used them metacognitively to think about which topics they had mastered and which they had not. Nearly half (13 students – 46%) used the summative end-of-module quizzes often to study for exams, while another 39% (11 students) used them sometimes. 39% (11 students) indicated that thought more about how well they were learning using Waymaker compared to a traditional textbook. 25% (7 students) said they thought less about how well they were learning using Waymaker.

Less	Same	More
5	17	6	How much time did you put into this course compared to others you have taken at UMW?

Easier	Same	Harder
0	7	16	How did this course compare with the others you have taken at UMW?

A majority of students (57%) found the course harder than others at UMW, but only 26% put more time into the course, which seems odd to me.

What tentative conclusions have I drawn from these survey responses?

Some students liked Waymaker; some students didn’t. What was the difference? Was it merely a question of taste, or something more substantive that could be improved?

Weaker students seemed to not like Waymaker, but was this cause or effect?

Some students didn’t seem to take the Waymaker features very seriously or they didn’t give them a serious try. For example, a number of students didn’t do much with the videos or the animations/simulations. Look back at the Video & Animation/Simulation results above. Students who responded None to the video or animation/simulation questions obviously didn’t benefit from them, but I suspect that many of those who responded Some/Some/No were also disengaged.

At the same time, 50% of the respondents said that the videos were useful or very useful for learning concepts or models, while only 25% said they were not useful. Similarly, 35% said the animations/simulations were useful or very useful, while 27% said they were not.

My impression is that some students, especially first semester first year, or transfer students, don’t seem to understand learning at the university level. Some don’t know how to study deeply and effectively. They have adopted the “school is about grades” motivation, and their approach to studying is to skim the readings looking for key terms to memorize. I know not all FY students or transfers are like this, but it’s a pattern I saw in the survey.

Waymaker was designed for deep learning; it was designed to mimic Socratic questioning of the student. It’s not perfect yet, but those were some of the goals.

Two common themes in the student comments were time management/self-discipline and the greater time required compared to a face-to-face course. The first theme is common to online courses. As one student explained it, “Being responsible and getting assignments done without having the professor in class reminding you.“

The second theme was more interesting. A number of students objected to the time required to work through the modules. But compared to what? The time spent reading a traditional text? But that’s not the right comparison. Waymaker tries to provide the structure students get in a face-to-face course. So the proper comparison should be time in the classroom plus time spent reading the text. I don’t think most students understood that.

Several students mentioned that the course required them to teach themselves! But what do they mean by that, and how is it different in a face-to-face course? If they simply mean the lack of lectures meant they had to filter the content and discern meaning for themselves, that’s almost tautological. And some research suggests that might actually enhance learning, for those that do it seriously.

But what if part of the pushback was students’ lack to experience with studying in depth? Then if they had bought into the program they could be more successful, but if they didn’t buy in they could be frustrated and unable to see the superficiality of their study approach, possibly leading to giving up on Waymaker. We might be able to test this if we have time-on-task data or some other measure of engagement.

I’m ending up with at least as many questions as answers here. Much to ponder. Next fall, when I teach both face-to-face and online sections using Waymaker in both, perhaps I’ll be able to tease out some more answers.