Qualitative Analysis of Econ201online

Last semester (as readers of this blog know), I taught my first fully online course, Principles of Macroeconomics.  I also taught a second section in my traditional face-to-face mode.  As part of the assessment of my online course, I asked students to complete a course evaluation, which they turned in to my department chair who gave me the results after the semester was over.  As a control, I gave the same assessment (except for a couple of online-specific questions) to my face-to-face course.  The response rate from the online class was quite good:  12/15 students.  The response rate from the face-to-face course was less impressive; only 8 of 21 students turned it in.  I’m aware that the latter may not be representative of the course experience, but I’m going to treat it as such anyway.

The mean grade on the final exam was one letter grade lower for the online class than for the face-to-face class.  In previous posts, I’ve alluded to why I think that might be the case, but it’s not yet clear that the cause was the modality of the course.  Let’s look at the survey results:

What year in school are you?

The two course sections didn’t have the same demographics.  There were more lower-level students in the online course, due probably to the fact that my department’s courses fill up quickly during registration. The modal student in the online course was a first semester freshman (5/12 students).  There were also 4/12 sophomores.  By contrast, the students in the face-to-face course were more uniformly distributed with two first years, two sophomores, three juniors (the mode) and one senior.  The less college experience of the online students could be part of the reason for the difference in exam scores.

Where are you living?

More students in the online course lived at a distance from campus, including three who live more than an hour away.  Still, more than 50% (7/12) of the online students lived on campus.  This is pretty common for liberal arts schools with online programs.

Did you read the syllabus?

Nearly everyone in the online course (10/12) read “All of the syllabus.” One student indicated they had read “Some of the syllabus” and one student had no response.

By contrast, the median student in the face-to-face course read “Some” of the syllabus.

Did you watch the Course Videos?

These four questions were exclusive to the online section.  The online course included two types of videos:  introductory videos to each section of the course (including three introductory videos to the course itself) and video excerpts from my face-to-face lectures.  These were limited to the last couple of topics which are highly analytical.

Three quarters of the online students said they watched “all” the videos.  The remainder indicated that they watched “some” of the videos.  Two thirds of the students said they watched the video lectures, and one third said they did not.  All the students indicated that when they watched a video, they watched it in its entirety.  Similarly, two thirds said they watched videos more than once.  It sounds like the students thought the videos were useful.  Next year, I plan to add more videos (for the analytic topics early in the course).

Did you read the blog posts on the website?   

In the online course, 7/12 students read “all” the posts, three read “some” posts, one read “few or none,” and one did not respond.

In the f2f course, half the respondents read “some of the posts,” half read “few or none”.  No one read all.

(There may have been confusion in the online class about what a post was, e.g. posts on the course website vs. posts on the discussion board.)

Did you read and use the Learning Objectives?

In the online course, 10/12 students indicated that they “Read and used the learning objectives to assess their learning,” one “Read but didn’t use the learning objectives,” and one did neither.

In the face-to-face course, 2/8 students “Read & used the learning objectives,” 5/8 “Read but didn’t use” them, 1/8 “didn’t read or use” and one student didn’t respond.

These questions suggest that the online students made greater use of the course syllabus & website, including announcements and learning objectives, than the face-to-face class did.  This is not surprising since the latter had class sessions with me three times a week which they could use for the same purpose.

How much time did you put into this course compared to others at UMW?

The face-to-face students overwhelmingly put the same time into this course as their other courses, and rated it the same difficulty as their others. By contrast, half the online students put less time into the course than their other (mostly f2f) courses, one student put more time in and five put in the same time.

More than half (7/12) of the online students said the course was harder than their other courses.  In addition, half the online students said the course was harder to keep up in than their other courses.

Let me repeat that:  Half the online students rated the online course as harder than their other courses, yet they put less time into the course.  Is this a reflection of the lower experience of the online students or was it something else?

Was the instructor accessible for help in this course?  

There was not much difference between the two courses on this question; the majority of both said the instructor was accessible.

On a scale of 1 to 5 (where 1 indicates “very connected” and 5 indicates “very disconnected), compared to your other courses at UMW, how “connected” did you feel in this course?

The mean score for the online students was 3.42 and the median score was 3.5.  For the face-to-face students, the mean score was 2.69 while the median was 2.75.   The online students felt less connected!  That’s not great news, but it’s not unexpected.

The remaining questions in the survey were more open-ended, which means they have the potential for richer information, but they are also harder to process.

What was the most difficult or challenging aspect of taking this course? 

The responses to this question from the online class were very different from the face-to-face class (the latter of which showed no pattern and familiar from past years).

There were five mentions of the need for independent learning. Four students indicated this was a negative, though one said, “Different learning techniques did have to be developed because a lot of an online course work and learning is done solely on the student’s own time.  Some people have difficulty understanding this but it isn’t something that cannot be fixed if it becomes an issue.  Note that was the student who put more effort into the online course than his other courses.

There were two comments along the lines of “not having any lecture materials.  Notes, videos, class meetings, or guides prior to doing each topic.  During each one, I felt very confused and not sure what I was doing each time.“  This is accurate at some level, but baffling nonetheless.  For each topic, the course website provides readings, learning objectives, and for the more analytic topics, video excerpts from my face-to-face lectures.  What I didn’t provide was my lecture notes or any other distillations of my thinking, because I think that students should construct those themselves, that such construction is necessary for deep learning.  Some students seemed to think that learning in this course meant memorizing my thinking and parroting it back to me on exams.  Again, this may reflect the relative inexperience of the online group.

There were three mentions of loss of opportunity to ask questions “right away”.  That is the nature of asynchronous learning, but it doesn’t imply no opportunity to ask questions at all.  I offered online office hours using Skype Sunday through Thursday evenings, but had very few takers.  Students were also aware of my regular, face-to-face office hours.  I was also accessible via twitter and of course email, though only a few students contacted me.  This was also inconsistent with the survey responses about instructor accessibility, so I wonder what concern this question reflects?

There were five mentions of difficulty keeping up with the deadlines, including two mentions of “not knowing when assignments were due” because dates were not on hard wired in the syllabus, since deadlines depended on the class’ progress thru the material.  All assignments were announced either by twitter, email, or text or video announcements on the course website.  Most were announced multiple ways.  I plan to create an Upcoming Deadlines link, prominently displayed on the course homepage, for next year’s class.

There were two mentions of the difficulty of the exams, which was the only thing in common with the face-to-face class.

Which essay was your favorite?  Why? 

The responses to this question revealed no pattern within or between classes.

What is the most important thing that you learned in this course?

The responses to this question revealed no pattern within or between classes.

What suggestions do you have for improving student learning in this course?

Three students asked for more online discussions. That was interesting, because I didn’t get a lot of participation in those and ended up not doing them after the middle of the semester.  I guess I pulled the plug too early.  I’ve given some thought to a better way to organize online discussions which I will try out next time.

Three students asked for more video lecture excerpts, especially early in the course.  I had already planned on doing that.  I have the raw video from a year ago, and I can easily edit that to fit.

Two students suggested synchronous meetings online once a week.  I’m not sure what need that would solve, though I plan on experimenting with Google Hangout as a better platform for online meetings.

Two students asked me to use our institution’s LMS (Canvas) “like their other courses do.”  Alternatively, they asked for a “more organized website.”  I’ll have to think about what that means since.

What advice would you give future students who wanted to know if they should take this course?

Four students mentioned the need for Time Management.  Four also mentioned the need to be self-directed, or else “you should take the course face-to-face.”

The other comments showed no particular pattern.

There was also no correspondence between the comments from the online class and the comments from the face-to-face class.  Two of the face-to-face students urged students to make sure you read the chapter before class time.  Two students suggested paying attention in class.  And two said students should study hard for exams.


This entry was posted in Teaching and Learning, The Experiment. Bookmark the permalink.

4 Responses to Qualitative Analysis of Econ201online

  1. Robert Rycroft says:

    Online courses have never seemed like fun to me. And they don’t seem to work very well. If your students can come to class why would you voluntarily teach online?

  2. sgreenla says:

    That’s kind of a non-sequitur for a couple of reasons. The decision to teach online is like the decision to teach at a particular time of day. If students can come to class at 8:00am, why would I voluntarily teach at 2:00pm? A couple of reasons come to mind. Maybe it’s inconvenient for students to come to class at 8:00am. Maybe they live more than a hour from campus, as three of my students did, and taking an online course allows them to avoid coming to campus two or three days a week. Maybe it’s summer and the students have returned home to New York or Massachusetts or maybe they’re taking a trip abroad and they can’t come to class. In short, the decision to teaching online is due in part to the desire to make students’ course scheduling more flexible. In principle, we could also reach students who want to take a course, but aren’t matriculating at our institution.

    I’m also intrigued by the possibility that learning online could work well for some people. I just need to figure out how to structure my course that way.

  3. Robert Rycroft says:

    I think you missed the point big time. What I take from this is that while Online courses might be a necessary option for students with irregular schedules or who live far away etc., they do not appear to offer a superior or even equivalent academic experience. They are a second best experience. So I wouldn’t want to teach that way if I didn’t have to. (And you didn’t get trashed.)

  4. Wendy Drexler says:

    I commend you for taking the time to experiment in both environments. There are many reasons why the demographic in your online course is different. I think you hit on key differences. In reply to Robert, some of my deepest learning experiences have taken place in well-designed online courses. Rather than question the environment, we should focus our attention on instructional design that makes the most of the opportunities afforded by the technology. In as much as we need to do a better job preparing faculty to teach online, we must consider the implications for learning in this environment. Typically, very little is done to prepare students to learn effectively online. Most are accustomed to passive learning (lecture/discussion/test). Active learning requires skills and strategies that can empower the learner, but require a completely different paradigm of learning. Thanks again for sharing your experience.

Leave a Reply

Your email address will not be published. Required fields are marked *