You may recall from my earlier posts that I was concerned that students were learning substantially less in my online course than the parallel face-to-face course. The difference in final exam scores was one full letter grade! My hypothesis was that this difference was due to the fact that the online course enrolled mostly first years and sophomores, while the face-to-face course was mostly juniors and seniors. To test this hypothesis, I conducted a regression analysis using a fairly standard “educational production” function. The function assumes that learning is produced by combining inputs of student ability, student effort, the quality of the learning environment, plus some standard demographic controls.
I collected the data fairly early in the spring semester, but my busy life being what it is I didn’t get around to running the regression until I found a reason for it: I teach regression analysis in my introductory research methodology class, and I find the students are always more engaged when I illustrate with “real” research results than made-up ones. So when it was time to talk regression in the research class, I did the analysis and presented the results to my research class.
The sample size was 35: 21 from my face-to-face course and 14 online. Yes, the sample size is small, but both courses are writing-intensive, so that gives me the “luxury” of teaching small classes. The bottom line is that it is what it is. I did a couple different versions of the model with raw (uncurved) final exam scores as the dependent variable, and the following as explanatory variables:
- Student GPA (less their grade in my course) — this was to capture some measure of how bright each student was, and to a certain extent, how hard they work.
- Credit Hours Earned — this was to control for how much experience each student had in university coursework.
- Whether the Section was Online or Face-to-Face (i.e. the treatment variable)
- Whether or not the course was required for the student’s major
- Gender — the literature says that women do less well in economics than men.
The results were consistent across the different model specifications: The medium of the course had no statistically significant influence on the final exam score. Instead, the most important determinant was Student GPA, followed by Credit Hours Earned. (Neither Required for the Major nor Gender mattered either.)
In short, I may have done a less than optimal job of teaching the online course my first time, but it didn’t seem to have a significant impact on student learning (subject to all the normal caveats of regression analysis).
I have fiddled with the registration permissions so that when I teach the course again this fall, I will have a more equal mix of lower level and upper level students in each class. Plus I’ll have a year of experience under my belt. We’ll see how it goes. Stay tuned!