How Good is a Free, Open Source Text: Part 2?

This post is where I go off the reservation.  It is the third in a series of posts about the process of creating a free, open source textbook.  I ended the previous post with a question:  Is the book I helped create for OpenStaxCollege the best principles of economics book on the market? I would like to suggest that this is a more complicated question than it seems.

5370119909_6e647e87af

Beauty Contest

If you asked a Ph.D. economist whether Greg Mankiw’s Principles text is better than the OpenStax principles text, most economists would probably say yes.  Mankiw’s book is a good one.  I’m singling it out only because it is the market leader, the prototype for the current crop of commercial text books.  [On the other hand, I know of no assessment evidence that evaluates Mankiw’s book vis a vis other intro texts, but that’s another issue.]

But the real question is not how good Ph.D.s think a textbook is, but rather how much students learn from it.  In a way, this is a variation on John Maynard Keynes’ beauty contest metaphor for the stock market. Keynes observed that the way to make money in the stock market is not to invest in the best companies, but rather to invest in what the majority of other people think are the best companies.

If students neither buy nor read the text because it is expensive, they aren’t learning anything from it.  Thus, a “great” book that isn’t read by students is not better than a good book that students actually read!

So where does that leave us?  I believe that introductory textbooks these days are 3084239579_f1813df8cf_zcommodities.  They may not be identical, but they are close enough substitutes that any experienced faculty member can teach their intro course with pretty much any textbook on the subject.  Publishers don’t want you to believe this, but I think it’s true.  Think about how little time instructors devote to textbook selection.  They certainly act as if the choice doesn’t matter.

Half of undergraduates do not buy the text.  They make this decision for one of two related reasons: either they don’t see the value given the price, or they don’t think they can afford it.

What is the value-added in an intro textbook?  Students often see textbooks as substitutes for class lectures.  If they go to class, why do they need the book?  I don’t agree with this logic, but many students think his way.  To defeat this logic, we need to raise the value/price calculation in students’ eyes.  If you accept that intro texts are commodities, it’s going to be very difficult to raise the value.  The much more promising approach is to lower the price.  And free is a very attractive price.

In my view, while textbooks have value, the real value-added is in the ancilliary products, specifically online problems, quizzes and other activities that allow students to work with the course content until they master it.  This is not just my idea.  The latest issue of Campus Technology has an article subtitled, “Technology is moving the digital textbook from print look-alike to next-generation learning platform.”  Subjects like economics really benefit from students regularly working with the content in thoughtful, interactive, adaptive and low-stakes ways.  At present, the quality of these exercises lags behind the technology with which students can work them, but that’s an issue for another post. Suffice it to say we need to get beyond “drill & kill.”  Before digital, such practice was limited by the instructor’s (or teaching assistants’) ability to grade and return student work quickly (tasks which are ideally suited to computers).   Which is to say, such practice was very limited.  Note that the real benefit in this ancilliary software is self-directed learning along with self-assessment of mastery.  It is not primarily course management and grades.

Think about how commercial textbooks are priced.   Commercial publishers put a high price on the textbooks, but then a relatively low price on the ancilliary products, which have been perceived, after all, as ancilliary.  In my view, this is exactly backwards since the value to students is no longer in the course content, but in the digital learning environment around the content.  This is the reason why OpenStaxCollege has teamed up with commercial aftermarket producers, including Sapling Learning, and WebAssign which have at least some of the features I’m describing, at really modest prices, currently between $30 and $40, including the text. Disclaimer: I have no financial interest in these companies, but I think they offer some really interesting products.   If you genuinely care about your students learning, I urge you to take a look at these products. Faculty at nearly 100 schools to date have adopted OpenStax’ Principles of Economics.  While commercial publishers are also moving in these directions, they can’t give away the content for free, like OpenStaxCollege does.

Photo credits:

  • JanWillemsen via flickr
  • Ben.Gallagher via flickr

 

Posted in Uncategorized | Leave a comment

How Good Can a Free, Open Source Book Really Be?

This post is a continuation of the story of how I helped create a free, open source text book. The first post is here.

When one hears the words “free text book,” it is natural to question the quality of the freeproduct.  I get that; I even wondered myself.  So let me explain the process we went through in developing the OpenStax Principles of Economics book.

The process was unlike any other writing project I’ve done before.  First, it was fast.  My first book took three years to write and then was two more years in production.  Admittedly, this project was more like a revision, but by incorporating a team approach, the project took less than one year from start to finish.  A large part of this was due to the excellent people I worked with, especially freelance development editor, Audrey Regan Solarino, who basically kept me sane during the process.  We started with an initial manuscript—Tim Tayler’s Principles of Economics (2nd Edition).  Using an existing manuscript has both pros and cons.  First, it saved a significant amount of time.  We had a structure and basic core content to work from.   On the other hand, there were many changes necessary to make it a better book, one that fit into the vision of OpenStaxCollege.

The initial manuscript had few real features, so we had to change that.   We framed each medium_Economics_700x906chapter with a case study.  The case was presented at the beginning of each chapter, and the solution, if you will, was explained at the end of each chapter based on the content in the chapter.  This may not have been an original idea, but it made for a better book.

We attempted to make the book a more immersive experience for students by dividing each chapter into modules.  Each module includes interactive features in the body and at the end.   The idea was to embed student active learning rather than relegate it to the end of the chapter, where it can be ignored.  We wanted to make students think about what they were reading as they were reading it.  Taylor’s manuscript included “Clear it Up” features, which explain a concept that students typically misunderstand, in the text.    We added “Work it Out” features, heavily scaffolded examples or problems for the student to work through to understand a complex concept.  Mostly, these were analytical things that require practice to perfect.  Imagine the kind of conversation during office hours you would have to help a student work through a problem.  That’s what the “Work it Outs” were designed to mimic.

We then added several additional types of end-of-module questions. The original text had included standard review questions that required students to look up the answer in the text.  Think definitions.  We augmented these with “Self-Check” and “Critical Thinking” questions and “Numerical Problems”.   Unlike the “Review Questions” which were factual, the “Self-Check” questions were applications of concepts explained in the text, but in different contexts.  The Self-Check questions have “Click-to-Reveal” answers in the electronic formats of the book.  The “Critical Thinking” questions are open-ended questions, requiring out-of-the-box thinking: Taking a concept and applying it to a very different context or asking students to apply it in an extreme case.  The purpose is of the “Critical Thinking” questions is less about the answers and more about getting students to think more deeply about the content.   The “Numerical Problems” were just that:  numerical applications of the concepts in the module.   These features were the result of feedback derived from an initial survey conducted with approximately 40 economics faculty from around the country, which is a great segue to the next point about how the book was developed:  The process was driven by peer review.

Now every publisher says their books are peer-reviewed.  But what exactly does that 3157621454_902378aa2f_mmean?  For my first book, I wrote a couple of sample chapters which I shopped around with my book proposal to publishers.   When I finally found a publisher willing to take a risk on the book, the chapters were sent to three reviewers, I think, who read them and gave the thumbs up to the publisher.  That was the extent of the peer-review.

The OpenStax process was quite different.  Every chapter was peer-reviewed at multiple stages.  As mentioned above, the original Taylor manuscript was sent out to about 40 faculty for a high level review.  For each chapter, reviewers were asked: What did you like?  What did you dislike?  What improvements can you suggest?  The review comments were compiled by chapter and became the basis for my work.

I read each Taylor chapter, considered the review comments and made revisions.  In some cases, I essentially rewrote the chapter.  In others, the revisions were less dramatic, updating data, changing wording in some cases, and of course, adding the new features.  In parallel, Audrey, my development editor, read each chapter, giving it the same attention it would have received from an in-house development editor at a major publishing house.  Once we were each satisfied, the revised chapter went out for review by a half dozen faculty experts from a variety of different types of institutions around the country.  The reviews were collated and then I made revisions based on the consensus of the reviewers and editors (Audrey and others) I was working with.

After the first few months, it became clear that I would not be able to complete the book on the timeline required by OpenStax.  As a result, we brought in several other economists to revise chapters.  I ended up doing about two thirds of the book (generally the introductory chapters and the macro portion), while the other economists did the remaining third.   I reviewed each of their chapters and decided when it was complete, so in some ways my role was like the editor of a volume of published papers.

The last stage in the OpenStax process was “accuracy review” in which each chapter was sent to a separate group of faculty experts to insure that everything written was accurate.  I responded to their comments with one final set of revisions.  Many of the reviewers really took ownership of their task, and the result was a much better book than what I initially drafted.

One of the advantages of a digital text is the ability to revise after publication.  With a traditional print text, revisions must await new “editions,” but something close to continuous revision is built into the OpenStax process.   Users are encouraged to report errors and suggestions for improvement.   Every month or two, I receive the list of suggestions and determine what corrections to make.  I just finished my first iteration of this and I must say that many of the suggestions were quite good, leading to a better, more usable text.

Is this the best principles of economics book on the market?  For my answer to that question, you’ll have to wait for my next post.

Image Credits:

  1. Daiji Hirata via Flickr.com
  2. OpenStaxCollege
  3. Gideon Burton via Flickr.com
Posted in Uncategorized | 1 Comment

Writing Principles of Economics: an open source textbook

Last spring, I received an invitation to become the lead content expert for OpenStax logoOpenStaxCollege’s principles of economics book . OpenStaxCollege is the new name for Connexions, a project I first heard about when founder Rich Baraniuk spoke at the 2005 National Learning Infrastructure Initiative, an organization that became the Educause Learning Initiative. (Thanks again, @GardnerCampbell!) .

When I got the invitation, I was intrigued.   I had been somewhat aware of the concept of open educational resources.   Many people I follow online had either been involved in or had talked about OER.   Steven Ovadia describes OER as freely available course content, including syllabi, assignments, and even textbooks. As he explains:

Open education resources (OER) are an attempt to solve the textbook pricing problem by giving students and faculty great content at more reasonable prices — even free, which many consider to be the most reasonable price point of all.

Which brings us back to OpenStax. The goal of OpenStax is to make higher education more affordable by creating free, open source textbooks for many of the introductory courses undergraduates take.  OpenStax currently has texts in Statistics, Pre-Calculus, Physics, Biology, Sociology, Economics, and soon History, Chemistry and Psychology.   Take a look!

If you haven’t purchased a college textbook lately, you may be surprised to discover how much they cost. New books for my courses can cost more than $150 each, and I understand that at least one popular intro text for principles of economics has breached the $300 level. These costs are pretty typical in higher education today.   If a student takes five courses per semester that works out to $750.

OpenStax has a different business model than traditional publishers. OpenStax obtains grant funding from the Gates Foundation, the Hewlett Foundation, 20 Million Minds, the Kazanjian Foundation, the Arnold Foundation and the Maxwell Foundation. They use the funding to create the books, paying for content experts, editors, programmers, all the production expenses. In contrast to the traditional publishing models, all the expenses are paid up front. What OpenStax doesn’t have is a large sales force, which is a major way they keep their costs down. The books are available free on the web, or as e-books, or as downloadable pdfs. There’s even a phone app for the principles of economics book, which is the way I use it most frequently. OpenStax books are also available in print at a modest price.  The Principles of Economics book is currently priced under $40.

These books are free and customizable. Instructors can add content, delete content, and because the texts follow a modular design, move content around.  For example, next semester when I teach principles of micro, I will be adding a chapter that wasn’t in the original version, but which fits the way I teach the course. My amended version of the book will be available from the OpenStax website in the same formats as the original. Students will get only what they need for a specific course. And since the books are free, they won’t object to not being able to resell them, which is a significant problem for customized books from traditional publishers.

The cost of traditional textbooks may explain why only about half of undergraduates purchase the texts.   And if only half your students are buying the book, how many do you think are reading it?  Don’t you think a free text might help?  If you care about your students learning, isn’t it worth considering OpenStax?

You’re probably thinking “how good can a free book actually be?”  In the next post, I will describe the quality assurance process for OpenStax texts.

 

Posted in Uncategorized | Leave a comment

Thoughts about Student Success

Over the last year, I have fallen prey to a common ailment of bloggers—wanting to get 7510769972_7a212fcfe5_myour ideas just right before they are published.  Putting your ideas out there is, after all, risky.  Especially when those ideas may be half-baked.  Of course, that’s contrary to the way some view blogging.  Gardner Campbell recently wrote “Almost all the time, I write my posts in one sitting and publish them right away.”  Half-baked or not, it’s almost certainly better to post those ideas; otherwise, they may never get published at all.  So here goes!

In recent years, I’ve become interested in student success at a systematic level.  Last Saturday, I participated in our First Year Honors Common Reading Group.  Afterwards the faculty discussion leaders went out to debrief.  We were universally pleased with how well it went, with how easy it was to lead the discussion with students who had read the book and were clearly interested in discussing it.  The hardest part was reining in the dominant speakers (of whom we had three in a group of eight) to provide the space for the other students to talk.

You might retort, well of course the book discussions went well since you had bright students, but I think it was more than that.  Students are successful and earn good grades not merely because they are intelligent, but also because of behaviors, habits of the mind, that they practice.  So it’s not just who they are, but what they do that matters.  The honors students were prepared for the discussion and they also wanted to learn from it.  They were intellectuals in the sense that they wanted to discuss and debate the ideas in the book so they could learn from it.  The prospect of learning seemed to motivate them.

I had two takeaways from this about student success more generally:  First, I think it may be important that students find a sense of belonging, for example, by becoming part of a group with common interests, especially one where some sense of peer pressure motivates them.  On athletic teams, there can be a sense of wanting to do well academically so you don’t let the team down.  Academic majors probably are another example of students working together in parallel towards common goals.  Could this be one reason why students who fail to declare a major until very late find it difficult to complete the degree?

The second takeaway is the following.  I wonder if productive academic behaviors can be taught to students who aren’t honors students, especially in the context of their group.  What are good practices for economics majors?  What are the same for Historic Preservation?  I’m sure there is some overlap across departments, but I suspect there are also differences.

I also think that being part of a cohort (not just part of a group) can be important.  A sophomore economics major probably has less in common than a senior economics major.  Two sophomores are more likely to be taking the same courses.

Groups tend to have leaders or at least faculty responsible for the groups.  This could be a way to establish the “one faculty or staff member who cares about your academic success.”  The director of our Honors Program struck me as similar to the “house mother” in dormitories of old.  Who is offering the same caring and attention to our non-honors FY and older students?

Posted in Uncategorized | Leave a comment

Is teaching more than content delivery?

Good teaching should be.

As I prep for classes this year, I feel meta—I feel like I may be seeing or understanding things more clearly than before.

Student and instructor enthusiasm is high at the beginning of the term.  As an instructor, I provide lots of scaffolding, e.g. motivation for the class, explanation how the course will work, suggestions for how students can be successful.  (I wrote myself a sticky note to remember:

  • NEED to Scaffold, Scaffold, Scaffold
  • NEED to Encourage, Encourage, Encourage
  • NEED to look for the struggling students )

And students make genuine efforts to follow through.

Then at some point (around mid time), the enthusiasm/thoughtfulness slips:  Students and faculty feel tired.  Faculty start to worry about “covering the content” – they focus their energies on “teaching” rather than on student learning.  Students start to worry about grades instead of learning. There doesn’t seem to be any time for thought anymore.  But can there any real learning without thought?  Have instructors forgotten that it’s not what they “cover” but what students learn that matters?  Some instructors seem to have the old school notion that their job is to push all the content out there, and then the students who get it (memorize it?) get rewarded, and those who don’t get it, well not everyone is cut out to be an [ insert discipline ] major.  Hey, not everyone is cut out to go to college.  I disagree.  Our job is to teach all our students, not simply filter out those who can’t learn  the material as fast as we deliver it.

Which brings me to my point:

Can we take a time out during the semester for thought?  Is there a way we can re- generate the scaffolding so that it lasts all term?  I’m looking for practical suggestions here.  I have one thought: to spend a class period after the midterm in my intermediate macro class reviewing some of the things we talked about the first week of the course.  In other words, not just going over the exam, but reminding students what the purpose of the course is?  What students really need to learn from the mass of content we study?  The best way to learn that.   How I am available for help, and how it’s not too late to get a decent grade in the course.

Any thoughts would be much appreciated!

 

Posted in Teaching and Learning | 1 Comment

The Future of Higher Education

2227374205_44520cfe70_nHaving nothing better to do (or nothing that I felt like doing) on a Sunday afternoon, I thought I’d respond to Bryan Alexander’s thought-provoking post on nothing less than the future of higher education.

First a caveat: I am not an expert in higher education finance, but as an economist I do have an opinion–no rejoinder necessary, Bryan ;-)

I don’t think Bryan’s analysis is quite complete, or at least it’s not representative of what I’m seeing.  The economist in me always asks, “What’s the alternative?” If consumers and families decide “that college costs shouldn’t rise any further,” that’s a wish or an emotion, but what does it mean?  What can families do about it?  Let’s assume we’re dealing with the majority of colleges and universities below the very top, where students and families actually have to pay for school.  The natural response is to downsize, either by choosing “cheaper” schools, say a public over a private, or spending the first year or two at community college and then finishing at a four year school.

There’s a lot of money to save with both options.  The average cost per year at a private four year institution is more than $40,000. The corresponding cost at an in-state public is about half that.  (Even the “net cost” at a private is higher than at a public.) So we should expect to see more students choosing the University of Virginia (in my state) over say NYU.  We should also expect to see more students staying in state, choosing UVa over say Penn State.  Community colleges are cheaper even than public four year schools.  When you include the fact that most community college students live at home so they don’t pay room and board, community colleges become significantly cheaper.  So Bryan’s first scenario may be normal looking for higher education as a whole, but it’s not normal (even new normal) for individual schools, who may be facing extraordinary stresses.

Bryan’s second scenario feels right to me, at least the symptoms seem familiar.  But what’s not right is it seems too surface level to me.  To make sense out of what’s going on, one has to start with a theory of what’s been driving the costs of higher ed.  The theory that makes the most sense to me is Baumol’s cost disease.  The best explanation of this is Archibald & Feldman’s Why Does College Cost So Much?

If Archibald and Feldman are right about what’s driving costs, many of the surface symptoms are only minor embellishments on the cost train.  The question again is what can consumers do about it and what can schools do?  Consumers may be “outraged a[t] rising prices,” but their options here are the same as in scenario 1 above.  “State and national politicians of all stripes [may] join their chorus.  They [may] demand more for less.”  But what does that mean operationally?   If schools do not have the endowment growth to cover the increasing costs and/or the reduction in state support, then (whatever people think) prices will have to continue to rise and the scenario 1 options will prevail.  I see this as the likely short run outcome.  But this is not a stable outcome.   As consumers downsize to other schools, the schools they would have attended will run into problems on the revenue side.  See, for example, St. Mary’s College of Maryland and Loyola of New Orleans.  For now, those schools are holding their programs together with the fiscal equivalent of duct tape, but if the trend continues, at some point a tipping point will be reached resulting in catastrophic failures.

The only solution that I see, long term, is to change the rules of the game.  MOOCs are an example of an attempt to do this, but I don’t see that MOOCs as they currently exist are the answer.  For a school to be viable, it’s going to need to clearly identify its value-added and focus its efforts in that area.  Schools that continue to offer programs that are only adequate are going to face increased revenue shorfalls.  Our school is in the early stages of attempting to play a different game by looking for new, innovative programs to invest in, and by making difficult choices about long time programs to reduce or close.  We’ll see how that works out.  My guess is that in the coming years we will see an increase in diversity of what a college education looks like, as schools experiment to find a strategy that works for their context.  Many of these experiments will fail, but hopefully some will succeed.  I haven’t thought enough about this yet, but I could imagine faculty whose disciplinary departments in small liberal arts colleges are being downsized moving into interdisciplinary departments to teach common courses in a new university curriculum.  A very different but promising example of a sustainable strategy could be Mike Caulfield’s idea for residential online education.  I would hate to see the alternative of a two tiered system of higher ed:  Residential education for the few and something like career-oriented online education for the many.

One thing is clear: the higher ed system has not reached an equilibrium yet.

Posted in Uncategorized | 4 Comments

Quantitative Analysis of ECON201Online

You may recall from my earlier posts that I was concerned that students were learning substantially less in my online course than the parallel face-to-face course.  The difference in final exam scores was one full letter grade!  My hypothesis was that this difference was due to the fact that the online course enrolled mostly first years and sophomores, while the face-to-face course was mostly juniors and seniors. To test this hypothesis, I conducted a regression analysis using a fairly standard “educational production” function.  The function assumes that learning is produced by combining inputs of  student ability, student effort, the quality of the learning environment, plus some standard demographic controls.

I collected the data fairly early in the spring semester, but my busy life being what it is I didn’t get around to running the regression until I found a reason for it:  I teach regression analysis in my introductory research methodology class, and I find the students are always more engaged when I illustrate with “real” research results than made-up ones.  So when it was time to talk regression in the research class, I did the analysis and presented the results to my research class.

The sample size was 35: 21 from my face-to-face course and 14 online.  Yes, the sample size is small, but both courses are writing-intensive, so that gives me the “luxury” of teaching small classes.  The bottom line is that it is what it is.   I did a couple different versions of the model with raw (uncurved) final exam scores as the dependent variable, and the following as explanatory variables:

  • Student GPA (less their grade in my course) — this was to capture some measure of how bright each student was, and to a certain extent, how hard they work.
  • Credit Hours Earned — this was to control for how much experience each student had in university coursework.
  • Whether the Section was Online or Face-to-Face (i.e. the treatment variable)
  • Whether or not the course was required for the student’s major
  • Gender — the literature says that women do less well in economics than men.

The results were consistent across the different model specifications: The medium of the course had no statistically significant influence on the final exam score.  Instead, the most important determinant was Student GPA, followed by Credit Hours Earned.  (Neither Required for the Major nor Gender mattered either.)

In short, I may have done a less than optimal job of teaching the online course my first time, but it didn’t seem to have a significant impact on student learning (subject to all the normal caveats of regression analysis).

I have fiddled with the registration permissions so that when I teach the course again this fall, I will have a more equal mix of lower level and upper level students in each class.  Plus I’ll have a year of experience under my belt.  We’ll see how it goes.  Stay tuned!

Posted in Uncategorized | Leave a comment

Why take #DS106?

6635587829_c89c7936bbI’ve decided to embark on the roadtrip called DS106.  This week as I’ve pondered my discussion, I joined the twitter conversation about this iteration of the course.  To get my feet wet, I tweeted:

@sgreenla: Wondering whether I have the creative chops for #ds106.  Guess that’s part of why I’ve put off taking it.

This casual tweet set off a series of responses (slightly edited):

Christina Hendricks@clhendricksbc  Me too, so need practice.

Alan Levine@cogdog 

@clhendricksbc @sgreenla This intrigues me- the barrier for entry to #ds106 could not be lower. No requirements, no fixed schedule, no grade

Christina Hendricks ‏@clhendricksbc

@cogdog Agreed! But there are so many creative ppl doing amazing things; I feel a bit intimidated. That’s probably stupid, though @sgreenla

Alan Levine ‏@cogdog

@clhendricksbc @sgreenla It makes me wonder about how we mature to beings that self-talk ourselves out of trying (I’m guilty of it myself)

Alan Levine ‏@cogdog

@clhendricksbc @sgreenla It’s not stupid, its human. Only compare yourself to yourself. It’s like taking up running at Olympian pace.

Christina Hendricks ‏@clhendricksbc

@cogdog @sgreenla Yeah, the social comparison thing is hard, esp. re: creative stuff. Decided to ditch it and just have fun. Seems like such a great group of ppl doing ds106 that I want to talk to/connect with too. Bonus!

 

So this got me thinking.   It’s the social pressure that’s the barrier.  As Christina says, this doesn’t feel like an entry level course where learners are expected to know little about the subject coming in.  There are so many creative people doing amazing things.  I barely know what some of the things are (e.g. animated gifs) much less know how to create them.  What if I embarrass myself?  What if I can’t live up to the standard set by the rest of the class?

I’ve thought about “taking” DS106 for a longtime.  I’m not sure that’s the right verb, though.  My sense is you can’t “take” DS106 in the same way that I’ve taken the three MOOCs I’ve attempted.  Rather, DS106 seems to demand a level of engagement or commitment that goes far beyond what a normal class requires.  It feels like a cult, almost.  This is not a bad thing—I’d love to be able to create course environments in my classes that would make students feel like this.  But it is a hurdle for bringing new people in.

The thing is, my “ought to take DS106 desire” wasn’t enough to make me take the plunge.  There are a lot of things that professionally I “ought” to do, but I don’t have the time or emotional energy to do them all.  In fact, things I put in that category are generally things I don’t do.

So what’s different this time?   I have a real reason for taking DS106.  Last year, I taught my first fully online course.  It was a great experience and, at the same time, it was a humbling experience.  I failed my students in some important ways.  The course was not the best learning experience it could be.  Teaching online is a different animal than teaching face-to-face, and though I thought I understood that, in retrospect deep down I still thought I could hit it out of the park the first time.  I couldn’t.   I’ve spent six months, since the end of that course thinking about how to improve it, thinking about what I did wrong the last time.  I am anxious do to it again.

 

I’ve gotten a lot of help along the way.   Wendy Drexler (@wendydrexler) made an extraordinary effort to meet with me to give me just the right kind of pep talk.  She had been following my blog posts about my online course, and decided correctly that I needed an intervention. Her effort involved an hour long drive, a car breakdown, which made the trip *much* longer, all to help someone she’d never met in person before.

I also participated in Dan Ariely’s A Beginner’s Guide to Irrational Behavior MOOC.  I haven’t completed it yet, but I did achieve my goal which was to get a richer sense of online learning, something I can use to improve my own course.

In short, I want to take DS106 because I think it will help me make my online course a better learning experience.  So I’m not taking DS106 because it’s a “requirement” in some sense (“I ought to take this for professional reasons.”)  Rather, I’m taking it because I want to, because I see a genuine need this course can help to address.  Perhaps that’s the difference between a novice view of learning and a professional, mature view.

Maybe the best help came from Christina Hendricks (@clhendricksbc) who showed me that I’m not alone in my inexperience and anxieties.  So Christina, if you’re willing to do this with me, I’m willing to do it with you.

Posted in Uncategorized | Tagged | 5 Comments

Qualitative Analysis of Econ201online

Last semester (as readers of this blog know), I taught my first fully online course, Principles of Macroeconomics.  I also taught a second section in my traditional face-to-face mode.  As part of the assessment of my online course, I asked students to complete a course evaluation, which they turned in to my department chair who gave me the results after the semester was over.  As a control, I gave the same assessment (except for a couple of online-specific questions) to my face-to-face course.  The response rate from the online class was quite good:  12/15 students.  The response rate from the face-to-face course was less impressive; only 8 of 21 students turned it in.  I’m aware that the latter may not be representative of the course experience, but I’m going to treat it as such anyway.

The mean grade on the final exam was one letter grade lower for the online class than for the face-to-face class.  In previous posts, I’ve alluded to why I think that might be the case, but it’s not yet clear that the cause was the modality of the course.  Let’s look at the survey results:

What year in school are you?

The two course sections didn’t have the same demographics.  There were more lower-level students in the online course, due probably to the fact that my department’s courses fill up quickly during registration. The modal student in the online course was a first semester freshman (5/12 students).  There were also 4/12 sophomores.  By contrast, the students in the face-to-face course were more uniformly distributed with two first years, two sophomores, three juniors (the mode) and one senior.  The less college experience of the online students could be part of the reason for the difference in exam scores.

Where are you living?

More students in the online course lived at a distance from campus, including three who live more than an hour away.  Still, more than 50% (7/12) of the online students lived on campus.  This is pretty common for liberal arts schools with online programs.

Did you read the syllabus?

Nearly everyone in the online course (10/12) read “All of the syllabus.” One student indicated they had read “Some of the syllabus” and one student had no response.

By contrast, the median student in the face-to-face course read “Some” of the syllabus.

Did you watch the Course Videos?

These four questions were exclusive to the online section.  The online course included two types of videos:  introductory videos to each section of the course (including three introductory videos to the course itself) and video excerpts from my face-to-face lectures.  These were limited to the last couple of topics which are highly analytical.

Three quarters of the online students said they watched “all” the videos.  The remainder indicated that they watched “some” of the videos.  Two thirds of the students said they watched the video lectures, and one third said they did not.  All the students indicated that when they watched a video, they watched it in its entirety.  Similarly, two thirds said they watched videos more than once.  It sounds like the students thought the videos were useful.  Next year, I plan to add more videos (for the analytic topics early in the course).

Did you read the blog posts on the website?   

In the online course, 7/12 students read “all” the posts, three read “some” posts, one read “few or none,” and one did not respond.

In the f2f course, half the respondents read “some of the posts,” half read “few or none”.  No one read all.

(There may have been confusion in the online class about what a post was, e.g. posts on the course website vs. posts on the discussion board.)

Did you read and use the Learning Objectives?

In the online course, 10/12 students indicated that they “Read and used the learning objectives to assess their learning,” one “Read but didn’t use the learning objectives,” and one did neither.

In the face-to-face course, 2/8 students “Read & used the learning objectives,” 5/8 “Read but didn’t use” them, 1/8 “didn’t read or use” and one student didn’t respond.

These questions suggest that the online students made greater use of the course syllabus & website, including announcements and learning objectives, than the face-to-face class did.  This is not surprising since the latter had class sessions with me three times a week which they could use for the same purpose.

How much time did you put into this course compared to others at UMW?

The face-to-face students overwhelmingly put the same time into this course as their other courses, and rated it the same difficulty as their others. By contrast, half the online students put less time into the course than their other (mostly f2f) courses, one student put more time in and five put in the same time.

More than half (7/12) of the online students said the course was harder than their other courses.  In addition, half the online students said the course was harder to keep up in than their other courses.

Let me repeat that:  Half the online students rated the online course as harder than their other courses, yet they put less time into the course.  Is this a reflection of the lower experience of the online students or was it something else?

Was the instructor accessible for help in this course?  

There was not much difference between the two courses on this question; the majority of both said the instructor was accessible.

On a scale of 1 to 5 (where 1 indicates “very connected” and 5 indicates “very disconnected), compared to your other courses at UMW, how “connected” did you feel in this course?

The mean score for the online students was 3.42 and the median score was 3.5.  For the face-to-face students, the mean score was 2.69 while the median was 2.75.   The online students felt less connected!  That’s not great news, but it’s not unexpected.

The remaining questions in the survey were more open-ended, which means they have the potential for richer information, but they are also harder to process.

What was the most difficult or challenging aspect of taking this course? 

The responses to this question from the online class were very different from the face-to-face class (the latter of which showed no pattern and familiar from past years).

There were five mentions of the need for independent learning. Four students indicated this was a negative, though one said, “Different learning techniques did have to be developed because a lot of an online course work and learning is done solely on the student’s own time.  Some people have difficulty understanding this but it isn’t something that cannot be fixed if it becomes an issue.  Note that was the student who put more effort into the online course than his other courses.

There were two comments along the lines of “not having any lecture materials.  Notes, videos, class meetings, or guides prior to doing each topic.  During each one, I felt very confused and not sure what I was doing each time.“  This is accurate at some level, but baffling nonetheless.  For each topic, the course website provides readings, learning objectives, and for the more analytic topics, video excerpts from my face-to-face lectures.  What I didn’t provide was my lecture notes or any other distillations of my thinking, because I think that students should construct those themselves, that such construction is necessary for deep learning.  Some students seemed to think that learning in this course meant memorizing my thinking and parroting it back to me on exams.  Again, this may reflect the relative inexperience of the online group.

There were three mentions of loss of opportunity to ask questions “right away”.  That is the nature of asynchronous learning, but it doesn’t imply no opportunity to ask questions at all.  I offered online office hours using Skype Sunday through Thursday evenings, but had very few takers.  Students were also aware of my regular, face-to-face office hours.  I was also accessible via twitter and of course email, though only a few students contacted me.  This was also inconsistent with the survey responses about instructor accessibility, so I wonder what concern this question reflects?

There were five mentions of difficulty keeping up with the deadlines, including two mentions of “not knowing when assignments were due” because dates were not on hard wired in the syllabus, since deadlines depended on the class’ progress thru the material.  All assignments were announced either by twitter, email, or text or video announcements on the course website.  Most were announced multiple ways.  I plan to create an Upcoming Deadlines link, prominently displayed on the course homepage, for next year’s class.

There were two mentions of the difficulty of the exams, which was the only thing in common with the face-to-face class.

Which essay was your favorite?  Why? 

The responses to this question revealed no pattern within or between classes.

What is the most important thing that you learned in this course?

The responses to this question revealed no pattern within or between classes.

What suggestions do you have for improving student learning in this course?

Three students asked for more online discussions. That was interesting, because I didn’t get a lot of participation in those and ended up not doing them after the middle of the semester.  I guess I pulled the plug too early.  I’ve given some thought to a better way to organize online discussions which I will try out next time.

Three students asked for more video lecture excerpts, especially early in the course.  I had already planned on doing that.  I have the raw video from a year ago, and I can easily edit that to fit.

Two students suggested synchronous meetings online once a week.  I’m not sure what need that would solve, though I plan on experimenting with Google Hangout as a better platform for online meetings.

Two students asked me to use our institution’s LMS (Canvas) “like their other courses do.”  Alternatively, they asked for a “more organized website.”  I’ll have to think about what that means since.

What advice would you give future students who wanted to know if they should take this course?

Four students mentioned the need for Time Management.  Four also mentioned the need to be self-directed, or else “you should take the course face-to-face.”

The other comments showed no particular pattern.

There was also no correspondence between the comments from the online class and the comments from the face-to-face class.  Two of the face-to-face students urged students to make sure you read the chapter before class time.  Two students suggested paying attention in class.  And two said students should study hard for exams.

 

Posted in Teaching and Learning, The Experiment | 4 Comments

Online Teaching & Learning: It’s harder than it looks

It has been said that no plan survives first contact and that has certainly been my experience teaching online this semester.  I thought I was well prepared to teach this course, as well prepared as anyone can be who has never taught a fully online course before.  I have explored teaching with different digital technologies in my otherwise face-to-face courses since about 1992.  I have used groupware, discussion boards, wikis, blogs, and other tools.  Admittedly, I am writing this during final exam week, so I may not have enough distance yet for an objective look, but looking back it appears that many/most of the components of my course that were intended to differentiate it and make it a genuinely liberal arts & sciences experience were ultimately jettisoned in an effort to save time and cover the content.

Let’s start with my plan to use twitter to create social context for the course.  I figured this would be easier than my earlier experiments with twitter since Millenials have now “discovered” twitter, but that didn’t seem to be the case.  Early on it seemed clear that my students didn’t really want to use twitter.  Some even told me so.

Most students tweeted using the class hashtag no more than five times over the course of the semester.   Only one student really used twitter daily while a few more did weekly.   That doesn’t seem like much of a social presence to me.   After a couple of weeks, I became worried that my students were not getting the basic communications about the course.   This inference could have been unwarranted, since posting tweets is neither necessary nor sufficient to guarantee following twitter.  In any case, I began to follow important tweets (after a couple of hours) with duplicate email messages. Then I stopped the tweets pretty much altogether.

I had great hopes for using an online discussion board to take the place of class sessions.  It didn’t work out well in practice.  I never got the majority of the students to participate in the online discussions we had.  Most students seemed unable to do the type of critical thinking necessary for using the schema to categorize content in each topic—perhaps the approach was too abstract for them.   I never got a majority of students to participate, even when I changed the task to asking them to “summarize” a reading for the course.  They appeared to see the discussion board as something extra, not something central to the course.  This could have been because, though I gave students credit for participating, I gave no explicit grades on the participation.  I made note of the quantity and quality of student posts, but I didn’t provide that to students.  This is something I could improve on in the future.  It’s also possible I needed to spend (more) time training the students in what I wanted them to do with the discussion board.  The students seemed to want me to “give them” what they needed to know, rather than asking them to dig it out for themselves.

Finally, the macromooc was a fail.  I think the concept had legs, but the timing didn’t work out.  By the time we had covered the course content needed to evaluate the presidential candidates’ economic policies, the election was upon us so we had no time to do what I had planned.  I think I will try something more limited next year.

What was left to distinguish the course was the writing assignments which worked fairly well at getting the students to explore the content in more detail than they would in a lecture-course.  Some students, though, seemed to see them as independent of the course content.  Several asked, “Will what we did on the essays be on the midterm?”

The online class ran consistently about a week behind the face-to-face class.  This lag was due to the first week in which we spent team-building and getting used to the online features.  Note that I tried to do this before the “first” day of class but didn’t get enough participation.  It was fortunate that the final exam for the online course was the last day of the exam period.

Students didn’t do a good enough job of “following” the course.  A number missed Essay 5, which was given around the same time as Essay 4.  Essay 5 was announced via an all-class email.  One student emailed me, “How did you notify us that it was due? I was talking to some classmates and none of us had any clue that there was an essay due.”  Eight students had already turned the assignment in at that point.

I knew coming in to this course that there were certain analytical topics that students have traditionally had difficulty learning on their own.  I planned to video my in-class lectures on those topics and edit the videos down to bite-sized chunks for the online course.  The first of these topics, Supply & Demand, occurred early in the term.  I found it difficult to come up with the video equipment by the time the lectures started, so I didn’t video this topic.  The online students did noticeably worse than the f2f students on the first exam questions involving S&D.  They also did very poorly on the second exam, which prompted me to find a friend from whom to borrow the video equipment.  From that point on, I lugged the camera, tripod and power cords to and from class each session.  I had to arrive early to set everything up, and depart later to break everything down.  This was tedious.  I learned that video editing can take *hours and hours*, though I did get better at it by the end of the semester.  I did find it very interesting to see myself lecture.  I also got a good sense of how my interactive lecture style is very different (less efficient, time-wise perhaps) than a direct content delivery lecture would be.  In some cases, the pared down videos were only 10 minutes in length for a 50 minute lecture.  I did make sure that none of the clips were longer than 10 minutes and most were closer to 5 minutes.  They are all up on YouTube if you’d like to look at them.  A number of students told me they used the videos to study for the final and that they were very useful, but I think by that point it may have been too late.  I need to think more about how to build the interactive elements of my face-to-face course (e.g. small group problems to work out and present) into the online version.

In retrospect, the students never seemed to buy in to my vision for the course, but neither did they seem to have the self-discipline and metacognition to do the work, and thus the learning, on their own.  Perhaps undergraduates (first and second years and upper level students with other majors) do not have what it takes to be successful in an online course.  Or perhaps the failure was due to my inexperience as an online teacher.

In sum, I found this online course a tremendous amount of work and more than once asked myself why I was killing myself to do this when teaching face-to-face is so much easier and fun.  Nonetheless, I think I should teach online at least once more, since there clearly is a learning curve.

I have two more tasks to do to complete the evaluation of this course.  I asked students in both my face-to-face and online sections to do a course evaluation that I designed.  I need to review and reflect on those evaluations.   I also plan to do a careful statistical analysis of exam grades in the two sections, controlling for student ability, credit hours completed and a couple of other standard variables.  To get that data I need to talk to the Registrar, so both of these tasks will have to wait until I return to school next week.

 

Photo Credit: http://www.flickr.com/photos/ladybugblue/295252840/sizes/m/ “Harder than it looks”

Posted in Teaching and Learning, The Experiment | 2 Comments