Part 2: We must define and measure online learning

Part 1 of Andy Frost's post on whether online learning is disrupting education can be found here Definition and measurement are critical (and they’re HARD)

As a first-time sponsor of the Keeping Pace report, I was surprised by how much of our time we spent defining things. What’s blended learning? What’s online learning? What qualifies as a virtual school? These deceptively simple questions took hours and hours of conversation over several months to answer in a way that met the needs of the report. And once consensus definitions were created, the critical task was to measure, in a meaningful way, growth and activity in each of these areas.

Two things made the measurement problem significantly challenging: moving from the state to the district and the growth of blended learning.

Question 1: How do you measure district activity?

When the Keeping Pace report started eight years ago, one of its primary focuses was to report on activity at the state level.  There are 50 states – a finite and known universe.  Doing a full census at the state level (while anything but easy and still hugely valuable) was a completely tractable problem.  You need to get 50 people to return your phone call.  That’s a huge task (especially finding the RIGHT 50 people), but it’s a doable one.  You can interview the same 50 people the same way with the same set of questions and get a very clear picture of the status of online learning at the state level. Which states are operating statewide virtual schools? What are the state policies related to online learning? How many full-time virtual schools (multi-district, single district, charters, etc) exist in the state? These are important questions to answer and Evergreen has gotten very good at answering them with the Keeping Pace report.

Districts, however, are increasingly creating their own online learning programs for their students.  The universe to survey has now gone from 50 states to well over 15,000 districts and intermediate service units (BOCES, County Offices, and other similar organizations). You can’t fully quantify the universe of districts with phone calls. Even surveys like the ones done by Simba have had a hard time providing a definitive number on things like the percentage of high school courses delivered online. The Keeping Pace team made some impressive and important strides with the 2010 Report. Still, the techniques for measuring and quantifying what’s happening in every  school district in the country still needs to be developed further in order to fully understand the impact of online learning in K-12 education.

Question 2: How do you define blended learning?

The other key to measuring the accuracy of Christensen’s 2019 prediction is to define blended learning.  There are two reasons that this is critical: 1) the prediction itself includes blended learning (Christensen, p. 115-116n13?) and 2) while growth in online learning remains explosive, most experts (including the authors) agree that the bulk of online learning will, in fact, be blended learning. The importance of blended learning is underscored by the recent release of a whitepaper by Michal Horn (a Disrupting Class co-author), called The Rise of K-12 Blended Learning. In this paper, Horn defines blended learning as, “any time a student learns at least in part at a supervised brick-and-mortar location away from home and at least in part through online delivery with some element of student control over time, place, path, and/or pace.” There are many different types of programs that fit this definition, and Horn outlines six models or categories that fit the definition. Whether it’s this definition, or the definition referenced in Disrupting Class (“30 to 80 percent of instruction is delivered on the Internet,”), defining blended learning is a prerequisite for measuring the district activity that everyone agrees is exploding around the country.


So, the answer to the question “Is online learning disrupting education” is “maybe, but it depends on how online learning is defined and measured.  And we won’t know for sure for another eight years.”  In the final analysis, though, the question (and even its answer) is probably less important than the lessons we learn from figuring them out.  And at least two of the key lessons are: 1) disruptive deployment (or competing against non-consumption) is an important method for implementing change via online learning and 2) understanding and implementing blended learning will be one of the most important challenges for educational leaders in the near future.