A week ago I mentioned that two studies had recently come out that were worth a review. The second of these is a study by RAND Education looking at Effectiveness of Cognitive Tutor Algebra I at Scale. The RAND study has received some attention (see the Christensen Institute for example) because it shows positive results from a study of more than 25,000 students using the “technology-based algebra curriculum” across 147 middle and high schools in 52 school districts in seven states. Cognitive Tutor is published by Carnegie Learning, Inc. The main conclusion has been reported fairly widely:

“This large-scale effectiveness trial of Cognitive Tutor Algebra I finds a significant positive effect in high schools in the second year of implementation, relative to similar schools that continued to implement a variety of existing textbook-based algebra curricula. The effect size of approximately 0.20 is educationally meaningful – equivalent to moving an algebra I student from the 50th to the 58th percentile.”

There is also one very important element of the finding of positive results: it didn’t show up until the second year of implementation. This findings mirrors our experience in working with schools that are implementing blended learning. When the goal is to increase student achievement in existing courses like algebra, it is reasonable to expect that the improvement will require two or more years to be seen.

The report’s  difficulties in doing the study are worth reviewing as well. There are quite a few caveats, such as “It is necessary to be cautious in interpreting these results because mean student pretest scores were lower in the treatment group than the control group.” At least one of these issues “raises concern about the validity of the middle school experiment.” And the challenges in identifying schools willing to take part in the study, and other issues, show just how difficult this type of research is.

My takeaway: This is an important study,  as educators and policymakers consistently ask “do we have evidence that blended learning works?” The study provides a data point showing that it can work.

But, the caveats are non-trivial, and the time and expense required of this study show that it is simply not feasible to conduct this level of effort with all blended learning models, instructional approaches, or specific courseware or providers.