A shift toward efficacy

I recently had time to peruse Pearson’s report The Incomplete Guide to Delivering Learning Outcomes, which “outlines Pearson’s own efficacy programme and shares the company’s strategy and initiatives in its first phase.” Although the report was published in 2013 and has received some attention, the depth at which Pearson explores efficacy and examines how the concept applies to its products is notable both for the company, and for its applications to education and policy more broadly. For example, the title of the introduction—From Inputs to Outcomes—is almost exactly the same as the title of a report that we did with iNACOL a couple of years ago (Measuring Quality From Inputs to Outcomes), suggesting that these are issues that apply to schools, policymakers, and many stakeholders in addition to companies.

The report has depth and makes many points that are worth consideration, starting with its definition of efficacy in the education context as requiring “a measurable impact on improving people’s lives through learning.” As the dictionary definition of efficacy is “the power to produce an effect,” Pearson’s addition (“improving people’s lives”) is not a trivial distinction. The company is going further than the usual definition and saying the effect must be a measurable improvement on a learner’s life. The fact that this definition of efficacy requires a measurement of an outcome, and that the outcome must be related to the learner’s life and not simply be any grade or test score, is critical. Some test scores, such as SATs, are likely to have an impact on the student’s life, while others, such as a state assessment, probably will not.

The report goes on to say that Pearson is beginning to evaluate all of its products and potential acquisitions with efficacy as a chief measure. If the largest education company in the world does this successfully, it may have an effect throughout many of the areas in education (e.g. content, technology systems) that are provided by vendors. And if that happens, perhaps the same standard could be applied in other sectors as well. For example, a key tenet of charter school proponents has been that successful charter schools should grow, and unsuccessful ones should be shuttered. But it has proven difficult to close unsuccessful charter schools, so a key component of the promise of charters has not been realized.

It may be easy to overlook just how profound an impact a true focus on efficacy across educational organizations could be. In a previous career I spent many years as an environmental consultant, working with several national non-profit organizations based in Washington, DC. One was a well-funded think tank that published influential reports on a variety of topics, convened meetings and working groups with policymakers and companies, and generally worked within the system to create changes to improve environmental performance. After I had worked with this organization for a couple of years, the board and executive management decreed that all future projects would be evaluated based on the change that they would produce in the world. The organization explicitly defined change as a measurable outcome, and stated that a report or a meeting was not a “change in the world.” The result over time was a more effective organization focused on real-world outcomes. Reports were still published, and meetings convened, but in all cases a clear connection to a further outcome was required.

Based on that experience years ago, and others since, I appreciate the move that Pearson is making to focus on efficacy. I also believe that every school that is moving to digital learning should be asking a similar set of questions: what change is this shift going to produce, how will we measure it, and how will we know if our desired outcome has been achieved? If a school can’t answer those questions, it likely should delay its expansion of digital learning until it can.