“The problem with growth scores”

In late 2012 the Evergreen research team worked with iNACOL to explore issues surrounding measuring quality and outcomes of online schools and courses. That project led to the publication of Measuring Quality From Inputs to Outcomes: Creating Student Learning Performance Metrics and Quality Assurance for Online Schools, and accountability continues to be an issue that we are exploring in Keeping Pace. In Keeping Pace 2012, on this subject we said “Many states recognize the limitations of proficiency measures, and are using student growth. Growth measures ideally look at how much a student has learned in a given period of time. A measure of growth is necessary because proficiency measures alone will reward schools whose students arrive above grade level, and penalize schools whose students arrive below grade level. This is of particular concern to online schools because they often are the option of last resort for students who are at-risk, under-credit, or otherwise not successful in a physical school. Student growth calculations, however, are complex and vary in significant ways.”

How complex are growth measures? A good example is Colorado’s growth model, which is generally well regarded and considered an example for other states. A practical and well-written review of the complexity of the model, as well as its benefits and limitations, is available at the problem with growth scores, a blog post from Alexander Ooms, a senior fellow in education with the Donnell-Kay Foundation in Denver (although his blog does not necessarily reflect DKs views).

Ooms lists these key issues:

“The baseline is inadequate to the goals.

Growth percentiles are a norm-referenced measure: every year, the Colorado Department of Education calculates a median growth percentile of 50 for every grade and subject…But a median growth percentile of 50 reflects a trajectory of student outcomes that are wholly inadequate: in 2013 just half of Colorado 10th grade students were proficient, a result not significantly different that it was a decade ago.”

“Growth cannot solve the equation most people ask it to solve. 

Many people believe that growth percentiles can be used to measure both progress and performance: progress to determine how quickly a group (a class, school, or district) is moving towards a fixed goal, and performance to compare this metric to other groups. This is a common assumption, and it is false.”

“Growth slightly above the median does not mean proficiency will increase.”

“Growth is means, not end. 

If a school or district believes that its primary goal is relative — to outperform other schools and districts in Colorado — the growth model works just fine. But recognize that this is placing in a race where roughly 50% of the participants finish short of basic proficiency goals. If schools and districts believe their primary goal is absolute — to prepare their students for college or career success — they need to use proficiency criteria.  In my opinion, a primary goal of schools and districts should be exit-level proficiency. And I believe that, used properly, growth percentiles are a useful metric to help us reach this goal.  But growth percentiles are a means, not an end.”

Ooms closes with a reminder that he believes that “Growth scores are important and worth our attention” – he is simply saying that they need to be used as one tool, and with knowledge of what the tool can, and cannot, do.

My experience in researching growth models a year ago reflected many of the issues that Ooms discusses. My initial knowledge was rudimentary—although I did not realize it at the time. I didn’t know what I didn’t know. As our research progressed and we spoke with many educators, policymakers, and other stakeholders, it became clear to me that much of the discussion about growth and other accountability measures did not reflect the complexity of the issues.

These issues are critical to all schools, because they are being measured based on growth and other metrics. But they are especially significant for online schools and students, because the states that are restricting the expansion of online schools are often citing student test results (using proficiency most often, and sometimes growth scores) as the reason. Including growth scores in school measures is an improvement over using just proficiency, but the benefits and limitations of each state’s growth measure must be taken into account.