Wallethub hits the “liquify” button on their K-12 credibility again in 2020

Highly respectable judges of school quality like the National Assessment of Educational Progress (NAEP) and Stanford University’s Education Opportunity Project give Arizona schools high marks. The online finance website however ranks Arizona 49^th out of 50states.

Measuring the quality of a K-12 system is a complex endeavor, but not nearly as much as the marketing firm Wallethub makes it. For a succinct explanation of what Wallethub does and the basic problem with their K-12 rankings, I cannot improve upon that provided by Robert Robb, editorial columnist for the Arizona Republic regarding a previous Wallethub state K-12 ranking:

WalletHub is a personal finance website that makes its money from advertising and premium listings on the site. It has come up with a brilliant marketing strategy. It collects data from other sources, tosses it into a blender, and spits out a list of the best and worst states and cities about various things…WalletHub, however, isn’t a serious social science research outfit, as this report amply demonstrates.

If you peer into the blender of Wallethub’s 2020 K-12 ranking methods, you once again see a goulash of weird ingredients that don’t really belong together.

Imagine looking into a blender to see grapes, ice cream, vegetable stew and Purina monkey chow with a cup of chocolate syrup and you might get in the neighborhood. Specifically, the Wallethub rankings include a mixture of measures that are inputs rather than outputs, others that are projections about the future, and others that are merely regulatory in nature and have nothing to do with school quality.

Direct measures of academic achievement are few and far between, and predictably the bartender has thrown the wrong ones into the mix. Most analysts agree that the best measure of the quality of a school system is not the overall levels of academic achievement, but rather academic growth over time.

Academic achievement is important, but it is also highly correlated with socio-economic characteristics. We know for instance that upper quintile families spend approximately five times as much as bottom quintile families on student enrichment activities. If the higher income kids show higher levels of math achievement than low income students, is it because they attend higher quality schools, or is it because they go to Kumon? We cannot be entirely sure.

Academic growth on the other hand is not nearly as correlated with socio-economic status. Measuring academic improvement over time tells us where students start and how much they learn over time. Kids can have different starting and ending points but a measure of academic growth can tell us how much they learned between those two points in time.

Stanford University’s Educational Opportunity Project linked state academic exams across the country to give both a comparable growth and achievement score for Grades 3-8 in the country. The below chart shows academic growth (the dotted east-west line) by academic achievement (north-south line) for public schools in Arizona. Each dot represents an Arizona public school. Note that the dotted east-west line represents the national average for academic growth:

The first thing you should notice here is that far more schools are above the dotted line than below it. That is because Arizona schools lead the nation in academic growth during this period.

Arizona not only has many above-average-growth schools, it also has some of the highest growth schools in the country. The highest performing schools rank among the highest performing schools in the nation whether you use growth, proficiency or both.

The National Assessment of Educational Progress (NAEP) also shows Arizona students made stronger than average academic progress across a variety of 4^th and 8^th grade exams since 2009.

Arizona schools, like schools in all other states, must continue to improve. Our schools, however, are a strength, not a weakness. Arizona families can access world class public schools through both charter school and open enrollment mechanisms. Arizona not only has fantastic schools, families do not have to buy their way into their attendance boundaries in order to access them.