Over 15,000 educators subscribe to my free weekly newsletter. Click here to sign up.

Are We Measuring the Wrong Things?

By Dave Stuart Jr.

It is entirely possible that your school or state or country is making dangerous assumptions about what should be measured (and therefore improved) and what shouldn't.

Kirabo Jackson is an economist at Northwestern University. He used a database of North Carolina students — 464,502 students, according to Paul Tough's Helping Children Succeed — to examine the long-term impact of doing well on statewide standardized tests and the long-term impact of a proxy for noncognitive abilities (specifically: attendance, suspensions, on-time grade progression, and overall GPA). Jackson discovered something fascinating.

Jackson found that his noncognitive proxy measurement was a better predictor of select post-graduation outcomes than statewide standardized test results. In other words and more specifically, if you had better attendance, less suspensions, better on-time grade progression, and a better GPA — if you scored well on that proxy measurement for the noncognitive factors that affect these four things —  you were more likely to attend college, earn better adult wages, and avoid future arrests than if you simply scored well on statewide standardized tests.

We can stop there for a minute to reflect on the fact that in the USA, I would conservatively estimate that we are spending millions of hours of educator labor on improving statewide standardized test outcomes. In my own district, which I consider to be fairly forward-thinking and a great place to work as a teacher and learn as a student, I've been in at least ten hours' worth of meetings analyzing or discussing the results of the statewide test my world history students take two years after they've had me as a teacher. When we compute that there are at least ten other educators in these meetings, that's at least 100 hours of teacher labor spent toward statewide standardized tests. If, for the sake of being as conservative as possible, we assume that my district is far more test-obsessed than the average school — again, I highly doubt that given the forward-thinking nature of my district — and that therefore the average USA teacher spends a tenth of the time we do working on, toward, or in response to statewide standardized tests, then we're looking at one hour per every one of the 3.5 million USA teachers — 3.5 million teacher hours.

In reality, the average educator probably does spend at least ten hours per year laboring toward statewide standardized tests, which means, before we even calculate the hours spent creating these tests or the hours students spend taking them, we are looking at, at minimum, 35 million educator hours per year aimed at these results.

And just so I don't give you the wrong impression, I want to be clear about something: I'm not one to auto-disdain all standardized tests. For example, I really appreciate the AP World History test as an effective measure of a student's achievement in the study of world history. Here are some of the reasons:

  • It is vetted by real-life disciplinary and pedagogical experts;
  • It balances the important tension in world history between knowledge (and it organizes that knowledge thematically, geographically, chronologically, and conceptually) and skill (which it condenses into a very manageable nine historical thinking skills);
  • It asks students to show mastery through multiple choice, short answer, and essay questions;
  • It is accompanied by ample guidance for success, such as sample syllabi, course guides, full versions of past tests, descriptive scoring guidelines from past tests, and sample student work from past tests;
  • Its maker, the College Board, provides certified professional development; and
  • There is “something in it” for all stakeholders: students can earn college credit; parents who plan to pay for a students' college can support their student in earning college credit; and teachers get information on how their students perform on an actual, quality, end-of-course assessment.

In short, the AP World History exam provides high challenge with high support. It's a good test.

The trouble with most state-wide standardized tests is that they often fall far, far short of this. They too often are not vetted widely or rigorously, not created by experts in the field, not accompanied by coherent support materials, not extensively field-tested, not recognized by post-secondary institutions, not incentivized by something positive for kids and parents, not proven to correlate to long-term flourishing outcomes, and not accompanied by high-quality professional development.

Instead, they boast about things like “new drag-and-drop functionality.” Is this real?

And so even while I believe that we must not turn off our brains by auto-disdaining all standardized tests just because they are standardized tests, I also believe we must not turn off our brains by auto-accepting all standardized tests. They aren't all created equal — not even close.

Practical application

Here's what I try to do in light of these things:

  • Support my administrators, who face the brunt of the pressure to make state-wide standardized test scores improve. Don't conflate your administrator with your policymakers.
  • Advocate for students, specifically for setting higher, saner targets for student achievement than the latest state-level accountability model and its accompanying tests. Striving to increase the volume of reading or writing my students do will likely satisfy state-level targets and, of vastly greater importance, promote the long-term flourishing of kids. I don't insist on my way with this — after all, I could very well be wrong — but I do, in my turn, try to advocate for more enduring high-level targets. In my school, we use the phrase “AP Down,” aiming to design our non-AP courses to achieve outcomes that would lead one to be better prepared for successful Advanced Placement work later on.
  • Do the right work in the classroom. Even when I taught in Baltimore in a “failing school” at the height of No Child Left Behind, I would take every chance I had to rush through the scripted curriculum and do legitimate, Atwell-driven reading/writing workshop work.
  • Be mindful of Jackson's noncognitive proxy measures. I regularly talk to my kids about attendance, discipline, credit, and GPA. They might not be part of official accountability measures for me, but they apparently matter for long-term outcomes.

Even though we might be measuring the wrong things at a national or state-level, we still have agency. I would still be happy if my children someday became teachers; it is still a noble profession.

No comments yet.

Leave a Reply