Is Education Reform Effective? Depends on the Definition.
Here’s the dilemma for people who write about education: Certain critical principles need to be mentioned again and again because policymakers persist in ignoring them, yet faithful readers eventually tire of the repetition.
Consider, for example, the reminder that schooling isn’t necessarily better just because it’s more “rigorous.” Or that standardized test results are such a misleading indicator of teaching or learning that raising scores can actually lower the quality of students’ education. Or that using rewards or punishments to control students inevitably backfires in multiple ways.
Education policymakers have turned a blind eye to solid evidence supporting each of these points for decades, yet still call their work "reform." Hence the dilemma: Will explaining in yet another book, article, or blog post why their premises are dead wrong have any effect other than eliciting grumbles that the author is starting to sound like a broken record?
Another axiom that has been offered many times to no apparent effect is that it means very little to say that a given intervention is “effective”—at least until we’ve asked “Effective at what?” and determined that the criterion in question is meaningful. Lots of educators cheerfully declare that they don’t care about theories; they just want something that works. But this begs the (unavoidably theoretical) question: What do you mean by “works”?
Once you’ve asked that, you’re obligated to remain skeptical about simple-minded demands for evidence, data, or research-based policies. At its best, research can only show us that doing A has a reasonably good chance of producing result B. It can’t tell us whether B is a good idea, and we’re less likely to talk about that if the details of B aren’t even clearly spelled out.
To wit: Several studies demonstrate the effectiveness of certain classroom management strategies, most of which require the teacher to exercise firm control from the first day of school. But how many readers of this research, including teacher educators and their students, interrupt the lengthy discussion of those strategies to ask what exactly is meant by “effectiveness”?
The answer, it turns out, is generally some variation on compliance. If a teacher does this, this, and this, it's more likely that his or her students will do whatever they’re told. Make that explicit, and we must ask whether compliance is really the paramount goal. If, on reflection, a teacher decides that it’s most important for students to become critical thinkers, enthusiastic learners, ethical decision-makers, or generous and responsible members of a democratic community, then the basic finding—and all the evidence behind it—is worth very little. Indeed, it may turn out that proven classroom management techniques designed to elicit obedience actually undermine the realization of more ambitious goals.
An even more common example of this general point concerns academic outcomes. In scholarly journals, media coverage, professional development workshops, any number of techniques are described as more or less beneficial—again, with scant attention paid to the outcome. The discussion of “promising results” is admirably precise about what steps achieved the results, while swiftly passing over the fact that those results consist of nothing more than scores on standardized tests.
We’re back, then, to one of those aforementioned key principles that are so often ignored. Standardized tests tend to measure what matters least about intellectual proficiency, so it makes absolutely no sense to judge curricula, teaching strategies, or the quality of educators or schools on the results of those tests. Indeed, as I’ve reported elsewhere, test scores have actually been shown to be inversely related to deep thinking.
Thus, “evidence” may demonstrate beyond a doubt that a certain teaching strategy is effective, but it isn’t until you remember to press for the working definition of effectiveness that you realize the teaching strategy (and all the impressive-sounding data that support it) are worthless because there’s no evidence that it improves learning instead of just test scores.
Which leads me to a report published earlier this year in the Journal of Educational Psychology. A group of researchers at the City University of New York and Kingston University in London performed two meta-analyses, statistically combining studies to quantify the overall result. The title of the article was “Does Discovery-Based Instruction Enhance Learning?”—a question of interest to many educators.
The first review, of 580 comparisons from 108 studies, showed that unassisted discovery learning is less effective than “explicit teaching methods.” The second review, of 360 comparisons from 56 studies, showed that various “enhanced” forms of discovery learning work best of all. In other words, students learn better when they are guided by a teacher.
There are many possible responses to this news, ranging from “Duh" to “Tell me more about those enhanced forms, and which of them is most effective" to "How much more effective are we talking about?” since a statistically significant difference can be functionally meaningless if the effect size is low.
I took my own advice and asked “What the hell did all those researchers, whose cooking was tossed into a single giant pot, mean by ‘effective’?” It’s astonishing how little this crucial definition appeared to matter to the review’s authors. There was no discussion of what effectiveness means in the article’s lengthy introduction or in the concluding discussion section. There wasn’t a word to describe, let alone analyze, what all the researchers were looking for. Did they want to see how these different types of instruction affect kids' scores on tests of basic recall? Their ability to generalize principles to novel problems? Their creativity? (There’s no point in wondering about the impact on kids’ interest in learning—that almost never figures in these studies.)
Papers like this one are peer-reviewed and, as was the case here, are often sent back for revision based on reviewers' comments. Yet apparently no one thought to ask these authors to take a step back and consider what kind of educational outcomes are really at stake when comparing different instructional strategies.
In fact, the desired outcome in education studies is often quite superficial, consisting only of standardized test scores or a metric like number of items taught that were correctly recalled. And if one of these studies makes it into the popular press, an examination of desired outcomes probably won’t. In January I wrote about widespread media coverage of a study that supposedly proved one should, to quote The New York Times article, “Take a Test to Really Learn, Research Suggests.” One had to carefully read the study itself to discover that “really learn” just meant “cram more facts into your short-term memory.”
But the problem isn’t just an over-reliance on outcome measures—rote recall, test scores, or obedience—that some of us regard as shrug-worthy and a distraction from the intellectual and moral characteristics that should be occupying us instead. The problem is that researchers are, as a journalist might put it, burying the lede. And too many educators don’t seem to notice.
Photo via ameshistoricalsociety.org
So You Think You’re a Foodie? Pop culture was onto these trends way before you were. A sampling of the screwball comedies, sob stories, and sci-fis that anticipated our culinary moment
Dear Nine-Year-Old Me The transition is going to be difficult for you, but whenever you feel a little lonely and left out, take comfort in the knowledge that you are honing one of your greatest superpowers.
What to Do When Your Country is Drowning The wild and desperate ways island nations are fighting the effects of climate change
The Rise of Drone Pizza Delivery Why the skies will soon be filled with flying, snack-bearing robots
How Helsinki Became a Public Transporation Paradise One European city plans to make car ownership obsolete within a decade.
Follow the Crowd NanoCrafter and the rise of group intelligence Why online gaming may just be the future of science
The Empathy Mirror Neurofeedback enables us to better see ourselves in the other. Recent discoveries in neurofeedback can teach you to be less of a dick.
Robots On Ice Probe the Arctic Why a team of research robots is investigating disappearing sea ice, and why you should care
Don’t Turn Away Colin Finlay photographs the consequences of climate change. You will never see more beautiful photos of the deteriorating state of our planet than the ones in this photo feature.
Puppy Love How dogecoin spawned an improbable community of giving What a canine-emblazoned cryptocurrency can teach about philanthropy
Positive In, Positive Out: How a USC Alumna is Coping with Lymphoma Coast Guard Reserves member Cassie Sulfridge, 28, had just graduated from MSW@USC, the Southern California university’s web-based Master of Social Work program, and was working two jobs when her life was turned upside down.
Politics by Yummier Means An Israeli-Palestinian popup restaurant and the precarious art of gastric diplomacy Two chefs win over hearts, minds, and stomachs in Jerusalem.