Note: the original version of this post included a section on individual teacher ratings. I could not find public references for the statements I made, and have deleted these two paragraphs. My apologies!
I recently placed a copy of How to Lie with Statistics in a little free library near campus. Not because I want people to be more deceitful – if you don’t understand how to trick others, then you yourself will be easy game. Numbers sound like facts. They can be used for malicious ends.
Consider medical ratings. These are ostensibly beneficial – prospective patients get to learn how well-trained their doctors are!
Saurabh Jha wrote an excellent essay explaining why these rankings are misleading, “When a Bad Surgeon Is the One You Want.” In brief, doctors who take easy cases will improve their ratings – their patients are more likely to have good outcomes. When doctors are assessed on their patients’ outcomes, then the doctors who take hard cases will appear to be incompetent. Even if they are much better at their craft than others.
The same phenomenon holds in teaching. Schools and teachers are often evaluated based on their students’ performance, without normalizing for the unique challenges faced by different populations of kids.
This week, the Indiana Department of Education released federal evaluations of local schools.
The elementary school located amidst our town’s most expensive houses, at which the lowest percentage of students receive free or reduced-price lunch, was rated as “exceeding expectations.”
The elementary schools that serve our town’s most disadvantaged students – one of which holds bilingual classes in English and American Sign Language to support deaf children, and has 86% of students receiving free or reduced-priced lunch – were rated as not meeting expectations.
My spouse and I are sending our own children to one of the schools that was rated as not meeting expectations. We know a fair bit about education – among other things, my spouse is the editor-in-chief of a national journal of teacher writing. I’ve observed classrooms in this low-rated school, and they are excellent.
But teacher morale is low, because the teachers are continually evaluated as being sub-par, despite the fact that they have chosen to work harder than others. Our school district is mandating that teachers in the low-rated schools spend time on unfulfilling test-prep regimes, even though these practices are known to further alienate under-resourced students.
Our nation’s school administrators ought to read How to Lie with Statistics, it seems. They’ve looked at a set of numbers and allowed themselves to be misled. Which bodes ill for the learners in their care.
But I understand that, for students less recalcitrant than I was, the ones who might actually learn something during regular instructional days, standardized tests waste time. And the current barrage of tests don’t even fulfill their purported goal.
The example I like to start with is shipping. Apples at an apple orchard might be worth two dollars a pound. Anybody who wants an apple has to go to all the trouble of driving there. But if someone loads them into a truck and brings them to a grocery store near people’s houses, the apples might be worth three dollars a pound. Transporting apples from where they grow to where people eat adds value.
If we don’t care about how well kids learn to fill in bubbles nice and dark with a #2 pencil, then what should we value? Well, we might care about workforce productivity, in which case your “value added” metric should track students’ eventual salaries or lifetime earnings. Maybe we want to make people into better citizens, in which case we should measure how often people volunteer, or how often they vote, or what percent of students stay out of jail. Maybe we care about something as ethereal and hippy-dippy as happiness, in which case we could use surveys to assess well-being, or look at how many former students are married, or track how many commit suicide.
A student with learning disabilities will probably gain less than average each year, independent of teacher quality. A gifted student will probably gain more than average, again independent of teacher quality. The teachers do matter, of course. If you gave both Pablo Picasso and me some crayons and a piece of construction paper, his drawing would probably be better than mine. But he’d add less value to that piece of construction paper than he would’ve been able to add to a canvas, if you instead gave him a canvas and some oil paints.
One difficulty in sorting people this way is determining what matters. What attributes define a student’s type? Do you include parental income as a variable? A near-meaningless childhood IQ test? Do you sequence every student’s genome and include genetic factors (Good Lord I hope not — even including ethnicity seems politically suspect — but that’s the sort of thing you’d want to consider)?
Frank Brown Cloud 