How much damage can we do by misusing metrics?

Quite a lot. And this time, we’re not talking about a multi-million dollar project, that’s peanuts.

How about setting the education system of an entire country on a multi-year track to failure?

Last week, the Israeli government finally changed a policy of rating the schools in the country, that started back in 2002. The rating system, based on the (loosely translated from Hebrew): School Effectiveness and Improvement Metrics (SEIM) tests, was managed by the National Authority for Education Measurement and Evaluation (NAEDE).  I know it sounds like it came directly from the Dilbert Management Dictionary, but I’m not making that up (this is the Hebrew Wikipedia page).

Measurement was simple: All pupils at the same level got tested, and the schools were rated according to the marks. The ratings were public, along with local and national averages.

Sounds very innocent, even sensible: When schools get a higher grade, they become more attractive. When cities get a higher local average grade they get more attractive. And when the grades are low, they will try harder, right?

Not exactly.

Schools asked the less capable kids not to come on the day of the test. They focused on teaching the test material, and less on the material tests didn’t cover. Teachers were encouraged to get the questions right, rather than teach how to solve problems. 

And to make things worse, the SEIM rating became the most important rating of all, so schools threw time, money and teachers at the problem to get better marks. After all, nobody likes to lose this competition.

That’s finally over. It took more than a decade to change, and I’m sure someone will come up with a new and improved metric. We’ll see how that goes in 10 years.

“I’m not bad, I’m just drawn that way” – Jessica Rabbit

Metrics are not inherently wrong (or right). But like in the Heisenberg Uncertainty Principal, when you start measuring, you’re changing the system. In a complex system, (and if the Israeli education system is not complex, I don’t know what is), you can’t really know in advance how a change might impact the future. Slow processes, like these take years to understand, and maybe the time it took to understand it was short.

When we come to measure something, we need to ask: Am I measuring the right thing? Or am I measuring the easy thing?

Are system-wide tests easy to conduct? There’s a lot of time, money and people involved. But the metrics are simple,  and boy, do we love simple. Students have been taking tests for centuries, the method must work!

And when we already have marks, we can improve them (by cheating, if we have enough incentive).

Enough ranting for now.

Make sure the metrics you measure align with your goals. And don’t wait for a decade to pass before checking that there are no side effects to the metrics.

PS. Did you ever think why still work with the grading system, when it obviously doesn’t produce great results? Later on that.