Articles
In his book “The Art of Doing Science and Engineering”, Richard Hamming wrote “what gets measured gets better, and what’s difficult to measure suffers”.
In our era of quantitative accountability, measuring finances, assessment grades, strategic goals, learning outcomes, curriculum compliance, demographic changes, admission numbers – and lots more – has never been more important in schools. Of course, it doesn’t necessarily follow from this that Hamming’s claim of “what gets measured gets better” is true for schools. Nonetheless, the conventional wisdom of many bureaucrats and school boards seems to be that Hamming’s words are indeed accurate – measuring things makes them better.
In schools, and elsewhere, great care must be taken in determining what is to be measured and what is not. For school leaders and their boards, this often translates into a set of KPIs (key performance indicators) which are regularly presented in the form of dashboards – easy to understand visual representations of a variety of statistical data.
Determining the combination of factors which are important to measure is a key decision for school leaders and boards. It is necessary to remember the caution inherent in Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure”. In school environments, this implies that when a measure is chosen to assess people’s performance, they will find a way to “game the system”.
The classic (non-school) example of this is a nail factory in the former Soviet Union. The goal of the government’s central planners was to measure the performance of this (and every other) factory in the USSR, and so the factory manager was given targets (KPIs) based upon the number of nails produced. In order to meet and exceed the targets, the factory manager ordered the factory to produce millions of tiny, useless nails. The central planners reacted by changing the target to measure the total weight of nails produced, and so the manager shifted operations to produce several enormous, heavy and useless nails.
(I have not been able to verify the name or location of the nail factory, and indeed it may be apocryphal. Nonetheless it was celebrated in the cartoon shown here that was published in the Soviet satirical magazine “Krokodil” in 1922. The embedded caption translates as “Who needs such a nail?” “That's not important. What’s important is that we immediately fulfilled the plan for nails!”. Apocryphal or not, I think it illustrates Goodhart’s law brilliantly).
It is easy to see how such targets can distort a school’s culture. If enrolment targets are set to maximise new admissions, students who are not well suited to the school and who would not previously have been considered suitable are very likely to be admitted. If enrolment targets are set to emphasise high academic standards, sports-oriented and artistic students may be turned away, or even worse, students with special needs might be rejected.
When KPIs are set, it is essential that consistency is maintained, or at least changes are thoroughly discussed and negotiated before being introduced. I know of one school where the board set an ambitious enrolment target – a net gain of 50 students in the coming year - which the Principal had to achieve as a KPI. By the end of the year, there was a net gain of almost 100 students. Notwithstanding this achievement, the board claimed the Principal had failed to achieve the KPI because many of the additional students were from poorer families and were thus entitled to subsidised fees under the school’s policies. Consequently, the increase in fees income had been less than the board’s treasurer wished, even though income from fees was never mentioned as a KPI. Not unexpectedly, that Principal was leading a different school within two years.
An uncharitable interpretation of this Principal’s initiative to secure additional enrolments could be to label it as an example of the Cobra Effect. The Cobra Effect arises when a well-intentioned initiative backfires and achieves the opposite effect to the one intended. It is named after another possibly apocryphal event in India during the period of British colonisation.
The story is that the British authorities decreed that there were too many cobras around Delhi, so they introduced a cash reward for anyone who brought in a dead cobra. Apparently, this incentivised some enterprising folk to start breeding cobras to collect the bounty. When the British authorities realised what was happening, they discontinued the scheme. This resulted in the closure of the cobra farms and the owners releasing the now-worthless cobras, which bred profusely, thus greatly increasing the number of cobras in the wild.
Whether or not this story is true, there have been numerous documented examples of the Cobra Effect. In Hanoi (Vietnam), a bounty on rats’ tails followed an almost-identical track as the cobras in Delhi, while a measure in Bogotá (Colombia) to reduce traffic congestion by restricting each car from driving one day each week resulted in residents going out and buying second cars, thus increasing traffic congestion.
Unintended consequences like the Cobra Effect also affect schools. I have seen well-intentioned school leaders who conflate student achievement with student effort, and specify additional homework or other requirements such as formal study times to increase students’ effort. While this did work for some students, many students simply spent more time in activities that superficially resembled studying (such as quiet daydreaming). The Cobra Effect was operating when students pretended to be studying in ways the school demanded rather than in ways that genuinely worked for them, causing their grades to fall rather than rise. Sadly, many well-intentioned but ill-informed bureaucratic initiatives from government authorities and school boards can result in similar ‘Cobra Effect’ consequences in schools.
Neither the Cobra Effect nor Goodhart’s Law should deter schools from making decisions based on evidence and data – indeed, it would be foolish (and in some cases, possibly illegal) not to do so. Nonetheless, wisdom and prudence are required if Hamming’s claim is to be realised so that “what gets measured gets better.”
-Dr Stephen Codrington
Hamming, R (2020) “The Art of Doing Science and Engineering: Learning to Learn”, San Francisco: Stripe Press.
We help school boards and leaders decide upon and prepare straightforward, relevant KPIs that feed into an efficient, easy-to-understand Principal’s report to the Board.
Further information on many aspects of effective board meetings and reporting is provided in the book “Optimal School Governance", which can be ordered directly through Pronins.
You may also be interested in previous articles which are archived at https://optimalschool.com/articles.html. You can subscribe to receive future articles by e-mail using the red button below.
