Measurement Fetishism By Steven Klees

By Steven Klees, University of Maryland.

At my first university job, in the early 1970s, a professor whose expertise was statistics and research methods said, “If you can’t measure it, it doesn’t exist.”  I thought, then and now, that that statement is both extreme and absurd.  More recently, in a World Bank blog, Harry Patrinos quoted management guru Peter Drucker as saying, “If you can’t measure it, you can’t manage it.”  While I am afraid that, to many, this statement sounds more reasonable than the first, I think it just as extreme and absurd.  Unfortunately, we live in a world of measurement fetishism, much to our detriment.

To begin with, you can manage things you don’t measure.  You have to.  Everyone does.  Patrinos tries to make the opposite sound like both common sense and proven by research.  It is neither.  We all manage our households every day of our lives.  Our time, our children.  With outputs much more unmeasurable than measurable.

Business, especially large ones, have a lot of distance between what their employees do and the final products produced.  As many management gurus will tell you, management is more art than science.  Supervisors have to evaluate the performance of subordinates every day, when the impact of the subordinate’s work on final products is tenuous at best.  All of these examples involve human judgment.

Sometimes measuring some outcomes can improve some assessments, but many times it can misdirect attention to what may be measurable but not most important.  If businesses had to manage solely or even primarily by focusing on measurable outputs, most would have gone out of business long ago.  There is a whole literature in the field of public administration pointing out the many circumstances in which focusing management on measurable output leads to distorted and inefficient decisions.

This is equally true in education.  Neoliberal, market fundamentalists like Patrinos have tried to convince us for 30+ years that testing and measurement is a cheap solution to most of our educational problems.  For 30+ years, we have increased educational testing and measurement around the world and our educational problems remain, or have increased.  No Child Left Behind (NCLB) raised measurement to a fetish in the U.S., but, as with similar attempts elsewhere around the world, all the resources go to building a better thermometer with little attention to the causes of the illness or resources to do something about it.  Moreover, the testing and measurement fetish has distorted education towards simplistic measurements of language and math achievement, neglect of other subjects, rampant teacher dissatisfaction, and damage to our children.

Of course, in some instances, measurements can be useful.  But they are far from necessary, and when useful, what is measured should usually only be a small piece of what is assessed.  Before neoliberal dominance took hold, in the U.S. there was a strong movement towards portfolios of student work as the essential ingredients for assessment, much of which required qualitative judgment to assess, not relying on test scores or even grades necessarily.  Universities did and still do, in some places, accept portfolios in order to make admission decisions.  Colleges like Reed and Antioch in the U.S. do not give grades, yet they manage very well, as do universities that offer admission to their graduates based on qualitative assessments.  In vaunted Finland, classroom teachers control assessment; many use portfolio assessment and do not give grades or tests, yet they turn out some of the best test-takers in the world.  There is nothing cost-effective about testing.  It is another “cheap” reform brought to you by people who are unwilling to put in the resources needed to improve education.  To argue how cost-effective even high-stakes testing is also neglects the significant psychological and material damage it does to so many children.

While testing is a relatively low-cost reform, compared to the investment in teachers, learning materials, principals, and schools that are really needed to improve education, the costs of testing add up to big business for business.  So there is a huge lobbying effort by firms like Pearson to make sure quantitative assessments are an integral part of a renewed EFA and the newly developing Sustainable Development Goals (SDGs).  Of course, measurement fetishism goes way beyond education.

Currently the SDGs include 17 goals and 169 targets.  Each target may have multiple indicators.  Each indicator must be decomposed into many sub-indicators that show the extent to which the indicator is realized for multiple marginalized groups.  This is less a system for sane management and more likely to be a welfare program for researchers.  So many resources will be spent on trying to develop, measure, and collect data on what may literally be thousands of indicators that there may be few resources and little attention left over to actually do something about the problems indicated.

Diane Ravitch, the noted U.S. education historian and policy analyst asks:  “How did our elected officials become convinced that measurement and data would fix the public schools?”  In a later book she blames the business model that has so dominated education reform in the U.S. and worldwide:  “It means an obsessive devotion to testing, accountability, and data.  Devotees of the business approach like to say, ‘You measure what you treasure.’  Believing this, they have fastened a pitiless regime of testing on the nation’s schools that now reaches as low as kindergarten…[and even] prekindergarten.”  Ravitch goes on to argue that we can’t measure what we treasure most – such as human relationships or love of art and music – and education has been distorted by a narrow focus on measurement and accountability.  While other countries have not gone as far as NCLB, this is the direction in which we are heading.

Don’t get me wrong.  Indicators can be useful.  But our fetishism with measuring everything has gotten in the way of emphasizing what is needed even more than indicators.  The legitimacy of the processes, from the local to the global, in which quantitative indicators and qualitative information and judgments are used is key.  We too often treat policy analysis and policy making as a technical process.  It should not be.  Decision-making should rest on very messy, very participative, democratic processes that engage people from the local to the global.  We need to have more deliberative democratic approaches in which thoughtful human beings are able to combine quantitative and qualitative judgements to make sensible decisions, often in situations of conflicting interests.  Of course, measurement has a place, but to argue ‘if you can’t measure it, you can’t manage it” is belied by experience, research, and common sense.

Steven J. Klees is the R. W. Benjamin Professor of International and Comparative Education at the University of Maryland. Email: sklees@umd.edu

This blog first appeared on 16th July 2015 on Education International’s ‘Education in Crisis’ Blog

Disclaimer: The views given in this blog are those of the author alone and should not be attributed to NORRAG or its members. Readers are invited to comment below. 

>>View all NORRAG Blogs on Sustainable Development Goals

NORRAG (Network for International Policies and Cooperation in Education and Training) is an internationally recognised, multi-stakeholder network which has been seeking to inform, challenge and influence international education and training policies and cooperation for almost 30 years. NORRAG has more than 4,200 registered members worldwide and is free to join. Not a member? Join free here.