Lessons From Chengdu (2): The Political Economy of Learning Metrics By Trey Menefee

By Trey Menefee , Hong Kong Institute of Education

I argued in my previous NORRAG blog post, Lessons From Chengdu: The Case For ‘Open-Source’ Learning Metric Methods, that easily accessible non-proprietary learning metric methodologies would be a valuable asset to the international education community. What I argue here is that metrics themselves are not the problem. We are familiar with the adage that we cannot manage what we cannot measure, but to this we should add that metrics make for terrible managers. However, metrics go beyond just saying something – they seem to tell us what to do. An antidote to this is employing what might be called the ‘Scott Audit’, which answers three questions James C. Scott (2012) has proposed as a framework for navigating the politics and anti-politics of metrics. This post will use this auditing technique on our Chengdu research and conclude by showing how the issues addressed link directly with problems in the technical and vocational skills development (TVSD) sector.

1. What is the relationship between the proposed quantitative index and the construct – the thing in the world – it is supposed to measure?

There is a risk that the metrics are measuring against the wrong norms or that they carry more symbolic weight than they can or should carry. Many rightly scoff at league tables associated with PISA which aim to offer a specific number showing how one school system is better than another. From another perspective, we see the Brookings-UIS Africa Learning Barometer doing an excellent job using learning metrics to expose the shallowness of another: enrollment rates. Their work shows that “37 million African children will learn so little while in they are in school that they will not be much better off than those kids who never attend school.”

In the Chengdu study, one of the earliest issues encountered was that Chinese English tests scores often have a low correspondence with a student’s actual ability to communicate in English. These tests measure what might be called “virtual English” – a reorganization of communicative English in ways that made it more accommodating to rote memorization and assessment. This fosters a justified concern amongst educators that focusing on communicative language teaching, usually considered best practice for language acquisition, might hinder student test scores.

Further, the merging of value-for-money approaches and performance-based development metrics lends itself toward breeding inequality within the unequal. There were substantial inequalities both between the private schools for migrants in Chengdu and within the classes. The easiest way for an NGO, school, or teacher to get measurable impact and ‘bang for the buck’ would be to ignore the struggling students and concentrate resources on the higher performing schools and students. A shallow analysis of learning outcome improvements, for instance an increase in the number of students ‘passing’ the Zhongkao English section, runs a high risk of identifying the most inequitable interventions as having the highest impact.

2. Is a political question being hidden or evaded under the guise of quantification?

Knowledge-based barriers to social mobility might be thought of as coming in two forms: intrinsic and extrinsic. Illiteracy is an intrinsic barrier when someone can’t read the instructions, signposts, or maps that might lead to a better livelihood. By and large, white collar jobs exclude the illiterate because literacy is genuinely necessary to perform those jobs competently. On the other hand, my hometown denied many African-Americans the right to vote through the use of literacy tests until it was outlawed in 1965. It would be morally indefensible to approach this problem with the solution of increasing literacy rates, no matter how valuable literacy might otherwise be. This is amongst the most difficult but important issues when working with learning metrics: determining whether the metrics are genuinely of intrinsic usefulness or whether they are instead being used extrinsically as tools for social closure.

A primary mechanism of social closure with learning metrics is a very specific type of equality – an equality in expectations. In context to English, it begins with unrealistic and largely uncontested curricular assumptions that all students are performing at roughly the same level. These expectations move into the classroom and create an environment where every student studies the same material at the same time, regardless of their command of the subject. The idea of equality is contorted, such that many schools and parents believe that offering remedial classes would be inequitable.

In practice, this amounts to the academic equivalent of a death march. Students furthest away from this arbitrarily invented academic norm find themselves abandoned by their schools, teachers, and textbooks a little bit more every day. There is little sense of progress for many students: the curriculum appears to march three steps forward with every stride they take. English, in this view, is deployed in China as a tool of social closure more than a subject of intrinsic value. These Chengdu students were failing by design. There are lessons here for any post-2015 goal requiring minimum learning outcomes. Who, exactly, will pay the price for failing to meet these new standards?

3. What are the possibilities for colonization or subversion of the index, such as misreporting, feedback effects, or the prejudicing of other substantive goals?

Theodore Porter (1996, p. 43) poignantly noted that, “quantitative technologies work best… if the world they aim to describe can be remade in their own image.” There is a constant and largely unacknowledged dialectic that prioritizing specific outputs transforms the inputs. Whenever new learning outcomes are deemed important for success we find teachers and administrators transforming curricula to meet these expectations. Scott (2012, p. 116) aptly calls these ‘colonizing metrics.’

The importance of English in Chinese academic testing elevates it to compulsory, rather than an elective, subject. It joins a handful of academic skills that become ever more condensed as students matriculate. Questions mostly revolve around how to increase quality and improve scores, rather than challenging the fundamentals of the situation. It is through this mechanism that students, parents, teachers, and civil society get trapped in playing perpetual catch-up in a game that is designed to exclude three out four Chinese youth from higher education access.

While parents and students should be expected to adapt to these situations, researchers and institutions involved in collecting and analyzing learning metrics are in a position to identify and critique situations of education-based opportunity hoarding. At this, we too often fail. The conceptual purpose of PISA, TIMMS-PIRLS, or SAQMEQ scores rarely seem to rise above showing how much a country, district, or a school deviates from others. There is a danger that the post-2015 learning targets, should they come to pass, might begin a process of transforming schools systems around the world into machines that produce higher scores of questionable value that attract more aid dollars, higher league table rankings, and electoral advantage. The metrics themselves too readily sideline contentious politics and become managers.

Collecting internationally comparable learning metrics are a valuable tool for understanding educational inequalities if it can put politics front and center. They are dangerous when uncritically mixed with positivism and teleological assumptions about performativity. It would have been problematic if the primary rationale for collecting our Chengdu data was to only answer how scores might be improved, rather than to look at how learning outcomes were being used against students. The ‘Scott Audit’ used here is one way of ensuring that the politics of educational performance and development stay in focus.

This analytical lens highlights the proverbial elephant in the room: while there are two streams of secondary education in China, only one is seen as viable: the college preparatory track. Problems in China’s second stream, vocational  education, make it unattractive to even the most marginalized families. In this view, the problem was not low English scores but a lack of educational options for the 75% of students who won’t make it to university. What our learning metrics data provided was snapshot showing how many students were likely to fall off and the limitations of improving academic performance to improve equity. The better question to take from our data is not how to universalize high quality English education for a hundred million children, but “what’s an equitable Plan B?”

Trey Menefee is Lecturer at the Hong Kong Institute of Education and is currently working on the primary statistical report for the 19th Conference of Commonwealth Education Ministers, which focuses on educational quality and inequality. Email: trey333@gmail.com