What Learning is Important? How should it be Measured? How can such Measurement Improve Quality?: An Update of the Learning Metrics Task Force

By Kate Anderson Simons.

I was pleased to read about the rich discussions NORRAG members are having on education and the post-2015 agenda. Heikki Holmås’ call for “an open and inclusive process where all stakeholders, especially partners in the South, take actively part in the discussions” really resonated with me, as this has been a driving principle of an effort my colleagues and I are working on to answer some of the difficult questions around how learning should be measured at the global level.

A global dialogue on learning metrics

To define global ambition on improving learning and inform the post-2015 global development policy discourse, we at the Center for Universal Education (CUE) at the Brookings Institution, together with the UNESCO Institute for Statistics (UIS), co-convened the Learning Metrics Task Force in mid-2012. This 18-month-long project aims to harness the collective expertise of the global education community in order to identify common learning goals and targets.

Members of the task force include national governments, EFA-convening agencies, regional political bodies, civil society, and donor agencies, with balanced representation from the global north and south. Three working groups, comprised of technical experts on various topics, convene sequentially to make recommendations for the task force to deliberate and decide upon. Through a globally consultative process, more than 800 interested stakeholders in approximately 60 countries have provided feedback to date on draft documents; the feedback is then used to refine the proposed approaches.

The task force seeks to propose a set of approaches that address the following sequence of questions:

1. What learning is important globally?
2. How should it be measured?
3. How can measurement of learning improve education quality?

What learning is important globally?

As of January 2013, the task force is midway through its deliberations. The first working group was tasked with defining areas of learning that are important globally, going beyond what is currently measured in international assessments. In response to the working group and public consultation recommendations, the task force decided on a broad definition of learning encompassing seven domains, beginning in early childhood and extending through the transition to work and life:

• Physical well-being
• Social and emotional
• Culture and the arts
• Literacy and communication
• Learning approaches and cognition
• Numeracy and mathematics
• Science and technology

How should learning be measured across these domains?

The task force is now in the process of developing recommendations to address the second question on how learning in these domains should be measured. The second working group on measures and methods was charged with developing a framework for measuring learning that is both informative to global policy dialogues and relevant to national education goals, preventing a global “one-size-fits-all” approach. This group’s initial proposal, “Prototype Framework for Measuring Learning Outcomes” released for consultation in December 2012, presented a tiered framework, with some indicators recommended for measurement at the global level and others more suitable for measurement at the country level. The group clarified that measuring a subset of domains at a global level should not imply that some domains are more important than others, but that it is more pragmatic to measure them at a global level given current measurement efforts.

The Measures and Methods Working Group also tested the idea of two levels of measuring progress toward global goals—a global minimum indicator and a global desirable indicator for each of three levels: early childhood, primary, and post-primary. In this approach, the target would be for all countries to meet the minimum level and also show progress toward the desirable goal. We found that there was generally support for having a multi-leveled approach. Setting only one global benchmark would be too high for some countries to reasonably achieve in 15 years and too low for other countries to care about measuring. Working group members and consultation participants pointed out that benchmarking country progress over time or reduction in learning gaps for marginalized groups is an alternative way of measuring progress at the global level, rather than defining an international minimum or desirable level of learning, which could be somewhat arbitrary. This equity approach to measuring learning is also proposed as a way to measure progress in post-2015 frameworks by Save the Children and the Commonwealth Education Ministers.

The working group is finalizing its recommendations to propose to the task force at their February 2013 meeting in Dubai. The working group’s proposal to the task force builds on numerous existing efforts to measure learning at the national, regional, and international levels and will propose several approaches for how existing measures can inform a global learning goal, or could achieve that purpose by implementing some modifications or garnering more resources. The group acknowledges that the overwhelming majority of learning assessments used around the world measure the domains of “literacy and communication” and “numeracy and mathematics”, and that from a pragmatic lens these may be the domains to start measuring in the short term. However, the working group recommends that even if global measurement starts there, the education community should at the same time build capacity for measuring other areas. The working group will also outline a process for improving measurement capacity in the domains of physical well-being, social and emotional, culture and the arts, learning approaches and cognition, and science and technology.

Ongoing debates

There are still several questions that must be addressed before deciding on any single approach.

First, do we think that using a learning measure that is internationally benchmarked (e.g. PIRLS, TIMSS, PISA) strengthens the case for education’s place on the post-2015 agenda vis-à-vis other priorities? If we choose to measure progress toward a global goal in a way that is not internationally comparable, are we willing to risk having learning passed up in favor of goals more systematically measured? As Kenneth King asked in an earlier NORRAG post, what impact would setting one single goal (e.g., proficiency in early grade reading) have on all the other primary school subjects?

There is also a question of measuring learning when access and completion goals have still not been met. Can we propose a global learning goal for lower-secondary learning when schooling is not compulsory at this level in 1 out of 4 countries? Should the learning of out-of-school children be a measurement focus at a global level, or should there instead be a global target of universal lower-secondary education by 2030 as the 2012 GMR proposes?

Finally, although the task force is developing recommendations through a fairly robust globally consultative process, we are still not reaching every stakeholder in the education community. How can we make sure the recommendations are accepted and taken up by national-level decision makers, global and regional bodies, civil society, donors, and teachers, and others whom these recommendations will affect?

Next steps

The working group will approach their revised approach to the task force in February, and the task force will deliberate and decide on a final recommendation. After this, the LMTF Secretariat will release a summary of decisions to date. The third and final phase of the project will focus on the critical question of implementation, with recommendations for practical actions to deliver and measure progress toward improved learning outcomes.

We are accepting applications for this working group, and we look forward to the participation of NORRAG members. Visit brookings.edu/learningmetrics to learn more.

Kate Anderson Simons is the Technical Lead, Learning Metrics Task Force, Center for Universal Education at the Brookings Institution. Email: ksimons@brookings.edu