Can the Measurement of Learning Outcomes Lead to Quality Education for All? By Pablo Zoido, Michael Ward, Kelly Makowiecki, Lauren Miller, and Catalina Covacevich

By Pablo Zoido (former OECD), Michael Ward, Kelly Makowiecki, Lauren Miller, Catalina Covacevich (OECD)

This is one of the complex questions that kicked off last month’s, NORRAG-Brookings event, “Learning From Learning Assessments: The Politics and Policies of Attaining Quality Education”, which brought together targeted stakeholders with expertise in learning assessments, education policy-making and classroom experience.

Pablo Zoido on our team (until just recently) had the pleasure of participating in a plenary session discussion about how we move from theory to practice in using assessments models to improve education policy-making and delivery in a country. Representing the OECD, he shared our experience with the PISA for Development (PISA-D) initiative.

The Programme for International Student Assessment (PISA) is the OECD’s triennial international survey which aims to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. PISA began in 2000, and as more middle-income and low-income countries took part in the assessment over the years, it became more and more pressing for PISA survey instruments, methods and analyses to be relevant and useful to a broad set of countries. PISA-D was thus initiated in 2013 to help middle-income and low-income countries maximise their use of PISA for monitoring progress towards nationally-set targets for improvement, for the analysis of student learning outcomes, particularly for vulnerable populations, for institutional capacity-building, and for tracking international educational targets in the UN-led Sustainable Development Goal (SDG) for Education.

PISA-D participating countries include Cambodia, Ecuador, Guatemala, Honduras, Paraguay, Senegal and Zambia. Panama, a PISA 2018 participant, has also joined PISA-D for the out-of-school component and to benefit from the project’s capacity building activities. Why are these countries participating and what do they get out of it? The selection was demand driven and subject to countries having access to the necessary funds for ensuring successful participation – this often involved the support of a development partner. These countries have signed up to PISA test instruments that capture a wider range of performance levels and contextual questionnaires meant to effectively capture the diverse situations in their countries. PISA-D countries are also pioneering new methods and approaches to include out-of-school youth in the assessment.

PISA-D countries have joined the initiative because they recognize that PISA has the potential to serve as a powerful tool for policy making in their context. PISA assesses competences in reading, mathematics and science in a way that is not linked to the school curriculum. The assessment is designed to measure to what extent students at the end of compulsory education can apply their knowledge to real-life situations and be equipped for full participation in society. The information collected through background questionnaires also provides valuable context. The PISA-D survey will produce results that are on the same scale as the main PISA assessment, and help participating countries see where they stand in comparison to their regional and global peers, and to learn from each other’s experiences.

A crucial element of PISA-D is that countries receive support to build their capacity for carrying out and using the assessment. The OECD conducts a capacity needs analysis to determine the capacities that need to be strengthened in order to successfully manage a large-scale assessment. Then the organisation works with the country to establish a capacity building plan as well as a project implementation plan to ensure that the capacities are developed and the PISA-D national team is prepared to adhere to the assessment timeline and standards. The team members go through rigorous trainings on a variety of technical topics, such as sampling, translation/adaptation of survey instruments, data management, coding assessment responses, and data analysis. These activities help them implement the assessment successfully and fully benefit from their participation to strengthen national and regional evaluation efforts.

Another valuable feature of PISA-D is the peer-to-peer learning that takes place throughout all phases of the project, which include: i) design and planning; ii) technical development; iii) field trials and in-country data collection; iv) analysis and reporting; and, v) report production, dissemination and post-pilot governance. PISA-D countries enter a partnership with other PISA countries for peer learning and technical support. The main PISA national teams have valuable lessons to share with the newly appointed PISA-D national teams such as how to effectively engage with stakeholders, embed PISA in a broader national discussion of the value and standards of assessment, and prepare to report and disseminate the assessment results.

PISA countries like Brazil and Peru, which are also serving as peer-to-peer learning countries for PISA-D participating countries, have shown how valuable PISA can be by using the surveys to set quality-of-learning benchmarks and monitor progress against these over time. Their PISA results have shown that diverse countries have managed to raise the quality of educational outcomes substantially, despite starting from different points.

As the theory behind this collaborative effort is put into practice, we are seeing PISA-D countries giving new shape to national and international policy dialogues on how to improve the quality of education. After the field trial later this year, main survey data collection will take place in 2017, and the results will be reported in 2018, at which point the PISA-D countries will have data to help policy makers adjust current benchmarks and set new ones to monitor and gradually improve learning outcomes. It is already proving to be a very interesting process to watch as it develops from one phase to the next.

The UN-led SDG Education 2030 agenda emphasises the quality, equity, and measurement of learning outcomes for young children through to working adults. The challenge is to define global learning indicators that can be measured and tracked on a global scale over time, and the OECD’s PISA-D initiative is helping inform and support the SDG Education discussions and strategy. PISA is an invaluable tool for data collection, and as more and more countries use it to measure learning outcomes, we have the potential to make improvements to education systems worldwide and attain the goal of providing quality education for all.

Pablo Zoido was an analyst at the OECD until mid-July 2016, working on PISA for Development. He has since joined the Inter-American Development Bank. Email: pzoido@gmail.com

Michael Ward is in the OECD Development Co-operation Directorate. Email: Michael.Ward@oecd.org

Kelly Makowiecki, Lauren Miller and Catalina Covacevich all work on PISA for Development at the OECD. Emails: Kelly.Makowiecki@oecd.org; Lauren.Miller@oecd.org; Catalina.Covacevich@oecd.org

Other NORRAG Blogs about PISA for Development:

• PISA for Development: Expanding the Global Education Community Esperanto or Developing a Dialect? – By Camilla Addey, Humboldt University in Berlin (28^th July, 2016)
• Expanding PISA: The OECD and Global Governance in Education – By Sam Sellar and Bob Lingard, School of Education, The University of Queensland. (June 20^th, 2014)
• PISA for Development: One World, One Measure for Learning? – By Angeline M. Barrett, University of Bristol. (January 13^th, 2014)
• PISA in Low and Middle Income Countries – By Simone Bloem, formerly OECD. (January 10^th, 2014)
• PISA for Development and the Post-2015 Agenda – By Michael Davidson, Michael Ward and Alejandro Gomez Palma, OECD. (January 8^th, 2014)

>> View full list of related NORRAG blogs about learning assessments, global governance, data.

NORRAG (Network for International Policies and Cooperation in Education and Training) is an internationally recognised, multi-stakeholder network which has been seeking to inform, challenge and influence international education and training policies and cooperation for almost 30 years. NORRAG has more than 4,500 registered members worldwide and is free to join. Not a member? Join free here.