PISA for Development: Redefining Out-of-school Youth for Success

Abstract: This NORRAG Highlights was submitted by guest-author, Xiaomin Li, a doctoral candidate at the University College London (UCL) Institute of Education. Li finds that out-of-school youth are continually excluded from OECD activities. The non-inclusive definition of ‘youth’ used in the recent PISA-D Assessment leads Li to question the legitimacy of the data. She tells us to proceed with caution before formulating educational responses based on PISA-D results. 

The results of PISA for Development (PISA-D) out-of-school assessment were released by the OECD in December 2020. Before the release of this report, on 3 December, the OECD held an online PISA-D Out-of-school Assessment – International Technical Workshop. Together, they marked the closure of this pioneering project (2013–2020) which aims to make PISA more accessible and relevant to low- and middle-income countries. Technically, PISA-D focused on three strands:

Strand A composes test items to target at the lower end of the performance spectrum by adding lower rungs, such as Level 1a, 1b, 1c, Level 2a, 2b, 2c.

Strand B adjusts contextual questionnaires to capture the situations in low- and middle-income countries by endorsing the Educational Prosperity framework.

Strand C develops methods and/or approaches to incorporate out-of-school youth into the assessment.

The focus of this post is on Strand C that developed methods to include out-of-school youth in the assessment, known as the out-of-school assessment or surveys. I argue that this supposed inclusion of out-of-school youth has been misrepresented by the OECD, and risks being misinterpreted by policymakers and national systems who are encouraged to opt into future PISA rounds following this ‘successful’ pilot.

Source: OECD

Among the nine countries which participated in the PISA-D school-based assessment, only five of them conducted the out-of-school component, namely Guatemala, Honduras, Panama, Paraguay and Senegal. In the newly released report, the OECD has acknowledged the countries’ bravery in ‘comparing themselves internationally’ and ‘going an extra mile’ to capture ‘the skills and circumstances of the most disadvantaged children and youth in their populations’. It also confirmed that, through PISA-D, the OECD has ensured that out-of-school youth are no longer beyond the reach of large-scale learning assessments and, therefore, it has allegedly succeeded in filling the data gap for the monitoring of the Education Sustainable Development Goal (SDG 4).

The achievements of PISA-D out-of-school assessment appear novel and significant, given the size of the problem, which is currently being amplified by the COVID-19 pandemic. According to UNESCO Institute for Statistics (UIS) data for the school year ending in 2018, about 62 million youth at lower secondary school age (about 12–14 years old) are out-of-school. This group of population constitutes a large proportion of the ‘poorest of the poor’ (Carr-Hill 2013) and they often reside at ‘the bottom of the pyramid’ (Wagner, Wolf and Robert 2018). However, so far there has been a lack of adequate instruments to identity and approach out-of-school youth, measure the scope of and assess the reasons for exclusion, and assess the possible levels of skills they have to inform policy and planning. The PISA-D out-of-school component was designed to channel through this bottleneck.

The OECD contracted Educational Testing Service (ETS), a US-based school testing service that had never carried out a household survey, to deliver these tasks. To help obtain a sufficient number of youth samples, the OECD decided to enlarge the target population age range for the out-of-school surveys from 15-year-old to 14–16-year-old. This decision was made on the recommendation of an independent expert – Roy Carr-Hill – who helped to identify the challenges of counting and locating a single year age group in developing country contexts. In particular, based on demographic statistics, 15-year-olds in the piloting countries are about 2.2% (i.e. 1 in 45) of the population and with average household size between 4 and 5, which means it would need to visit around 10 households to find one 15-year-old. Meanwhile, it is likely that most of the marginalised will be found in the isolated rural poor and informal settlements/urban slums (see Carr-Hill 2015).

Extending the target population age range to 14–16-year-old seems reasonable, the OECD, however, took a shortcut to avoid addressing directly the challenges highlighted by Carr-Hill in the initial working paper. To elaborate on this point, it is helpful to look at the resulting coverage of the out-of-school youth samples presented in the newly released report.

The six zones of exclusion considered in PISA-D include 14–16-year-olds who have:

• never enrolled in school (zone 1, “never enrolled”)
• dropped out of school in early primary grades (zone 2, “primary drop-outs”)
• remained at school but are currently in grade 6 or below (zone 3, “grade 6 or below”)
• dropped out after completing primary school (zone 4, “primary leavers”)
• dropped out in lower secondary school (zone 5, “secondary drop-outs”)
• remained at school in grade 7 or above but are not attending regularly (zone 6, “fading out”)

Although the OECD claimed that the categorisation was employed in accordance with UNESCO’s and UNICEF’s definition of out-of-school youth who are excluded from education opportunities, the third and last categories of youth aged 14–16 who have repeated a grade and who are not attending regularly were not justified by their approach. They even contradict the ideal of UNESCO and UNICEF that emphasises the rights of children having access to basic education. Thus those who are enrolled in school but not in the appropriate grade for the PISA target age (grade 7) or not attending regularly should not be considered as ‘out-of-school’.

Albeit inappropriate, by including these sub-groups ‘remained at school’ accounting for almost one third (27%) of the total out-of-school target population, ETS can increase access to youth samples that are more readily available. In Senegal, in particular, the percentage of 14–16-year-olds enrolled below grade 7 is 42% of the PISA-D out-of-school target population. Moreover, incorporating the student samples can also help to validate the out-of-school surveys, which use the same test items and contextual questionnaires as the school-based assessment. It is suspectable otherwise that there would be sufficient common items between Strand C and the main PISA instruments, as there is no reason to conceptualise out-of-school youth in terms of school-based concepts. Whereas through this approach, the OECD can achieve a minimum degree of comparability between the results of in school and out of school, which is critical for legitimating PISA as the universal metric for SDG 4. For example, the findings show that, in reading, less that 2% of 14–16-year-old ‘out-of-school’ youth achieved Level 2 (considered as the basic proficiency level indicated by SDG 4.1.1[c]), compared to 27.2% of 15-year-olds who took part in the school-based assessment; in mathematics, the percentage is 1.1% versus 12.2%.

Essentially, I am arguing that the OECD has sought to redefine the meaning of ‘out-of-school’ to ensure the experimental success of PISA-D, and in this way, to avoid resolving the challenges of reaching the most marginalised in the piloting countries. The UNESCO/UNICEF approach does not address the issue of defining current status vis-à-vis school in a study dedicated to identifying and persuading specifically 14–16-year-old out-of-school youth to take a test. The problem was indicated when Cambodia decided to opt out from the out-of-school component, even with ‘substantial interest’ at the outset. The country’s Capacity Building Plan raised concerns about the suitability and utility of the PISA-D out-of-school target population definition and its implementation:

Specifically, given the challenges with identifying a target population that was both meaningful for research purposes and relevant to national interests, the consensus within the Ministry of Education, Youth and Sports is that the implementation of Strand C in Cambodia would be too expensive and not have sufficient utility.

I suggest that this strategy of redefining what it means to be ‘out-of-school’, along with the use of ‘pilot’ to locate PISA-D within a process of investigative science to overcome resistance (Auld, Li and Morris 2020), illustrates the OECD’s self-serving tendency and the desire to expand its reach (Auld, Rappleye and Morris 2018; Li and Auld 2020). This misinterpretation of what can actually be inferred from the data also echoes a critique recently sounded regarding the release of the results of the PISA 2018 global competence assessment.

In the current world heavily affected by the global pandemic, although the OECD stresses the importance of including out-of-school youth, it has ignored them in its recent educational initiatives, such as the PISA Global Crises Module that aims to capture students’ learning experience during the lockdown. In this post, by challenging the way the OECD defines out-of-school youth and by highlighting its continual exclusion of this group from the recent activities, I question the legitimacy of the OECD via the attempt in PISA-D to position PISA as a barometer of inclusion and a reliable source of data for the monitoring of SDG 4. Countries having embarked on the development path indicated by PISA-D should treat the data with caution before formulating educational responses and allocating scarce resources.

About the author: Xiaomin Li is a doctoral candidate at the UCL Institute of Education. Her PhD thesis focuses on the OECD’s PISA for Development project. Email: xiaomin.li.15@ucl.ac.uk