Fit for purpose household survey tools in the SDG era
This NORRAG Highlights post is published in connection with “SDG4 Data Week: Understanding the Monitoring of SDG 4: Targets, Actors, Data and Resources”, held on 11 and 13 July 2018 at the Graduate Institute, Geneva with the support of the Education Network of the Swiss Agency for Development and Cooperation. In this post, Sheena Bell discusses recent developments for education researchers in monitoring SDG4 with household surveys. This post also explains new innovations in education data available in the latest round of the Multiple Indicator Cluster Surveys, developed by UNICEF. Sheena is currently an Education Specialist with the UNICEF Regional Office for Europe and Central Asia, and from 2010-2015, a focal point for household survey analysis with the UNESCO Institute for Statistics.
The principle of leave no one behind underpins the SDG agenda (UN 2015). If we are to realize progress toward inclusive and equitable quality education for every child, particularly the most marginalized, we need comprehensive, comparable, and disaggregated information on who they are, where they live, and what barriers they face. Better data is, in fact, an SDG target (17.18) aiming toward the “full disaggregation” of all relevant SDG indicators. This is especially important for education. SDG target 4.5 focuses on eliminating gender disparities in education and ensuring equal access to the most vulnerable, including persons with disabilities, indigenous peoples and children in vulnerable situations. However, analysis by UIS reveals substantial gaps in the crucial disaggregated data to monitor SDG4 globally (2016).
Household surveys are a rich source of data for monitoring SDG4. They complement the two other main sources: administrative data (such as school records) and learning assessment data. Unlike these two sources, survey data collected in the home cover the demand side of education, including both children in and out of school and household members. Household survey data also typically include a wide range of individual and socio-economic characteristics of respondents, providing a more nuanced picture of which children and households fall through the cracks of the education system. One can use an example from Serbia: administrative data from UIS show that in Serbia 99% of primary and lower secondary age children and 88% of upper secondary age adolescents are in school. A Multiple Indicator Cluster Survey (MICS), developed by UNICEF, undertaken in Serbia Roma settlements tells a different story. Primary-school age enrollment is only at 85% and secondary-age enrollment at a dismal 36%, for the same academic year, enabling us to pinpoint the high education exclusion for this group (Statistical Office of the Republic of Serbia and UNICEF 2014). Certainly, household surveys have disadvantages – they occur only every few years, are costly, sample based and omit children not living in households (such as those in institutions, or refugee camps). But despite these limitations, they remain a powerful tool to shed light on the education experience of some of the most vulnerable children.
Household surveys are increasingly accessible for the education practitioner and researcher. For the practitioner, key education indicators from international household surveys such as the Multiple Indicator Cluster Survey (MICS) developed by UNICEF, Demographic and Health Survey (DHS) supported by USAID, the Living Standards Measurement Survey (LSMS) led by the World Bank, are available online in survey reports. These surveys are conducted in the countries with the relevant national statistical offices and line ministries. Education indicators are also compiled in major publications and databases, such as the UNESCO Institute for Statistics’ SDG4 database, UNESCO GEMR’s WIDE database, World Bank EdStats and UNICEF Education database. Data extraction is increasingly efficient through the availability of application program interface (API) functionality in UIS and World Bank databases.
However, what is published represents only a fraction of what is possible. For the researcher, datasets from the MICS, DHS and LSMS surveys can be downloaded for free. With a little effort, country datasets from these survey programs can be standardized and rendered comparable, by applying international indicator definitions and ISCED (International Classification of Education) (UIS 2011). These standardized surveys can then be combined into a master database to calculate a range of interesting indicators related to SDG4 and beyond.
Household surveys designed for the SDG era: new education data available in MICS6
The SDG focus on equity has resulted in new household survey methodologies and tools (UIS 2017). One prominent example is the latest round of MICS (MICS6), which includes data to monitor 15 SDG goals and 34 SDG indicators. Sixty MICS6 surveys are planned in 54 countries between 2018 and 2020.
Crucially, MICS6 includes several new modules that provide a unique source of data to monitor SDG4 globally. These modules go beyond the typical school attendance and attainment questions, and measure the home learning context and parental involvement in education, ICT and mass-media activities, child functioning (to capture children with disabilities) and early-grade learning in reading and mathematics. Taken together, these modules capture indicators for monitoring five SDG4 targets (4.1, 4.2, 4.3, 4.4, 4.5 and 4.6). Let’s look deeper into each of these new modules.
To fill a crucial data gap in earlygrade literacy and numeracy (SDG indicator 4.1.1.a), MICS6 includes a new module called Foundational Learning Skills which measures basic literacy and numeracy skills for children aged 7-14 years (primary grades 2-3). The reading assessment includes reading a short story calibrated to an appropriate textbook in the country (typically at grade 2), and features questions on both literal and inferential comprehension. In the numeracy assessment, the module covers number recognition, number discrimination, addition, and pattern recognition and completion. More information can be found in the methodological paper and the related tools (UNICEF 2017). The first results are already out. The rate of 7-14-year-olds with foundational reading skills in DPR Korea is 95%, falling slightly to 82% for foundational numeracy skills. In Sierra Leone, only about 16% and 12% demonstrated foundational reading and numeracy skills, respectively, for the same age group. The rates were similar between boys and girls, in both countries. However, in Sierra Leone, the data show a stark urban-rural divide in the attainment of foundational early-grade skills: in literacy there was a 25 percentage point gap (30% urban vs. 5% rural), and in numeracy a 17 percentage point gap (22% urban vs. 5% rural) (Central Bureau of Statistics of the DPR Korea and UNICEF 2017, and Statistics Sierra Leone, 2018).
Internationally-comparable data on education demand and the home learning context are scarce. The last time comparative education researchers had such household survey data was from the DHS EdData surveys. These surveys provided nationally-representative data on school absenteeism, household expenditures and other contributions to schooling, and parents’/guardians’ perceptions of schooling, among other topics. These USAID-supported surveys were implemented largely in the 1990s and 2000s, in only a small number of countries in Africa (USAID 2016).
In contrast, the new MICS6 Parental involvement data is likely to be widely available in many of the more than 60 planned MICS6 surveys. It contains useful data on home-based activities (including questions about the availability of reading materials, reading practices and languages, and homework) as well as school-based activities (whether parents have information on school performance, discuss progress with teachers, attend meetings, school events, and school management processes) (UNICEF 2017).
ICT skills are also featured in MICS6, linked to the global indicator SDG 4.4.1. Nine ICT skills among youth and adults 15-49 years are measured. These include self-reports on whether the person has written a computer program, transferred a file, or found, downloaded, installed and configured software within the last three months. Recent data from the Iraq MICS6 show that only 8% of adolescent girls 15-19 years old can perform at least one of the nine ICT skills (See survey results snapshot here).
Lastly, children with disabilities are one of the most marginalized groups and largely left behind in terms of education access and learning. What’s more, these children are often invisible in data collection. Even if identified, definitions vary and questionnaires tend toward an overly medical categorization (e.g. diagnosis) which does not provide information on functional difficulties, which are much more informative to design services and support (UNICEF 2013 and UIS 2018). MICS6 features Child Functioning modules aimed at children aged 2-17 years to fill this data gap. They were developed by UNICEF and the Washington Group on disability statistics and are in line with the United Nations Convention on the Rights of Persons with Disabilities and the International Classification of Functioning for Children and Youth (ICF-CY). The module assesses functional difficulties in the domains of speech and language, hearing, vision, learning, mobility and motor skills, and emotions (UNICEF 2016). The questionnaire can be downloaded online. What’s more, the data on children with disabilities can be used to do further analysis cross-analyzing the modules discussed above (Foundational Learning, ICT Skills, Parental Involvement in Education) as well as the data on school attendance and attainment.
So far, four countries have published MICS6 results: Democratic People’s Republic of Korea, Iraq, Lao PDR, and Sierra Leone (See the latest data available here). In 2019 most of the sixty planned MICS6 surveys will go to the field or publish the reports and datasets. The increased availability of household survey data on education is itself notable. However, if it is going to have any appreciable impact on our understanding of the situation of children in relation to SDG4, then it is our challenge to ensure these data are widely analyzed, interpreted and used both in research, and policy and program decision-making.
Contribute: The NORRAG Blog provides a platform for debate and ideas exchange for education stakeholders. Therefore, if you would like to contribute to the discussion by writing your own blog post please visit our dedicated contribute page for detailed instructions on how to submit.
Disclaimer: NORRAG’s blog offers a space for dialogue about issues, research and opinion on education and development. The views and factual claims made in NORRAG posts are the responsibility of their authors and are not necessarily representative of NORRAG’s opinion, policy or activities.