By Andrés Sandoval-Hernández, University of Bath, and Diego Carrasco, Pontificia Universidad Católica de Chile

When UN Member States adopted the 2030 Agenda and its 17 Sustainable Development Goals (SDGs), there was not much discussion about how these goals were going to be measured. As we enter the Decade of Action, deciding on a measurement strategy for all SDGs and their targets has become a pressing issue.

We live in very challenging times. The rapid influx of immigrants, refugees and asylum seekers, along with increasing intolerance, social exclusion and feelings of alienation, extremism among young people, and the ongoing climate crisis, pose complex challenges. To face this global environment, we need information that enables us to think critically, connect our actions with their impacts, and act as empowered, active global citizens.

When looking specifically at SDG 4 for education, Target 4.7 asks Member States to “ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development.”

Today, the 60^th anniversary of the UNESCO Convention Against Discrimination in Education, we are reminded more than ever of the values of inclusion embodied in this target. While we say no to discrimination in education, let us say yes to inclusivity, respect for our differences and the right of all children to a quality education.

In this blog post we describe a recently developed strategy for assessing two indicators that embody tolerance, respect and sustainable development:

Indicator 4.7.4: Percentage of students by age group (or education level) showing adequate understanding of issues relating to global citizenship and sustainability.

Indicator 4.7.5: Percentage of 15-year-old students showing proficiency in knowledge of environmental science and geoscience.

Indicators 4.7.4 and 4.7.5 speak to empowering and enabling students to be active agents of positive change, while taking action to meet the other goals.

Using UNESCO Institute for Statistics (UIS) data, we are preparing an open-source, robust and easy-to-use document containing detailed technical guidelines for countries and other interested parties to collect the data necessary to produce the scales we discuss below.

Our measurement strategy is based on International Large-Scale Assessments (ILSAs) in education ((Sandoval-Hernández, Isac, & Miranda, 2019; Sandoval-Hernández & Carrasco, 2020)). In our view, ILSAs are a natural fit for assessing these particular thematic indicators because existing studies have already collected much of the relevant information. Our strategy includes a proposed conceptual framework, measurement models, a process to generate proficiency scores, and a method for establishing a threshold of ‘adequate’ and ‘satisfactory’ performance for Indicators 4.7.4 and 4.7.5, respectively.

This measurement strategy has been reviewed and endorsed by the UIS’ Technical Cooperation Group on the Indicators for SDG 4-Education 2030 (TCG), which is responsible for the development and maintenance of the thematic indicator framework for the follow-up and review of SDG 4. The scores are available on the UIS database.

A global content framework

We first identified a global content framework based on UIS data for defining and operationalizing Global Citizenship Education (GCED) and Education for Sustainable Development (ESD) (Hoskins, 2016; IBE, 2016). While there is no universal agreement on how to define or operationalize these concepts, it is possible to identify a set of guiding principles and themes.

We then carried out a mapping exercise to evaluate how to measure concepts in the framework using instruments and procedures of existing ILSAs. To do this, we identified the International Civic and Citizenship Study (ICCS) as the most valuable source of information for Indicator 4.7.4; and the Trends in Mathematics and Science Study (TIMSS) and the Programme for International Student Assessment (PISA) as the most informative for Indicator 4.7.5. These studies have the highest coverage of topics relevant to the SDGs, and a high potential to inform long-term monitoring. The results of this mapping exercise can be found here.

Producing scores

The analytical strategy to estimate the percentage of students who show ‘adequate’ and ‘satisfactory’ performance in each indicator had four main steps:

verifying the availability of observed responses to the items proposed by the mapping exercise,
testing the uni-dimensionality of the intended constructs,
fitting the corresponding measurement models to obtain scores for the categories and sub-categories of each indicator, and
establishing the cut-off points to identify ‘adequate’ or ‘satisfactory’ performance assuming a common measurement model.

A list of the categories and sub-categories of scores produced for Indicator 4.7.4, and a description of student knowledge at the cut-off points are shown in Figure 1.

Figure 1. Categories and descriptions for SDG 4.7.4

The percentage of students showing an ‘adequate’ understanding of issues related to global citizenship and sustainability (Indicator 4.7.4) – according to the scores and cut-off points – is shown in Figure 2. In this waffle plot, each square dot represents 1% of students reaching a certain standard (category or subcategory). As can be observed, there is an important variation in the proportion of students reaching the different standards both across countries and across standards. If we look at the cognitive standard, Latin American countries have lower proportions of students being able to make connections between processes related to global citizenship and sustainability and the legal and institutional mechanisms used to control them; while these proportions are generally higher in northern European and Asian countries (e.g. 19% and 32% in Dominican Republic and Peru vs 77% and 78% in Finland and Chinese Taipei).

Figure 2. Proportion of students reaching the SDG 4.7.4 standards in each country.

Note: Data is presented for all countries for which data is available in ICCS 2016. BFL= Belgium (Flemish); BGR= Bulgaria; CHL= Chile; TWN= Chinese Taipei; COL= Colombia; HRV= Croatia; DNK= Denmark; DOM= Dominican Republic; EST= Estonia; FIN= Finland; HKG= Hong Kong SAR; ITA= Italy; KOR= Korea, Republic of; LVA= Latvia; LTU= Lithuania; MLT= Malta; MEX= Mexico; NLD= Netherlands; DNW= North Rhine-Westphalia; NOR= Norway; PER= Peru; RUS= Russian Federation; SVN= Slovenia; SWE= Sweden.

As with global citizenship and sustainability, the results for Indicator 4.7.5 for environment science and geoscience, also show interesting variations both across countries and standards. A list of the categories and sub-categories for which scores were produce for Indicator 4.7.5 and a description of what students know or can do at the established cut-off points and are shown in Figure 3.

Figure 3. Categories and descriptions for SDG 4.7.5

The percentage of students showing ‘proficiency’ in environmental science and geoscience (Indicator 4.7.5) according to the scores and cut-off points is shown in Figure 4. As in Figure 2, each square dot represents 1% of students reaching a given standard.

When looking at the cognitive standard, Asian countries report the highest proportions of students who are able to apply and communicate concepts related to environmental science in everyday situations (e.g. Singapore, 59%). However, this pattern does not hold for the non-cognitive standards, where Botswanan and Kuwaiti students are the ones who report the highest enjoyment and confidence in learning science (51% and 39% respectively). Nevertheless, we insist that the real value of these measures is that given their reliability, relevance and timeliness, they can be used to inform the development of strategies to reach the targets included in Target 4.7. A full description of the thresholds used to set these standards and the items and methodology used to produce the respective scores can be consulted here.

Figure 4. Proportion of students reaching the SDG 4.7.5 standards in each country

Note: Data is presented for all countries for which data is available in TIMSS 2015. AAD= Abu Dhabi, UAE; ARM= Armenia; AUS= Australia; BHR= Bahrain; BWA= Botswana; ABA= Buenos Aires, Argentina; CAN= Canada; CHL= Chile; TWN= Chinese Taipei; ADU= Dubai, UAE; EGY= Egypt; ENG= England; GEO= Georgia; HKG= Hong Kong, SAR; HUN= Hungary; IRN= Iran, Islamic Rep. of; IRL= Ireland; ISR= Israel; ITA= Italy; JPN= Japan; JOR= Jordan; KAZ= Kazakhstan; KOR= Korea, Rep. of; KWT= Kuwait; LBN= Lebanon; LTU= Lithuania; MYS= Malaysia; MLT= Malta; MAR= Morocco; NZL= New Zealand; NOR= Norway; OMN= Oman; COT= Ontario, Canada; QAT= Qatar; CQU= Quebec, Canada; RUS= Russian Federation; SAU= Saudi Arabia; SGP= Singapore; SVN= Slovenia; ZAF= South Africa; SWE= Sweden; THA= Thailand; TUR= Turkey; ARE= United Arab Emirates; USA= United States.

Conclusions, limitations and suggestions for monitoring these indicators

We believe that studies like TIMSS, ICCS and PISA are well suited for providing at least a proxy measurement of Indicators 4.7.4 and 4.7.5. These ILSAs provide high coverage for the GCED and ESD themes, incorporate these topics naturally in their frameworks, collect comparable data consistently (allowing long-term monitoring), and have unrivalled data quality assurance mechanisms in place (ensuring data accuracy, validity and comparability).

It is, however, important to consider the limitations of this measurement strategy. For example, the data is confined to a specific level of education or student population (e.g. Grade 8 for ICCS and TIMSS; 15-year-old students for PISA). Another limitation is country coverage. The information available in the last cycles of TIMSS and ICCS allowed us to produce scores for 60 countries. While this is a significant number of countries, it is important to acknowledge that two-thirds of UN members do not participate in these studies.

Nevertheless, we believe that with a coordinated effort and support by all stakeholders, many more countries can collect the data for this measurement strategy so that we can work toward the elimination of discrimination in schools and create a more equal and just society.

Sources:

By Silvia Montoya, Director, UNESCO Institute for Statistics (UIS) and Manos Antoninis, Director, Global Education Monitoring Report

New global indicator will provide a simple, comprehensive measure of progress towards the education goal, SDG4.

The Sustainable Development Goal (SDG 4) for education recognizes that all children deserve, and have the right to, a quality education. Over the last three decades, enrollment has risen to historic highs, though school disruptions and the economic implications of COVID-19 will offset some of these gains. But enrollment is only a part of what children need. For children to be fully prepared for the future, they need to complete their education, and emerge having learnt at least the basics. The new global indicator will combine all these critical factors to provide a snapshot of progress towards SDG 4.

Completing and learning are critical elements of a quality education

Unfortunately, in some of the poorest regions where children are most in need of a high-quality education to get ahead, poor learning outcomes often result in higher drop-out rates with large numbers of children not completing school at all – or completing it when more than five years older than the intended graduation age for that level. In sub-Saharan Africa, for example, where 82% of primary aged children are enrolled in school at the right grade for their age, just 62% graduate from this level on time. When children don’t finish school, it is hard, if not impossible, for any more learning to happen.

Even where education is free, poor families still pay for books and uniforms, and there is the perceived “opportunity cost” of lost income or help with household chores, while children are in school. If parents don’t see a pay-off from their investment in education, children can be pulled out of school before completing a level, or when transitioning between levels, from say, primary school, to lower secondary.

Introducing the new indicator: Prepared for the Future

To underline the need for countries to prioritize school completion, the UNESCO Institute for Statistics (UIS), working through the Global Alliance to Monitor Learning (GAML), is proposing a new, holistic, indicator that will track both completion and learning combined. The aim is to ensure that global leaders and education policy makers have the evidence they need to zero in on where they stand on their SDG 4 commitments. With ten targets and 42 indicators in the monitoring framework for SDG 4, some might argue that it is hard to quickly grasp where countries stand in their progress towards the goal. With so many touch-points, it risks calls for change being watered down. This new indicator will hopefully answer these concerns, providing a simple rallying reference point for all education actors to lobby for improvement, ensuring all children are prepared for the future.

Continue reading →

Data for Sustainable Development

UNESCO Institute for Statistics Blog

Tag Archives: Sustainable Development Goals

Ensuring the Next Generation are Global Citizens and Stewards of Sustainable Development: Why Monitoring SDG 4.7 is Essential