Methods Summary and Data Tables

Part 4

4.1 Interpreting the SDG Index and Dashboards results

The Sustainable Development Report 2021 describes countries' progress towards achieving the SDGs and indicates areas requiring faster progress. The overall SDG Index score and scores for individual SDGs can be interpreted as a percentage of optimal performance. The difference between a country's score and 100 is therefore the distance, in percentage points, that needs to be overcome to reach optimum SDG performance. The same basket of indicators is used for all countries to generate the SDG Index score and rankings.

Substantial differences in rankings may be due to small differences in aggregate SDG Index scores. Differences of two or three places between countries should not be interpreted as "significant", whereas differences of 10 places or more can show a meaningful difference. For details, see the statistical audit by Papadimitriou et al. (2019) conducted on behalf of the Joint Research Centre (JRC) of the European Commission.

The SDG Dashboards provide a visual representation of each country's performance on the 17 goals. The "traffic light" color scheme (green, yellow, orange, and red) illustrates how far a country is from achieving a particular goal. As in previous years, the Dashboards and country profiles for OECD countries include additional metrics that are not available for non-OECD countries.

The SDG Trend Dashboards indicate whether a country is on track to achieve a particular goal by 2030, based on recent performance on a given indicator. Indicator trends are then aggregated at the goal level to give a trend indication of how the country is progressing towards that SDG.

This section provides a brief summary of the method used to compute the SDG Index and Dashboards. A detailed methodology paper is also accessible online (Lafortune et al., 2018). The European Commission Joint Research Centre (JRC) conducted an independent statistical audit of the report's methodology and results in 2019, appraising the conceptual and statistical coherence of the index structure. The detailed statistical audit report and additional data tables are available on our website: www.sdgindex.org

4.2 Changes to the 2021 edition and main limitations

Changes to the 2021 SDG Index and Dashboards

The 2021 SDG Index covers 165 countries, one fewer country than last year due to missing data on Comoros. Several new indicators have been introduced to address key data gaps (table 4.1). This table also identifies indicators that have been replaced or modified due to changes in the methodology and estimates produced by data providers. The data for this year's edition were extracted between February and April 2021.

Table 4.1 | New indicators and modifications

Table 4.1 | New indicators and modifications

Source: Authors

Limitations and data gaps

Due to changes in the indicators and some refinements to the methodology, SDG Index rankings and scores cannot be compared with the results from previous years. Section 2 provides an assessment of trends over time, calculated retroactively using the data from the indicators included in this year's report.

Most global indicators are not yet available for 2020 due to time lags in data reporting. The impact that COVID-19 has had on the SDGs is therefore not fully captured in this year's SDG Index and Dashboards. Table 4.2 displays the list of indicators by percentage of 2020 data points for countries. For certain international indices, the date of publication may not represent the year of reference (the year when the data were collected).

Table 4.2 | Indicators by percentage of 2020 data points

Table 4.2 | Indicators by percentage of 2020 data points

Source: Authors' analysis

Despite our best efforts to identify data for the SDGs, several indicator and data gaps persist at the international level (table 4.3). Governments and the international community must increase investments in SDG data and monitoring systems and build strong data partnerships to support informed SDG decisions and strategies.

Table 4.3 | Major indicator and data gaps for the SDGs

Table 4.3 | Major indicator and data gaps for the SDGs

Source: Authors

To ensure maximum comparability, we only use data from internationally comparable sources. The providers of this data may adjust national figures to ensure international comparability. As a result, some data points presented in this report may differ from those of national statistical offices or other national sources. Moreover, the length of validation processes can lead to significant delays in publishing some data from international organizations. National statistical offices may therefore have more recent data for some indicators than presented in this report.

4.3 Methodology (overview)

The Sustainable Development Report 2021 (SDR2021) provides a comprehensive assessment of how close countries are to achieving the SDG targets based on the most up-to-date data available for all 193 UN Member States. This year's report includes 91 global indicators as well as 30 additional indicators for OECD countries, due to better data coverage.

Below is an overview of our methodology for indicator selection, normalization and aggregation and for generating indications on trends. Raw data, additional data tables, and sensitivity tests are available online.

A. Data selection

Where possible, the SDR2021 uses official SDG indicators endorsed by the UN Statistical Commission. Where insufficient data are available for an official indicator, or to close data gaps, we include other metrics from official and unofficial providers. Five criteria for indicator selection were used to determine suitable metrics for inclusion in the report:

1. Global relevance and applicability to a broad range of country settings.

2. Statistical adequacy: the indicators selected represent valid and reliable measures.

3. Timeliness: the indicators selected are up to date and published on a reasonably prompt schedule.

4. Coverage: data must be available for at least 80 percent of UN Member States with a population of more than a million people.1

5. Capacity to measure distance to targets (optimal performance can be determined).

Data sources

The data included in the SDR2021 come from a mix of official and non-official data sources. Most of the data (around two thirds) is developed by international organizations (World Bank, OECD, WHO, FAO, ILO, UNICEF, other), which have extensive and rigorous data validation processes. Other less traditional statistical sources used (accounting for around a third of our data) include household surveys (Gallup World Poll), data from civil society organizations and networks (among others, Oxfam, Tax Justice Network, World Justice Project, Reporters without Borders) and peer-reviewed journals (to track international spillovers, for example). The full list of indicators and data sources is available online.

B. Missing data and imputations

The purpose of the SDR2021 is to guide countries in discussing their current SDG priorities based on available and robust data. To minimize biases from missing data, the SDG Index only includes countries that have data for at least 80 percent of the variables included in the global SDG Index. The list of countries not included in the SDG Index due to insufficient data availability is presented in table 4.4 below. We include all UN Member States in the SDG Dashboards and country profiles, which also indicates gaps in available SDG data for countries.

Considering that many SDG priorities lack widely accepted statistical models for imputing country-level data, we generally did not impute or model any missing data, apart from a few exceptional circumstances. The list of indicators where imputations are performed is available online.

Table 4.4 | Countries excluded from the 2021 SDG Index due to insufficient data

Table 4.4 | Countries excluded from the 2021 SDG Index due to insufficient data

Source: Authors' analysis

C. Method for constructing the SDG Index and Dashboards

Calculating the SDG Index comprises three steps: (i) establish performance thresholds and censor extreme values from the distribution of each indicator; (ii) rescale the data to ensure comparability across indicators (normalization); (iii) aggregate the indicators within and across SDGs.

Establishing Performance thresholds

To make the data comparable across indicators, each variable was rescaled from 0 to 100, with 0 denoting worst possible performance and 100 describing optimum performance. Rescaling is usually very sensitive to the choice of limits and extreme values (outliers) at both tails of the distribution. The latter may become unintended thresholds and introduce spurious variability in the data. Consequently, the choice of upper and lower bounds can affect the relative ranking of countries in the index.

The upper bound for each indicator was determined using a five-step decision tree:

1. Use absolute quantitative thresholds in SDGs and targets: e.g., zero poverty, universal school completion, universal access to water and sanitation, full gender equality.

2. Where no explicit SDG target is available, apply the principle of "leave no one behind" to set the upper bound to universal access or zero deprivation.

3. Where science-based targets exist that must be achieved by 2030 or a later date, use these to set the 100 percent upper bound (e.g., zero greenhouse gas emissions from CO₂ as required by no later than 2050 to stay within the 1.5°C target, or 100 percent sustainable management of fisheries).

4. Where several countries already exceed an SDG target, use the average of the top 5 performers (e.g., child mortality).

5. For all other indicators, use the average of the top performers.

These principles interpret the SDGs as "stretch targets" and focus attention on the indicators on which a country is lagging behind. The lower bound was defined at the 2.5th percentile of the distribution. Each indicator distribution was censored, so that all values exceeding the upper bound scored 100, and values below the lower bound scored 0.

Normalization

After establishing the upper and lower bounds, variables were transformed linearly to a scale between 0 and 100 using the following rescaling formula for the range [0; 100]:

where x is the raw data value; max/min denote the upper and lower bounds, respectively; and x' is the normalized value after rescaling.

The rescaling equation ensured that all rescaled variables were expressed as ascending variables (i.e., higher values denoted better performance). In this way, the rescaled data became easy to interpret and compare across all indicators: a country that scores 50 on a variable is halfway towards achieving the optimum value; a country with a score of 75 has covered three quarters of the distance from worst to best.

Weighting and Aggregation

Several rounds of expert consultations on earlier drafts of the SDG Index made it clear that there was no consensus across different epistemic communities on assigning higher weights to some SDGs over others. As a normative assumption, we therefore opted to give fixed, equal weight to every SDG, reflecting the commitment of policymakers to treat all SDGs equally as part of an integrated and indivisible set of goals. To improve their SDG Index score, countries need to place attention on all goals, albeit with a particular focus on those they are furthest from achieving and where incremental progress might be expected to be fastest.

To compute the SDG Index, we first estimate scores for each goal using the arithmetic mean of indicators for that goal. These goal scores are then averaged across all 17 SDGs to obtain the SDG Index score. Various sensitivity tests are made available online: including comparisons of arithmetic mean versus geometric mean, and Monte-Carlo simulations at the Index and Goal level. Monte-Carlo simulations call for prudence in interpreting small differences in the Index scores and rankings between countries, as those may be sensitive to the weighting scheme.

Dashboards

We introduced additional quantitative thresholds for each indicator to group countries in a "traffic light" table. Thresholds have been established using statistical techniques and through various rounds of consultations with experts conducted since 2016.

Averaging across all indicators for each SDG risks masking areas of policy concern if a country is performing well on most indicators for a goal but faces serious shortfalls on one or two of its metrics (often called the "substitutability" or "compensation" issue). This applies particularly to high-income and upper-middle-income countries that have made significant progress on many SDG dimensions but may face serious shortfalls in relation to individual variables. As a result, the SDG Dashboards focus exclusively on the two variables on which a country performs worst, with the added rule that a red rating is applied only where the country scores red on both of these worst-performing indicators. Similarly, to score green, both indicators must be green. More details on the construction of the Dashboards are accessible online.

SDG Trends

Using historic data, we estimate how fast a country has been progressing towards an SDG and determine whether – if extrapolated into the future – this pace will be sufficient to achieve the SDG by 2030. For each indicator, SDG achievement is defined by the green threshold set for the SDG Dashboards. The difference in percentage points between the green threshold and the normalized country score denotes the gap that must be closed to meet that goal. To estimate trends at the indicator level, we calculated the linear annual growth rates (annual percentage improvements) needed to achieve the target by 2030 (from 2015–2030), which we compared to the average annual growth rate over the most recent period (usually 2015–2019). Progress towards achievement on a particular indicator is described using a 4-arrow system (figure 4.1). Figure 4.2 illustrates the methodology graphically.

Figure 4.1 | The Four‑arrow system for denoting SDG trends

Figure 4.1 | The Four‑arrow system for denoting SDG trends

Figure 4.2 | Graphic representation of the SDG trends methodology

Figure 4.2 | Graphic representation of the SDG trends methodology

Since projections are based on past growth rates, over several years, a country may have observed a decline in performance over the past year (for instance due to the impact of COVID-19) but still be considered as being on track. This methodology emphasizes long-term structural changes over time since the adoption of the SDGs in 2015, with less emphasis on annual changes that may be cyclical or temporary.