On this page
The methodology for the Blavatnik Index of Public Administration is inspired by that of the previous International Civil Service Effectives (InCiSE) Index. This page provides a brief overview of the methodology, for more detail visit the methodology section.
Defining public administration, selection of sources and metrics
Owing to the range of different sources used in the calculation of the Index, a pragmatic approach has been taken to define the unit(s) of analysis as the executive/ administration activities and characteristics of national level governments. The Index also explicitly excludes the measurement of policy outcomes (e.g. life expectancy, literacy rates, unemployment, economic growth etc), and wider aspects of public governance (the functioning of the legislature, judiciary, rule of law, media/academic freedom etc).
The Index is intended to be a practical tool that helps officials, politicians and others to understand how different countries’ public administrations compare. This goal has guided our approach to reviewing and selecting data sources for inclusion. Sources have been selected for inclusion if they meet four criteria:
- Open access – the data source and its methodology must be published online in a free-to-access form.
- Actionable – the data must measure some quality or component that officials or ministers can act on to improve performance/practice.
- Quantifiable – the data must be something that can be represented numerically either as a quantity or an ordinal scale.
- Recency – the data should have been updated after 1 January 2019.
A total of 36 sources were reviewed in-depth for inclusion in the Index, of which 17 have been used to calculate the 2024 Index results, details of the sources included are provided on page 26. The data comes from a mix of multilateral institutions (such as the United Nations or the World Bank), academic projects (such as the University of Gothenburg), and non-government researchers (such as the Data 4 Development Network). The source data can be categorised into three different types:
- Statistical data – either official statistics or administrative data from governments.
- Assessment data – assessments of government policies and practices, either compiled by external experts or responses by government officials to surveys.
- Opinion data – responses to rating exercises or opinion surveys by professional experts, or general perception surveys of business and the public at large.
In addition to these criteria, sources were also reviewed for their country coverage. In contrast to the previous InCiSE Index which covered 38 OECD/EU countries in its 2019 edition, the Blavatnik Index of Public Administration has sought from the outset to develop a more global coverage.
From these 17 sources, 82 metrics were identified for inclusion in the Index. After selection each metric was allocated to one of the 20 themes defined in the Index’s conceptual framework. While some metrics are strongly aligned with only a single theme (e.g. effective enforcement of regulations with the regulation theme), some metrics align with multiple themes and in this case a pragmatic decision has been taken to ensure broad coverage of the Index’s framework. For example, the metric on the use of innovative technologies and practices by tax administrations has been included in the innovation theme rather than the tax administration theme since there were several other metrics for tax administration and only two others for the innovation theme. Ultimately, at least one metric has been identified for 16 of the 20 themes defined in the Index’s framework.
Country coverage
While the 17 sources used by the Index were in part selected for their broad country coverage, each source has different coverage and only one country has data for all 82 of the metrics used to calculated the Index. Like many other comparative analyses of international data, the Index must actively consider how it handles missing data.
A simple approach to country coverage would be to just use the overall percentage of metrics a country has data for. However, the metrics that make up the Index are not evenly distributed across the framework. Some themes have ten or more metrics while others have just one or two metrics. A data coverage algorithm, adapted from the chi-square test, that assesses not only the amount of data a country has but how it is spread across the Index framework has been used to give each country a data coverage score. Countries were selected for inclusion if they had a data coverage score of at least half the theoretical maximum (i.e. if they had data for all 82 metrics) and at least two-thirds of metrics overall. To help users interpret country data coverage, countries have also been given a grade of A to D based on both the data coverage score and the overall percentage of metrics.
Data processing, aggregation and calculation of the Index
The production of the Index follows a five step process: (i) collation, (ii) extraction and standardisation, (iii) country selection, (iv) normalisation, and (v) aggregation.
First, the data for each source is downloaded, collated and catalogued. Each source is then processed to extract the data for each metric in a standardised format. Country coverage is then determined through the data coverage assessment described above.
Once the country coverage has been determined, the source data is subset to the countries of interest and normalised. Normalisation converts the values for each metric to a common 0-1 scale, where 0 represents the ‘lowest’ score of the observed data for the metric and 1 represents the ‘highest’ score of the of the observed data for the metric. For the majority of metrics the maximum value represents the ‘highest’ score, but in some metrics the source data needs to be inverted (i.e. where lower scores indicate better performance, such as the level of tax arrears) or the transformation is based on a reference point (e.g. for the gender equality measures a score of 1 represents the country closest to women being 50% of the workforce).
After normalisation the data is then aggregated according to the Index’s data model. As described above, the 82 metrics extracted from the source data are allocated to one of the framework’s 20 themes (which themselves are organised in 4 domains). As an intermediate tier in the data model, metrics that measure similar concepts within a theme are grouped together as an indicator. For example, the eight metrics that contribute to the openness and communications theme are grouped into three indicators: right to information; open government; and engagement and feedback. The aggregation of data is undertaken for each tier by calculating the unweighted (mean) average of its constituent parts – an indicator’s score is the average of the metrics assigned to it; each theme’s score is the average of the indicators within the theme; each domain’s score is an average of its four themes; and, finally, the Index itself is an average of the score for each of the four domains. To keep the methodology simple and transparent, in addition to not weighting the data, there is no active imputation of missing data at any stage of aggregation.
Mapping of data sources to the Index’s domains and themes
The Blavatnik Index of Public Administration 2024 draws on 17 separate data sources, set out below is how these sources contribute to each of the Index’s four domains and 16 themes.
Strategy and Leadership domain
- Strategic capacity
- Bertelsmann Transformation Index 2024
- Sustainable Governance Indicators 2022
- Openness and communications
- Bertelsmann Transformation Index 2024
- GovTech Maturity Index 2022
- Rule of Law Index 2023
- Sustainable Governance Indicators 2022
- Integrity
- Bertelsmann Transformation Index 2024
- Global Corruption Barometer 2019-2021
- Global Data Barometer 2021
- Quality of Government Expert Survey 2020
- Rule of Law Index 2023
- Varieties of Democracy Dataset v14 (2023)
- Innovation
- Bertelsmann Transformation Index 2024
- GovTech Maturity Index 2022
- International Survey of Revenue Administration 2021
Public Policy domain
- Policy making
- Bertelsmann Transformation Index 2024
- Sustainable Governance Indicators 2022
- Regulation
- Rule of Law Index 2023
- Crisis and risk management
- Global Cybersecurity Index 2020
- Sendai Framework Monitor 2023
- Use of data
- Global Data Barometer 2021
- Open Data Inventory 2022
- PARIS21 Statistical Capacity Monitor 2020
National Delivery domain
- System oversight
- Bertelsmann Transformation Index 2024
- Sustainable Governance Indicators 2022
- Digital Services
- GovTech Maturity Index 2022
- Tax administration
- Doming Business 2020
- International Survey of Revenue Administration 2021
- Border services
- Logistics Performance Index 2022
People and Processes domain
- Diversity and inclusion
- Gender Statistics Database 2023
- ILOSTAT 2023
- International Survey of Revenue Administration 2021
- Varieties of Democracy Dataset v14 (2023)
- HR management
- Quality of Government Expert Survey 2020
- Varieties of Democracy Dataset v14 (2023)
- Procurement
- Global Data Barometer 2021
- Technology and workplaces
- GovTech Maturity Index 2022
Data source reference list
Source | Published | Type of data collection | Reference year(s) | Website |
---|---|---|---|---|
Bertelsmann Transformation Index 2024 | Bertelsmann Stiftung | Opinion (expert) | 2023 | https://bti-project.org/ |
Doing Business 2020 | World Bank | Opinion (business) | 2020 | https://www.worldbank.org/en/businessready/doing-business-legacy |
Gender Statistics Database | European Institute for Gender Equality | Statistical data (official statistics) | 2023 | https://eige.europa.eu/genderstatistics/dgs |
Global Corruption Barometer | Transparency International | Opinion (general public) | 2019-2021 | https://www.transparency.org/en/gcb |
Global Cybersecurity Index 2020 | International Telecommunications Union | Assessment (self-assessment) | 2020 | https://www.itu.int/epublications/publication/D-STR-GCI.01-2021-HTM-E/ |
Global Data Barometer | Data for Development Network | Assessment (external) | 2021 | https://globaldatabarometer.org/ |
GovTech Maturity Index 2022 | World Bank | Assessment (self-assessment) | 2022 | https://www.worldbank.org/en/programs/govtech/gtmi |
ILOSTAT | International Labor Organisation | Statistical data (official statistics) | 2023 | https://ilostat.ilo.org/ |
International Survey of Revenue Administration 2021 | CIAT, IMF, IOTA and OECD | Statistical data (administrative data return) | 2021 | https://data.rafit.org/ |
Logistics Performance Index | World Bank | Opinion (business) | 2022 | https://lpi.worldbank.org |
Open Data Inventory 2022-23 | Open Data Watch | Assessment (external) | 2022 | https://odin.opendatawatch.com/ |
Quality of Government Expert Survey 2020 | Quality of Government Institute, University of Gothenburg | Opinion (expert) | 2020 | http://qog.pol.gu.se |
Rule of Law Index 2023 | World Justice Project | Opinion (expert and general public) | 2016-2022 | https://worldjusticeproject.org/rule-of-law-index/ |
Sendai Framework Monitor | UN | Assessment (self-assessment) | 2017-2023 | https://unstats.un.org/sdgs/dataportal |
Statistical Capacity Monitor | PARIS21 (UN, EC, OECD, IMF and WB) | Assessment (expert) | 2019-2020 | https://statisticalcapacitymonitor.org/ |
Sustainable Governance Indicators 2022 | Bertelsmann Stiftung | Opinion (expert) | 2022 | http://sgi-network.org |
Varieties of Democracy dataset version 14 | V-Dem Institute, University of Gothenburg | Opinion (expert) | 2023 | https://www.v-dem.net |