Last weekend (30-31st October 2009), the American Political Science Association (APSA) organized a conference on Democratic Audits and Governmental Indicators, hosted by the Goldman School of Public Policy at the University of California, Berkeley. This meeting was held at the initiative of Professor Henry Brady, President-elect of APSA and Dean of the Goldman School, and it is the start of developing a year-long APSA taskforce on this topic. The meeting generated lively debate among leading scholars, experts, and representatives from some of the major multilateral organizations and advocacy groups engaged in generating and using governance indicators.
The focus of the APSA initiative is timely since the third wave era of democratization has seen a burgeoning array of diagnostic tools designed to monitor the quality of democratic governance. Political indicators are now widely used by the international community, by national governments and by advocacy groups to evaluate needs and determine policy priorities, to highlight problems and identify benchmark practices, and to evaluate the effectiveness of programmatic interventions.
The most commonly-used indicators focus on issues such as the existence of democracy, civil liberties and political rights, the extent of 'good governance', respect for human rights, perceptions of corruption, the degree of women's empowerment, levels of civic engagement, the extent of conflict, the distribution of social capital, and many other related topics.
The expansion of indicators has been accompanied by important gains in the level of conceptual sophistication, methodological transparency, scope, and geographic coverage of all these measures. Nevertheless there remain considerable concern that the proliferation of indices has not been accompanied by sufficient quality checks, for example through the transparent publications of technical notes about how each of these measures are constructed, the degree of measurement error, and the sources of funding and accountability.
Types of indicators
Four types of indicators have become common:
- Elite-level perceptual judgments are an important source of information. Organizations commonly use standardized questionnaires distributed to 'expert' sources, typically including scholars and researchers, country analysts, journalists, lawyers, business executives, independent consultants, human rights observers, and NGO staff. This approach is exemplified by Freedom House's process employing teams of researchers to assess the state of civil liberties and political rights worldwide, by Transparency International use of experts to construct the Corruption Perceptions Index, and by Reporters without Borders survey used to generate its annual Press Freedom index.
- Many indicators are also compiled from official data based on administrative records collected by public officials, national statistical offices, and multinational organizations, for example concerning levels of voter turnout in national elections (collated by International IDEA), public access to the news media and new ICTs (UNESCO/ITU), and the proportion of women in national parliaments (IPU).
- Recent decades have also witnessed a substantial growth in the range and scope of cross-national public opinion surveys. Major cross-national and time-series surveys of public opinion include the Euro-barometer and related EU surveys (which started in 1970), the European Election Study (1979), the European Values Survey and the World Values Survey (1981), the International Social Survey Programme (1985), the Global Barometers (1990 and various), the Comparative National Elections Project (1990), the European Voter and the Comparative Study of Electoral Systems (1995), the European Social Survey (2002), the Transatlantic Trends survey (2002), the Pew Global Attitudes project (2002), and the Gallup World Poll (2005). These surveys allow a representative sample of the general public to express their views about the quality of democracy in their own country, as well as to assess the performance of their government leaders, institutions, and policies, confidence in government institutions and satisfaction with democracy, patterns of political behavior and civic engagement, and social and political values.
- Lastly, qualitative audits have also become increasingly popular, whether concerning the state of democracy in a country or more narrowly focused upon specific sectors, such as media audits.
The use of different data sources, and their combination in certain indicators, raises many important and complex questions. In particular, do indicators based on mass or elite surveys coincide or do they diverge? Where perceptions differ, which source provides the most legitimate, useful, valid, and reliable benchmarks for both scholars and practitioners? Elite evaluations are commonly assumed, usually implicitly, to provide the factually 'correct' and more reliable assessment. By contrast the public is often usually believed to be mistaken, unaware, ill-informed, or simply misguided in their judgments.
Clearly public evaluations can be flawed – as can expert perceptions – for multiple reasons. Instead of assuming that expert perceptual indicators always provide the most valid and reliable measurement, it is more appropriate to conclude, more agnostically, that no single best measure or indicator of democratic governance exists for all purposes; instead, as Collier and Adcock suggest, specific choices are best justified pragmatically by the theoretical framework and analytical goals used in any study.
The most prudent strategy is to compare the results of alternative indicators at both mass and elite levels, including those available from cross-national public opinion surveys, to see if the findings remain robust and consistent irrespective of the specific measures employed for analysis. If so, then this generates greater confidence in the reliability of the results, since the main generalizations hold irrespective of the particular measures which are used. If not, then we need to consider how far any differences in the results can be attributed to the underlying concepts and methodologies which differ among these measures.
Moreover the available elite-level perceptual indicators are relatively blunt instruments, whether for assessing the quality of democracy, for program performance evaluation, or for diagnosing specific problems of concern within any particular state. For policymakers, in some circumstances, the legitimacy derived from professionally-conducted public opinion polls may also make these preferable indicators of democratic governance.
Therefore, social surveys can play an important role when seeking to diagnose, monitor, and strengthen the quality of democratic governance across and within many states worldwide. Representative and professional surveys of public opinion are likely to generate results which prove more politically legitimate, disaggregated, and useful for policymakers, practitioners and research analysts than reliance upon expert perceptions alone.
In thinking about how APSA could take this work forward, many fruitful ideas were suggested by participants. The following six-step strategy represents some of the ways that APSA can contribute to strengthening the quality of democratic governance indicators.
Stimulate scholarly debate & collaboration
Develop consultancy directory
Recommend authoritative scientific standards
One stop data shop
Develop innovative indices
Commission meta-analysis thematic articles
Online directory of expert consultants available for practitioners
Establish authoritative standards/code for methods, data collection, validity, reliability, transparency
Develop integrated annual x-sectional and time-series datasets
Pre-APSA workshops for practitioners and younger scholars
Incentivize APSA sections to develop innovative indices; use membership networks for expert surveys
Useful scholarly reviews
Service for members and practitioners
Improve practices; voluntary code; rating of standards
Facilitate indicator comparisons and simplify for users
Draws on APSA skills and expertise ; capacity development
Generate unique data; already used for ideological party scales
Minimal costs; rarely reaches practitioners
Minimal costs; needs public evaluation feedback to be useful
Fund report workgroup meetings
Fund RAs; Reinvent wheel? QoG Institute
Fund 2-3 section-organized annual workshops awards
Special issue PS
Publish and disseminate APSA Report and guidelines
Online annual datasets
Workshops; training guidelines; video?
New disaggregated indices of democratic governance
For more details about the meeting, including the full list of participants, slide presentations, bibliographies, and further resources, see