Pippa Norris's Weblog

Changing media, changing statistics

It is a cliché to acknowledge the rapidly changing world of the mass media, from a static form of transmission from wireless and television sets in the living room, to communications arriving via a plethora of multimedia platforms, where my cable TV comes to me over WiFi to watch on my smart phone, radio is delivered on my IPod, and headlines from the news media are updated 24/7 online. All these developments complicate but also enrich the media environment. What is less often acknowledged is how far behind we are in monitoring use of news and information from these channels through internationally-standardized communication statistics.

UNESCO is the primary agency worldwide with the responsibility to standardize, gather and disseminate communication statistics. For decades national statistical offices counted newspaper circulation figures, and the distribution of television and radio sets, and related data, and UNESCO disseminated the information through their annual yearbook. But the limitations of this traditional approach have rapidly become apparent. At the same time, the international community has rediscovered the important economic and cultural role of 'creative industries', covering everything from music, dance and crafts through fashion, architecture and ceramics to movies, television, and the online world. Moreover the role of the media for development has also attracted new attention, with international agreement in March 2008 to adopt the UNESCO Media Development Indicators .

To catch up with developments, the UNESCO Institute of Statistics in Montreal is in the process of developing and piloting new surveys monitoring newspapers, broadcasting (television and radio), and movies. This week, UNESCO organized an Experts Workgroup meeting from 17^th-19^th November 2009 in San Jose, Costa Rica. The aim was to review the draft questionnaire, along with the accompanying technical notes for the survey completion and the eventual analysis of the data. After pilot tests, the survey is planned for roll out in 2010, with data available the following year.

Once finalized, the survey instrument will be distributed to national statistical offices for the annual collection of statistics. This may sound dry as dust. But in fact this ambitious initiative opens exciting new opportunities for the policy and research communities to understand the diffusion of the new media environment worldwide and the persistence of some old problems facing the role of the media for development. The UNESCO Media Development Indicators emphasize that media development needs to meet certain normative standards:

1. A system of legal regulations conducive to freedom of expression, pluralism and diversity;
2. Plurality and diversity of media ownership;
3. Media should serve as a platform for democratic discourse, reflecting the diversity of views and interests in society;
4. Professional capacity building and civic society institutions underpinning freedom of expression, pluralism and diversity;
5. Infrastructure capacity sufficient to support an independent and pluralistic media.

The planned new UNESCO surveys on newspapers and broadcasting will not provide indicators capable of monitoring and comparing all these standards, but they will go a lot further than is possible at present, especially in terms of assessing the balance of private v. public sectors in broadcasting and newspapers. For the first time, the surveys will also facilitate a global comparison of number of hours devoted to certain types of broadcasting programs (such as news, education, sports, and drama) shown in different countries, as well as the gender diversity of the media workforce, the proportion of domestic and foreign ownership, and the availability of linguistic broadcasting, amongst other aspects.

The workgroup considered the proposed survey instruments and focused attention upon the technical notes which accompany it, grappling with many complex and detailed issues about how best to relate the new data to broader notions of media pluralism and diversity, reflected in the Media Development Indicators. The new media environment is so complicated today that any 'top down' approach to assess its structure can often only grasp at aspects which are inevitably partial. Hence we can measure the number of terrestrial TV channels broadcasting in a country, for example, but does that actually tell us anything in a world where national TV is supplemented by Hulu, You Tube and similar broadband streamlining sources, along with cable and satellite channels. We can measure the number of national daily newspapers with mainly (51%+) public, private or foreign ownership – but does this provide important insights about the primary sources of information available in any country where surfing the New York Times or listening to my local NPR station every morning is as easy in Boston as it is in San Jose or Shanghai? Only a 'bottom-up' approach --where representative sample surveys assess what information people use as their regular sources of information --can start to evaluate the diversity and pluralism of media access in any country with any accuracy. Individual-level data on media uses is increasingly collected for many nations worldwide in surveys such as the Gallup World Poll (covering 140 nations) and The World Values Survey (covering 90+). Expanding upon these sources, by matching them with the structural national data, would be very worthwhile for future research.

UNESCO will work with national statistical offices to develop the capacity to generate the new data. If this initiative succeeds, so that a comprehensive and accurate picture of the media environment can be assembled, the international data will be widely welcomed. What remains to be done, however, for a richer understanding of pluralism and diversity, is to match the new national-level data on the media ownership and structure more closely with additional datasets. This would allow analysts to compare systematically the typical contents of the newspapers and broadcast media, as well as individual-level patterns of usage. If this approach remains outside of UNESCO's role and capacities, it is hoped that other non-profit agencies and independent researchers concerned with media for development will utilize the new data. We urgently need to develop robust and comparable international indicators which combine the analysis of the media environment with an understanding of the media contents and audience.

Measuring democratic governance

Last weekend (30-31^st October 2009), the American Political Science Association (APSA) organized a conference on Democratic Audits and Governmental Indicators, hosted by the Goldman School of Public Policy at the University of California, Berkeley. This meeting was held at the initiative of Professor Henry Brady, President-elect of APSA and Dean of the Goldman School, and it is the start of developing a year-long APSA taskforce on this topic. The meeting generated lively debate among leading scholars, experts, and representatives from some of the major multilateral organizations and advocacy groups engaged in generating and using governance indicators.

The focus of the APSA initiative is timely since the third wave era of democratization has seen a burgeoning array of diagnostic tools designed to monitor the quality of democratic governance. Political indicators are now widely used by the international community, by national governments and by advocacy groups to evaluate needs and determine policy priorities, to highlight problems and identify benchmark practices, and to evaluate the effectiveness of programmatic interventions.

The most commonly-used indicators focus on issues such as the existence of democracy, civil liberties and political rights, the extent of 'good governance', respect for human rights, perceptions of corruption, the degree of women's empowerment, levels of civic engagement, the extent of conflict, the distribution of social capital, and many other related topics.

The expansion of indicators has been accompanied by important gains in the level of conceptual sophistication, methodological transparency, scope, and geographic coverage of all these measures. Nevertheless there remain considerable concern that the proliferation of indices has not been accompanied by sufficient quality checks, for example through the transparent publications of technical notes about how each of these measures are constructed, the degree of measurement error, and the sources of funding and accountability.

Types of indicators

Four types of indicators have become common:

1. Elite-level perceptual judgments are an important source of information. Organizations commonly use standardized questionnaires distributed to 'expert' sources, typically including scholars and researchers, country analysts, journalists, lawyers, business executives, independent consultants, human rights observers, and NGO staff. This approach is exemplified by Freedom House's process employing teams of researchers to assess the state of civil liberties and political rights worldwide, by Transparency International use of experts to construct the Corruption Perceptions Index, and by Reporters without Borders survey used to generate its annual Press Freedom index.
2. Many indicators are also compiled from official data based on administrative records collected by public officials, national statistical offices, and multinational organizations, for example concerning levels of voter turnout in national elections (collated by International IDEA), public access to the news media and new ICTs (UNESCO/ITU), and the proportion of women in national parliaments (IPU).
3. Recent decades have also witnessed a substantial growth in the range and scope of cross-national public opinion surveys. Major cross-national and time-series surveys of public opinion include the Euro-barometer and related EU surveys (which started in 1970), the European Election Study (1979), the European Values Survey and the World Values Survey (1981), the International Social Survey Programme (1985), the Global Barometers (1990 and various), the Comparative National Elections Project (1990), the European Voter and the Comparative Study of Electoral Systems (1995), the European Social Survey (2002), the Transatlantic Trends survey (2002), the Pew Global Attitudes project (2002), and the Gallup World Poll (2005). These surveys allow a representative sample of the general public to express their views about the quality of democracy in their own country, as well as to assess the performance of their government leaders, institutions, and policies, confidence in government institutions and satisfaction with democracy, patterns of political behavior and civic engagement, and social and political values.
4. Lastly, qualitative audits have also become increasingly popular, whether concerning the state of democracy in a country or more narrowly focused upon specific sectors, such as media audits.

The use of different data sources, and their combination in certain indicators, raises many important and complex questions. In particular, do indicators based on mass or elite surveys coincide or do they diverge? Where perceptions differ, which source provides the most legitimate, useful, valid, and reliable benchmarks for both scholars and practitioners? Elite evaluations are commonly assumed, usually implicitly, to provide the factually 'correct' and more reliable assessment. By contrast the public is often usually believed to be mistaken, unaware, ill-informed, or simply misguided in their judgments.

Clearly public evaluations can be flawed – as can expert perceptions – for multiple reasons. Instead of assuming that expert perceptual indicators always provide the most valid and reliable measurement, it is more appropriate to conclude, more agnostically, that no single best measure or indicator of democratic governance exists for all purposes; instead, as Collier and Adcock suggest, specific choices are best justified pragmatically by the theoretical framework and analytical goals used in any study.

The most prudent strategy is to compare the results of alternative indicators at both mass and elite levels, including those available from cross-national public opinion surveys, to see if the findings remain robust and consistent irrespective of the specific measures employed for analysis. If so, then this generates greater confidence in the reliability of the results, since the main generalizations hold irrespective of the particular measures which are used. If not, then we need to consider how far any differences in the results can be attributed to the underlying concepts and methodologies which differ among these measures.

Moreover the available elite-level perceptual indicators are relatively blunt instruments, whether for assessing the quality of democracy, for program performance evaluation, or for diagnosing specific problems of concern within any particular state. For policymakers, in some circumstances, the legitimacy derived from professionally-conducted public opinion polls may also make these preferable indicators of democratic governance.

Therefore, social surveys can play an important role when seeking to diagnose, monitor, and strengthen the quality of democratic governance across and within many states worldwide. Representative and professional surveys of public opinion are likely to generate results which prove more politically legitimate, disaggregated, and useful for policymakers, practitioners and research analysts than reliance upon expert perceptions alone.

Next steps

In thinking about how APSA could take this work forward, many fruitful ideas were suggested by participants. The following six-step strategy represents some of the ways that APSA can contribute to strengthening the quality of democratic governance indicators.

Strategic Options	(i) Stimulate scholarly debate & collaboration	(ii) Develop consultancy directory	(iii) Recommend authoritative scientific standards	(iv) One stop data shop	(v) Train users	(vi) Develop innovative indices
Activities	Commission meta-analysis thematic articles	Online directory of expert consultants available for practitioners	Establish authoritative standards/code for methods, data collection, validity, reliability, transparency	Develop integrated annual x-sectional and time-series datasets	Pre-APSA workshops for practitioners and younger scholars	Incentivize APSA sections to develop innovative indices; use membership networks for expert surveys
Pros	Useful scholarly reviews	Service for members and practitioners	Improve practices; voluntary code; rating of standards	Facilitate indicator comparisons and simplify for users	Draws on APSA skills and expertise ; capacity development	Generate unique data; already used for ideological party scales
Costs/ limits	Minimal costs; rarely reaches practitioners	Minimal costs; needs public evaluation feedback to be useful	Fund report workgroup meetings	Fund RAs; Reinvent wheel? QoG Institute	Minimal costs	Fund 2-3 section-organized annual workshops awards
Products	Special issue PS	Online directory	Publish and disseminate APSA Report and guidelines	Online annual datasets	Workshops; training guidelines; video?	New disaggregated indices of democratic governance

 

More information

For more details about the meeting, including the full list of participants, slide presentations, bibliographies, and further resources, see

http://sites.google.com/site/democracyaudit/