The Accreditation of Data in the Information Society
humanITy Briefing Paper 10
Date: 02/07/2003
Article
1. Introduction
1.1 In the analogue age you could, in spite of the popular saying, know the contents of a book by looking at its cover. The cost of publishing, cartels in broadcasting, the scale of academic peer review, were all controlling factors in helping users to weigh and evaluate data. There was an intuitively recognised hierarchy which ran, approximately, in descending order: Academic peer review; academic book, public service broadcasting, broadsheet press, magazines, tabloid press.
1.2 Clearly, the ability to publish and/or broadcast digitally has massively reduced production costs and increased dissemination but users do not have their traditional analogue rules of thumb for weighting and evaluation.
1.3 This Paper looks at a variety of aspects of data weighting and evaluation before suggesting an institutional framework for helping users:
- Peer Review (Section 2)
- Surveys and Sponsorship (Section 3)
- Frequency Analysis (Section 4)
- Reporting and Public domain Documents (Section 5)
- A New Institutional Framework (Section 6).
2. Peer Review
2.1 The volume of academic publication, the pressure to be first and the need to obtain publicity for research in such areas as medicine, nutrition, safety and trend analysis have led to a serious breakdown of the rigidities of academic peer review.
2.2 Apart from ethical problems with publication prior to appropriate peer review, unreviewed research leaves users in a 'twilight zone' where it is easy to exaggerate significance. Unreferred lurid findings anticipate and then relegate properly peer-reviewed research. This is not specifically a digital information problem but the speed of dissemination, the appetite for apparent survey data and ranking systems each make this a problem for Internet users.
2.3 Peer reviewed information should be fronted by a template which includes the salient details of the research and the review, together with a global authenticity mark. It is vital that such a system is internationally agreed and not a unilaterally declared United States Standard.
3. Surveys and Sponsorship
3.1 The most abused form of information dissemination is the survey.
3.2 Surveys are used to mislead in three significant ways: first, sponsorship of surveys often becomes detached from data publication; secondly, the methodology for choosing and framing questions is almost never published; thirdly, sample size, significance and margins of error are often omitted.
3.3 Those involved in creditable market and opinion research should protect themselves and users by publishing a standard format for reporting survey data which includes:
- Criteria and methodology of question selection
- Range of parties consulted in formulation
- Sample size
- Results
- Significance
- Margin of error
4. Frequency Analysis
4.1 One common methodology for ranking information is frequency analysis, not dissimilar to the ranking of pop and rock music through sales charts.
4.2 Frequency ratings tempt information suppliers to rig the rankings but they also imply that there is some intrinsic information value according to the ranking. These two factors in turn force a radical split between information ranked on the first page of a search engine report and all other information. Because search engines do not differentiate on the basis of quality (not least because there are as yet no adequate quality markers as those recommended in Sections 2. and 3. above), unreliable information can be ranked much more highly than reliable information; this is a particular problem when a major new topic comes to public attention because bogus research and opinion are much easier to formulate and publish than verifiable research.
5. Reporting and Public Domain Documents
5.1 A key area of concern to all information creators is the way that their material is edited and reported so that end users cannot trace the source.
5.2 In all reputable publishing and broadcasting output the source for any material should be clearly attributed so that the user can find the original from which an intermediate has quoted or arrived at an interpretation.
5.3 The process of understanding the relationship of an original document to interpretation or simplification would be greatly facilitated by two strategies:
- First, the enforcement of attribution separate from the assertion of intellectual property rights
- Secondly, the use and declaration of the Public Domain Document.
5.4 The Public Domain Document (PDD) is an intellectual artefact which consists of data and a set of simplification tools. The user may be in receipt of a simplified version of a PDD but has access to the publicly declared rules by which simplifying tools operate and has one click access to the original unmediated by the tools.
6. A New Institutional Framework
6.1 As peer review, public service broadcasting, market and opinion research and Government information are all suffering from a lack of credibility and are all being exploited for a variety of reasons to improve personal, political or commercial advantage, we need to create a new institutional framework for information kite marking.
6.2 We have already made some suggestions (Sections 2. and 3.) on international standards in some areas but we also need domestic and European Union structures. While it could be argued that the aggregation of institutions suffering from declining credibility will not improve the situation, there is ample evidence in the UK at least that inter-sectoral self-regulation, in such areas as advertising standards, has some merit. Public serviced broadcasters, universities, opinion and market research organisations and lay persons appointed according to the Nolan criteria should establish an Institute of Data Validation; it should not seek to censor under any circumstances but it should rate information according to publicly derived criteria.
7. Resources
- Arms, W. Y. (1992) "The Institutional Implications of Electronic Information." Presented at Technology, Scholarship, and the Humanities: The Implications of Electronic Information (September 30 - October 2, 1992; U of California, Irvine). Rptd. Coalition for Networked Information. http://www.cni.org/docs/tsh/Arms.html
- Beattie, D. and McCallum, D. (1998) “Electronic Publishing Initiatives at Industry Canada.” Beyond Print: Scholarly Publishing and Communication in the Electronic Environment. Online Posting. September 26-27. http://citd.scar.utoronto.ca/EPub/talks/IC-EPPP.html
- Chan, L.and Barek, W. (1998) Beyond Print: Scholarly Publishing and Communication in the Electronic Environment. Online Proceedings. Sept. 30, 1998. E-Publishing Symposium: Sept. 26-27, 1997. 6/15/00. http://citd.scar.utoronto.ca/epub/1997.html.
- European Broadcasting Union, www.ebu.ch
- OSCE (2003) Public Service Broadcasting: New Challenges, New Solutions. http://www.osce.org/documents/rfm/2003/03/26_en.pdf
- Standler, R. (2003) Evaluating Credibility of Information on the Internet. http://www.rbs0.com/credible.pdf
- Guédon, JC with Siemens, R (2001) The Credibility of Electronic Publishing: Peer Review and Imprint. http://web.mala.bc.ca/hssfc/Final/PeerReview.htm