... the case for doing some scientific projects in a radically open manner, supported by an Open Science Protocol
Article by Alan Cottey, late 2001
SummaryFour Levels of OpennessThe Argument for Radical Openness in Some CasesTowards an Open Science ProtocolThe FenceOpen and Comprehensive RefereeingObjections and AnswersExample on Open Evidence: Forensic Explosives Lab's Contaminated CentrifugeExample on Open Funding: Bristol University Vet School ControversyTaking Advantage of Information TechnologyMore Details About Some of These Topics ...
The Open Science Proposal is that it would be useful for the scientific community to have a widely recognised Protocol of Openness, applying to individual scientific projects. Only a small fraction of all scientific projects would be done according to the required standard of radical openness, but those projects could claim a 'gold standard' of openness. Such projects would permit detailed scrutiny and criticism, and so furnish an efficient route to socially established reliable knowledge. Not all science is open, or even nearly open, nor is there any prospect of this. Four levels of openness are identified. If these four levels were borne in mind, controversial truth claims in science could be debated with more light and less heat.
There are places in society for such contrary qualities as reserve, discretion, tact and secrecy. The word science covers a wide range of activities, and these activities are not merely linked with society, they are embedded in society. Given this situation, we need to step back from what the philosopher Sissela Bok has called the ritualistic denunciation of secrecy in science.
I use the following terms to describe four levels of openness in science. The first three already exist. The fourth may come to be, if we wish it ...
The Argument for Radical Openness in Some Cases
Openness is an essential element of science. The detailed basis of any scientific truth claim must be open to scrutiny, criticism, repetition and refutation. The weaker this openness, the weaker is the claim. There may be socially valid reasons, such as politeness, privacy, commercial interest or national security, for a lack of complete openness. So be it. The scientific truth claim is weakened, though it may not be annihilated. Openness is not 'all or nothing', it exists on a continuous scale.
The prestige of well-established scientific findings (what some call 'objective facts' and science-studies specialists often call 'reliable knowledge') is very high. When we consider the accuracy, reproducibility and predictive power of some of the more definite branches of science, such as celestial mechanics, we can understand why everyone would like to have science backing up their pronouncements.
Very well. Why not? Anyone can make a truth claim. The crunch comes when others ask 'What is the basis of your claim?' Then the claimer must neither swing rank, nor clam up, nor obfuscate, nor bluster, nor cite commercial interest or national security. S/he must show all of the arguments and data that might possibly be relevant.
This description of science is very familiar. Perhaps it is too familiar, so that, like wallpaper, it is not seen. For the kind of openness described is very unusual, compared with human behaviour in most situations. And every branch of science is intimately connected with social affairs. Number theory is connected with commerce and the military, through cryptography. Cosmology affects how we think of ourselves, our lives and our culture.
Another aspect of science which is relevant for the level of openness is the trend, in recent decades, towards 'managed science'. Contrary to what many scientists think, this has not had a wholly negative effect on openness. True, commercial considerations inhibit openness more than in earlier times. On the other hand, public accountability requires a higher level of a certain kind of openness (which is not wholly subverted by 'public relations').
Thus, since science is embedded in society, unqualified calls for more openness in science are too simplistic. A more nuanced approach is needed. The present proposal calls for the openness principle of science to be taken more seriously. This does not, however, mean that all research should reach a very high standard of openness. To do so would be to declare that most of the work that is currently said to be science would henceforth be declared to be non-science. This approach (quite apart from being extremely unpopular!) would not serve human culture well.
It would be better if all involved in the assessment of a scientific truth claim would start by considering the four levels of openness, discussed in the previous section.
Towards an Open Science Protocol
The core of the Open Science Protocol is that the following stages of an Open Science project are all open, as they occur ...
Some aspects of this Protocol are elaborated in the other sections of this exposition, including the section Objections and Answers.
Certain exceptions to the extreme of complete, real-time openness are acceptable. Three examples are - some aspects of project staff records; some information about subjects in research on humans; temporary secrecy in certain 'blind' and 'double-blind' investigations.
Even so, the required standard of openness is high. It is expected that only a small fraction of all scientific projects will ever be done to this standard of openness. Nearly all applied work will not be done as Open Science. Nevertheless, those projects which are done in the open way will have a special significance, transcending their substantive results, by showing something new about the nature of scientific knowledge. Precisely what will emerge from the completions and non-completions of, say, the first dozen such projects cannot be known ahead of time.
'The Fence' need not be, and indeed usually is not, the physical fence that surrounds, for example, a military establishment. In general the Fence is any artificial impediment to the free flow of scientific information from 'inside' to 'outside'. Within the fence, there is a peer group of highly qualified specialists. These peers have much in common, and they communicate fairly freely with each other. They have shared assumptions, career-interests, and blind-spots. They are a group of humans like other groups, with features special to their particular situation. Such a group may do work which conforms to the accepted norms of production of reliable knowledge within the fence.That is, the knowledge has validity for the group.
The problem comes when the establishment wishes to export its truth claims to the wider world. The laboratory may be prestigious and well-funded. Its staff may include many experts loaded with qualifications and honours. If outsiders cannot examine the basis of the claims, then those claims do not even get to square one. The question of scientific truth or falsity cannot even arise. The claims, in the outer world, are general claims. They have the same status as any other claims whose basis is prestige or power or anything other than full access to the evidence.
Here, in order to make the principle clear, the distinction between open science and fenced science has been expressed starkly, in black and white. In society as it is, truth judgements are made in a holistic way, not by a computer algorithm capable of weighing all the evidence and concluding 'true' or 'false'. There can be no perfectly open science, nor is the boundary between 'inside' and 'outside' perfectly sharp and leak-proof. Even so, there is a very great difference between the claims of an open science project and the claims presented to the public by a fenced establishment.
Open and Comprehensive Refereeing
Refereeing is an important element of the overall critical process to which scientific knowledge claims must be subjected. Scientists attach much importance to peer review. It is, however, usually conducted very discreetly. Referees are nearly always anonymous, and their reports are seen only by a few insiders - authors, editors and adjudicators. Most scientists are keenly aware of the merits of peer review. They are, however, blind to fact that, when there is conflict between the peer group's interest and the public interest, discreet review by peers alone produces conclusions which favour the peer group. For this reason, peer review should be broadened and opened up.
In an O S project, all needed review processes (ethical, safety, funding and formal publication) would be conducted much more openly. Scientists from contiguous specialities (with considerable expertise and more 'objective distance'), together with persons acting as guardians of the public interest should be involved, complementing the narrow interests of the peer group.
Here are the questions raised most immediately by scientists on hearing the O S proposal ...
What of the risk of a smart competitor taking your promising idea and pipping you at the post of definitive proof?
Answer:There is this risk, but most of us have an inflated conception of how interesting our latest bright ideas are to others. A much more common problem is to get others to take them seriously! And by posting one's progress, essentially in real time, one is putting down markers of possible priority much more efficiently than happens with the current (publish-when-everything-is-complete-and-reviewed) practice.
Why would any scientists go to the extra trouble and 'risk' of doing a project in a completely open way?
Answer:There would be an important incentive - the enhanced credibility of scientific work done in a fully open way and withstanding the scrutiny of all who wish to scrutinise.
Would not such a warts-and-all exposure be horribly embarrassing?
Answer:Disclosure in a secretive milieu has a disproportionate impact ...
FEAR OF OPENNESS --> SECRECY
By contrast, in the OS way, objections of a named referee, or disagreements on research in progress, would be no great deal - merely single items in a large amount of available information.
Wouldn't this scheme bog science down in even more bureaucracy?
Answer:Less formal research would not be prevented. The proposals on open science are not meant to create a monolithic or bureaucratic science. Projects done to less high standards of openness would still be done, for various reasons including cost, speed and scientist's temperament. Other things being equal, the claims of such work would require more checking than those of fully open projects.
Example on Open Evidence: the UK Forensic Explosives Laboratory's Contaminated Centrifuge
In 1996 it became publicly known that the UK Forensic Explosives Laboratory had supplied forensic evidence in criminal trials on the basis of evidence deriving from the use of a centrifuge which was later discovered to be contaminated with an explosives indicator.
The essential point is that informed criticism from outside the fence was not possible. The effective way to avoid the contaminated centrifuge case would have been to have a level playing field for all the litigants in respect of forensic evidence. If the defence had equal access to samples to be tested for the explosives indicator, the court would have been considering scientific evidence, rather than unchallengeable claims.
Example on Open Funding: The Bristol University Vet School Controversy
This case started in 1988 when some staff of the School discovered that veterinary research had been 'bent' towards the biological warfare area in order to attract funding from the Ministry of Defence, civil funding having become unobtainable. This only became generally known in the School after the grant had been awarded. A bitter dispute followed and the ethical problems were discussed intensively in the university and the media. The project did, however, go ahead. In the financial climate that existed at the time, the cards were heavily stacked against the option of returning the substantial grant to the MoD. Since scientific research institutions will always find it difficult to be picky after a large grant has been won, openness at the application for funding stage is essential if a project is not to become a fait accompli, subjected to later 'review' which is in reality only cosmetic.
Taking Advantage of Information Technology
Doing a project the Open Science way will obviously involve a large amount of recording, storing and accessing of information. Is it practicable? Before the IT era, the answer would be 'hardly'. IT has transformed the situation. The Open Science way could take its place alongside the three traditional ways of doing science (Secret, Restricted and Circumspect).
The earliest OS projects - and none have been attempted to date - should be simple and short. The volume of data preserved in the O S archive of an O S project need not be unmanageable, since front-end reduction and parameterisation of raw data, in ways that do not compromise the principle of openness, will be possible and indeed recommended. Naturally the procedures and algorithms used will be part of the O S record. Analysis further down the line, which is subject to more assumptions, will be recorded and stored in more detail.
More problematic than recording and storing information is accessing it. When we consider the large number of specialised subjects in science, each with particular software needs and favourite packages (including ad hoc program suites) for the technical work, and yet other software used for recording and processing administrative matters, we see that a large variety of different software packages will be used by investigators. One way of rendering the O S records accessible to all who are interested, would be to require investigators to export reduced technical data and administrative data to applications selected from a short list of widely known and durable software packages.
IT has a bearing on the choice of OS infrastructure. One, centralised, model would require an Open Science Institute to keep a fairly tight rein on OS practice. The more attractive alternative would be to follow the philosophy of the World Wide Web. Successive versions of basic OS standards, principally the OS Protocol, would be published as a consensus by those interested. Then any investigators could unilaterally declare themselves as doing a project the OS way. There would be no formal, centralised monitoring but all outsiders could judge for themselves the quality of adherence to the OS way, and the quality of the work itself, including the quality of any open refereeing that had taken place. The build-up (or decay!) of acceptance of a project's claims would occur partly by open comment on the Web. OS would become a visible college.
The OS Protocol should include a requirement of arrangements to archive for a fixed period of time. Thirty years should be sufficient to permit most work to be absorbed into the canon of reliable scientific knowledge, or rejected, or quietly forgotten. In principle, an archive review would take place after 30 years, so that projects of continuing interest could be preserved longer.
Technology drift is sometimes mentioned as a problem with archiving - if data are recorded using hardware or software which becomes obsolete, they may become effectively unreadable or lost. Although this was a substantial problem in the early years of IT, there is today more awareness of archivists' needs. IT is still developing rapidly, but if O S recommendations are worked out with the advice of professional archivists, OS records should be readily accessed for 30 years, with the option, if required, for exporting the records to a new storage system at the end of that period.
More Details About Some of These Topics ...
may be found in two articles by Alan Cottey -
Open Science and Practicable Politics (pp 88 - 96 of Green Politics in Grey Times, Ecopolitics XI Conference Proceedings, ed C Star, pub University of Melbourne, 1998)
and
SGR Newsletter, Autumn 2000, No 21, p 8 - 9. Information technology: the key to taking Open Science forward. (A slightly edited version of this article may also be seen on AC's UEA Personal Pages )