Reinhold Kliegl
The Berlin-Brandenburg Academy of Sciences and Humanities (BBAW) has published Recommendations on the Future of Scholarly Publishing. These recommendations represent a core set of policies upon which the BBAW as a whole has officially agreed, albeit not without heavy and lingering dissent among its members. At the outset, I was sceptical that the interdisciplinary research group drafting these recommendations would find any common ground – for the diversity of the publication formats and models and the heterogeneity of the disciplines and their representatives was immense. I was also wondering whether it is even desirable or necessary. Why not let every discipline find its own way? Fortunately, as it stands, this initiative and many similar ones prove an important point. The common goal of all sciences and humanities to achieve a better understanding of the world and to share this knowledge universally with as little cost as possible to the individual is still strong enough to motivate joint action on how we want to document and facilitate growth in scientific knowledge in the future. Scientific knowledge must be reliable, open to scrutiny and criticism. It is the result of global cooperation, extending also across earlier generations. Its growth is facilitated by competitive claim to fame, based on the desire to be the first to report an important advance of knowledge (for example, Merton 1973).
Publications are the primary format to document the corpus of scientific knowledge in almost all disciplines. Given their central role, it is not surprising that side-effects have evolved that to some degree undermine the common goal. For example, the number of publications and the number of citations of these publications serve as convenient indicators of scholarly reputation. These indicators are instrumental in allocating positions, obtaining awards, raising research funds and, yes, writing even more publications. Thus, in a way, publications have achieved the status of a ‘common currency’ in the scholarly system. In principle, there is nothing wrong with this development. Indeed, I suspect, for example, that the development of objective indicators has been an effective strategy to counteract nepotism in old-boy networks. Obviously, we need to improve, not abandon, objective indicators. In particular, we need to address the problem of their reactivity (i.e. the measure itself influences the object of measurement) (see Espeland & Sauder 2007; see also Weingart 2015, for a general discussion). The main and very serious problem of current indicators is that they have become an end in themselves to the extent that for some colleagues, they are taken to be more important than the substance they are supposed to indicate. These issues were elaborated in sections 5.4 and 5.5 and translated into a set of sensible specific policies in sections 6.8 to 6.11 of the Recommendations.
Building on these recommendations, but also going beyond them, I will briefly highlight two issues that derive from or at least are intertwined with current problems of the scholarly publication system. I will then argue that these problems can be addressed effectively if we support and embrace open science initiatives.
Problems with journals
The traditional journal-based publication system trades off speed of growth in knowledge against quality control. For an excellent summary of the many issues, I refer to Krumholz’s (2015) editorial perspective in Circulation: Cardiovascular Quality and Outcomes. He succinctly characterises journals as too slow, too limited, too unreliable, too focused on the wrong metrics, too powerful, too parochial, too static, and too dependent on a flawed business model. These are actually the headings of the editorial paragraphs. He concludes (2015: 2):
We have arrived at the juncture where medicine and science need new vehicles for the dissemination of knowledge. [...] The question for all of us [...] is how that would best be accomplished in a new world that is flat, digital, and transparent.
Indeed, it is not long ago that our scholarly publication system began to implement the opportunities afforded by technological developments related to digital publication, storage space for data, instant and global availability of knowledge. The problems have been recognised and constructive proposals for their resolution are actively worked on (see relevant sections in Fiedler et al. 2016; The Royal Society 2015). A very promising approach are so-called ‘overlay’ journals where the traditional peer-review process uses submissions on the green access arXiv pre-print server for initial reviews, revisions and final publication (Ball 2015). Thus, journals such as Discrete Analysis or The Open Journal of Astrophysics exist only as virtual layers on arXiv.org. I will sidestep these developments to highlight below what I perceive to be a principled, even more encompassing answer.
Competition dominates cooperation
Growth in knowledge is driven, in part, by the dynamics of competition and cooperation between researchers or research groups. I suspect that the current reward system, which is strongly linked to maximising the number and citations of publications and to questionable indicators of journal impact, has led to an imbalance of the dynamics of competition and cooperation between scientists (or groups of scientists). Arguably, the increase in the number of publications and the immediate and global availability of associated statistics (such as impact factors of journals or the h-index) have reinforced competition, at the cost of cooperation and even some of the joy in science.1 I admit that this claim is based on a gut feeling about how scientific disciplines have changed over the last decades rather than on solid empirical evidence. I am ready to stand corrected, but I am not alone with this sentiment. For example,
In the last 50 years, there have been many changes to the substance, conduct, and style of research. Many of these changes have proved disastrous to the life of scientists and to science itself. As a consequence, the near-romantic spirit of adventure and exploration that inspired young scientists of my own and earlier generations has become tarnished. Now, many of us feel beleaguered by bureaucrats and by politicians: they affect our lives profoundly, apparently without an understanding of the way discoveries are made or of the nature of science itself. (Lawrence 2016: 1)
Open science
To overcome these shortcomings, we must re-evaluate the future of scholarly publications in the broader context of the future of scholarly communication to re-align our scientific goals with scientific practice. For me, the relevance of publications was put into perspective by the following quote from a paper which introduced the concept ‘reproducible research’:
An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures (Buckheit & Donoho 1995: 5, citing an internal report by Clearbout 1994).
Publications are advertisements! Nowadays this is often no longer a figurative, but a literal description of articles in high-impact journals where, with an eye towards citation statistics, hype often trumps substance and where the technical details about the research are relegated to supplements, if they are included at all. Indeed, on the basis of an analysis of journal rank, Brembs et al. (2013: 5) conclude:
(1) journal rank is a weak to moderate predictor of utility and perceived importance; (2) journal rank is a moderate to strong predictor of both intentional and unintentional scientific unreliability; (3) journal rank is expensive, delays science and frustrates researchers; and, (4) journal rank as established by [impact factor] violates even the most basic scientific standards, but predicts subjective judgments of journal quality.
Brembs et al. (2013) recommend abandoning journals in favour of setting up a new communication system in line with the above proposal for reproducible research.
Reproducible research is the precursor of what is nowadays usually referred to as ‘open science’. For a state-of-the-art comprehensive review of this initiative, I refer to a recent Organisation for Economic Co-operation and Development (OECD). According to this report,
[open science] refers to efforts by researchers, governments, research funding agencies or the scientific community itself to make the primary outputs of publicly funded research results – publications and the research data – publicly accessible in digital format with no or minimal restriction as a means for accelerating research; these efforts are in the interest of enhancing transparency and collaboration, and fostering innovation. (OECD 2015: 5)
As elaborated in the next paragraph, this definition explicitly also encompasses, among others, open source software and open collaboration through tools of information communication technology (ICT). Given the heavy dependency on the web and modern software tools, it is also not surprising that the initial development occurred primarily in the domain of computer science. Thus, what has emerged as a broad and growing movement during the last years has been around for some time and has spread to other disciplines since (for example, psychology and linguistics).
Free software development as role model
The first initiative that made output of research available to the scientific community as early and as completely as possible was the Free Software Foundation (Stallman 1985). This initiative started with the GNU General Public Licence four ‘freedoms’ with respect to software:
• freedom to run the programme;
• freedom to study the programme;
• freedom to redistribute; and
• freedom to distribute copies of modified versions.
Those of us who use Emacs, Linux, gcc, etc. are still profiting directly from the GNU project established in this context.
The second and third initiatives that had a profound effect on many natural and social sciences are the R Project for Statistical Computing (founded in 1993 as part of the Free Software Foundation’s GNU project) and the Git project (a member of Software Freedom Conservancy, also a not-for-profit organisation). The R Project is now the de facto standard software for instruction in statistics in psychology, displacing commercial alternatives. Git (available since 2008) is a system for the cooperative and simultaneous development of software by an in principle unlimited number of contributors. The entire development process is recorded in detail and previous states of the software can be restored. Thus, there is maximal transparency about who contributed what and when to the project.
The basic idea for both projects is very simple. The development of new software occurs in the public domain. The source code of computer programs is already available during their development, and the community is invited to help improve the code by fixing bugs, implementing new features, writing or translating documentation, or by beta-testing the program and reporting errors. Obviously, the software is debugged much more quickly than software developed in a closed shop.2
Cooperation and competition: striking a new balance
Is this approach therefore ready to be used in non-computing disciplines? A very successful example was provided by the Open Science Foundation (OSF), which published the replication of 100 psychological experiments of which only roughly one third were judged successful (Open Science Collaboration 2015). Much could be and has been written about how this low rate relates to the problems with current research practices as described above. The point here, however, is that everything about these 100 experiments (correspondence with original authors, data, analysis scripts, etc.) is documented in a transparent and accessible way on the OSF platform. Moreover, the platform is available to everybody. It provides tools for cooperation, and users can also set different degrees of privacy for documents in the repository. A similar platform, DataWiz, is under development at the Leibniz Centre for Psychology Information and Documentation.
Carrying out one’s research in the public domain is radically different from the past practice of science. How can we convince the scientific community to join this initiative? The answer is that the benefits for scientists’ workflow must outweigh the cost of learning to handle this software. This is a big hurdle because our colleagues do not have time; they need to publish articles. We will need all the support we can get to develop user interfaces that respond to users’ intuitions. Users will join if they can publish articles faster this way than via the traditional way.
We will also need a change in mind-set. This can be illustrated best by how one handles errors in research. In a highly competitive environment, errors are associated with a fear of loss of status. There will be an inclination to cover it up. Obviously, this slows down the accumulation of reliable knowledge. However, the open source community programmers welcome reports about errors in their software, for errors need to be eliminated as quickly as possible. To witness their attitude and cooperative spirit, I recommend simply following exchanges on a relevant Google group for some time. Moreover, this kind of constructive and supportive behaviour is obviously spilling over into general help groups, most notable stackexchange.com. These exchanges cut across disciplines and across countries and everybody has a good time. We need to foster such cooperative environments within the disciplines.
Finally, scientific societies, foundations and academies might want to reevaluate their awards. By far the most of them honour individual achievements. We could reverse this ratio, at least for a number of years, and recognise collaborative projects. In this context, adversarial collaboration deserves special attention (Fiedler et al. 2016). Let us also rethink how individuals earn reputation in this context. Consistently contributing problem solutions on stackexchange.com at international level should be worth a few publications when this person applies for a job. Similarly, helping to debug software that is used by many and across many disciplines is an important contribution. Such activities also serve the primary goal of science, namely the accumulation of reliable knowledge.
Much of the debate about the scholarly publication system focuses on issues within the confines of the current system, such as costs and benefits of gold versus green open access to publications. We need to resolve these issues, but I suggest we also aim higher. If reproducible research as practiced in software development is integrated into the regular workflow of research projects in the natural and social sciences and humanities (at least a part of them, for example, digital humanities), then many of the highly controversial topics will dissipate. For example, until recently, I never felt bad about granting copyright for an article to a publisher. Possibly, I intuitively felt that this is advertising. I do not think I would ever hand over to a publisher my data or computer programs. They are the foundation of my research and I love to share them with my colleagues. Of course, one size does not fit all; reproducibility of experimental or empirical research is not important in some disciplines. The general point is that to think about how transparency and cooperation could be increased might serve as a productive starting point to tackle analogous problems in any discipline.
So, to end on a slightly (meta-)competitive note, as senior members of an academy, we owe it to the younger generation to help re-engineer the scholarly communication system in such a way that the motives that had us enter the field of science regain ground again in the future. And I think we are winning already.
References
Ball, P. 2015. The journal that publishes no papers. Nature, 526: 146.
Brembs, B., Button, K. & Munafò, M. 2013. Deep impact: Unintended consequences of journal rank. Frontiers in Human Neuroscience, 7(291):1–12.
Buckheit, J.B. & Donoho, D.L. 1995. Wavelab and reproducible research. In A. Antoniadis (ed.). Wavelets and statistics. New York, NY: Springer, 55–81.
Espeland, W.N. & Sauder, M.M. 2007. Ranking and reactivity: How public measures recreate social worlds. American Journal of Sociology, 113:1–40.
Fiedler, K., Elbert, T., Erdfelder, E., Freund, A., Kliegl, R. & Stahl, C. 2016. Empfehlungen der DGPs-Kommission ‘Qualität der psychologischen Forschung’. Psychologische Rundschau, 67:59–74.
Krumholz, H.M. 2015. The end of journals. Circadian Cardiovascular Quality and Outcomes. doi:10.1161/CIRCOUTCOMES.115.002415
Lawrence, P.A. 2016. The last 50 years: Mismeasurement and mismanagement are impeding scientific research. Current Topics in Developmental Biology, 116:617–631. doi:10.1016/bs.ctdb.2015.12.013
Merton, R.K. 1973. The normative structure of science. In R.K. Merton (ed.). The Sociology of Science: Theoretical and empirical investigations. Chicago, IL: The University of Chicago Press, 267–286.
OECD (Organisation for Economic Co-operation and Development). 2015. Making open science a reality. In OECD Science, Technology and Industry Policy Papers, 25. Paris: OECD Publishing. doi:10.1787/5jrs2f963zs1-en
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349(6251).
Stallman, R. 1985. The GNU Manifesto. Retrieved from http://www.gnu.org/gnu/manifesto.html [Accessed 1 February 2016].
The Royal Society. 2015. The future of scholarly scientific communication. Retrieved from https://royalsociety.org/events/2015/04/future-of-scholarly-scientificcommunication-part-1/ [1 February 2016].
Weingart, P. 2015. Nostalgia for the world without numbers. Soziale Welt, 66:2.
1 There is, of course, variance between scientists in whether they are primarily guided by power or achievement motives. There may be joy for those seeking power in the current situation.
2 The main drawback of open source projects is a much larger variance associated with development and support of open source compared to commercial software.