Metascience is often defined as the scientific investigation of science itself with the aim to improve science. This ‘improving’ part of metascience has been called a reform movement. While the intention to improve science is generally laudable, metascience and the associated movement(s) are not without their critics. The metascience reform movement has for instance been characterized as among other things, neoliberal, positivistic, atheoretical, technological, moralizing, bureaucratic, homogenizing, mechanistic, quantitative, psychological (psychologizing), social, civilizing, bullying, exclusive, coercive, activist, and normative. In this symposium some of those critical perspectives will be presented and discussed.

Welcome

6:00 am PST / 9:00 am EST / 3:00 pm CEST

Opening remarks from Tim Errington, COS’s Senior Director of Research, and Sven Ulpts, PhD Fellow, Aarhus University, Department of Political Science – Danish Centre for Studies in Research and Research Policy.

Shamed Into Good Science

6:10 am PST / 9:10 am EST / 3:10 pm CESTScientific reform has proposed various improvements to the research process, some of which have been put into place. These proposals, or innovations, include preregistration, registered reports, post-registration, (data) sharing infrastructures and -platforms and more. However, beyond the ‘process innovations’, scientific reform also generated new forms of collaboration towards knowledge-making in which novel approaches to harnessing trust and distrust are built into the collaboration, including but not limited to ManyLabs approaches. The many “ManyLabs” in existence display diverse and experimental epistemic and moral strategies and ideals for scientific organisation and practice. I will speak about the social processes that drive these new processes and colaborations.

The Invisible Workload of Open Research

6:35 am PST / 9:35 am EST / 3:35 pm CEST

It is acknowledged that conducting open research requires additional time and effort compared to conducting ‘closed’ research. However, this additional work is often discussed only in abstract terms, a discourse which ignores the practicalities of how researchers are expected to find the time to engage with these practices in the context of their broader role as multifaceted academics. There is a clear danger that additional expectations to engage in open practices add to the workload of often already overburdened academics and research professionals. Using theories of administrative burden and workload creep, I discuss individual and systematic responsibilities to mitigate this.

Why Replication Is Underrated

7:00 am PST / 10:00 am EST / 4:00 pm CEST

I will argue that replication is underrated because it can do more epistemic work than is usually acknowledged. In the current philosophical and metascientific literature replication is depicted as a backward-looking tool: it is used to increase trust in published data. I will argue that replication also plays an important forward-looking role, as it can help researchers to build trust in novel data that is being generated. To understand this forward-looking role it is important to build a better understanding of the ways in which replication is deeply integrated into everyday research practice (Guttinger 2019; Peterson and Panofsky 2021).

Communities of Practice in the Science Reform Movement – Recognizing and Mobilizing Diverse Contributions to Science Reform

7:30 am PST / 10:30 am EST / 4:30 pm CEST

When people write and talk about science reform, they refer to a movement, or a single community of practice centered on improving how science is done and shared. An ethnography and subsequent network analysis I conducted on the online part of the movement reveals evidence of a much more complex research and activism landscape, however. My work indicated that while there indeed exists a singular purpose, or ‘joint enterprise’ which unites much of the science reform’s activity, the group of people driving the movement tend to organise into smaller clusters within the broader landscape. They group together based on different priorities, methodologies and even philosophies of science; a constellation of small communities of practice. My talk presents this preliminary research, and outlines some interesting implications it might have for considering the future of the reform movement.

The Decline of Scientific Individualism?

7:55 am PST / 10:55 am EST / 4:55 pm CEST

A number of metascience innovations—such as the Psych Science Accelerator, Many Analysts projects, the Multi-PART study, and cloud labs—involve an explicit move away from individuals or small teams as the basic unit of scientific research. This talk will ask whether metascience as a movement might be hastening the decline of scientific individualism (if such a decline is indeed taking place!), and what epistemic and social risks might be associated with these changing work practices.

Meta Makeover: Ditching Dogmas and Raising Standards of Evidence

8:20 am PST / 11:20 am EST / 5:20 pm CEST

Metascience and scientific reform have initiated a social movement toward procedural changes in scientific practice. Prioritization of prescriptive recommendations (e.g., preregistration, replication) over theoretical understanding or evidence-based solutions has led to uncritical, even dogmatic interpretations of reforms. Evidential standards are relaxed when evaluating metascientific results that endorse reforms. Self-serving and self-congratulatory narratives have led to premature metascience hubris. Without intellectual humility, however, there is no hope for meaningful progress. I suggest it is time to course correct, time for (meta)scientists to ditch dogmas and heuristics, and refocus on science.