From data sharing mandates to clinical trial registration, Open Science (OS) policies for biomedical research are in no short supply. But ensuring those policies become real-world practices can be a challenge—particularly when there’s no simple way to measure success. Launched in Fall 2021, the Biomedical Research OS Dashboard project hopes to overcome this barrier by helping research organizations track the use of the OS practices that matter most to their communities.  

“The reality is that just creating a policy for an open science practice isn’t sufficient in terms of its actual implementation,” explains Kelly Cobey, a Scientist at University of Ottawa Heart Institute and primary investigator on the project. “A policy is only as good as the adherence to it, so we want to make sure that we have a way for organizations to actually monitor that compliance.”

---

The Wellcome-funded project is a collaboration involving 80 researchers and stakeholder participants from 20 institutions around the world. It is led by an international team of researchers that includes ScholCommLab co-directors Stefanie Haustein and Juan Pablo Alperin, as well as leading scholars such as David Moher, Cameron Neylon, Emily Sena, Justin Presseau, Lars Hemkens, Rodrigo Costas, Thed van Leeuwen, Ulrich Dirnagl, and Daniel Strech. The project takes a novel approach that is part research, part action. The goal is to create a community-informed automatic dashboard that will allow institutions to track how consistently members are using recommended OS practices. Beyond measuring progress, the dashboard could be used to identify areas in which more support or training is needed. 

“Right now it seems like funders or policymakers come up with a policy and then tell institutions: ‘Okay, you have to do this now.’ Then institutions say: ‘Okay, researchers, you have to do this now.’ But the researchers are left in a situation where they may not be aware of why or how to comply with the OS practice and may not feel supported,” Cobey explains, “No one coaches them through it. That’s why you see the gap between policy and implementation.”

---

The project builds on the foundations laid out by Charité’s custom OS dashboard and Curtin Open Knowledge Initiative’s (COKI) open access (OA) dashboard. It will monitor a range of OS practices, much like the Charité dashboard, but will be developed, tested, and refined to reflect those practices that are seen as most valuable to stakeholders at biomedical institutions across the world. It’s being created by the community, for the community. 

“A lot of times people just get an idea and make a prototype, but after investing all that time and money and funding, they find that the tool doesn’t resonate with the community,” Cobey says, “We’re trying to circumvent that by involving them in the process and the user-design.” 

To identify the OS practices that should be included in the final dashboard, the project employs what Cobey calls an “integrated knowledge mobilization” approach. This includes using the Delphi survey method, which allows researchers to iteratively gather and integrate feedback from stakeholders throughout the research and prototyping process. This process involves a cycle of surveying stakeholders, synthesizing key findings and presenting them to participants, gathering feedback, and finally integrating that feedback into the next set of survey questions. The Biomedical Research OS Dashboard project team is currently knee-deep in the survey cycle, but hopes to publish a preprint of the final results in summer 2022. 

---

To ensure the final dashboard is useful across multiple biomedical research contexts, Delphi participants were recruited to support broad representation of geographies, genders, and career stages. Cobey says that the diversity of participating stakeholders has already yielded important insights. “It’s been really eye opening,” she says, “you have all these people with good intentions thinking about public accessibility, but not every intention is feasible in every local context.” 

Take, for example, the question of whether to include Open Access (OA) publishing in the dashboard. While OA compliance seemed like an obvious activity to include, Cobey says deciding on the right metric to track was complicated by international differences in policies and resources. In line with the recommendations laid out in Plan S, many European participants indicated in round 1 and 2 surveys that they wanted to track Gold OA (i.e., publication of articles in openly available journals). But scholars from regions with different OA policies, such as Nigeria and Latin America, felt that this would be inequitable. In biomedicine, publishing Gold OA typically requires researchers to pay high Article Processing Charges (APCs), which can be inaccessible for scholars with fewer resources. Scholars from these regions made it clear that they would not want to be benchmarked against the European standard. 

“Had I, from here in Canada, just come up with a metric, I might not have had that perspective,” Cobey reflects. “That’s why I think it’s so important to have a diversity of views.”  

“As researchers and research institutions, there’s the responsibility to maintain these best practices. We’re excited to provide a tool that makes that responsibility a little bit easier.”

Kelly Cobey

It’s still early days for the project, and there’s lots of work to be done. But Cobey is hopeful about what lies ahead. “We know that Open Science practices can increase the usability of research, support reproducibility, and reduce bias” she says. “At the end of the day, if we don’t publish our results, or we report them in a way that’s biased, it’s unethical. As researchers and research institutions, there’s the responsibility to maintain these best practices. We’re excited to provide a tool that makes that responsibility a little bit easier.”

To stay up to date with the Biomedical Research OS Dashboard project, sign up for the ScholCommLab newsletter.