This blog post is the third in a four part series based on the keynote presentation by Stefanie Haustein at the Swiss Year of Scientometrics lecture and workshop series at ETH Zurich on June 7, 2023. Stay tuned for our next post, in which we’ll share opportunities and challenges of scientometrics, with a focus on the responsible use of metrics and conclusions.

Diversification of applications

I now want to talk about opportunities and challenges of scientometrics with regards to applications of scientometrics methods.

Collection management

Let’s go back to the roots of scientometrics… One of the first ever citation analyses was done by Gross and Gross, two librarians who decided which journals to subscribe to for a chemistry library.

To quote their 1927 paper in Science:

“One way to answer this question would be merely to sit down and compile a list of those journals which one considers indispensable. Such a procedure might prove eminently successful in certain cases, but it seems reasonably certain that often the result would be seasoned too much by the needs, likes and dislikes of the compiler.

In casting around for a better method of arriving at the answer, the writers decided to seek an arbitrary standard of some kind by which to measure the desirability of purchasing a particular journal. If one grants, to avoid argument, that the department is trying to train men, first, to understand the science of chemistry (including, of course, the methods and means of advancing the frontiers of the science) and, second, to be able actually to contribute to this progress, then it seems inquiry should be made into the library tools which men are using who are now doing just this.

With this purpose in mind, it was decided to tabulate the references in a single volume of The Journal of the American Chemical Society. This journal was chosen as the most representative of American chemistry. It is believed that the results of such a tabulation can be considered statistically and used with certain reservations to predict the future needs for a period, let us say, of ten years. The most recent complete volume (1926) of this journal has been chosen and the results tabulated in such a way that the relative importance of any single periodical for any five-year period can be seen. This is very important when one considers that only relatively few libraries can afford complete files of journals which have been published continuously for a century or more.”

Thus, citation analysis, but also usage analysis based on downloads, can inform collection management.

This is particularly true when it comes to canceling big deals, where having an evidence base of what content contained in bundles is actually used puts librarians in better positions when negotiating with large publishers—particularly those for-profit companies that control more than half of journal publishing.

Unpaywall, and their service Unsub, can be used to investigate whether journal subscriptions are still necessary if more and more publications are available open access.

With the shift to open access, bibliometrics has now become an important tool to track the cost of article processing charges. In a 2023 preprint (under review at Quantitative Science Studies), we estimated that, globally, the academic community paid more than one billion U.S. dollars to publish open access to the five-big publishers.

Publication statistics are also used in the context of transformative agreements. For instance, it allows publishers to determine flat rate fees and justify price increases, and librarians to monitor the use of agreements. 

Open science monitoring

Another new application area of scientometrics is open science monitoring. With an increasing number of open science policies, funders and policy makers are interested in tracking compliance. From open access, we know that rates of compliance with mandates are significantly higher when they are monitored.

Influenced by developments in open infrastructure, more and more dashboards are being developed to track open practices such as open access publishing and data sharing.

Above, I am showing examples of two dashboards: the Curtin Open Knowledge Initiative Open Access Dashboard and the Charité Berlin Dashboard on Responsible Research. There are many more examples, including the French Open Science Monitor and the European Open Science Cloud.

Most dashboards are similar to bibliometrics in the past: very data driven, tracking what’s available. We are taking a different user-centred approach with the open science dashboard for biomedical institutions. Instead of including what’s easily trackable, we first conducted a Delphi study to ask the community what they consider important to open science practices and what to track in a dashboard.

This is certainly challenging since many outputs—especially those that aren’t publications—lack the necessary metadata (e.g., institutional affiliations) to generate all indicators that the community deemed important.

Stay tuned for our next post, in which we’ll share opportunities and challenges of scientometrics, with a focus on the responsible use of metrics and conclusions. Check out the Swiss Year of Scientometrics blog for recordings of Stefanie’s lecture and workshop.

---

References

Gross, P.L.K, & Gross, E.M. College Libraries and Chemical Education. Science, 66, 385-389 (1927).