Nature: la valutazione bibliometrica rende la scienza “less efficient and reliable”

«… when researchers apply strategies that boost individual and institutional performance metrics, by publishing papers as often as they can in high-profile journals, scientific research as a whole becomes less efficient and reliable». Su Nature, Philip Ball  commenta alcuni articoli che studiano, anche quantitativamente, come i ricercatori si adattano agli incentivi che ricevono, tra cui, in particolare, quelli legati a valutazioni bibliometriche. La conclusione, va esattamente nella direzione opposta rispetto alle politiche della valutazione della ricerca attuate in Italia negli ultimi cinque anni:  «Whenever quantitative metrics are used as proxies to evaluate and reward scientists,» scrivono Smaldino and McElreath, «those metrics become open to exploitation if it is easier to do so than to directly improve the quality of research.» Se uno degli articoli citati si intitola “The natural selection of bad science”, bisogna dire che in Italia siamo andati persino oltre. Se consideriamo il ricorso sistematico a metodi notoriamente pseudoscientifici (uno tra tutti: la somma di numeri ordinali)  e l’uso di metriche screditate e dannose (l’Impact Factor, per esempio), da noi c’è ben poco di naturale e si può tranquillamente parlare di “The planned selection of bad science”.

«… when researchers apply strategies that boost individual and institutional performance metrics, by publishing papers as often as they can in high-profile journals, scientific research as a whole becomes less efficient and reliable». Su Nature, Philip Ball (“The mathematics of science’s broken reward system“)  commenta alcuni  articoli che studiano, anche quantitativamente, come i ricercatori si adattano agli incentivi che ricevono, tra cui, in particolare, quelli legati a valutazioni bibliometriche: «Bibliometric data sets mean that scientific success can be quantified with achievement metrics such as the h-index — a quantitative measure of the number of highly cited papers that an academic produces — or the journal impact factor, which proposes to identify the most visible journals». E non mancano le preoccupazioni:

Many researchers have warned that the availability of these metrics, which are supposed to make the management of science and funding more systematic and objective, may be changing the nature of science. They are starting to dominate how science is structured and steered and place great pressure on researchers, especially in the earlier stages of their career, to publish often and prominently. As a result, understanding science as a social phenomenon has become a matter of some urgency.

And by quantifying goals and rewards, metrics transform science into the kind of game-theoretical play of utility maximization and cost-benefit analyses with which economists and ecologists have long become familiar. It is only, I think, because science is now so driven by metrics-based incentives that such models and analyses can meaningfully be applied at all.

La conclusione, va esattamente nella direzione opposta rispetto alle politiche della valutazione della ricerca attuate in Italia negli ultimi cinque anni

“Whenever quantitative metrics are used as proxies to evaluate and reward scientists,” write Smaldino and McElreath, “those metrics become open to exploitation if it is easier to do so than to directly improve the quality of research.” That’s basically a statement of Goodhart’s law, familiar to economists: when a measure of success becomes a target, it loses its worth.

Se uno degli articoli citati si intitola The natural selection of bad science, bisogna dire che in Italia siamo andati persino oltre. Se consideriamo il ricorso sistematico a metodi notoriamente pseudoscientifici (uno tra tutti: la somma di numeri ordinali)  e l’uso di metriche screditate e dannose (l’Impact Factor, per esempio), da noi c’è ben poco di naturale e si può tranquillamente parlare di “The planned selection of bad science”.

Sempre su Nature:

Fewer numbers, better science

Scientific quality is hard to define, and numbers are easy to look at. But bibliometrics are warping science — encouraging quantity over quality. Leaders at two research institutions describe how they do things differently.