The Responsible Researcher Reinforcement Index

Now that I’ve finished my ARC Discovery proposal I have time to write a blog post, and Angela Moles’ tweet today made up my mind on the topic.

The tweet in question referred to an editor at a Biology journal, who went through 18 potential referees, all of whom said no to even reviewing it, before giving up in frustration and reviewing it themselves.

I find this sad and shocking, but unfortunately not surprising. I have seen this first hand, both through involvement with journals and from talking to other editors, and knowing some scientists personally (who I won’t name here) that literally boast about the fact that they referee nothing except for the one or two papers that, in all proper ethics, they shouldn’t review because they are conflicted, but do because it gives them control (and I’ve seen both positive and negative conflict here).

I have also seen people carry well more than their fair share of peer review, mostly out of responsibility and generosity and a desire to advance science over themselves.

The sad thing is, refereeing actually carries little reward beyond the warm glow some take from doing the right thing. And worse, those that dodge it actually get ahead, because we all have the same amount of time in our day — 24 hours — and those who don’t referee can spend their time on other things like writing more papers that need someone else’s peer review.

Remarkably, in my experience, some of those who referee least publish the most, and vice versa (n.b., I said some, this intrinsically means not all, ok). In this sense there is an inherent unfairness in the system. There is also the delays incurred by authors who can’t obtain timely, quality peer review, along with the frustration for editors, many of whom do their jobs pro bono.

In the modern era of worldwide online systems like Orcid and ResearcherID, I think there is an easy solution, and I call it the Responsible Researcher Reinforcement index, or RRR for short. The idea works as follows. All journals will only accept submissions from authors registered on and who have positive RRR credit. A researcher earns that credit by reviewing papers for journals, grants for funding bodies or for editorial tasks. There are fixed credits for these tasks, but they also carry a separate quality score (could be as simple as a reddit style thumbs up or thumbs down based on a good job or an acceptable job). The database is open and searchable, a la rate my professor, so that promotion committees, funding agencies, etc can see it, and the credits are directly transferrable for papers…

So, on publishing, each paper needs two referees, right. This means each paper requires two RRR credits, which can come from any (one or two) authors on the paper. This enables Ph.D. students and ECR’s who might be RRR credit poor to be supported by more senior candidates to some extent. Need to appeal to third or fourth referee, then that’s another one or two points. Ditto goes for funding applications; you pay for your reviews in RRR credits. In this way people who use the system the most have to contribute in a commensurate fashion. Those who publish less than they review are at least publically acknowledged at, and perhaps large credit balances can be used to offset open access fees down the line. The quality score helps to distinguish good referees and see them acknowledged in a way that’s disconnected from the exact paper/grant they reviewed. It also adds tangibility to the claims in CVs and grant applications about how much ‘regular refereeing for prestigious journals’ actually really gets done (is it one paper a year or fifty?). Note that the quality score is not determined by the authors or fund applicants receiving the review, it is decided by the editor and grant panel/manager, making it both objective and anonymous.

And this one is actually pretty easy to implement, because it’s in the best interests of all journals and all grant funding agencies, because they finally have a real carrot and a real stick to make refereeing happen in a good, timely manner. All it takes is enough cooperation to establish a site like Orcid, and a critical mass of key journals and agencies to back it. With the outrageous profit margins most journals have these days, the cost would be a small imposition.

Long term, one could even consider ‘floating’ the RRR credit, let’s call it the ₩ for convenience, like a currency. Reviewing long review papers could pay more ₩ than short papers, journals struggling for reviewers can pay more ₩ for reviews, etc. Turning this into a research currency might entail its own issues that should be considered separately and deeply, but as a basic concept, I can’t think of anything better to make the responsibility of refereeing get taken more seriously by researchers.

About these ads

2 thoughts on “The Responsible Researcher Reinforcement Index

  1. Great post Adam!

    I love the idea of making people’s contributions public.

    One small point: I think it needs to be three RRR credits per submission, because an editor has to read the ms, write comments and make a decision too (I think handling mss as an editor should also earn credits).

  2. Yep, sure, three credits is fine. These things would all shake out in the implementation anyway, the tricky thing here would be getting a publisher to actually take this concept up and use it, but it makes sense for them, as they’ve got the hardest job of all, finding people to actually do their refereeing!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s