This post follows an interesting recent job as an adjudicating editor for a paper that underwent two rounds of review, was rejected and sent back under appeal. The first round was: reconsider and submit elsewhere. The second round was: submit elsewhere, accept and submit elsewhere. This scenario is most authors’ worst nightmare: almost there but not quite. The appeal was textbook “How to not convince an editor to overturn the referees”, which is sad because it can be done, but it takes a very calm head and good tactical approach. It’s far from the first time I’ve seen cases where the authors’ have written an appeal that’s about as successful as the Vasa as an adjudicating referee or editor, so I figured I’d post some useful observations for how to or not to deal with referees when you don’t have a full hand of accept or accept with very minor revisions come back. (n.b., points below are not limited to that most recent paper or entirely motivated by it).
Being annoyed is ok, just don’t let it show: Everyone hates rejection. The key is to not show it. If that means sitting on the reports for two weeks fine. If that means walking around the corridor telling everyone how much those comments suck so they can remind you they get the same too, that’s also fine. Running, boxing, smashing plates, whatever it takes, expel the rage, get a zen like calm, then move on.
Ad hominem attacks on referees never work: Seems obvious I know, but you see them again and again. The editor is always going to side with their referees unless they’ve said something mind-blowingly stupid or inappropriate. Even if they do, you can be tactful about it. Questioning the referee’s competence, haste, or mental state is never helpful. Accusing them of impropriety, deception, etc. likewise.
Forget guessing who they are: You are always wrong. I’ve only ever seen one exception. It was clear the authors thought Person X was Referee F. Person X was certainly a referee, but they were Referee Q instead, and actually on their side. Some referees are good at masking themselves. They have the advantage in that. Don’t play this game, you never win.
Be unemotional in your response: A good editor will know you’re angry and just ignore any emotion and vitriol in your argument. A bad editor will let it colour you. Either way, it’s not the shown emotion that’s your worst enemy here. It’s the fact that the emotion kills your ability to carry off a very cool objective argument. If you have a strong point, you can back it with facts. You don’t need emotion. If anything, you can be completely concessionary and your argument will still draw blood if it’s sharp.
If you disagree, back it with facts and literature: The editor is not a fool. They will back their referees as an a priori position, always, but they are open to logical argument. Make sure that argument is as clear for a non-expert as you can make it but absolutely backable with evidence. It can help to point out what the referee is right about along with what they are not right about, because it’s rare that someone is 100% wrong. Be reasonable.
Make sure your logic is bulletproof: Try to anticipate counterarguments: where can you be defeated? Avoid things that are easily shot down. For example, saying that the fact that a report was returned within a few days means a proper job hasn’t been done is dangerous. Does the fact that a referee took 3 months mean they thus spent the entire 3 months on the report and it’s intrinsically better as a result? Probably not. Argument killed. Don’t put forward an argument that you cannot logically defend.
Always look to make concessions: Taking up “The referees are stupid, we’re changing nothing…” heel-dragging stance will always get you nowhere. In contrast, pointing out that a referee is correct or perhaps not completely correct, but you can see why they got confused and made changes x, y and z to make sure no one else gets confused the same way will get you everywhere. Editors love the hell out of this. The referees can be wrong and the editors will still find their value. They are the canary in the coal mine for your paper. They point out the obvious flaws could statistically hit 1/n of your readers (where n = number of referees) based on a small discrete sample. Fix the holes.
No tinfoil hats: Do not ever go down the path of accusing the whole set of referees of some kind of intrinsic collective bias against your field or against experimental/theoretical papers or against your work. The referees don’t know who the other referees are. They cannot be part of some crazy Illuminati-style conspiracy against you. Even if you suspect it, just don’t even go there…
Look where the train crashes: If a referee gets hung up at some aspect of the paper and then says they can’t see the significance or impact or whatever, it’s usually a sign that you lost them. A lost referee is rarely going to write an eight word report “I don’t get any of this, reject it”. What they normally do is pick it apart on the bits they know, and then reject the paper on the criteria. This will look like they only made technical arguments and have no justification to reject. The actuality for an editor, who knows who the referees are remember, is that it points out how broad your audience can be, because you can rely on people the same distance from your field as that referee having equal struggles with the paper. The clue for you is that the hang-ups point out where you start doing a bad job keeping your readers. This can be a powerful tool for improvements. Don’t think ‘what’s wrong with the referees?’, think instead ‘what do their comments tell me about how far they got in before I lost them?’. It could well be the first column, in which case you have a lot of work to do. All is not lost though — I have seen this fixed and turned into acceptances before (and done it myself too). It just takes some very clever mastery of techniques for handling referees.
The higher the journal, the broader the referees: It stuns me how many people submit to high impact journals somehow expecting that the set of referees will only contain experts in their tiny little subtopic. And then decide the way to rebut referees is to attack their technical expertise level. The higher journals never give you this set of referees, they want to find out what will happen to this paper in a diverse(read as non-expert) audience. That doesn’t mean they’ll send you to people who are clueless about the topic. But they will send you to people that are far enough away that they’ll get lost if you haven’t written your introduction and conclusions exceptionally well. If your referees don’t get it, I’m sorry but it’s not because they aren’t expert enough to appreciate it, it’s because you haven’t done a good enough job at writing for a broad audience. The only way to make amends in an editor’s eyes is to resubmit with major improvements on this side, no matter what round of revision it is. If you don’t, an editor can rarely help you because the referees’ comments stand and there’s really no way out of the fact that if they didn’t get it, the readership probably won’t either.
Just because you made revisions, doesn’t mean referee won’t still reject: Another flawed expectation from some authors. Some referees have a high bar, particularly in the top journals. Sometimes the paper is improved, but it’s still just not there. Some authors have a lottery approach to this, they grudgingly give concessions and go through rounds of revise and resubmit or revise and submit elsewhere until they find a set of referees that will take the work. The better approach is to keep working on improving your writing and upping the effort you put into making revisions. On the latter, always give 110% to revisions, go the extra mile if you can. Referees and editors are so used to crappy rebuttals and pathetic changes that even moderate efforts can have massive positive impact.
In the end, key points here, respect the referees, look for the hidden clues about how to make improvements, if you must argue then back it with facts, and sell on the improvements you’ve made rather than going head to head. I’ve never seen anyone go from reject to accept simply by going to war with the referees. It’s a losing strategy 100% of the time.