Lessons learned about peer review from the Remote Sensing affair

This is the second entry here in The Uncertaintist concerning the resignation by Wolfgang Wagner from the editorship of Remote Sensing. Wagner explains his decision here,

http://www.mdpi.com/2072-4292/3/9/2002/pdf

The earlier entry,

https://uncertaintist.wordpress.com/2011/09/30/an-%e2%80%9coff-topic%e2%80%9d-exploit-against-a-peer-reviewed-journal/

showed that one concern about a controversial paper which Wagner published was misplaced. There appear to have been other problems, however, both with the paper and with the journal’s standard operating procedures for review..

The controversial paper was not an “off-topic publishing” exploit. However, that the authors disclosed their intentions to promote a viewpoint about global warming underlies one concern about how the paper was reviewed.

That concern is not that a non-specialist journal like Remote Sensing ought to have avoided completely papers in a controversial specialized field like global warming. Of course its editors lacked domain-specific expertise. Remote Sensing’s subject is inherently interdisciplinary. For its editors to evaluate papers in its announced field, they will necessarily venture into areas where the editors lack special expertise.

That is an argument for the editors to take appropriate precautions to locate suitably qualified reviewers, not to avoid controversial application papers altogether. And so arises the valid concern. If the authors said what their paper was about, then wasn’t there an opportunity to ensure that at least one reviewer was a neutral or even hostile expert in global climate change, someone who would have anticipated the poor reception paper would receive?

 

The chief complaint

The main shortcoming of the paper, according to Wagner, is that its authors used methods to analyze satellite sensor data that were similar to what other climate-change “sceptics” had already published, and which had been roundly criticized by the larger climate-change community. The authors did not address these criticisms when they presented their analysis, although the criticisms were applicable to them.

If so, then that is a flaw whether or not the authors are right about the merits of their model. That is a flaw which has nothing to do with the authors’ position on global warming.

Why weren’t the application community’s concerns about the papers’ methods revealed by the reviewers? Supposedly, all three reviewers happened to have been sympathetic to the authors’ viewpoint. Well, how could that happen, that a minority opinion is the only one represented among the reviewers? Luck of the draw?

Not necessarily. Two possible policy mechanisms might account for the lapse. Authors sometimes spontaneously nominate potential reviewers, and editors sometimes request that authors propose possible reviewers. Obviously, this practice creates risks if the paper is controversial in its application community. Ironically, the less an editor knows about a field of application, the more he or she might appreciate some leads in the search for reviewers.

Wagner says only that “The managing editor of Remote Sensing selected” the three reviewers, and doesn’t say how the managing editor got the reviewers’ names. Once the reviewers were chosen, no “alarm bells” went off until after the paper appeared in the journal. The editor was impressed with the scientists’ senior positions at reputable institutions and their extensive publication records. Their reviews seemed to be of good technical quality, and weren’t “gimmes,” fawning endorsements of the authors’ position. There was a “good spread” in the action recommendations: one each of: accept as is, require a minor revision, and require a major revision.

There is a second mechanism, other than author nomination, that could account for the failure of the reviewers to disclose that the authors hadn’t addressed known criticisms of their approach. Editors very often develop leads for potential reviews from the literature that the submission cites. The authors in this case cited 14 earlier works, many of them multiply authored. Excluding the authors themselves, their colleagues at the same institution, and maybe known past collaborators leaves a rich list of apparently promising candidate reviewers.

In the scientific research publication, authors usually only cite work which they actually use and discuss in their paper. There isn’t any tradition of proposing a wide ranging, general purpose bibliography covering the field except in so-called “survey” papers. Since the authors did not discuss criticisms of the methods they used, the authors of the work they did cite might be unrepresentative of those active in the field.

Searching for reviewers among those whose work the authors cite would increase the risk of engaging reviewers who might not object to the authors’ way of telling the story. The risk may be comparable with that of accepting nominations from the authors outright.

One lesson learned, then, is not to rely exclusively on the authors for leads to reviewer candidates, not explicitly and not implicitly. Author nominations do have value. In the event of conflict with the authors about the reviews. It is helpful to be able to say to the editor in chief or publisher that the authors “chose” some of the reviewers, either directly or by relying on their work.

The power to choose the reviewers, or even influence the choice of reviewers is a great lever in the peer review system. Giving up that power entirely to the authors is dangerous.

 

Procedural rigidity and alternatives

Wagner concludes his description of the reviewing history with the following sentence.

The authors revised their paper according to the comments made by the reviewers and, consequently, the editorial board member who handled this paper accepted the paper (and could in fact not have done otherwise).

Why not? Fine, if the editorial board member who was assigned the manuscript had no reason to overrule the judgment of the reviewers. Maybe that was the case here, and so the authors’ compliance with the reviewers’ suggestions settled the matter. But Wagner says the review overseer “could not” do other than to accept the paper. That discloses a flaw in the journals’ procedure.

For example, suppose late in the process, the editorial board member had visited the blog of one of the authors, or seen the name of one of the authors mentioned in a news story about the controversy over global warming. “Hey,” says the member to herself, “None of the reviewers said anything about this paper containing anything controversial. That’s odd.” And suppose further the member visited the reviewers’ web sites and caught on that the authors and the reviewers, however this may have happened, were all on the same side in a hard fight.

Could not the editorial board member have done otherwise than to accept the paper? At the very least, the member should have been able to ask for an additional reviewer. Under the circumstances, and especially if the existing reviewers had been nominated by the authors or were all represented among their citations, the board member could communicate the newly discovered concerns to the authors while asking their patience with the delay another review would require.

Even stronger measures might be taken. Wagner now understands the seriousness of the nondisclosure of known applicable criticism in the literature. He probably would have understood at the time, too, if he had been made aware of the situation. Would he not have been receptive had the editorial board member recommended overruling the reviewers?

It is true that accepting such a hypothetical recommendation would have created a problem at Remote Sensing. In order to pay its expenses, Remote Sensing expects its authors to pay 500 Swiss francs per article in fees, plus up to 250 additional Swiss francs if additional editorial work is required. At this writing, a Swiss franc costs about 1.11 United States dollars.

The fee is conditional on acceptance. Since Remote Sensing is electronically distributed, the work needed to review a paper may be a greater proportion of the total expense and effort of publishing a paper than at a conventional paper-and-ink journal.

Nothing in the above should be taken to imply that Remote Sensing makes its publishing decisions based on selling space in its virtual pages. It is simply a recognition that a policy about editorial board member prerogatives might need to accommodate a range of concerns.

In an earlier article about a peer review dust-up at Science, I remarked that that journal’s on-line prepublication procedures offered an opportunity for the community to “second guess” peer review before papers appeared in the final archival printed-on-paper edition. In the case studied, the reviewers’ decision was ultimately reversed (by the authors’ withdrawal of the paper) because new information was revealed after the paper was available for general public inspection.

Clearly, an online journal could easily add a pre-publication community-comment component to its existing review procedures. Authors’ fees could be assessed at that stage, in whole or part, to ensure that the journal is fairly compensated for its effort to bring papers along that far.

 

Recovery

Now that the paper has appeared, what can the journal do about repairing its archival integrity? The editor’s resignation does wonders for the community’s perception of his good intentions. It is unclear that such a gesture does anything for the journal, except possibly to spare the editorial board member whose hands were apparently, and maybe unwisely, tied.

An obvious remedy to any damage done would be to invite a rebuttal paper or guest editorial that explains the community’s concerns with the modeling approach taken by the authors.

In fact, the journal has chosen to do that, in spades. Submission are currently being solicited for a special issue of Remote Sensing devoted entirely to the subject of what role human activity plays in global environmental change, including climate change,

http://www.mdpi.com/journal/remotesensing/special_issues/human_global_change/

The guest editor, Prof. Dr. B. L. Turner II of Arizona State University, appears twice on a list of scholars of whom a climate change sceptics’ site says

All these people and many more, must be held accountable for corrupting the minds of our children. It is a major and unpardonable crime.

http://www.seafriends.org.nz/issues/global/hall_of_shame.htm

One senses that a disturbance in the force may be about to be balanced.

Leave a comment

Filed under Scientific publishing

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s