Is peer review just a crapshoot?

Posted on March 31, 2015 by

0


Elsevier (31 March 2015); by Richard B. Primack, Ahimsa Campos Arceiz, and Lian Pin Koh — How do reviewer recommendations influence editor decisions? And are Chinese authors treated fairly?

An important concern in the scientific publication process is how well reviewers evaluate the quality of papers and how their recommendations influence editors’ decisions to accept or reject papers. Are there some comprehensible patterns in the review process or is it just a crapshoot — a random process with reviewers and editors making arbitrary decisions?

Additionally, a particular concern for researchers in China and other countries with rapidly developing scientific communities is whether there are potential biases affecting their manuscripts in the review process. Do editors treat authors from China fairly, or are their papers given an especially harsh treatment?

In our study, which was recently published in the journal Biological Conservation, we examined 4,575 manuscripts submitted to Biological Conservation over the past 7 years,and a selected sub-set of 2,093 papers that were sent out for review.

Right from the beginning, we could see that Chinese authors face special difficulties in getting their papers published: 71% of the papers from China were rejected without review in contrast to a far lower 50% of papers from English-speaking countries. Although identifying the causes of the higher rejection rate was beyond the scope of our paper, the reasons are probably due to a combination of the topics being less relevant to the journal, lower quality of research work, and less effective presentation of the material.

How do reviewer recommendations affect editorial decisions?

Of the articles that were sent out for review, reviewer recommendations strongly influenced the outcome of the review process and the decisions by the editors. Papers having two reviewers’ recommendations to accept and minor revision have much greater chance of acceptance (more than 98%) than papers receiving two reviewers’ recommendations that include a major revision and a reject or two rejections (which results in an acceptance rate of less than 5%): (Figure 1).

Figure 1. The outcome of papers receiving particular pairs of reviewer recommendations. Bars represent 95% Confidence Intervals (CI) of the predicted probability of acceptance (note that CI values for accept-accept, accept-minor, and reject-reject could not be estimated because all manuscripts in these categories were either accepted or rejected).

Figure 1. The outcome of papers receiving particular pairs of reviewer recommendations. Bars represent 95% Confidence Intervals (CI) of the predicted probability of acceptance (note that CI values for accept-accept, accept-minor, and reject-reject could not be estimated because all manuscripts in these categories were either accepted or rejected).

When articles from China and English-speaking countries were compared, similar reviewer recommendations resulted in comparable outcomes. However, papers from China that were sent out for review had more negative reviewer recommendations of reject and major revision than did papers from English-speaking countries (Figure 2).

Figure 2. The percentage of pairwise reviewer recommendations for papers from English-speaking countries and China. Larger circles indicate a greater percentage of pairs in that category.

Figure 2. The percentage of pairwise reviewer recommendations for papers from English-speaking countries and China. Larger circles indicate a greater percentage of pairs in that category.

Of papers that were sent out for review, the acceptance rate is 51% for papers from English-speaking countries and 42% for papers from China. If we consider the fate of all the submissions, including the papers that were rejected without review, papers from English-speaking countries have about a 25% acceptance rate and papers from China have a 12% acceptance rate.

Reviewer rejection as the “kiss of death”

In considering the effects of reviewer recommendations on editors’ decisions to accept or reject a paper, it is surprising that even though there are four different categories of reviewer recommendations (accept, minor revision, major revision and reject), editors clearly do not use these categories on a simple linear scale. Reviewer recommendations of accept and minor revision are virtually identical in their positive effects on editor decisions; two such recommendations of Accept and Minor result in a greater than 98% of acceptance (Figure 1).

In contrast, papers receiving at least one recommendation of reject (“the kiss of death”) were almost always rejected; in fact receiving even one reviewer recommendation of reject lowers the probability of a paper being accepted by more than 50% when combined with a reviewer recommendation of accept or minor revision. Recommendations of major revision resulted in a greater than 16% decrease in the probability of the paper being accepted in contrast with a recommendation of a minor revision.

Variability of reviews has a negative outcome

Papers with more consistent reviews (e.g., both reviewers recommending a major revision) had a greater chance of acceptance than did papers with more variation (e.g. minor revision and reject). Similarly, papers with a minor and major recommendation had a great probability of acceptance than papers with reviewer recommendations of accept and a reject. These results are likely due to editors weighting negative recommendations more heavily than positive ones, especially recommendations of reject.

We found no evidence of editor bias against papers from China when receiving the same reviewer recommendation; however, reviewer recommendation for papers from China had a greater degree of agreement than did reviewers of papers from English-speaking countries (e.g., intra-class correlation of 0.25 vs. 0.55), due to reviewers of papers from China often agreeing that papers should be rejected or require major revision. The significant agreement between pairs of reviewers who are reviewing the same paper (as reflected by significant intra-class correlations) for papers from both China and from English-speaking countries indicates that reviewers are able to determine the quality of the paper using objective and repeatable criteria. However, the fact that these correlations are much lower than 1.00 also indicates that most of the time the two reviewers of the paper are coming up with different opinions on the quality of the paper.

Harsh reviewers from China

In their cover letter, authors from China sometimes request reviewers from China, presumably hoping that Chinese reviewers will provide more favorable or sympathetic reviews than will reviewers from outside of China. In fact, we found the opposite, with reviewers from China judging papers from China more harshly (43% recommending rejection) than did reviewers from other countries (with only 15 % of the reviewer from the UK recommending rejection).

Reviewers provide useful information, and editors make reasonable decisions

Our results demonstrate that the review process is not a crapshoot; reviewers are providing useful information and editors are using this information to make reasonable decisions. Authors from a non English-speaking country such as China have a tougher time getting their work published, but this is not due to a negative bias by reviewers from English-speaking countries or editors. The next step in evaluating the review process might be to examine the usefulness of the extensive comments of the reviewers in the decision-making progress. And also, more attention needs to be devoted to supporting scientists from China and other developing countries to help them reach their potential in carrying out and publishing their research.

Selected references

  • Baethge, C, Franklin, J, Mertens, S: “Substantial agreement of referee recommendations at a general medical journal – A peer review Evaluation at Deutsches Ärzteblatt International,” PLoS ONE (2013)
  • Bornmann, L., Mutz, R., Daniel, H-D: “A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants,” PLoS ONE (2010)
  • Lee, CJ, Sugimoto, C.R., Zhang, G., Cronin, B: “Bias in peer review,” Journal of the American Society for Information Science (2012)
  • Primack, RB:  “Editorial: Why did we reject your article?” Biological Conservation (2009)
  • Primack, RB, Ellwood, E, Miller-Rushing, AJ, Marrs, R, Mulligan, A: “Do gender, nationality, or academic age affect review decisions? An analysis of submissions to the journal Biological Conservation,” Biological Conservation (2009)
  • Primack, RBMarrs, R: “Editorial: Bias in the review process,” Biological Conservation (2008)
  • Xie, Y, Zhang, C, Lai, Q, “China’s rise as a major contributor to science and technology,” Proceedings of the National Academy of Sciences (2014)