What do authors and editors think about peer review? A cross sectional study in 12 journals across research fields

Objective: To assess satisfaction of authors and editors with the quality of reviews and reviewers’ recommendations in a pilot study of 12 Elsevier journals across four disciplines.

Design: Cross-sectional study in May-December 2014. Using convenience sampling of five manuscripts/week journal administrators submitted 1340 reviews containing reviewer-recommendations, and 1068 editor (80% response rate), and 336 author (25% response rate) perceptions regarding those reviews. In total, we had information from 794 different manuscripts. For 328 (41%) we had reviews from a single reviewer, for 399 (50%) from two, and for 69 (11%) from three or more. Two independent raters used modified version of Review Quality Instrument (RQI) to assess review quality. We also determined associations among: 1) authors’ perception of the reviews; 2) editors’ opinions regarding review timeliness; 3) editors’ opinion on review’s impact on decision; 4) review quality, measured by RQI; and 5) reviewers’ recommendation (accept, revise, rejected).
Results: Overall median RQI score of reviews was 18 (95% CI 17-18, IQR 15-22, range 10-40). Authors were more satisfied with reviews recommending rejection than for recommendations to revise or accept (Md = 4, IQR 4-5, χ2 = 41.7245, P<0.0001) and we found statistically significant inverse correlation between author satisfaction and review recommendation (Spearman's rank correlation coefficient (r) = -0.467, 95% CI -0.602 to -0.320, P<0.0001). Editors' opinion about influence of the review and RQI (r = 0.149, 95% CI 0.029-0.269, P=0.008) and review recommendation (r = 0.206, 95% CI 0.013-0.360, P=0.013) were positively correlated.

Conclusions: Paradoxically, authors were most satisfied with reviews recommending rejection perhaps reflecting their perception of the review as helpful regardless. Moderate correlations between the quality of the review, review recommendation, and editors’ opinion about the influence of the review indicate that better reviews are more helpful to editors. Our findings additionally emphasize the role of the editor to assess regularly both availability of reviewers who produce high quality reviews and authors’ perception of those reviews as well as highlight the importance of an objective instrument to assess review quality.