“This paper is absolutely ridiculous” – peer reviews that hurt.

April 8, 2020

HKAH Foundation Fellow Ken Hyland left Hong Kong in 2017 but continues to be a prolific researcher at University of East Anglia where he is Professor of Applied Linguistics in Education. Ken’s overarching project is an investigation of how academic writing has changed in the last 50 years as a consequence of the increasingly competitive pressures of publication and a numbers-driven appraisal culture. Ken gives us insight into his current research, drawing on work out this year in the Journals of English for Academic Purposes and  English for Research Publication Purposes on harsh peer reviews and the need for objective screening.

Ken Hyland

Ken Hyland

Despite the massive changes in academic publishing in recent years, one thing remains more or less constant: the disdain many academics feel for peer review, and perhaps for peer reviewers themselves.  A means of separating the sweet from the sour, peer review acts as a filter for readers, a litmus test for writers, and a screening process for editors. But this is possibly the most contentious and secretive practice in our academic lives, reviled and tolerated in equal measure but rarely loved.  Slow, biased, contradictory, hurtful or wilfully obtuse, reviewers come in for a lot of stick.  For many of us, then, with careers increasingly tied to whether (and where) we publish, peer review can be a fraught experience with professional lives hanging on decisions.  But while no one enjoys having their work critically dismembered, sometimes it is not just the rejection that stings the most, or even the extent of revisions that reviewers demand, but the reviewer comments themselves.

 With my collaborator Kevin Jiang, I decided to explore what authors seemed to find most wounding in reviews by examining a corpus of 850 excerpts posted by authors on the shitmyreviewerssay website hosted on tumblr (link). The site states that these extracts are

“a celebration of the harsh, the weird, the passive aggressive, the active aggressive and the downright mean comments practicing scientists receive on a regular basis”. (link)

What we found was that while some comments were directed at the argument or at language failings, most gave a damning assessment of the work as a whole, as here:

  • This work is antithetical to the spirit of research and will impede potentially important developments.

  • Unfortunately, I cannot recommend this paper for publication because its contents violate the laws of physics.

  • The writing and data presentation are so bad that I had to leave work and go home early and spend time to wonder what life is about.

These unequivocal statements on the value of the paper as a whole are damning, but perhaps equally wounding for authors is personal criticism of their competence, comprising a quarter of all the items in the corpus:

  • It is clear that the author has read way too much and understood way too little.

  • The authors are amateurs.

  • In addition to tough comments, these evaluations were expressed unequivocally using a range of boosters and attitudinal markers

  • This paper must be rejected, because the work it describes is clearly impossible.

  • The writer of the manuscript is utterly ridiculous and appears to believe they will solve poverty through radio astronomy.

  • I have rarely read a more blown-up and annoying paper 

I don’t want to contribute to the reviewer-bashing widely seen on the internet and in university corridors and don’t believe these texts are typical of the thousands of peer reviews written every day in every discipline.  Our study didn’t differentiate by discipline, but my impression is that we see less of these types of comments in the humanities, perhaps because we are more sensitive to the socio-pragmatic impact of language, but mainly, I would guess, because of the fact that the top journals in our fields accept only 10% of submissions while the sciences have a ‘bias to publish’ policy so that less gets desk rejected.  In 15 years of editing two SSCI journals I came across only a handful of caustic reviews and a recent study shows that the more revisions a paper undergoes, the more citations it gets, indicating that reviewers actively contribute to the text.  But obviously not all reviewers are mentors, working with authors to bring manuscripts to publication. The fact is that some reviews are caustic and unhelpful, and while anonymity might help prevent personal bias, it can make reviewers less accountable.

The main problem lies in the fact that publishing now drives research, rather than the other way around.  Some eight million academics across the world now produce over 2.5 million new peer reviewed articles a year, so just one publisher, Elsevier, made use of 700,000 peer reviewers in 2018 alone to conduct 1.8 million reviews. But while the population of researchers is growing, the pool of potential reviewers may be shrinking, certainly the proportion of writers to reviewers is shifting significantly. Reviewer shortages are, in addition, happening at a time when reviewing has become a marginalised part of an academic’s role, so that even willing reviewers are often forced to do a hurried job.  Reviewing is done without reward or credit, and so time for it suffers as universities demand more teaching, more admin, more outreach, more research and more everything else. This requirement for ever-increasing numbers of reviewers, moreover, has also occurred alongside an ever-narrowing coverage of journals, so there are now over 700 language and linguistics journals and 1000 history journals on the SJR list, for example.  This increases the demand for reviewers with specific expertise, making it harder for editors to find suitable readers. Editors therefore turn to untried individuals and the quality of reviews may suffer. 

Our objective in writing the paper I am discussing here was not only to describe the ways reviewers expressed hurtful comments, but also to contribute to a wider conversation about the feedback academics receive on their work to encourage more mentoring and formative practices. In the end, peer review is the value added to a paper by careful checking, but it is also the source of the academic community’s sense of fairness and self-reflection.  Negotiating a paper to publication is a way, albeit often a fraught and distressing way, for writers to stay in the professional game.

Keep reading:

Hong Kong Academy of Humanities HKAH