Peer Review, A Tarnished “Gold Standard” | Eastern North Carolina Now

Publisher's Note: This post appears here courtesy of The James G. Martin Center. The author of this post is Richard Phelps.

    I recently submitted a manuscript to an education journal, a review essay of another scholar's work. It opened with a compliment of the author's "highly-praised and influential work." To that statement, one reviewer of my manuscript asserted that I used "emotionally loaded language of incredulity, dismissiveness, and hyperbole."

    More "tone policing" comments riddled the review, suggesting that even when my words might sound benign or complimentary, what I really meant was malevolent. There were several examples of another curious critique type as well: Raw declarations that my claims could not possibly be true, without any effort having been made to follow my footnotes to the evidence.

    Few of my manuscript submissions to education journals have been reviewed substantively. For those familiar with Paul Graham's hierarchy of disagreement, the reviews tend to lack a refutation of the central point, refutation in general, or counterargument. Instead, most consist of responding to (perceived) tone or being ad hominem (even anonymously) attacks.

    I consider it likely that this reviewer knew that I had written the manuscript. And that may have motivated the decision to volunteer. If I read the editor's reviewer numbering system correctly, three others had agreed to review but then failed to follow through. Who knows how many were originally asked?

    As it is, the traditional peer review system relies on unpaid volunteers from a small population of very busy people to perform an intensive and time-consuming task. Two types of scholars feel most compelled to review papers: Those intrinsically motivated (not necessarily for noble reasons) such as the aforementioned reviewer, and those who "have to," such as graduate students and not-yet-tenured professors.

    Assume an editor with a new manuscript chooses wisely and picks two eminently qualified reviewers, they accept the invitations, and then review the manuscript responsibly. Nonetheless, they are still just two people in an intellectual circle of thousands. The odds are small that their consideration of the manuscript will bring to bear all the relevant knowledge or even a representative sample of it. I have written elsewhere about the widespread plague of "dismissive literature reviews," time-saving and self-aggrandizing claims made by manuscript authors that no previous research exists on their topic. Those reviewing manuscript submissions generally lack the time to thoroughly review the new research presented, much less verify claims about entire research literatures.

    Scholars have tabled many proposals to tweak the current system. Two widely discussed changes are to pay the reviewers, or open the process by eliminating anonymity in reviews (and thus discouraging the self-dealing and cruelty toward authors it now makes possible).

    But these proposals maintain the traditional set-up: submit a manuscript, get it reviewed, then it's rejected or published. If rejected, the same hit-or-miss process starts at another journal. The traditional review process can take months, or even years.

    University and faculty raters construct their metrics in part from peer review results. Gaming these metrics corrupts the process with fake co-authors or reviews, citation cartels, co-citation agreements between journals or universities, reviewer- or editor-coerced citations, or purchased citations or by-lines.

    Some published articles may later be retracted, but they represent a small proportion of those that probably should be, and even they are largely limited to the hard sciences. Retractions in the social sciences and humanities are far less common and, in education, less common still.

    Given the well-deserved reputation of inferior quality in education research, its lower rate of retraction is more worrisome than reassuring.

    One of the more intriguing solutions to peer review's limitations and other faults would be to publish and then peer review.

    With "post-publication review," a submitted manuscript that meets the many other standards for publication (e.g., readable, understandable, relevant), is posted online as a temporary preprint and then critiques are welcomed via a broadcasted invitation to a wider circle of potential reviewers. (Of course, depending on how open the process is, some of the reviews might be irrelevant or unhelpful, just like the comments posted to many popular websites.)

    PubPeer offers this process to a large extent and has already altered the fate of some articles. But it remains limited in popularity and seems to be most commonly used as an addendum to the traditional process. PubPeer's allowance for anonymous reviews also engenders controversy.

    Advantages of post-publication review include:

  • Increasing the size of the pool of intrinsically motivated potential reviewers manyfold
  • Reducing the reliance on overworked junior faculty, who may reallocate their time to reviewing articles of more interest
  • Reducing the time and expense of the back-and-forth searching for and corresponding with potential reviewers.
  • Getting new research findings out into the world more quickly
  • Increasing process transparency
  • Reducing opportunities for corruption among authors, editors, and reviewers
  • Reducing opportunities for dismissive literature reviews (as those knowledgeable of prior relevant studies are provided the opportunity to contradict them publicly)

    The traditional peer review process demands enormous amounts of time while providing only hit-or-miss and superficial appraisals of research quality. The result is a proliferation of scholarship of widely varying quality and much duplication of effort. This regime benefits scholars individually by thickening their curriculum vitae.

    We may forget, however, that citation and publication numbers are not supposed to be ends in themselves but, rather, just proxies for adding useful and accurate information to society's store of knowledge. Society might be better served by fewer scholarly journals publishing fewer articles that have been much more widely and thoroughly vetted.

    Richard P. Phelps is founder of the Nonpartisan Education Group, editor of the Nonpartisan Education Review, a Fulbright Scholar, and a fellow of the Psychophysics Laboratory. He has authored, edited, and co-authored books on standardized testing, learning, and psychology.
Go Back


Leave a Guest Comment

Your Name or Alias
Your Email Address ( your email address will not be published)
Enter Your Comment ( no code or urls allowed, text only please )




What They Are Saying James G. Martin Center for Academic Renewal, Editorials, Op-Ed & Politics Emergency Meeting Of the Beaufort County Board Of Elections at 5 PM Today, Friday, July 16th


HbAD0

Latest Op-Ed & Politics

President Joe Biden took direct aim at Supreme Court Justice Clarence Thomas during a recent interview, referring to him simply as “the guy who likes to spend a lot of time on yachts.”
The best way the county and city can help hold down inflation is to resist all tax increases
Pope Francis lambasted leftist gender ideology during an address this week, warning that it presented an extreme danger to mankind.
amnesty would just encourage more illegal aliens to storm our borders
The Christmas candy was barely off the shelves when the Valentine’s candy appeared. Red and pink hearts with caramel and nut-filled chocolate goodness caught our eye. We are reminded of how we love love. Young love, especially.
far left sugar daddy has also funded anti-Israel groups and politicians in US
Be careful what you wish for, you may get it
America needs to wake up and get its priorities right

HbAD1

 
Back to Top