What editors look for in referee reports

A recent discussion in orgtheory.net sparked some interest. I was particularly interested in the views from editors of various journals such as Organization Science, Organization Studies, Academy of Management Journal, Management Science, American Sociological Review, and Administrative Science Quarterly.

If you are interested in improving the way that you carry out reviews, there is some excellent guidance there. I added a note, as follows:

This discussion provides an excellent presentation of the diverse and multiple aspects of a good review. In Construction Management and Economics, we have also been dealing with how best to advise referees. One thing of vital importance, as mentioned by several of the editors, is to ensure that the referees deal with the content rather than the presentation/style of the paper. To help focus attention on what matters, I have recently developed a more explicit description of what I mean when I say “scientific content” because in an applied and practical field like construction management, it can be difficult for some people to distinguish research from practice, let alone content from style.

I am increasingly asking referees to consider this definition of the scientific approach when the write their reviews: “The observation of certain specific phenomena within a theoretical framework in order to develop better explanations that improve our collective understanding.”

Specifically, I want to avoid papers that merely report on specific phenomena. I want referees to tell me whether the papers they review are explicit about the extent to which a paper deals with the phenomena being observed, the underlying concepts and theoretical framework being used, the existing explanations being critiqued or found wanting, and the new explanations being proffered that add to our pool of knowledge. And all of this has to be done within the conventions of what a paper in this field looks like.

This is a complicated and difficult issue, and the way we put our requests to referees is constantly developing. What makes a paper acceptable is conformance with the customs and practice of the particular field in terms of: how arguments are presented, how data is used,how research is carried out, formatting of papers. The overriding criterion is clarity. These are the issues that I wish referees to guide me on when I am deciding the fate of a paper.

Crucially, reviewing a paper for a journal is clearly not the same thing as writing a critique for a literature review.