What editors look for in referee reports

A recent discussion in orgtheory.net sparked some interest. I was particularly interested in the views from editors of various journals such as Organization Science, Organization Studies, Academy of Management Journal, Management Science, American Sociological Review, and Administrative Science Quarterly.

If you are interested in improving the way that you carry out reviews, there is some excellent guidance there. I added a note, as follows:

This discussion provides an excellent presentation of the diverse and multiple aspects of a good review. In Construction Management and Economics, we have also been dealing with how best to advise referees. One thing of vital importance, as mentioned by several of the editors, is to ensure that the referees deal with the content rather than the presentation/style of the paper. To help focus attention on what matters, I have recently developed a more explicit description of what I mean when I say “scientific content” because in an applied and practical field like construction management, it can be difficult for some people to distinguish research from practice, let alone content from style.

I am increasingly asking referees to consider this definition of the scientific approach when the write their reviews: “The observation of certain specific phenomena within a theoretical framework in order to develop better explanations that improve our collective understanding.”

Specifically, I want to avoid papers that merely report on specific phenomena. I want referees to tell me whether the papers they review are explicit about the extent to which a paper deals with the phenomena being observed, the underlying concepts and theoretical framework being used, the existing explanations being critiqued or found wanting, and the new explanations being proffered that add to our pool of knowledge. And all of this has to be done within the conventions of what a paper in this field looks like.

This is a complicated and difficult issue, and the way we put our requests to referees is constantly developing. What makes a paper acceptable is conformance with the customs and practice of the particular field in terms of: how arguments are presented, how data is used,how research is carried out, formatting of papers. The overriding criterion is clarity. These are the issues that I wish referees to guide me on when I am deciding the fate of a paper.

Crucially, reviewing a paper for a journal is clearly not the same thing as writing a critique for a literature review.

Academics vs practitioners?

In discussing the opportunities for involving practitioners as authors in Construction Management and Economic a friend of mine challenged me because he got the impression that I was wanting to work in isolation from industry in case "they contaminated our minds and data"! I felt that he was misunderstanding my motives. Of course it would be disastrous if academics worked in isolation, I agree 100%. Our research is rightly grounded in the construction sector. It certainly is not the case that I think that practitioners contaminate our minds and data. I agreed 100%. "They" ARE our data! They are the source of the problems that we study, they may sometimes be the source of solutions that we seek to understand and, frequently, they are the very people that we study. No, there is no sense of isolation there.

My point was that our data subjects are not authors. It was to do with ensuring that our message is tailored to suit our audience. If the findings of our research are meaningful for industry, then we must, of course, present them to industry. But in the journal, we are academics talking to academics, in a fairly structured way guided by conventions that may not be appropriate for a wider audience. In the journal, we are focusing on theory-testing and/or theory-building, but not on the dissemination of our results to a wider audience. That is the crucial characteristic of an archival research journal like ours - to record advances in research. Other media already exist for recording and disseminating advances in practice. I think that is right, and it is helpful for everyone to have this distinction.

So, yes, it is precisely because practitioners define our field by their practices that I am interested in their contributions. But not because of their ability to carry out research projects with us. Of course, there are people who have a foot in both camps and I do come across practitioners who have done "proper" research that is reportable in our pages. But then they are generally writing as researchers, not as practitioners. So the boundary is blurred. The acid test for a research paper is simply this question: does it test or develop theory? If the answer is "yes", then we are interested. If it is "no", then we are not.

And, I also acknowledge that although this is the editorial policy now, it may not be the editorial policy forever, and it was not the policy many years ago. Everything is open to challenge and change!