What editors look for in referee reports

A recent discussion in orgtheory.net sparked some interest. I was particularly interested in the views from editors of various journals such as Organization Science, Organization Studies, Academy of Management Journal, Management Science, American Sociological Review, and Administrative Science Quarterly.

If you are interested in improving the way that you carry out reviews, there is some excellent guidance there. I added a note, as follows:

This discussion provides an excellent presentation of the diverse and multiple aspects of a good review. In Construction Management and Economics, we have also been dealing with how best to advise referees. One thing of vital importance, as mentioned by several of the editors, is to ensure that the referees deal with the content rather than the presentation/style of the paper. To help focus attention on what matters, I have recently developed a more explicit description of what I mean when I say “scientific content” because in an applied and practical field like construction management, it can be difficult for some people to distinguish research from practice, let alone content from style.

I am increasingly asking referees to consider this definition of the scientific approach when the write their reviews: “The observation of certain specific phenomena within a theoretical framework in order to develop better explanations that improve our collective understanding.”

Specifically, I want to avoid papers that merely report on specific phenomena. I want referees to tell me whether the papers they review are explicit about the extent to which a paper deals with the phenomena being observed, the underlying concepts and theoretical framework being used, the existing explanations being critiqued or found wanting, and the new explanations being proffered that add to our pool of knowledge. And all of this has to be done within the conventions of what a paper in this field looks like.

This is a complicated and difficult issue, and the way we put our requests to referees is constantly developing. What makes a paper acceptable is conformance with the customs and practice of the particular field in terms of: how arguments are presented, how data is used,how research is carried out, formatting of papers. The overriding criterion is clarity. These are the issues that I wish referees to guide me on when I am deciding the fate of a paper.

Crucially, reviewing a paper for a journal is clearly not the same thing as writing a critique for a literature review.

Academics vs practitioners?

In discussing the opportunities for involving practitioners as authors in Construction Management and Economic a friend of mine challenged me because he got the impression that I was wanting to work in isolation from industry in case "they contaminated our minds and data"! I felt that he was misunderstanding my motives. Of course it would be disastrous if academics worked in isolation, I agree 100%. Our research is rightly grounded in the construction sector. It certainly is not the case that I think that practitioners contaminate our minds and data. I agreed 100%. "They" ARE our data! They are the source of the problems that we study, they may sometimes be the source of solutions that we seek to understand and, frequently, they are the very people that we study. No, there is no sense of isolation there.

My point was that our data subjects are not authors. It was to do with ensuring that our message is tailored to suit our audience. If the findings of our research are meaningful for industry, then we must, of course, present them to industry. But in the journal, we are academics talking to academics, in a fairly structured way guided by conventions that may not be appropriate for a wider audience. In the journal, we are focusing on theory-testing and/or theory-building, but not on the dissemination of our results to a wider audience. That is the crucial characteristic of an archival research journal like ours - to record advances in research. Other media already exist for recording and disseminating advances in practice. I think that is right, and it is helpful for everyone to have this distinction.

So, yes, it is precisely because practitioners define our field by their practices that I am interested in their contributions. But not because of their ability to carry out research projects with us. Of course, there are people who have a foot in both camps and I do come across practitioners who have done "proper" research that is reportable in our pages. But then they are generally writing as researchers, not as practitioners. So the boundary is blurred. The acid test for a research paper is simply this question: does it test or develop theory? If the answer is "yes", then we are interested. If it is "no", then we are not.

And, I also acknowledge that although this is the editorial policy now, it may not be the editorial policy forever, and it was not the policy many years ago. Everything is open to challenge and change!

Multiple journal submissions

It is interesting how much there is to learn about publishing that appears not to be passed on by PhD supervisors to their students. Today I dealt with a problem author whose paper to Construction Management and Economics was not only 70% longer that the maximum we allow, but was also, as it turned out, very similar to another paper by the same author that I had dealt with only the day before. The similarities were obvious when I looked at them so close together, but it turned out that he had submitted three papers over a month or so, all drawing from his PhD, all far too long, and only differing in terms of the focus of the cases being presented in each paper. Each repeated the same mistakes as the other (not taking a critical stance in reviewing the literature and lacking in a certain punchiness to the conclusions).

When I challenged the author about cranking out so many papers from the same PhD, he pointed out that he had a further six papers in various states of progress, all reaching the same conclusion, but based on different case studies. I pointed out that this was not good academic practice and that we only wanted one paper that fell within our word limits. The author was very attentive to what I was saying, and it was clear that this was all new to him, having only recently completed his PhD. He suggested, respectfully, that maybe the published Instructions for Authors should make this point clear.

The problem with that suggestion is that the Instructions for Authors (IfA) are not meant to form general advisory page about the protocols of academic publishing, a topic about which much has been written in the past. Rather, the IfA are intended to highlight journal-specific matters. Moreover, we tend to see advice about general academic conduct as a primary responsibility of PhD supervisors, who usually make sure that they enlighten their PhD students about a much wider range of issues than multiple submissions. I have tried to make the general principles of authorship clear in publications of my own, in the past, for example:

Hughes, W.P. (2008) Getting your research published in refereed journals. In: Knight, A. and Ruddock, L. (eds). Advanced Research Methods in the Built Environment. Chichester: Wiley-Blackwell, 193-206. ISBN: 9781405161107.


But of course, I am not the only person who writes such papers, and it is quite important for new academics to learn about a whole range of issues relating to developing a career as a professional academic. And finely slicing a piece of research into a disproportionately large number of papers is only one of many pitfalls for the budding academic.

To withdraw or not to withdraw?

A colleague from another University asked me a question: he submitted a paper to an international journal and had received no response after 10 months. He tried to contact the editor and his secretary said that the review process has started but they have not received all the responses from reviewers. Did I think it is OK for him to write a formal letter to withdraw the paper now? Or should he wait until the decision is made, no matter how long it takes? I was quick to advise him as follows.

In my experience, the gathering of referee reports is a burdensome task for an editor and the editorial office. One of the reasons that it can take so many months to get reports is that referees often do not answer promptly, or they say they will carry out the task, then change their minds, or they are just slow because of all the other demands on their time. Often, over a period of many months, we might find that we have tried 20 different referees in order to end up with the requisite four reports.

After all this work it is very annoying to then discover that the author wants to withdraw the paper! However, if the paper is rejected, then there is no real problem. Alternatively, if the decision is Major, or Reject & Resubmit, then that is an opportunity for the author to say to the editor that the requirements are just too harsh, and that he or she would prefer to withdraw it. But if an editor calls for minor revisions, or simply accepts the paper, the author really should not withdraw it. It would be bad protocol, because the editorial office and the referees wouyld have been working through this process for no reason. Perhaps the question to ask is, why would an author want to withdraw a paper mid-way through a long refereeing process? I have come across situations where an author has realized, too late, that there are such significant flaws in the paper that it really should go no further. This would need some careful dialogue wit the editor, explaining why the paper cannot continue in the process.

I summed up my advice thus. Therefore, you need pretty strong reasons for withdrawal. Either the editor's requirements are too harsh, or you have discovered some aspect of the paper that renders it unpublishable. I don't think that you should withdraw for any other reason, unless you want to upset the editor!

Listing the 700 people who reviewed papers for CM&E this year

Each year, the editors of Construction Management and Economics like to compile a list of all the referees who have reviewed papers in the preceding 12 months, and thank them formally by publishing this list. Given that we have a database recording everyone's involvement, and their contact details, this should be, on the face of it, a simple task. On listing the people who had been active, I first noticed the obvious errors. Some people had been entered twice into the system, sometimes by themselves, sometimes by their co-authors. The system distinguished people by their e-mail addresses, but these change, or people use different ones for different purposes. Then I found that some people type their details all in lower case or all capitals, so I had to fix that, both in the outputted list and in the database. Then I found that a large number of people never bothered to enter their institution, so I had to figure it out from their e-mail addresses, or if that was not clear, find a paper they had authored, then inspect the cover page. Some people have middle initials, but there was no consistency as to whether they put a full-stop after their initial or not. I had to add one for every user who had omitted it, on the basis that if I subsequently wanted to remove it from any output for consistency, a global search and replace would have something to go at. Some users are unfamiliar with the name of their own university! Several had XYZ University, when it should be University of XYZ. Others had apparently not noticed that their University had changed its name, sometimes several years ago. Some had entered the University and part of their address, but not bothered changing the default country (UK), so I had to relocate them to their correct country. And then we noticed that strange thing had happened in the database (which is maintained by a software house in USA). Certain countries had been changed, arbitrarily. While Taiwan had changed to "Taiwan, Province of China", Iran had changed to "Iran, Islamic Republic of" and Hong Kong, SAR China had changed to just "Hong Kong". We decided that the change to Iran and Hong Kong were OK, but were aware of the sensitivities surrounding the status of Taiwan. All of the entries for Taiwan had to be edited manually, because countries are selected in the database from a drop-down list, which we cannot edit. Perhaps this was why some Taiwanese users had left the country as United Kingdom, because they preferred this kind of mistake over being categorized as part of China. After several hours of editing the list of 700 people, we finally had a list we could publish. Having put most of the errant records straight, and allocated someone to the task of checking the database more thoroughly over the next year, I am optimistic that next year, this will be a ten-minute job!