Reviewing research papers

The text below originally appeared in an ARCOM Newsletter. I reproduce a revised version here to make it more accessible to new researchers who might not be combing through back-numbers of ARCOM newsletters at www.arcom.ac.uk

There are many things that contribute to the quality of a research paper. Any report of research begins with a review of the relevant body of literature, and there is excellent guidance for undertaking a literature review in Silverman (2000: 12) and in Rudestam and Newton (1998: 50-51). Based upon this guidance, this paper offers a personal view of what we should be looking for in research papers, with the aim of prompting wider debate.

In all papers, a structure is required and the argument should flow from one section to the next. Obviously, clear English should be used throughout and jargon should be avoided. Good papers will move from the general to the particular and begin with the context of the work, move through the statement of the problem being investigated, deal with the empirical and/or analytical aspects of the work, then develop the discussion and draw conclusions based upon what has been covered in the paper, relating these back to the original context of the work. Generally, papers will either develop theory or test theory. A paper that does neither will not add to the sum of knowledge and therefore will not fall into the category of a research paper. Issues connected with style, structure and presentation are dealt with extensively elsewhere in the literature (for example, Turk and Kirkman 1982) and there is no need to reiterate that guidance here, other than to state that the easiest questions can be the most difficult to answer: what have you done, why is it important and how have you gone about it?.

The focus of this article is on whether the material that is included in a paper is suitable for a research paper, rather than whether it is well-written. Silverman’s (2000) headings form an excellent basis for a discussion of what constitutes a good research paper:

• Conceptualization and theoretical basis of the work
• Analytical framework and hypotheses
• Research design
• Results and discussion
• Conclusions of the paper

Conceptualization and theoretical basis of the work

The first questions to ask about a paper are connected with the problem or issue being investigated. There should be a clear statement near the beginning of a paper explaining what problem the paper seeks to resolve. Authors often leave this until half way through the paper. Indeed, some leave it out altogether, perhaps assuming that it is self-evident or simply not realizing that although it is obvious to the author, a reader with no previous knowledge of the work only has the paper to go on.

Any serious piece of research will involve concepts that are specific to the issue being investigated, or to the investigative approach that has been taken. These need to be summarized at the very least, and explained if they are not common within the field of the target audience. This is not just a case of explaining the concepts related to the particular construction industry phenomena under investigation, but, more importantly, to identify the methodological basis of the work. Thus, a research paper is not place for “text-book” explanations. Of course, the nature of the investigation is inevitably connected to some issue of relevance, but, while it may seem heretical to some, it is not necessary for a piece of construction management research to be practically relevant in industry. A piece of research may hold relevance only for other researchers, but that should not detract from our judgment of its value.

In assessing a report of research, it is always helpful to be able to see clearly how the reported work builds upon previous work. There should be explicit connections to an existing body of knowledge or body of theory, although these may not reside in the literature of construction management. Indeed, it is helpful if there are references to bodies of research and knowledge outside our own “domain”, since ours is not an academic discipline in its own right, with its own research techniques and theories (Hughes, 1999). While there are some emerging strands of theory that are unique to construction management or construction economics, most research in this area builds upon theoretical models developed elsewhere in the social sciences. These connections must be identified in order to make clear where a particular piece of work is rooted and to ensure that we are not simply re-inventing theories and models that are well-known in more mainstream disciplines. Without such connections, we run the risk of consigning our research to an academic backwater. With such connections, we may even be able to influence developments in mainstream thinking. In determining the theoretical basis of a piece of research, it is useful to think about knowledge domains. At the very least, it might be helpful to think about where would you expect to find a particular piece of work in a library so that connections and antecedents are clear.

These issues are important because progress in our understanding often depends upon our ability to generalize from specific examples. One question I often ask research students when they are trying to make these connections is “what is the general class of problem of which your chosen topic is a specific example?”. Understanding this enables some kind of view to be developed about the extent to which findings might be generalized into a wider context. Thus, good papers will begin with what is well-known and move gradually deeper into the less well-known (Latour, 1987: 57).

Analytical framework and hypotheses

All research papers have an analytical framework. Unfortunately, it is not always clearly articulated. The extent to which a particular approach is authoritative is often judged in terms of where it has come from. Thus, further connections to the research literature should be expected in the passages describing the analytical framework. When this is done well, it helps to establish the credibility of the paper by showing the usefulness of the particular approach, or approaches that precede it. The articulation of the analytical framework helps in judging the usefulness of the research question. It explains how the concepts and theories are being applied in this particular case.

One perennial problem with research papers in our field is the question of whether there should be hypotheses. They are certainly not a pre-requisite for a good research paper. In fact, they may not belong at all. The question about whether there should be hypotheses is, perhaps, a wrong question. Their presence or absence depends upon the methodological stance of the research. It is not intended to enter into the methodological debate here, other than to point out the dangers of not understanding the methodological implications of different approaches to research (see, for example, Seymour and Rooke, 1995). Given one methodological stance, hypotheses may be irrelevant. Given another, they may be indispensable.

If there are hypotheses, they should be clearly stated. If there are no hypotheses, then this, of itself, is not a problem, but it should be clear whether the paper is a review, a case study, a contribution to theory development or some other type of study. Without clear articulation, the reader stands no chance of determining the value of the contribution. In the presence of hypotheses, the relationships between the main variables should be explicit and reasonable. They should be stated in a way that makes them testable and the results, no matter what they are, interpretable. If the research is not built on hypotheses, the significance of the paper’s contribution to the development of theory must be explained.

Research design

In undertaking research, there are many methods that can be used to find answers to questions. Some are more suitable than others. In answering certain types of question, one particular method may be very powerful, but the same method may be weak in dealing with other types of question. Therefore, the relevance of the methods of research will be judged in terms of their appropriateness to the nature of the question being asked. Similarly, the sensitivity of the methods must match the needs of the research question. A good paper will make clear the type of research design, perhaps by reference to earlier, similar studies from different regions, different industries or different disciplines.

The research must be focused on an appropriate unit of analysis. It is useful to describe the criteria by which this was chosen, as well as the criteria by which the cases were chosen. For example, the unit of analysis could be a person, a finished building, a project, a firm, an industry or a country. Each would result in an entirely different study from the others. Moreover, cases might be selected from a large number of similar cases, which would imply one kind of approach, or the question might be framed in such a way that there is only one case, implying an entirely different approach. Neither, of itself, is more or less valid than the other. Indeed no judgement can be made about the validity of a piece of research simply by counting the cases or referring to the unit of analysis. Each characteristic depends on the other.

It is always important to address whether the research design isolates what is being measured from other effects, or, at the very least, identifies the inter-relationships between the effect under scrutiny and other effects. If the research design involves the identification of variables, they need to be clearly and reasonably operationalized (i.e. translated into simple descriptions of what is measured and how it is to be measured) and the reliability and validity of the measures should be discussed. Similarly, there will be issues related to the appropriateness of the population for the research question being studied, the sample size used and the extent to which the results can reasonably be generalized on the basis of this particular sample.

Again, not all research is as deterministic as this, but there are traditions in different types of work and if a phenomenological or ethnographic approach is being adopted, then the author should take this stance clearly and confidently and not try to dress it up in hypothetico-deductive clothes! These issues are well-articulated by Johnson and Duberley (2000), who warn against the dangers of not dealing with the epistemological positions that are implicit in different approaches to empirical research. It is important to emphasize that none of us wishes be prescriptive about the kind of research that is encouraged in ARCOM. We welcome research outputs that add to our understanding of construction management issues. But, the relevance and appropriateness of research outputs can only be evaluated in the light of the epistemological background to the research and this is why it is important to discuss the design of the research.

Results and discussion

Within the research paper, the data or evidence of the field-work must be present in some guise. But there are limits on the length of papers, whether for conferences or for journals. It is inevitable that the data will not be reported in their entirety through these outlets. Thus, one technique is to describe what the data is like, rather than what it is. Perhaps sample sections of interview transcripts can be given; examples of diaries or other source documents can be used to illustrate the approach taken. The full record of the data can be maintained elsewhere, perhaps in a departmental library or on the internet, so that the interested reader can interrogate the data further.

In any event, there must be sufficient information within the paper itself for the reader to evaluate whether the data were appropriate for the study and whether the data collection and record keeping were systematic. Similarly, the validity and robustness of the results of the study will depend upon whether the analytical techniques were appropriate and adequately described. Most importantly, there should be reference to accepted procedures for analysis. This helps the reader to understand what kind of tradition there is in the particular kind of analysis and how such research is generally reported.

In assessing how systematic the analysis has been, one of the main ideas is to persuade the reader that if he or she were to have done the same things, then the same conclusions would have been reached (Latour, 1987). Again, it is important that this very statement implies a certain epistemological stance, so the researcher and the reader need to be clear about whether they are working from the same basis in coming to their views about the results and their discussion.

Conclusions

Conclusions can be the most difficult part of a paper to write, particularly if the context and research design have not been addressed properly in the first place. It is often the case that those who have the greatest difficulty writing conclusions, can trace their difficulties to poor research planning. When research is well planned, the conclusions become obvious from the work that has been reported.

No new facts should be introduced in the conclusions. The conclusions of the study should be consistent with the results of the analysis. Where there is no numerical analysis, the conclusions should be consistent with, and follow from, the development of the argument in the paper. I am not quite sure why, but many authors commence their conclusions with a summary of the paper. This is usually not required, if there is already a summary at the beginning.

The conclusions may be expected to reconsider the purpose of the research, summarize what was discovered and provide a discussion about the implications of the findings. In many cases, conclusions can be bolstered by considering whether there are alternative conclusions that are consistent with the data or arguments that have been presented. Also, it is useful to consider both theoretical and practical implications of the results. If the research has been properly contextualized at the beginning of the paper, the theoretical implications of the reported research can be adequately connected to the literature discussed there. It is usual to include limitations of the study and future research needs, to the extent that the research has revealed further gaps in our collective knowledge. The limitations of the study should be noted in terms of the parameters of the research and applicability of the findings. Authors sometimes misinterpret the purpose of a section on limitations of the work and attempt to indulge in soul-searching self-criticism, identifying faults in the execution and reporting of their own work. This is simply not required. The section on limitations should make clear that, for example, the conclusions do not apply to all construction activity in all places at all times. The approach taken in the research enables certain generalizations to be made, but what are they?

Conclusions can also be bolstered by including discussion of the evidence for and against the researcher's arguments and making a clear distinction between the data and their interpretation.

Finally…

Any research paper is capable of being summarized succinctly. Papers are expected to include an abstract or summary at the beginning, especially in the cases of conferences and journals, but this should be the last thing to be written! Although abstracts may be invited for conferences before the paper is written, the abstract of the final paper ought to be written after the paper is finished, summarizing the paper that has been produced, rather than the paper that was originally intended.

In evaluating a research paper, it must be possible for the evaluator to summarize the paper, indicating an overall assessment of the adequacy of the study for exploring the research problem and an overall assessment of the contribution of the study to this area of research. Authors would do well to bear this in mind when writing their papers, so that they can provide the relevant cues that will lead a reader to conclude exactly what the author has concluded. Sometimes, there is simply not sufficient information in the paper to enable such statements to be articulated. In such cases, the paper should be revised.

References

Hughes, W.P. (1999) Construction research: a field of application. Australian Institute of Building Papers, 9, 51-58.
Johnson, P. and Duberley, J. (2000) Understanding management research: an introduction to epistemology.
London: Sage.
Latour, B. (1987) Science in action: how to follow scientists and engineers through society. Cambridge, Mass.: Harvard University Press.
Rudestam, K.E. and Newton, R.R. (1992) Surviving your dissertation: a comprehensive guide to content and process. London: Sage.
Seymour, D. E. and Rooke, J. (1995) The culture of the industry and the culture of research. Construction Management and Economics, 13(6), 511-523.
Silverman, D. (2000) Doing qualitative research. London: Sage.
Turk, C. and Kirkman, J. (1982) Effective writing: improving scientific, technical and business communication. London: Spon.

Checklist of questions to keep in mind while reviewing a research paper:

1 Conceptualization and theoretical basis of the work

(a) What is the problem or issue being investigated?
(b) What are the major concepts, and how clearly are they defined/explained?
(c) Is the connection to an existing body of knowledge or theory clear?
(d) Is there some practical relevance in this work (research practice or industrial practice)?
(e) What is the theoretical basis of this work, i.e. knowledge domain, for example, where would you expect to find this work in a library (classification number)?

2 Analytical framework and hypotheses

(a) Is there a clearly stated research question? (It might not be a research paper, as such)
(b) Are there hypotheses? Are they clearly stated? If there are not hypotheses, is the paper a review, case study, contribution to theory development or some other type of study?
(c) If there are hypotheses, are the relationships between the main variables explicit and reasonable? If there are not hypotheses, is there adequate development of theory?
(d) If there are hypotheses, are they stated in a way that makes them testable and the results, no matter what they are, interpretable? If there are not hypotheses, are there clear indications as to the significance to theoretical development?

3 Research design

(a) Are the methods of research appropriate to the nature of the question being asked?
(b) What is the type of research design?
(c) Could the design be improved? How?
(d) Is there a clear account of the criteria used for selecting the focus (unit) of analysis and the cases chosen?
(e) Does the research design isolate what is being measured from other effects? Are the variables clearly and reasonably operationalized (what is measured and how)? Are the reliability and validity of the measures discussed?
(f) Is the population appropriate for the research question being studied? Is the sample specified and appropriate? Can the results be reasonably generalized on the basis of this sample?

4 Results and discussion

(a) Are the data appropriate for the study? Was the data collection and record keeping systematic?
(b) Are the statistical techniques appropriate and adequately described? Is reference made to accepted procedures for analysis?
(c) Are the control variables adequately handled in the data analysis? Are there other control variables that were not considered but should have been?
(d) How systematic is the analysis?
(e) Is there adequate discussion of how themes, concepts and categories were derived from the data?

5 Conclusions

(a) Do the conclusions flow from the work that has been reported?
(b) Are the conclusions of the study consistent with the results of the analysis? (If there is no numerical analysis, are the conclusions consistent with the development of the argument in the paper?)
(c) Are alternative conclusions that are consistent with the data discussed and accounted for?
(d) Are the theoretical and practical implications of the results adequately discussed? Are the theoretical implications adequately connected to the literature discussed at the beginning of the paper?
(e) Are the limitations of the study noted (in terms of parameters of the research and applicability of the findings)?
(f) Is there adequate discussion of the evidence for and against the researcher's arguments?
(g) Is a clear distinction made between the data and their interpretation?

6 Summary

(a) What is your overall assessment of the adequacy of the study for exploring the research problem?
(b) What is your overall assessment of the contribution of the study to this area of research?

References

Rudestam, K.E. and Newton, R.R. (2007) Surviving your dissertation: a comprehensive guide to content and process. 3rd ed. London: Sage
Silverman, D. (2009) Doing qualitative research. 3rd ed. London: Sage.