Conceptualizing a research project

In my work as a supervisor of dissertations, whether BSc, MSc or PhD, I have developed an approach to help with research design and developing an outline of research.

The first question that needs to be settled is, what kind of science does the student want to do? Incidentally, social science is a kind of science in my mind. Who are the key researchers in the field that this students wants to base his/her work on? Some kind of conceptual model is usually required to make clear how the main concepts connect. The main concepts would be drawn from the research literature on the topic.

Second, where is the data from and how will it be analysed? Many students begin the dissertation process with a description of the kind of observations they wish to make, but this can only be part of the research design if it is placed in the context of the conceptual model, which itself is based on a theoretical position.

Third, if someone is working in a well-trodden academic discipline with a clear theoretical tradition that underpins it, then they typically do not explain their theoretical position, since understanding that is taken as a basic tenet in the discipline. So it is fair to expect that many papers will be silent about their theoretical perspective, even though it can be implied from the kind of question they are dealing with. In a multi-disciplinary or inter-disciplinary field like construction management, it is important to be explicit about the theoretical position. This does not always need a whole chapter. It is sometimes a few lines in the early part of the dissertation.

Following these three aspects, the initial work will rely heavily on literature review and will set up theory-concept-observation as an axis that leads to the research design. I tend to insist on a clear explanation of the connection between theory-concept-observations. Every dissertation student, I think, needs this to be clear in the write-up. This initial setting up will eventually form approximately half of the thesis followed, of course, by the second half; analysis-discussion-conclusions, which acts as a kind of mirror to the first half. These may be translated approximately into chapters: introduction, literature, methods. And it is useful to think how the analysis reflects the methods; the discussion reflects the literature (and conceptual model); the conclusions reflect the introduction.

With this conceptual model of a research dissertation in mind, I find supervision becomes much more transparent and students tend to see better where they are headed. There is no strict recipe for a dissertation, of course, and each student will change this initial framework as their confidence grows. But I find it forms a good starting point. Many different kinds of research can be covered by adapting this model to fit the kind of research.

I sketch this out frequently when talking to students, annotating it with keywords and ideas that relate to their specific interests and type of research. Sometimes it needs significant changes in order to make sense. But it still forms a good starting point for the early discussions when the student does not really understand what it means to do research and the supervisor does not really understand what the student wants to do. I have not yet had time to prepare a nice graphic with drafting software, but a pen-sketch is good enough:

The diagram shows how the conclusions relate back to the theory, how the discussion chapter relates to the conceptual model and how the analysis relates to the observation. It also shows how the aims inform the theory, the theory informs the objectives, the objectives drive the literature review to provide the conceptual model, the conceptual model leads to the research design and so on. Finally, we can see how the general issues lead to increasingly specific issues in the first half, and the second half involves moving back to generalized statements for the conclusions.

It must also be noted that not all research goes through this sequence. Ethnographic methods are often highly appropriate for construction management research, especially for those who have experience in practice. It is not always necessary to behave as if you have no experience or as if you were an outsider to the industry. If you are an insider, then look at ethnographic methods. These often involve immersing oneself in the field and then developing theory from the experience. For part-time MSc students, in particular, this is a very powerful way of construction a piece of valid research.

Reviewing research papers

There are several posts in this blog about reviewing research papers, and it is a theme that constantly crops up, both in relation to refereeing a journal paper for an editor of a journal, an in relation to building up the elemental parts of a literature review. As I have said before, these are two distinctive activities.

Many students struggle to make critical evaluations of published papers, and there are many reasons why they might find this difficult at first. One way of approaching the task could be to try to focus on whether a paper is making an interesting and useful contribution to our collective understanding. I think that what we are looking for is signs that author of a paper is clever enough to provide a persuasive case. What I mean by this is to look for evidence in a paper that the author is knowledgeable, has a good understanding of the issues, can put together an argument and can make judgements.

In order to be convinced that an author is knowledgeable, I would look for evidence that he/she knows about the generalities and specifics of the topic. To be convinced that the author has an understanding, I would try to find evidence of different views being weighed in the balance and ideas being applied to examples, especially I would look for the use of metaphors and analogies. In thinking about whether the paper has a strong argument, I would look for the use of logic and the sequence of ideas. The language of argument is interesting, and this is closely related to grammar, vocabulary and logic. I do not find the use of jargon or inflated diction particularly compelling. But plain language in straightforward words is often an element of a good argument. It should be clear. Finally, in looking to see whether an author is making judgements, there should be evidence of discrimination in the kind of sources being relied on to form the foundations for the work, and judgement in the nature of the conclusions as well as the all of the choices about what to include and what to exclude in putting the paper together.

Some common errors frequently reveal weak papers and unclear thinking. Papers typically follow this structure:

• Abstract - summarizing the whole paper including the specific findings
• Introduction - explaining what the paper is about
• Literature review - dealing with what we collectively already know, specifically, but not exclusively, in terms of past research
• Possibly some empirical work - not all papers fit into the category of empirically-based research
• Analysis and/or discussion - empirically-based papers would separate the analysis from the data, then all papers would include discussion of what all this leads us to
• Conclusions - explaining what this all means, perhaps with an outline of the parameters of the research and how that restricts the application of the findings, and usually with suggestions about what certain groups of people might do differently as a result of this research, whether these groups are scholars, practitioners or policy-makers

In weak papers authors often get the sections mixed up, probably because a weak paper will not really lead to any conclusions. Thus, I often find that the conclusions only really summarize the paper, the abstract introduces, rather than summarizes, and the introduction jumps straight into a general literature review before the detailed and focused part of the literature. As a result, in my editorial capacity, I often suggest to authors that they should move the introduction section to the literature review, move the abstract to the introduction, move the conclusions to the abstract, and then try to write some proper conclusions that convey something about what all this means. If they cannot do that, then the work is not really good enough to be called research, I would venture to suggest.

It might be an interesting exercise to see if you can find a paper where the sections are mixed up in this way. It would certainly give new researchers the confidence to take a critical stance when it comes to reviewing papers for their literature review!

What editors look for in referee reports

A recent discussion in orgtheory.net sparked some interest. I was particularly interested in the views from editors of various journals such as Organization Science, Organization Studies, Academy of Management Journal, Management Science, American Sociological Review, and Administrative Science Quarterly.

If you are interested in improving the way that you carry out reviews, there is some excellent guidance there. I added a note, as follows:

This discussion provides an excellent presentation of the diverse and multiple aspects of a good review. In Construction Management and Economics, we have also been dealing with how best to advise referees. One thing of vital importance, as mentioned by several of the editors, is to ensure that the referees deal with the content rather than the presentation/style of the paper. To help focus attention on what matters, I have recently developed a more explicit description of what I mean when I say “scientific content” because in an applied and practical field like construction management, it can be difficult for some people to distinguish research from practice, let alone content from style.

I am increasingly asking referees to consider this definition of the scientific approach when the write their reviews: “The observation of certain specific phenomena within a theoretical framework in order to develop better explanations that improve our collective understanding.”

Specifically, I want to avoid papers that merely report on specific phenomena. I want referees to tell me whether the papers they review are explicit about the extent to which a paper deals with the phenomena being observed, the underlying concepts and theoretical framework being used, the existing explanations being critiqued or found wanting, and the new explanations being proffered that add to our pool of knowledge. And all of this has to be done within the conventions of what a paper in this field looks like.

This is a complicated and difficult issue, and the way we put our requests to referees is constantly developing. What makes a paper acceptable is conformance with the customs and practice of the particular field in terms of: how arguments are presented, how data is used,how research is carried out, formatting of papers. The overriding criterion is clarity. These are the issues that I wish referees to guide me on when I am deciding the fate of a paper.

Crucially, reviewing a paper for a journal is clearly not the same thing as writing a critique for a literature review.

Literature searching

The most essential prerequisite for a successful piece of academic research is the establishment of what is already known about the topic. While it is important to be able to produce a critical review of a research paper (See "Reviewing Research Papers"), it is just as important to be able to string together all the bits of information from the individual readings into a coherent critique of what previous research has shown, and where we are up to, collectively, in our understanding of the issues.

In trying to develop pointers and hints to help students go about this important task, I recently came up the following summary about what is required:

• Catalogue: Keep track of every item you come across, and make a note of its status in your literature searching process, particularly whether it has been acquired, read, reviewed, critiqued, and so on.
• Categorize: Figure out for each item what keywords you would use to index it, in such a way that you can connect papers on similar topics. Include keywords about methods and data, not just about ideas, and develop your own view of the definitions of each term, always seeking to use dictionary definitions when they work, but defining terms more specifically where necessary. But dont re-invent the wheel! If some past researcher has a good definition, use it (and cite the source).
• Characterize: Figure out what each item is like, and explain in your notes what kind of paper it is, and the basic characteristics of the type of research and type of question. Make connections with the literature about research methodology.
• Conceptualize: Develop a framework of the broad concepts that emerge as being the basic buildings blocks of knowledge in your topic, as represented by the literature you have discovered.
  
• Recognize themes, resonances and contradictions: Look for overlaps and clashes, erecting new keywords as you go, and checking each time whether a new keyword should have been used as well for the papers you already looked at. Find out where past researchers are in agreement, and where there is controversy.
  
• Identify areas of interest: As you go through these steps, you will inevitably develop a clearer understanding of not only what has already been done, but what remains to be discovered or challenged.

I hope that these notes help researchers to develop their approach to literature searching and to structuring a logical and rational narrative about the research base that they are building on in their work.

Reviewing research papers

The text below originally appeared in an ARCOM Newsletter. I reproduce a revised version here to make it more accessible to new researchers who might not be combing through back-numbers of ARCOM newsletters at www.arcom.ac.uk

There are many things that contribute to the quality of a research paper. Any report of research begins with a review of the relevant body of literature, and there is excellent guidance for undertaking a literature review in Silverman (2000: 12) and in Rudestam and Newton (1998: 50-51). Based upon this guidance, this paper offers a personal view of what we should be looking for in research papers, with the aim of prompting wider debate.

In all papers, a structure is required and the argument should flow from one section to the next. Obviously, clear English should be used throughout and jargon should be avoided. Good papers will move from the general to the particular and begin with the context of the work, move through the statement of the problem being investigated, deal with the empirical and/or analytical aspects of the work, then develop the discussion and draw conclusions based upon what has been covered in the paper, relating these back to the original context of the work. Generally, papers will either develop theory or test theory. A paper that does neither will not add to the sum of knowledge and therefore will not fall into the category of a research paper. Issues connected with style, structure and presentation are dealt with extensively elsewhere in the literature (for example, Turk and Kirkman 1982) and there is no need to reiterate that guidance here, other than to state that the easiest questions can be the most difficult to answer: what have you done, why is it important and how have you gone about it?.

The focus of this article is on whether the material that is included in a paper is suitable for a research paper, rather than whether it is well-written. Silverman’s (2000) headings form an excellent basis for a discussion of what constitutes a good research paper:

• Conceptualization and theoretical basis of the work
• Analytical framework and hypotheses
• Research design
• Results and discussion
• Conclusions of the paper

Conceptualization and theoretical basis of the work

The first questions to ask about a paper are connected with the problem or issue being investigated. There should be a clear statement near the beginning of a paper explaining what problem the paper seeks to resolve. Authors often leave this until half way through the paper. Indeed, some leave it out altogether, perhaps assuming that it is self-evident or simply not realizing that although it is obvious to the author, a reader with no previous knowledge of the work only has the paper to go on.

Any serious piece of research will involve concepts that are specific to the issue being investigated, or to the investigative approach that has been taken. These need to be summarized at the very least, and explained if they are not common within the field of the target audience. This is not just a case of explaining the concepts related to the particular construction industry phenomena under investigation, but, more importantly, to identify the methodological basis of the work. Thus, a research paper is not place for “text-book” explanations. Of course, the nature of the investigation is inevitably connected to some issue of relevance, but, while it may seem heretical to some, it is not necessary for a piece of construction management research to be practically relevant in industry. A piece of research may hold relevance only for other researchers, but that should not detract from our judgment of its value.

In assessing a report of research, it is always helpful to be able to see clearly how the reported work builds upon previous work. There should be explicit connections to an existing body of knowledge or body of theory, although these may not reside in the literature of construction management. Indeed, it is helpful if there are references to bodies of research and knowledge outside our own “domain”, since ours is not an academic discipline in its own right, with its own research techniques and theories (Hughes, 1999). While there are some emerging strands of theory that are unique to construction management or construction economics, most research in this area builds upon theoretical models developed elsewhere in the social sciences. These connections must be identified in order to make clear where a particular piece of work is rooted and to ensure that we are not simply re-inventing theories and models that are well-known in more mainstream disciplines. Without such connections, we run the risk of consigning our research to an academic backwater. With such connections, we may even be able to influence developments in mainstream thinking. In determining the theoretical basis of a piece of research, it is useful to think about knowledge domains. At the very least, it might be helpful to think about where would you expect to find a particular piece of work in a library so that connections and antecedents are clear.

These issues are important because progress in our understanding often depends upon our ability to generalize from specific examples. One question I often ask research students when they are trying to make these connections is “what is the general class of problem of which your chosen topic is a specific example?”. Understanding this enables some kind of view to be developed about the extent to which findings might be generalized into a wider context. Thus, good papers will begin with what is well-known and move gradually deeper into the less well-known (Latour, 1987: 57).

Analytical framework and hypotheses

All research papers have an analytical framework. Unfortunately, it is not always clearly articulated. The extent to which a particular approach is authoritative is often judged in terms of where it has come from. Thus, further connections to the research literature should be expected in the passages describing the analytical framework. When this is done well, it helps to establish the credibility of the paper by showing the usefulness of the particular approach, or approaches that precede it. The articulation of the analytical framework helps in judging the usefulness of the research question. It explains how the concepts and theories are being applied in this particular case.

One perennial problem with research papers in our field is the question of whether there should be hypotheses. They are certainly not a pre-requisite for a good research paper. In fact, they may not belong at all. The question about whether there should be hypotheses is, perhaps, a wrong question. Their presence or absence depends upon the methodological stance of the research. It is not intended to enter into the methodological debate here, other than to point out the dangers of not understanding the methodological implications of different approaches to research (see, for example, Seymour and Rooke, 1995). Given one methodological stance, hypotheses may be irrelevant. Given another, they may be indispensable.

If there are hypotheses, they should be clearly stated. If there are no hypotheses, then this, of itself, is not a problem, but it should be clear whether the paper is a review, a case study, a contribution to theory development or some other type of study. Without clear articulation, the reader stands no chance of determining the value of the contribution. In the presence of hypotheses, the relationships between the main variables should be explicit and reasonable. They should be stated in a way that makes them testable and the results, no matter what they are, interpretable. If the research is not built on hypotheses, the significance of the paper’s contribution to the development of theory must be explained.

Research design

In undertaking research, there are many methods that can be used to find answers to questions. Some are more suitable than others. In answering certain types of question, one particular method may be very powerful, but the same method may be weak in dealing with other types of question. Therefore, the relevance of the methods of research will be judged in terms of their appropriateness to the nature of the question being asked. Similarly, the sensitivity of the methods must match the needs of the research question. A good paper will make clear the type of research design, perhaps by reference to earlier, similar studies from different regions, different industries or different disciplines.

The research must be focused on an appropriate unit of analysis. It is useful to describe the criteria by which this was chosen, as well as the criteria by which the cases were chosen. For example, the unit of analysis could be a person, a finished building, a project, a firm, an industry or a country. Each would result in an entirely different study from the others. Moreover, cases might be selected from a large number of similar cases, which would imply one kind of approach, or the question might be framed in such a way that there is only one case, implying an entirely different approach. Neither, of itself, is more or less valid than the other. Indeed no judgement can be made about the validity of a piece of research simply by counting the cases or referring to the unit of analysis. Each characteristic depends on the other.

It is always important to address whether the research design isolates what is being measured from other effects, or, at the very least, identifies the inter-relationships between the effect under scrutiny and other effects. If the research design involves the identification of variables, they need to be clearly and reasonably operationalized (i.e. translated into simple descriptions of what is measured and how it is to be measured) and the reliability and validity of the measures should be discussed. Similarly, there will be issues related to the appropriateness of the population for the research question being studied, the sample size used and the extent to which the results can reasonably be generalized on the basis of this particular sample.

Again, not all research is as deterministic as this, but there are traditions in different types of work and if a phenomenological or ethnographic approach is being adopted, then the author should take this stance clearly and confidently and not try to dress it up in hypothetico-deductive clothes! These issues are well-articulated by Johnson and Duberley (2000), who warn against the dangers of not dealing with the epistemological positions that are implicit in different approaches to empirical research. It is important to emphasize that none of us wishes be prescriptive about the kind of research that is encouraged in ARCOM. We welcome research outputs that add to our understanding of construction management issues. But, the relevance and appropriateness of research outputs can only be evaluated in the light of the epistemological background to the research and this is why it is important to discuss the design of the research.

Results and discussion

Within the research paper, the data or evidence of the field-work must be present in some guise. But there are limits on the length of papers, whether for conferences or for journals. It is inevitable that the data will not be reported in their entirety through these outlets. Thus, one technique is to describe what the data is like, rather than what it is. Perhaps sample sections of interview transcripts can be given; examples of diaries or other source documents can be used to illustrate the approach taken. The full record of the data can be maintained elsewhere, perhaps in a departmental library or on the internet, so that the interested reader can interrogate the data further.

In any event, there must be sufficient information within the paper itself for the reader to evaluate whether the data were appropriate for the study and whether the data collection and record keeping were systematic. Similarly, the validity and robustness of the results of the study will depend upon whether the analytical techniques were appropriate and adequately described. Most importantly, there should be reference to accepted procedures for analysis. This helps the reader to understand what kind of tradition there is in the particular kind of analysis and how such research is generally reported.

In assessing how systematic the analysis has been, one of the main ideas is to persuade the reader that if he or she were to have done the same things, then the same conclusions would have been reached (Latour, 1987). Again, it is important that this very statement implies a certain epistemological stance, so the researcher and the reader need to be clear about whether they are working from the same basis in coming to their views about the results and their discussion.

Conclusions

Conclusions can be the most difficult part of a paper to write, particularly if the context and research design have not been addressed properly in the first place. It is often the case that those who have the greatest difficulty writing conclusions, can trace their difficulties to poor research planning. When research is well planned, the conclusions become obvious from the work that has been reported.

No new facts should be introduced in the conclusions. The conclusions of the study should be consistent with the results of the analysis. Where there is no numerical analysis, the conclusions should be consistent with, and follow from, the development of the argument in the paper. I am not quite sure why, but many authors commence their conclusions with a summary of the paper. This is usually not required, if there is already a summary at the beginning.

The conclusions may be expected to reconsider the purpose of the research, summarize what was discovered and provide a discussion about the implications of the findings. In many cases, conclusions can be bolstered by considering whether there are alternative conclusions that are consistent with the data or arguments that have been presented. Also, it is useful to consider both theoretical and practical implications of the results. If the research has been properly contextualized at the beginning of the paper, the theoretical implications of the reported research can be adequately connected to the literature discussed there. It is usual to include limitations of the study and future research needs, to the extent that the research has revealed further gaps in our collective knowledge. The limitations of the study should be noted in terms of the parameters of the research and applicability of the findings. Authors sometimes misinterpret the purpose of a section on limitations of the work and attempt to indulge in soul-searching self-criticism, identifying faults in the execution and reporting of their own work. This is simply not required. The section on limitations should make clear that, for example, the conclusions do not apply to all construction activity in all places at all times. The approach taken in the research enables certain generalizations to be made, but what are they?

Conclusions can also be bolstered by including discussion of the evidence for and against the researcher's arguments and making a clear distinction between the data and their interpretation.

Finally…

Any research paper is capable of being summarized succinctly. Papers are expected to include an abstract or summary at the beginning, especially in the cases of conferences and journals, but this should be the last thing to be written! Although abstracts may be invited for conferences before the paper is written, the abstract of the final paper ought to be written after the paper is finished, summarizing the paper that has been produced, rather than the paper that was originally intended.

In evaluating a research paper, it must be possible for the evaluator to summarize the paper, indicating an overall assessment of the adequacy of the study for exploring the research problem and an overall assessment of the contribution of the study to this area of research. Authors would do well to bear this in mind when writing their papers, so that they can provide the relevant cues that will lead a reader to conclude exactly what the author has concluded. Sometimes, there is simply not sufficient information in the paper to enable such statements to be articulated. In such cases, the paper should be revised.

References

Hughes, W.P. (1999) Construction research: a field of application. Australian Institute of Building Papers, 9, 51-58.
Johnson, P. and Duberley, J. (2000) Understanding management research: an introduction to epistemology.
London: Sage.
Latour, B. (1987) Science in action: how to follow scientists and engineers through society. Cambridge, Mass.: Harvard University Press.
Rudestam, K.E. and Newton, R.R. (1992) Surviving your dissertation: a comprehensive guide to content and process. London: Sage.
Seymour, D. E. and Rooke, J. (1995) The culture of the industry and the culture of research. Construction Management and Economics, 13(6), 511-523.
Silverman, D. (2000) Doing qualitative research. London: Sage.
Turk, C. and Kirkman, J. (1982) Effective writing: improving scientific, technical and business communication. London: Spon.

Checklist of questions to keep in mind while reviewing a research paper:

1 Conceptualization and theoretical basis of the work

(a) What is the problem or issue being investigated?
(b) What are the major concepts, and how clearly are they defined/explained?
(c) Is the connection to an existing body of knowledge or theory clear?
(d) Is there some practical relevance in this work (research practice or industrial practice)?
(e) What is the theoretical basis of this work, i.e. knowledge domain, for example, where would you expect to find this work in a library (classification number)?

2 Analytical framework and hypotheses

(a) Is there a clearly stated research question? (It might not be a research paper, as such)
(b) Are there hypotheses? Are they clearly stated? If there are not hypotheses, is the paper a review, case study, contribution to theory development or some other type of study?
(c) If there are hypotheses, are the relationships between the main variables explicit and reasonable? If there are not hypotheses, is there adequate development of theory?
(d) If there are hypotheses, are they stated in a way that makes them testable and the results, no matter what they are, interpretable? If there are not hypotheses, are there clear indications as to the significance to theoretical development?

3 Research design

(a) Are the methods of research appropriate to the nature of the question being asked?
(b) What is the type of research design?
(c) Could the design be improved? How?
(d) Is there a clear account of the criteria used for selecting the focus (unit) of analysis and the cases chosen?
(e) Does the research design isolate what is being measured from other effects? Are the variables clearly and reasonably operationalized (what is measured and how)? Are the reliability and validity of the measures discussed?
(f) Is the population appropriate for the research question being studied? Is the sample specified and appropriate? Can the results be reasonably generalized on the basis of this sample?

4 Results and discussion

(a) Are the data appropriate for the study? Was the data collection and record keeping systematic?
(b) Are the statistical techniques appropriate and adequately described? Is reference made to accepted procedures for analysis?
(c) Are the control variables adequately handled in the data analysis? Are there other control variables that were not considered but should have been?
(d) How systematic is the analysis?
(e) Is there adequate discussion of how themes, concepts and categories were derived from the data?

5 Conclusions

(a) Do the conclusions flow from the work that has been reported?
(b) Are the conclusions of the study consistent with the results of the analysis? (If there is no numerical analysis, are the conclusions consistent with the development of the argument in the paper?)
(c) Are alternative conclusions that are consistent with the data discussed and accounted for?
(d) Are the theoretical and practical implications of the results adequately discussed? Are the theoretical implications adequately connected to the literature discussed at the beginning of the paper?
(e) Are the limitations of the study noted (in terms of parameters of the research and applicability of the findings)?
(f) Is there adequate discussion of the evidence for and against the researcher's arguments?
(g) Is a clear distinction made between the data and their interpretation?

6 Summary

(a) What is your overall assessment of the adequacy of the study for exploring the research problem?
(b) What is your overall assessment of the contribution of the study to this area of research?

References

Rudestam, K.E. and Newton, R.R. (2007) Surviving your dissertation: a comprehensive guide to content and process. 3rd ed. London: Sage
Silverman, D. (2009) Doing qualitative research. 3rd ed. London: Sage.