There are several posts in this blog about reviewing research papers, and it is a theme that constantly crops up, both in relation to refereeing a journal paper for an editor of a journal, an in relation to building up the elemental parts of a literature review. As I have said before, these are two distinctive activities.
Many students struggle to make critical evaluations of published papers, and there are many reasons why they might find this difficult at first. One way of approaching the task could be to try to focus on whether a paper is making an interesting and useful contribution to our collective understanding. I think that what we are looking for is signs that author of a paper is clever enough to provide a persuasive case. What I mean by this is to look for evidence in a paper that the author is knowledgeable, has a good understanding of the issues, can put together an argument and can make judgements.
In order to be convinced that an author is knowledgeable, I would look for evidence that he/she knows about the generalities and specifics of the topic. To be convinced that the author has an understanding, I would try to find evidence of different views being weighed in the balance and ideas being applied to examples, especially I would look for the use of metaphors and analogies. In thinking about whether the paper has a strong argument, I would look for the use of logic and the sequence of ideas. The language of argument is interesting, and this is closely related to grammar, vocabulary and logic. I do not find the use of jargon or inflated diction particularly compelling. But plain language in straightforward words is often an element of a good argument. It should be clear. Finally, in looking to see whether an author is making judgements, there should be evidence of discrimination in the kind of sources being relied on to form the foundations for the work, and judgement in the nature of the conclusions as well as the all of the choices about what to include and what to exclude in putting the paper together.
Some common errors frequently reveal weak papers and unclear thinking. Papers typically follow this structure:
- Abstract - summarizing the whole paper including the specific findings
- Introduction - explaining what the paper is about
- Literature review - dealing with what we collectively already know, specifically, but not exclusively, in terms of past research
- Possibly some empirical work - not all papers fit into the category of empirically-based research
- Analysis and/or discussion - empirically-based papers would separate the analysis from the data, then all papers would include discussion of what all this leads us to
- Conclusions - explaining what this all means, perhaps with an outline of the parameters of the research and how that restricts the application of the findings, and usually with suggestions about what certain groups of people might do differently as a result of this research, whether these groups are scholars, practitioners or policy-makers
In weak papers authors often get the sections mixed up, probably because a weak paper will not really lead to any conclusions. Thus, I often find that the conclusions only really summarize the paper, the abstract introduces, rather than summarizes, and the introduction jumps straight into a general literature review before the detailed and focused part of the literature. As a result, in my editorial capacity, I often suggest to authors that they should move the introduction section to the literature review, move the abstract to the introduction, move the conclusions to the abstract, and then try to write some proper conclusions that convey something about what all this means. If they cannot do that, then the work is not really good enough to be called research, I would venture to suggest.
It might be an interesting exercise to see if you can find a paper where the sections are mixed up in this way. It would certainly give new researchers the confidence to take a critical stance when it comes to reviewing papers for their literature review!
