What is over-research?
At this years’s RGS-IBG conference, I co-hosted a session on ‘over-researched places’ with Cat Button. It was nice to finally have the session as the idea had gathered attention elsewhere, and was sponsored by the Social & Cultural, Political and Participatory Geography Research Groups. We also got quite a bit of coverage before and during the conference on Twitter. I think I can see why ‘over-research’ struck a chord. Here, I want to collect some notes on what over-research is, why it comes to be, and also what we can do about it.
Basically over-research is the idea that certain research themes, case studies or locations become paradigmatic examples of a particular issues, to a less than helpful degree—there is a clear value judgment in the idea of OVER-research.
I suspect that almost all fields and subfields have an over-researched subject. My own focus on low carbon communities has more than its fair share of contenders. In the last ten years the Transition (Town) movement has exploded both on the ground and in terms of academic research articles, masters and PhDs. The Findhorn Foundation on the Moray Firth are usually expected in discussion of ecovillages, the Isle of Eigg in discussion of community energy, and in urban contexts LILAC in Leeds seem to have taken the place of BedZED as the go to example.
This is a UK-centred selection, but in each case, even only within the UK, there exists a wide range of examples and activities that are hidden, or obscured from view. It’s not that there’s necessarily a problem with Transition, Findhorn, Eigg or LILAC, or even with research on them, but that research on eco-communities has a lot more to say than can only be found in these high-profile cases.
Over-research is not only empirical though, but also theoretical. Sticking with low carbon community, there is also a narrow way in which the emergence and growth (although this is a contested term) of these examples has been understood. The major theory of change here has been through the cross-cutting platform of ‘Sustainability Transitions’, where a discourse of ‘niche’ initiatives growing into more established ‘regime’ actors dominates.
What’s wrong with over-research?
Part of the thinking behind the session on over-research is that the focus on totemic examples of any topic—which may or may not be fair representatives of the field—is a problem. First, and most basically, it’s a problem in the type of social research I carry out for those being researched. Basically, when one group or place becomes ‘The Place” where this research is done, it seems everyone wants a piece of the action, and those on the ground can become swamped. There are some ways around this, for example by using participative methods, and being as clear and up-front about one’s aims and objectives as possible. More and more tools and guidance is being produced to help in this regard. Indeed, Transition have described themselves as “overwhelmed” with researchers coming in, taking up time, and giving little in return—research becomes an unwanted distraction. Transition have even gone so far as to develop their own guidebook to prevent these types of problems. But it is unfair to expect each and every research subject to be this organised, especially research on/with more marginalised people and places without the resources to adequately hold off pushy researchers.
Over-research is not only a problem for those being researched, it is also a problem for those doing the research. An over-reliance of one or two case studies, can lead to distorting the accuracy of one’s data and results. It’s not that research cannot reach sound judgement on a given field from only a select case study, Flyvbjerg has comprehensively dealt with that concern. It’s more that the dominance of headline case studies often fails to represent a field adequately. There’s also the issue that certain examples are more preferable to academics, funders, and for gaining media traction, which of course are also partly influenced by a host of other biases: class, gender, language. In my field I think Transition are a pretty good proxy for collective action on environmental issues in the UK, but they are far from being the only one to the degree that research in this area seems to indicate. Over-research here is closely tied up with under-research—of other ideas and participants overshadowed by signature examples in the literature.
Over-research fails to accurately represent the field, but it can also worsen the quality of the data gathered. In qualitative research, interviewing, ethnographic and participative evidence and experience is dulled by being ‘yet another’ researcher among a steady stream. As the quantity of research increases, interviewees become jaded. Participants can also begin to try to work out what is expected of them from a researcher. I remember being surprised by one interviewee a couple of years ago who stopped me to ask “How is that? Am I doing ok?”. I had no idea what she meant, but thinking back upon it, this participant clearly had some notion of what counted as being a valid response to a university researcher.
A further, third, issue was raised in our session where the distorting effects of over-research were felt far away – a problem for those beyond those being researched or doing the research. When an example becomes paradigmatic of how an particular phenomena is carried out (empirically), or understood (theoretically), descriptive very easily slips to normative, and it becomes how things should be done. The idea of ‘best practice’ looms large here, where one approach begins to overshadow a wide variety of ways things can be carried out. One participant in our session pointed out that the over-reliance of certain cities as being the ‘greenest’—Freiburg or Vancouver are the usual suspects here—means that more ‘normal’ cities then try to shape their own efforts for sustainability to ape these paragons. Over-research is a sort of absent presence here. While researchers can be absent from these ‘normal’ cities, the effect of building up paradigms of sustainability such as Freiburg mean policymakers triangulate their efforts in response to what they see as the right, correct or best way to act. I have experienced something similar to this to a much smaller degree. When one of the councillors in my hometown in remote rural North Scotland heard my PhD had been on the Transition movement—he got in touch with me to “see about getting one going up here”: completely failing to recognise the citizen-led, emergent ethos behind such groups and movements. The idea of Transition Towns had become totemic, overshadowing any and all ways in which citizens can carry out environmental action collectively.
Why over-research? Bandwagoning, Price’s law and others.
So far, this is a fairly one-sided picture of over-research: Over research is bad! It’s damaging! And it’s not good research! Well, I think over-research is a negative thing, but I also think there are some understandable reasons why over-researched places come to dominate particular sub-fields. There are many surface reasons, like researcher’s conservatism in pushing new boundaries, or solid researchers sticking to tried, trusted and dependable examples: ‘what works’. Further to these, there are secondary surface reasons—reviewers of both papers and proposals immediately ‘get’ an example they’ve heard of before, but need to do much more thinking to get their heads around a new idea, or empirical example. New topics also need a lot more explaining which can take away from the flow and quick pace of writing. There is also the familiar rhetorical trick in social science of taking an example supposedly well-known, and giving a new hidden or unseen twist that brings it to life again—much harder to do when introducing something new in the first place. With a comparatively under-researched topic none of this is possible in the same way. This could very easily develop into a ‘rut’ in a given research track, which deepens and establishes particular examples as the traditional or established way to do research.
Understanding these mechanisms, and where the factors that produce over-research come from is important if we are to find a way past this impasse. Moving forward, I want to dig a bit deeper into the more sedimented reasons for over-research. Most of the above critiques fall into the broad category of bandwagoning. Bandwagoning is basically the idea that someone is more likely to say something, adopt a believe, or carry out a course of action, if it is already well-established, or legitimised. In methodology textbooks bandwagoning is normally discussed in focus groups: where one or two more dominant figures in the focus group declare something, making it far more likely that the rest of the group will follow or agree. Over-research can be seen as a side-effect of a general bandwagoning in any field of research.
The Matthew effect describes how being already successful academically is the best predictor for future success: those who are more likely to win the next grant already have one. Price’s law and Lotka’s law describe how in any given sub-field, or discipline, citation practices tend towards a hierarchy. Each of these is an attempt in some way to describe the gathering or bunching up that happens in various research fields. There are also ‘network effects’ where when one example or theory is already known another paper, research project on the same topic more easily fits into and benefits from the increased exposure to that issue.
In all of these there is a positive feedback mechanism at work, where researchers who are already cited or have won funding, are cited more and win more funding. These principles, model’s or laws are describing researchers, their grant income, or citations but each of these in turn are tightly interwoven with actual coal face research. These principles might help us explain why certain research themes and examples also bunch-up, and become over-researched. Processes like Price’s law and bandwagoning show what Deleuze might call ‘arborescent tendencies’. The challenge of over-research is to find a way beyond this arborescence or ‘bunching-up’, towards a far more representative distribution of our cases and theories.
Of course there are also factors in research which serve the opposite ends: the desire for novelty or the pursuit of originality. Fortunately too, we have tools at hand to tackle the tendencies that produce over-research. Bandwagoning is a well-established phenomena in focus groups, where one person can find themselves (perhaps inadvertently) leading the discussion down one particular path or viewpoint. The importance of good moderator is key here.
One of my highlights of the RGS-IBG conference where we had the over-research discussions was Will Eadson’s talk where he riffed off a Deleuze and Guattarri metaphor of a flock of birds, where no bird comes to predominate, rather each takes turns at the front, as a collective finding a way to move each individual from the centre towards the edge. Finding something similar in academic research choices would be a very useful way beyond over-research.