Tuesday, February 22, 2011

Does size matter?

I was talking to a student today who was getting exercised about response rate to his questionnaire. He is aiming at around 200 individuals and hoping for around a 70% response rate. I hope he gets it, I really do but.... if he doesn’t is it REALLY such a big deal?  Well, you might say yes, low response size means that the results are unreliable, we cannot extrapolate them to the wider population and that extreme responses loom larger than they would normally do and cause bias. I agree that all of these points matter if one is developing a cure for cancer or mapping the human genome to develop smart drugs but if all you want to know is how physiotherapists treat back pain, or what nurse prescribers think about their role, then surely then quality of information available is, as, if not more, important that the size of the sample?
I’m not the first to think this, in 1997, Templeton et.al were  making the case that as long as care is taken to ensure that the group surveyed is representative of the larger population, then sample size is not as important as we are lead to believe. Perhaps it is the case that educators and supervisors should spend more time with students discussing about how they are selecting their target sample than dismissing useful research data because “you only report a 25% response rate”.  Often such a comment highlights a lack of understanding in the supervisors or journal reviewers. I struggled to find a platform for a research paper that had a response rate of  16% - not great I admit - but that actually translated into well over 570 respondents, all of whom could be argued to be representative of the target population of interest. The study provided new insight in an under researched topic and could have disappeared into the  Great Academic Marsh of Indifference, had my co-author and I not persevered until we got it into print.
Chalmers enlarged upon this point in an editorial for the Royal Statistical Society in 2006 when he suggested that under reporting of so-called ‘poor response studies’ was leading to a publication bias. He made the point that much research (and I think this is particularly true of most postgraduate student, and Doctoral research) is done to increase knowledge on a specific topic and this information should be shared with the wider world. As we move more towards the publication of systematic reviews and increasing data synthesis across different disciplines the time may be right to start to focus upon the place of smaller studies in the wider knowledge pool. Even small response studies add crumbs to the knowledge table (forgive the mixed pool/table metaphors here – perhaps it’s a water table). Furthermore such ‘small response studies’ may encourage others to move the study design forward, replicate the work and validate it in that way.  Replication research has, to some extent become unfashionable and may be due for a renaissance.  Either way, educators and supervisors should not be discouraging fledgling research students by this obsession with sample size, they should, perhaps be focussing more upon what the study will do for the wider knowledge using community.

Sunday, February 13, 2011

Open to access

But are you also open to provide accessibility?

In the last few years the open access movement has grown stronger. This is a debate specially close to the heart of those who deeply believe knowledge is supposed to be shared and its advancement is related to the practice of opening up their ideas to a wider public. I’d also argue that it is about reaching the audiences the research focuses on and targets at. Some call it public and community engagement. Be that it may, it should be open too!
Further on that thought, I also believe that openness of research is more than putting a couple of peer reviewed papers online for easy download. Although that is already a major step and one that all researchers should be pushing forward, if it’s real impact we seek, then we need be looking beyond the citation metrics! We should be looking at making a difference both within our discipline and society. That, as Sarah Bodell so rightly hints at, involves us to revisit what we consider to be our professional obligation both before our professional bodies and the communities they are supposed to serve.

Although the Open Access Movement is still not a practice of the masses, I believe it has acquired status beyond ‘an ideal’. It is slowly penetrating academic practice. Now, we just need to make it widely and formally accepted. Yes, you read me. Despite all the institutional open access mandates across the world, I still don’t see it encouraging a real shift in practice, i.e., in terms of how academic publications are approached. Most repositories store bibliographical references of articles published in closed journals, instead of instigating researchers to publish in open access journals. I sometimes wonder how much of this is not free publicity for paid journals?! That’s the game we currently play. We want to implement a new idea (open access) in a rather traditionalist, elite-led frame. Something will have to give. So far, the old structure has not shaken. ...well, not enough to instigate a serious change in people’s epistemology of practice. (As I write this a new question arises to me which partly links with my current research: how many of the open access advocates see sharing, discussion and joint construction of knowledge as a key element of their practice? And how much of that philosophy is supported by the way we measure research?... are we really measuring research or the reputation of the places in which it is published?...)

One would think that given the current global economical climate the opportunity would be easy to spot, but I’m afraid to say that this is (still) not the case. I have high hopes - especially with the new communication channels available online -, but they still haven’t been fully materialised. There are forces that move against them. I feel those forces have more to do with tradition and reputation than with real impact!

Especially, in professional and applied sciences such as the case of health, education, environmental studies, social work, etc, what’s more important: to write a paper which features in a so called high impact journal, or in a place that other researchers and also other practitioners, and why not public, can access it to? With that comes another range of questions: should we just publish research in “academic language”? Should it be restricted to writing? Why can’t podcasts, short videos, newsletters, online discussions also serve as impact. They would probably be more accessible and generate different types of impact 'highly rated' journals haven’t been able to: not only in terms of being free and easy to get hold of, but also in terms of discourse and format.

Online technologies offer different conduits for true communication and dissemination of research, in which the researchers themselves can be directly involved in after they deliver their product (e.g: their publication). I’d even argue that the real strength of these new technologies is ‘participation‘. Hence, it’s not about delivery but rather about transactions of knowledge and ideas in progress. Funny enough, one of the first research journals, which would end up setting the tone for all the academic publications thereafter, was called exactly that: "Philosophical Transactions". It used, what I assume to have been seen as, the cutting edge technology of that time - printing press - to disseminate research.
I will also assume that, given that at the time literacy was a privilege of a few, there was no need to raise concerns about accessibility and whom the journal was read by. Today, however, the case is different, or at least I’d like to think so...but that’s probably another post!

So, all of this to say the following: Open Access is, in my opinion, more than the releasing of pre-prints and copies of articles published in closed journals. The effective transformation requires action and determination in publishing in and creating open access opportunities. Research Journals are still an important vehicle for communication of research, but they don’t have to be closed or managed with profit in mind. We already write the articles for free. Let’s make sure people don’t get charged to read them! Further to that, there are other forms of sharing our practice. Participatory media can provide alternatives. Let’s exploit them. Finally, we need to look at influencing policy. We need to instigate change on how our practice is appraised and how our research is measured. High impact journals aim at perpetuating an elitist style of communication. Times have changed. We need to think community if our goal is to impact on our society.

To hear a discussion about this topic visit the Research Zebra Chat site

Tuesday, February 1, 2011

Deep Impact?

If you were asked to name three pieces of  medical research that demonstrated "impact" what would you come up with? The discovery of penicillin?  Cloning? The mapping of the human genome? Advances in stem cell therapy? Now suppose I asked you to name three pieces of nursing research with similar impact? Anything...? 

In 2009 the Royal College of Nursing Research Society celebrated it's 50th anniversary by asking the same question. The names that garnered the most votes were Felicity Stockwell (1972), Patricia Benner (1984) and Jack Hayward (1975) and the overall winner was Florence Nightingale (1859). Thus, what is perceived as the most influential nursing research of the last fifty years is actually 150 years old. I'd even go so far as to suggest that what Nightingale's work has is longevity rather than influence. Are there no current nursing studies that can demonstrate impact? Doesn't look like it!

Why is this a problem? Across the UK academics are becoming increasingly exercised about this notion of impact since the next evaluation of research output (Research Excellence Framework) will be focusing upon this very thing.  This is a particular challenge for nursing since the high quality research from this discipline can take a considerable time to translate into changes in clinical practice.  In addition, it can take a long time for a piece of work to attain a critical mass of citations which are seen as further validation of the importance of the work. Publish a paper about smashing protons together in the LHC and your paper will achieve mega citations, come up with a useful way of assessing quality in an acute pain service and pray your citation rate reaches double figures before 2014.

This doesn't bode well for nursing research unless we find a way to redefine "impact"

For a complementary podcast go to our sister site http://researchzebrachat.podbean.com