Hughes, B., Joshi, I., Lemonde, H., & Wareham, J. (2009). Junior physician’s use of Web 2.0 for information seeking and medical education: A qualitative study International Journal of Medical Informatics DOI: 10.1016/j.ijmedinf.2009.04.008
I have to admit that when I first glanced at this paper I thought the methodology was good. There is talk of triangulation and inter-coder reliability etc. But when it is read more deeply much of it simply does not make sense because key concepts are so loosely defined. In the past few days I have seen this paper mentioned several times on twitter and in blogs, but there has been little or no mention of the poor quality of this study. Therefore I thought I should add my thoughts to the debate.
Method
The study took place in the NHS in England. The subjects were junior doctors. 55 were identified through a stratified sample (of 10 different specialties) from 300 graduating from a London medical school. 50 of these agreed to participate but only 35 completed all three stages. More demographic data on the participants would have been useful.
Next, they were given a questionnaire, used in previous research on this topic, and asked to keep a diary over at least 5 days of every website they accessed for work. Finally each participant was interviewed although themes were saturated after 20 interviews.
Results
From the survey data, 32 of 35 said they used web2.0 sites and of these 28 used wikis (read the content, only one doctor contributed to wikis). Next, looking at the diary data, confusingly, google.com is now referred to as web2.0 content, whilst in the survey it was not. 80% (28/35) of physicians used google during the five days. 25/35 reported using wikipedia. Smaller percentages used yahoo.com, doctors.net.uk and Facebook. This data is presented in chart form with percentages of physicians accessing each site (eg google, wikipedia, NICE). Presentation in tabular form with absolute numbers of accesses would have given more information.
The participants were asked to state for each of the 444 events where they accessed information online whether this was on a wed2.0 or user-generated content site, or hybrid, or traditional content site. The doctors said that on 235 occasions they were accessing web 2.0 content. However, the authors have classed the 142 uses of google and 115 uses of wikipedia (total 257) as web 2.0 content. No absolute numbers for the access of yahoo, facebook and doctors.net are given so the agreement between the authors and participants over what constitutes web 2.0 is not clear.
The authors then present themes from the interview data. Here "using the internet" is confusingly equated with web2.0 content. There is mention that doctors look things up online because it is easily accessible, and up to date, but at times they are uncertain about the quality or usefulness of the information found. The authors introduce a taxonomy of information needs from the interviews which they then use to analyse the information needs addressed inthe diaries, categorising 237 out of 444 internet accesses/information needs:
- "to solve an immediate defined problem" "to advance an immediate task in the clinical context and forms a closed question with a specific answer" "closed questions" 107 of total information needs, of which 90 addressed through use of "hybrid or best evidence tools" (these tools are not specified)
- "background reading on a subject" 130 information needs, of which 107 addressed through the use of "web 2.0"
Futher information is given on the way that doctors used google. 21 out of 35 mentioned using google as a way of navigating between trusted sites. It is not stated if these trusted sites were named in the diary.
There is then some discussion of how these (web 2.0) sites could be better used in clinical contexts. Doctors mentioned :
- patient education- comment is made of patient use of wikipedia and need to educate patients on different sites
- physician education- awareness of "web2.0 sites" as difficulty is in finding out about sites(wikipedia and google? or were they referring to some other web 2.0 sites? or to trusted web 1.0 sites?) , not much training necessary as sites so easy to use.
- remove blocks to web2.0 sites - it is reported that google is blocked in
I am not commenting on the discussion of the paper because I found the method and results section quite perplexing. No clear definition of web 2.0 content is given. It is not clear why the use of google is considered use of web 2.0. As Mark Hawker has pointed out google is a web 1.0 application. (Data is indexed by computers and pulled by humans. The content is not in any way user-generated or social. ) Previous researchers such as Sandars and Schroter, who this paper cite, did not consider google to be a web 2.0 application.
Because the authors did not use a clear definition of web 2.0 content this work can tell us very little about doctors use of web 2.0 content. It is possible that most doctors are using the same trusted websites that they have always uesd. They use wikipedia because it is easily accessible (free and no passwords needed) and is equivalent to an online textbook. The user-generated content of Wikipedia is not a factor for most doctors. Credibility of user-generated content for physicians did not emerge as a theme in the qualitative work. Instead they were concerned about how patients might use the same websites that they use.
Overall, I found the study very disappointing. We need debate and discussion on how best to address the informational needs of clinical staff. To me, the best description of these needs still seems to be Richard Smith's BMJ review in 1996. Now we should be asking, have the information needs of doctors changed in the last 13 years? How are these needs best addressed by current technologies and what tools should we be trying to develop.
What do you think? Am I being too harsh? Why did you like this paper?
Slightly too harsh, yes.
ReplyDelete"They use wikipedia because it is easily accessible (free and no passwords needed)"
Let's learn from that and build resources that professionals want to use.
Actually, that was my conclusion not theirs and I agree it is the lesson to be learned here.
ReplyDeleteI have been harsh on the authors who seem to have done a lot of work with no funding. But I have also been harsh on my fellow bloggers who I think have not been critical enough of the rigour of this paper.
We need to make accessing information as painless as possible. But let's not confuse things by dressing it up as 'web 2.0'.
Thanks AM, for an honest account.
ReplyDeleteAJC, I see what you're saying but how would these new resources be rolled-out to a local, national, or even global scale? Who would design, manage and govern them? What is currently wrong with Wikipedia, and how could we improve it using the technologies that are already there? Professionals want, and more importantly do, use Wikipedia. So how about we work with them, a freely editable resource, to make it better. That way, we're not re-inventing the wheel. We're enhancing a platform.
Just my 2p.
Kia ora Anne Marie
ReplyDeleteSo far as I can make out, there is no breakdown in your report in this post of what sort of information junior doctors were looking up.
My very early experience (1970s) in post grad work (not medical) was that all of my colleagues, including the 'doctors', always had resource books on hand. We were seldom judged by the cover of the resource we looked up for our day to day information needs, nor was there any judgement made on our need to look it up in the first place. This was in contrast to the need for citing reputable sources in submitted reports.
As far as I could make out (and my opinion has not changed over the years) this was all part of learning and getting things done. If the data required was some thermochemical constant that was important only insofar as its magnitude be known, it didn't matter if the researcher didn't have that committed to memory. What was important was that the researcher was able to ascertain whether data was appropriate, authoritative (if you like) and within the bounds of practical accuracies.
Now I'd use all the same sort of Internet-based sources you mention here, as well as printed resources for data searches.
Catchya later
from Middle-earth
Hi Anne Marie,
ReplyDeleteI enjoyed your take on this interesting paper. Clearly the paper is not the most scientifically rigorous article ever published...
You are highly critical of the lack of a strict definition of Web 2.0 in this paper. Difficulty defining web 2.0 is not a new problem - indeed Tim Berners-Lee has famously argued "sharing/ collaboration/ people to people" is what the web was all about all along. However some web-based tools and resources emphasize these qualities more than others. I agree the Google Search doesn't fall easily into the Web 2.0 category - but Web 2.0 resources certainly feature prominently in the search results and may be used to share findings (although I realize this is not a reported finding of the study).
Google and Wikipedia are easy alternatives to struggling with MESH terms and Boolean operators in trying to do a proper PubMed search (like we "should" be doing). This may be where easy-to-use alternative search engines (http://sandnsurf.medbrains.net/2009/06/medical-search-for-physicians/), with more characteristic Web 2.0 features, may gain a foothold.
On the topic of wikis, I think there is room for profession or specialty-orientated Wikis. Wikipedia is too general for specialist physician purposes. In the future I'd like to wikis made by the members of medical colleges, perhaps with topic editors overseeing the it's evolution.
PS.
My reaction to the paper was blogged here: http://sandnsurf.medbrains.net/2009/07/how-and-why-junior-docs-use-web-20/
@MarkH I'd prefer to go down the evolution rather than revolution route where Wikipedia is concerned, but Wikipedia has proved to be remarkably resistant to change. There's also the issue that it is forever tarnished as far as some people are concerned.
ReplyDeleteFor years the gold standard in science/medicine has been peer review (though it's faults are often overlooked). What we need to use Web2.0 for is to evolve "peer review" into "trusted peer networks".
Damn, another rogue apostrophe. I swear this keyboard is faulty! ;-)
ReplyDeleteI liked the article as well and have written a blog post about it mostly because it appealed to me. To my experience young doctors do use the Internet as described in the article, even I use it that way for some medical problems and I am not even a junior anymore.
ReplyDeleteProbably your right about google not being web 2.0 but it is still very much in use, new developments don't mean we should throw away the good from before.
To my opinion the biggest challenge for the future is ensuring that physicians have the skills to determine what credible internet information is. Peer review is an option although it takes a lot of effort.
By the way I mailed the author about the question hope he will answer them.
Kind regards Dr Shock
Agree with MarkH about not reinventing the wheel. Wikipedia is an amazing resource, and if all the doctors using it contributed just a few minutes of time to editing it, it would become a substantially better medical reference.
ReplyDeleteI think one of my roles as a medical teacher with an interest in webnerdery is to show docs not just how useful (or potentially useful) web2 tools are, but also how important it is that we contribute to them. Thus we can help maintain a high standard of information for professionals and patients (which helps everyone).
BTW, shouldn't you be working on your presentation for next week?...
@middle earth blogger (aka Ken)
ReplyDeleteThe information needs are introduced in the results section. I have no problem with using web-based resources. I couldn't survive without. But I object to the confusion caused by throwing around the web 2.0 term. But maybe I do object too much:)
@precordial thump I agree with pretty much everything you say. I use pubmed like google.... and it seems to work (just don't tell @laikas:) Maybe I am being a laggard about the newer searches.
I use google to navigate through trusted sites too. It is good for this. It learns what I want.
Thanks for your thoughts.
@drshock Thank you. I think understanding how health professionals gain information literacy is really important. What is evidence these days, anyway? :)
@colin Keep quiet about the presentation:) The more I think about it the more I like the idea of getting students to contribute to wikipedia.
I will think about this more.
Thanks again.
I'm glad to read a sceptical review of this paper - I started to think I'm the only one who thinks there's just no enough to cover a whooping 11 pages.
ReplyDeleteYou can't hide from me AnneMarie.....
ReplyDeleteI won't go into a discussion with you about how to search PubMed.... ;) Just want to say that I appreciate this critical review.
I agree that Google is not web 2.0 and that they should have been consistent in their classification. Perhaps they should have called it "Junior physician’s use of Internet instead of web 2.0. Still it is clear that INTERNET and some web 2.0 are used a lot, and especially for background (open) questions, which makes sense. The nice thing is that this qualitative study gives some insights into the motivation of the doctors for their choice.
Finally received the answers from the author of this work
ReplyDeleteAbout their definition of web 2.0:
"I published a definition of Medicine 2.0 in Hughes, B., Joshi, I.,
Wareham, J. 2008. Health 2.0 and Medicine 2.0: tensions and
controversies in the field. J Med Internet Res, 2008;10(3):e23, we use
this as a base definition in this more recent article."
About the search incidents
"We did classify all the search incidents, but only in a subset were
we able to identify both themes of open/closed question and Web
2.0/1.0 use. The use of hybrid sites, noted in the paper confuses
things. In most cases we could guess based on the diary information,
but we stuck to a clearly stated subset where we could be 100% sure -
a more transparent and conservative approach, which we felt made more
robust research."
About the sites accessed in the diary entries:
"We provided a list fo the top 20 sites in the paper. Somewhere in
an annex, published only online and not in the hard copy journal, the
full list of sites is given."
A little less disappointed?
Kind regards Dr Shock
sangambayard-c-m.com
ReplyDelete