Monday, 13 July 2009

Web 2.0 tools and medical education - more sceptical comments

Last week, @drves described me as a web2.0 skeptic. Those who know me in 'real-life' would certainly agree that I ask many questions, and may doubt received wisdom. Now it seems this facet of my personality is more apparent in the online world too!



Lemley, T., & Burnham, J. (2009). Web 2.0 tools in medical and nursing school curricula*EC Journal of the Medical Library Association : JMLA, 97 (1), 50-52 DOI: 10.3163/1536-5050.97.1.010

The above paper was published in January 2009. It has been talked about a lot on twitter today because it was mentioned in a student BMJ article, which was then picked up in a blog post by Dr Ves. The finding that '45% of medical schools use Web 2.0 tools in their curricula' is that most often cited in twitter and elsewhere. So what does this mean and how did the authors draw their conclusions?

Method
A survey was conducted using Survey Monkey. Participants were identified by emailing a link to the survey to 3 different email lists:
DR-ED (for those involved in medical education) 1383 subscribers
AACN (for those involved in nursing education) 150 subscribers
AAHSL (for academic health librarians- who were asked to forward the survey to those responsible for curricula in their institution) 146 subscribers

The questionnaire is given in an appendix.Although the title and background to the article talk about web 2.0, the first questions asks about use of the following 'web 2.o/social networking tools':
  • Blog
  • Del.icio.us or some type of social bookmarking resource
  • Flickr or some type of photo-sharing resource
  • Moodle
  • MySpace, Facebook, or some type of online community
  • Podcasts
  • Videocasts
  • Wiki
  • YouTube or some type of video-sharing resource

I don't know why Moodle, which is an open-source,and flexible learning management system (LMS) or virtual learning environment (VLE) is included as a web 2.0 or social networking tool. Moodle does support the use of web 2.0 tools, but so can other VLEs, so it is unclear why it is listed here.

Results

There was no way of tracking how those who responded to the survey found it, ie did they find it on a list themselves, or was it passed to them by a librarian? In any case there were responses from 36 individuals involved in medical school education, and 19 individuals involved in nursing school education.

The response rate from the medical school list is no higher than 36/1383 or 2.6%, amd possibly lower if some of the responses came via the librarians' list.

Several responses may have came from individuals in the same medical or nursing school as responses were anonymous.

Despite this the authors go on to report results as the percentage of medical schools which are engaging in the use of web 2.0 tools, rather than the percentage of medical educator respondents. The individuals who responded that they did not use these tools may work in institutions where many others do, and the individuals who responded positively may be the sole educators in the institution to use the tools out of several hundred or more.

The questionnaire did contain a question ("Please briefly describe how these tools are incorporated into your instruction.") which allowed free-text response and could have provided some information for a qualitative data analysis, but no results are given.

My conclusion

Does this paper tell us anything about the use of web 2.0 tools in medical and nursing schools in the US? No.


Is the author's justification of validity despite low response rate, because the study is "to gain insight into an issue", appropriate? No, because exclusively quantitative results are published.

This paper is short. It is open-access. I think that with a cursory look, most people would have reached similar conclusions to me. So why were so many people referring to this paper today without any criticism of the severe weaknesses in methodology?

Thanks to @drcolinmitchell for drawing my attention to this research.
He has also published a great post about this paper.

17 comments:

  1. Why? Because the vast majority of people never look below the surface, they just retweet!

    Funny, most people will agree it's a joke to say "It MUST be true, I saw it on the internet." But then they repeat-and-retweet.

    When I had my own adventure with publicity three months ago, after my post about moving my hospital data to Google Health, I was aghast at the idiocy that some people wrote. All they did was read the Boston Globe article ABOUT my post, and they went merrily about, saying all kinds of things that the Globe didn't say, never mind any question of whether *I* said it.

    My thoughts on this? For one thing, participatory medicine absolutely depends on having an keen and well-exercised crap detector. And, in a more or less friendly way, we need to blow the whistle on unidentified crap when we spot it, just as you've done.

    ReplyDelete
  2. Thanks for sharing this analysis with us. The numbers really don't make it statistically significant, why did they even publish it? Seems basic method is missing...

    ReplyDelete
  3. Thanks for this critical comment.

    Agree with e-patient Dave that ppl retweet to easily (without real reading) and I must admit I'm less thorough since I'm into web 2.0. Because there is so much information, but you can't read it all. I keep possibly interesting things in Google Notebook, which are automatically tweeted from there (i.e. like delicious).
    But I must say that I've got the impression that web 2.0-news seems to get disproportional attention from ppl on twitter and is less critical perceived than other news.
    The studies done often have a weak design.

    Btw you invalidated another popular item in the blogosphere: that "blogs are dead". Blogs are so useful to deal more in depth with a subject.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. The study itself seems to be a bit confused as to what it is investigating! Are they investigating the use of Web 2.0 or social networking? In my opinion the terms are not interchangeable: Web 2.0 encompasses social networking but is NOT exclusively social networking.

    I also found the numbers somewhat confusing. Are the podcasting and videocasting results inclusive of any podcasting and videocasting distributed through Moodle? What is 'academic support' and does it include Moodle? Why is Moodle separated from other course delivery systems? The whole study seems to be lacking an understanding of Web 2.0 and the various resources available.

    I agree with Laika that a lot of similar studies have a weak design. They often have sweeping verdicts based on very little, sometimes inaccurate, or incomplete data.

    ReplyDelete
  6. I am more inclined to be charitable to a truck driver who misquotes a Boston Globe article than to a medical school professor who lectures using a review based on research he has never read and certainly does not understand. But does that stop him from expounding on evidence-based medicine all day?

    ReplyDelete
  7. It certainly does like a pretty weak study. Many people don't realize the importance of not biasing your study by the medium through which it was communicated.

    ReplyDelete
  8. Wow, pretty weak study in actuality. Have to say I was one who retweeted without reading the background. I appreciate the analysis!

    ReplyDelete
  9. The research may be flawed or even misleading. But it does throw down the gauntlet. We can tell from our own experience that social media are playing a more important role in healthcare. We can tell from the fact that over 300 hospitals now follow our Twitter account, that in 3 months we have had over 12,000 page views on our blog (www.better-hospitals.com) that blogs are a growing source of healthcare informaton. We can also tell from the gradual increase in hospitals using social media (especially Twitter)as tracked by Ed Bennett at http://ebennett.org/hsnl that hospitals are experimenting with use of social media. So it would be surprising that physicians and nurses (especially younger ones) would want to learn how these media can be useful to them as a means of staying on top of issues or staying connected to patients and to other professionals, that medical schools and nursing schools should (if they are not yet doing so); that they should include social media in their curricula.

    So, yes, the study cited has flaws. But instead of attacking its veracity, maybe the real direction we should be taking is to push for more comprehensive research on this subject.

    -- Dale Wolf

    ReplyDelete
  10. Dale,
    If you read back through this blog you will see that I have use web 2.0 tools in my work as an educator. I try to share my learning.

    I am a physician and an educator and a scholar of education. I think it is pertinent that I comment on the veracity of research that I come across.

    If you look at Dr Colin Mitchell's blog about the same research you will see that we have had a discussion there about the way forward in learning about the use of web 2.0 tools in medical education.

    But we must retain our critical faculties when we come across research on this topic. I found the way in which this research was disseminated uncritically across the twitterverse disturbing. I believe that social media can be used to promote scholarship and appraising literature is essential to that task.

    Thanks for your comment,
    Anne Marie

    ReplyDelete
  11. I concur with your assertion that research be valid. And this is why the profession leans heavily on journals. And I think your postings regarding the need for valid research point out that what we all read on social media may not be true. To beat a dead horse, all I was positing is that instead of criticizing, we should also be posing solutions. There seems to me to be enough experiential evidence to encourage someone to do a valid project. Survey Monkey projects with unknown responders are always suspect, but may point to a need for real research. The service you have provided is to warn managers of nursing and medical schools not to rush to decisions regarding adding Web 2.0 and/or Social Media to curricula is a good counterpoint.
    Dr. Kevin Pho also touched on the subject of medical professionals using social media. Online publishing carries risk ... or to be more precise, publishing any medical information in any medium carries risk. We must be content-sensitive so as not to mislead or misinform, especially when using data rather than expressing opinions.
    Thank you for your reply to my comment.
    Dale Wolf
    Compass Clinical Consultants
    www.compass-clinical.com
    www.better-hospitals.com

    ReplyDelete
  12. Hi there, this is a great post!

    As we all know, Web 2.0 is a fantastic way of disseminating information and sharing content. But it is a broad term, and many would argue is more of a buzzword than an actual definition. I also agree with Mick in that this particular piece of research needed to differentiate between Web 2.0 and Social Networking - it did seem a tad confusing!

    We come across many medical educators using Web 2.0 applications to great affect, and not just for sharing information. For example, the use of Second Life to create virtual scenarios in which medical and health care students can come together for problem-based learning situations - the University of Sydney is currently using SL to provide 'virtual placements' for pharmacy students - not to replace the actual experience, but to prepare the students beforehand.

    The eViP programme places a big emphasis on evaluation, albeit on the value and effectiveness of Virtual Patients. It is important to determine that any new Internet-based tools are more than just a passing fad, and they are actually of value to the medical/healthcare community - obviously this goes without saying!

    Best wishes,

    eViP

    www.virtualpatients.eu

    ReplyDelete
  13. The fact that this unscientific "study" stimulated so much discussion provides a textbook case regarding the effectiveness of social media in distributing information. Information can be true, false, or questionable, and Web 2.0 won't be any better at policing accuracy and truthfulness than the first incarnation of the Internet was/is. Getting something into the public forum for discussion is valuable in and of itself, so veracity and usefulness can be determined by more than a select well-informed few. Some of us retweet things we find interesting or worth considering, and we post the URL so people can read and decide for themselves how valuable the information is. Web 2.0 is about community discussion, a completely unscientific process in the traditional view of science, yet worthwhile for education and progress.

    ReplyDelete
  14. Hi,
    My name is Annie Smith and I am the assistant editor of Medicalschool.org. I am contacting you today in hopes of developing a relationship with your website; we have seen your site and think your content is great. Medicalschool.org is a purely informational site dedicated to the general Public.

    I hope you show some interest in building relationship, please contact me at annie.medicalschool.org@gmail.com.

    ReplyDelete
  15. Thanks for sharing this analysis with us. The numbers really don't make it statistically significant. I'm using some web 2. tools which are very useful & effective for my educations by using this tools I get the answers of my questions and also help others to solve their problems also.

    Nursing programs

    ReplyDelete
  16. Web 2.0 technologies for undergraduate and postgraduate medical education: an online survey
    http://www.scribd.com/doc/45817302/Web-2-0-technologies-for-undergraduate-and-postgraduate-medical-education-an-online-survey

    ReplyDelete
  17. Instead of Survey Monkey, you could use Survey Prof It is designed for students, and in contrast to Survey Monkey, it offers all of its services for free.

    ReplyDelete

I am reintroducing word verification to cut back on spam posts. I'm sorry if you find it frustrating,