Rationale and Objectives
Emerging alternative metrics leverage social media and other online platforms to provide immediate measures of biomedical articles’ reach among diverse public audiences. We aimed to compare traditional citation and alternative impact metrics for articles in popular general radiology journals.
Materials and Methods
All 892 original investigations published in 2013 issues of Academic Radiology , American Journal of Roentgenology , Journal of the American College of Radiology , and Radiology were included. Each article’s content was classified as imaging vs nonimaging. Traditional journal citations to articles were obtained from Web of Science. Each article’s Altmetric Attention Score (Altmetric), representing weighted mentions across a variety of online platforms, was obtained from Altmetric.com . Statistical assessment included the McNemar test, the Mann-Whitney test, and the Pearson correlation.
Results
Mean and median traditional citation counts were 10.7 ± 15.4 and 5 vs 3.3 ± 13.3 and 0 for Altmetric. Among all articles, 96.4% had ≥1 traditional citation vs 41.8% for Altmetric ( P < 0.001). Online platforms for which at least 5% of the articles were represented included Mendeley (42.8%), Twitter (34.2%), Facebook (10.7%), and news outlets (8.4%). Citations and Altmetric were weakly correlated ( r = 0.20), with only a 25.0% overlap in terms of articles within their top 10th percentiles. Traditional citations were higher for articles with imaging vs nonimaging content (11.5 ± 16.2 vs 6.9 ± 9.8, P < 0.001), but Altmetric scores were higher in articles with nonimaging content (5.1 ± 11.1 vs 2.8 ± 13.7, P = 0.006).
Conclusions
Although overall online attention to radiology journal content was low, alternative metrics exhibited unique trends, particularly for nonclinical articles, and may provide a complementary measure of radiology research impact compared to traditional citation counts.
Introduction
Biomedical investigators seek to make an impact within their area of research . Measuring such impact is important for identifying the most meaningful research within a field, evaluating individual investigators’ academic performance, and guiding decisions regarding the awarding of promotions, tenure, and grant funding . However, an optimal method for assessing research impact remains elusive. Historically, the impact of individual research articles has been assessed by simple counts of citations to articles within the subsequent peer-reviewed literature . This approach, however, has been heavily criticized in recent years . First, conventional citations are very slow to accumulate. Months, if not years, are required for later studies referencing the article to be written, to proceed through the peer review and revision process, and then to undergo production and publication. This delay precludes any meaningful evaluation of impact based on citations within short time frames. In addition, conventional citations measure impact among essentially a single audience—that of other scientific investigators—and fail to reflect broader dissemination .
The advent of the Internet, and of social media in particular, has greatly promoted the diffusion of biomedical research among more diverse audiences . Allied health-care practitioners, patients, policy makers, the popular media, and other stakeholder groups may use a range of social media platforms to share and engage in dialog regarding research of their individual and unique greatest interests . The concept of “altmetrics” (referring to nontraditional alternative article-level metrics) was coined in 2010 in reference to new metrics that seek to capture the online “buzz” regarding individual articles as a measure of impact .
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Materials and Methods
Article Selection
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Article Characteristics
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Statistics
Get Radiology Tree app to read full this article<
Results
Get Radiology Tree app to read full this article<
Table 1
Summary of Citation Count and Altmetric Attention Score Among 892 Original Investigation Articles Published in Four General Radiology Journals in 2013
Measure Citation Count Altmetric Attention Score Range 0–193 0–264 Mean ± standard deviation 10.7 ± 15.4 3.3 ± 13.3 Median 5 0 Percentage of articles with score ≥1, ( n ) 96.4 (860) 41.8 (373) Percentage of articles with score ≥2, ( n ) 93.2 (831) 21.0 (187)
Table 2
Summary of Frequency of Included Original Investigations on Online Platforms Reported by Altmetric
Online Platform Percent of Articles with ≥1 Mention, ( n ) Maximum No. of Mentions Mendeley \* 42.8(382) 94 Twitter 34.2(305) 101 Facebook 10.7(95) 16 News 8.4(75) 24 CiteULike \* 4.0(36) 3 Blog 2.7(24) 3 F1000/Publons/Pubpear 1.3(12) 1 Google+ 0.8(7) 1 Video/YouTube 0.7(6) 1 Wikipedia 0.4(4) 1 Policy document 0.3(3) 1 Reddit/Pinterest 0.1(1) 1 Research highlight platform 0.1(1) 1 Peer review 0.1(1) 1 Sina Weibo 0.0(0) 0 Q&A 0.0(0) 0 LinkedIn 0.0(0) 0
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 3
Distribution of Citation Count and Altmetric Attention Score by Journal of Publication and Article Content, Along with Each Journal’s Impact Factor \*
n Citation Count \\ Altmetric Attention Score \\ Journal_Academic Radiology_ (IF = 1.966) 173 5.1 ± 5.2 0.7 ± 2.1American Journal of Roentgenology (IF = 2.660) 337 5.7 ± 7.7 1.3 ± 5.1Journal of American College of Radiology (IF = 2.929) 42 5.2 ± 4.6 6.4 ± 10.4Radiology (IF = 6.798) 340 19.2 ± 20.8 6.3 ± 20.1 Content Imaging 742 11.5 ± 16.2 2.8 ± 13.7 Nonimaging 150 6.9 ± 9.8 5.1 ± 11.1
IF, impact factor.
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Table 4
Correlation Coefficient ( r ) Between Citation Count and Altmetric-Based Measures, Along with Comparison of Correlations Between Articles with Imaging vs Nonimaging Content
Group 1 Group 2 All \* (892) Imaging \* (742) Nonimaging (150)P (Imaging vs Nonimaging) Citation count Altmetric Attention Score 0.20 0.19 0.47 <0.001 Citation count Number of Twitter mentions 0.17 0.13 0.62 <0.001 Citation count Number of Mendeley mentions 0.57 0.58 0.42 0.017
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Discussion
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
Get Radiology Tree app to read full this article<
References
1. Brigham T.J.: An introduction to altmetrics. Med Ref Serv Q 2014; 33: pp. 438-447.
2. Rezek I., McDonald R.J., Kallmes D.F.: Is the h-index predictive of greater NIH funding success among academic radiologists?. Acad Radiol 2011; 18: pp. 1337-1340.
3. Rad A.E., Brinjikji W., Cloft H.J., et. al.: The H-index in academic radiology. Acad Radiol 2010; 17: pp. 817-821.
4. Choudhri A.F., Siddiqui A., Khan N.R., et. al.: Understanding bibliometric parameters and analysis. Radiographics 2015; 35: pp. 736-746.
5. Barbic D., Tubman M., Lam H., et. al.: An analysis of altmetrics in emergency medicine. Acad Emerg Med 2016; 23: pp. 251-268.
6. Liu C.L., Xu Y.Q., Wu H., et. al.: Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: dimensions and structure. J Med Internet Res 2013; 15: pp. e259.
7. Thelwall M., Haustein S., Lariviere V., et. al.: Do altmetrics work? Twitter and ten other social web services. PLoS ONE 2013; 8: e64841
8. Trueger N.S., Thoma B., Hsu C.H., et. al.: The Altmetric Score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015; 66: pp. 549-553.
9. Haustein S., Costas R., Lariviere V.: Characterizing social media metrics of scholarly papers: the effect of document properties and collaboration patterns. PLoS ONE 2015; 10: pp. e0120495.
10. Duszak R.: The impact factory. Acad Radiol 2016; 23: pp. 659-660.
11. Eysenbach G.: Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res 2011; 13: pp. e123.
12. Melero R.: Altmetrics—a complement to conventional metrics. Biochem Med (Zagreb) 2015; 25: pp. 152-160.
13. Cress P.E.: Using altmetrics and social media to supplement impact factor: maximizing your article’s academic and societal impact. Aesthet Surg J. 2014; 34: pp. 1123-1126.
14. Citrome L.: Moving forward with article level metrics: introducing altmetrics. Int J Clin Pract 2015; 69: pp. 811.
15. Javitt M.C.: Section editor’s notebook: impact factor—an analog metric in a digital world. AJR Am J Roentgenol 2015; 205: pp. 1153-1154.
16. Altmetric : Who’s talking about your research?. Available at: https://www.Altmetric.com Accessed July 22, 2016
17. Robinson-García N., Torres-Salinas D., Zahedi Z., et. al.: New data, new possibilities: exploring the insides of Altmetric.com. El Profesional de La Informacion 2014; 23: pp. 359-366.
18. Altmetric : How is the Altmetric Attention Score calculated?. Available at: https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-attention-score-calculated- Accessed July 22, 2016
19. Scarlat M.M., Mavrogenis A.F., Pecina M., et. al.: Impact and alternative metrics for medical publishing: our experience with International Orthopaedics. Int Orthop 2015; 39: pp. 1459-1464.
20. Rosenkrantz A.B., Ayoola A.: The impact factor of radiological journals: associations with journal content and other characteristics over a recent 12-year period. Acad Radiol 2016; 23: pp. 661-668.
21. Hawkins C.M., Hillman B.J., Carlos R.C., et. al.: The impact of social media on readership of a peer-reviewed medical journal. J Am Coll Radiol 2014; 11: pp. 1038-1043.
22. Hawkins C.M., Duszak R., Rawson J.V.: Social media in radiology: early trends in Twitter microblogging at radiology’s largest international meeting. J Am Coll Radiol 2014; 11: pp. 387-390.
23. Altmetric : Bookmarklet for researchers. Available at: https://www.altmetric.com/products/free-tools/bookmarklet/ Accessed July 23, 2016
24. Thomson Reuters : Web of science. Available at: http://login.webofknowledge.com/ Accessed July 22, 2016
25. Diedenhofen B., Musch J.: cocor: a comprehensive solution for the statistical comparison of correlations. PLoS ONE 2015; 10: pp. e0121945.
26. Skaane P., Bandos A.I., Gullien R., et. al.: Comparison of digital mammography alone and digital mammography plus tomosynthesis in a population-based screening program. Radiology 2013; 267: pp. 47-56.
27. Lipton M.L., Kim N., Zimmerman M.E., et. al.: Soccer heading is associated with white matter microstructural and cognitive abnormalities. Radiology 2013; 268: pp. 850-857.
28. Mendeley : Your research, anywhere. Available at: https://www.mendeley.com/ Accessed July 23, 2016
29. Ferrada P., Suliburk J.W., Bryczkowski S.B., et. al.: The surgeon and social media: Twitter as a tool for practicing surgeons. Bull Am Coll Surg 2016; 101: pp. 19-24.
30. Wang F., Wang H., Xu K., et. al.: Regional level influenza study with geo-tagged twitter data. J Med Syst 2016; 40: pp. 189.
31. Waring M.E., Schneider K.L., Appelhans B.M., et. al.: Interest in a Twitter-delivered weight loss program among women of childbearing age. Transl Behav Med 2016; 6: pp. 277-284.
32. Prabhu V., Rosenkrantz A.B.: Enriched audience engagement through Twitter: should more academic radiology departments seize the opportunity?. J Am Coll Radiol 2015; 12: pp. 756-759.
33. Prabhu V., Rosenkrantz A.B.: Imbalance of opinions expressed on Twitter relating to CT radiation risk: an opportunity for increased radiologist representation. AJR Am J Roentgenol 2015; 204: pp. W48-W51.
34. American College of Radiology : Imaging 3.0TM. Available at: http://www.acr.org/advocacy/economics-health-policy/imaging-3 Accessed July 23, 2016
35. Hoang J.K., McCall J., Dixon A.F., et. al.: Using social media to share your radiology research: how effective is a blog post?. J Am Coll Radiol 2015; 12: pp. 760-765.
36. Dinsmore A., Allen L., Dolby K.: Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol 2014; 12: e1002003