Today's scientific community is remarkably diverse and only getting more so as new discoveries are made to advance our understanding of the world around us. With this diversity comes the natural process of subspecialization to maintain focus and develop expertise within a particular field when compared with other fields of scientific inquiry. This subspecialization leads to the development of “niche literature” where new journals are introduced to appeal to a particular subspecialty focus. Over the past century, the scientific community has maintained a 3.5% annual increase in the overall number of scientific journal titles published yearly,15 resulting in 25,400 peer-reviewed journal titles published worldwide in 2009.3 This growth rate provides evidence of the progressive development of niche literature as fields of science have morphed from those of macro study to those of micro study. As this coning-down process proceeds, the findings of a particular work or area become progressively less relevant to the greater body of science from which the subspecialty developed. How then do we assess the impact of a given scientific finding or work on the field at large (or another field to which the advance may also have relevance)?
Over the past 55 years, a number of bibliometric tools have been developed in an attempt to quantify bibliometric performance or value. These various tools include an article's overall citation count,7,20 a journal's impact factor,8 or an investigator's h-index11 as well as a myriad of normalized and standardized versions of each to increase applicability to different fields of research.2,4–6,12,13,20 Notably, each of these has particular application to a specific entity; each uniquely measures the performance of a journal, article, or author. We introduce here a novel bibliometric tool termed the reverberation index (r-index). We believe that there is value in understanding how a concept or finding in the literature is evaluated and applied to a certain scientific discipline. The extent to which such an entity surfaces and resurfaces (reverberates) in the literature of a given discipline is an indicator of its significance in the development of new knowledge and progress in patient care.
Methods and Examples
The r-index is calculated for a given search term using the online search tool Web of Science (Thompson Reuters). The search term may be an article title, author, or topical key word. This specifies the unique parameter whose reverberation is being investigated. A report of all indexed publications is generated for the given search term. In the case of a key word search, this publication report is frequently in excess of the maximum of 10,000 allowable publications from which to generate a citation report. In this case, the publication report can be culled down to the requisite 10,000 or fewer publications by filtering it for journal articles only and limiting it to English language publications from within a specific date range. Once this has been completed, a citation report is generated which represents the reverberation of the searched term within the scientific literature (or more specifically, the articles that have cited the search term's publications). From this citation report, the source titles are then retrieved; this provides a breakdown of which journals have published the citing articles in descending order by frequency. A rank factor (RF) of 1 to 10 is then appropriated to each journal based on where it falls within the rank order. This is completed as follows: the journal with the most citations receives an RF of 10, the next most frequently citing journal is given an RF of 9, the next an RF of 8, and so forth down to the 10th (or more) most frequently citing journal, all receiving an RF of 1. If fewer than 10 journals cite the search term, the corresponding RF assignment continues as previously described but terminates with the least most commonly citing journal in descending order. A given journal's RF is then multiplied by the total citation count for that journal. The result is then divided by 100 for ease of further analysis (Eq. 1). This functionally serves to create a weighted value representing that journal's citation pattern for that search term relative to the citation patterns of other journals frequently citing the same search term and represents a quantitative measure for the subjective concept of reverberation through a field of study, which is a fluidly defined term to be identified by the user of the r-index.
This process may be repeated for additional journal titles for purposes of comparison of r-indexes between or across fields of study. To compare a given search term's reverberation between 2 fields of scientific research, the r-index for the first field's primary journal titles is then compared against another field's journals' r-indexes. This may include selecting either a single or multiple journals for a given scientific field. If N journal titles are selected for the first scientific field, then N journal titles must be selected for all comparison scientific fields. Individual journal r-indexes are totaled for each field:
Examples of calculations comparing r-indexes between neurological surgery and neurological critical care with representative journal titles Journal of Neurosurgery and Neurosurgery and Brain Injury and Journal of Neurotrauma, respectively, follow.
Example of r-Index Calculation for a Journal Article
Journal article: Teasdale G, Jennett B. Assessment of coma and impaired consciousness—practical scale. Lancet 2(7872):81–84, 1974
Indexed publications: 1
Total number of citations (excluding self-citations): 6003
Field of Neurological Surgery
Number of citations in Journal of Neurosurgery, first most commonly citing journal: 332; RF = 10
Number of citations in Neurosurgery, fifth most commonly citing journal: 144; RF = 6
- Calculation of r-index for neurological surgery:
Field of Neurological Critical Care
Number of citations in Brain Injury, second most commonly citing journal: 292; RF = 9
Number of citations in Journal of Neurotrauma, fourth most commonly citing journal: 146; RF = 7
- Calculation of r-index for neurological critical care:
Interpretation
These results suggest that Teasdale and Jennett's seminal work proposing the Glasgow Coma Scale has reverberated similarly but somewhat more through the neurosurgical literature than it has through the neurological critical care literature.
Example of r-Index Calculation for a Topical Key Word
Key word or concept: traumatic brain injury
Indexed publications: 27,102
Indexed publications filtered for English language journal articles published since 2010: 9756
Total number of citations (excluding self-citations): 28,326
Field of Neurological Surgery
Number of citations in Journal of Neurosurgery, 17th most commonly citing journal: 126; RF = 1
Number of citations in Neurosurgery, 69th most commonly citing journal: 56; RF = 1
- Calculation of r-index for neurological surgery:
Field of Neurological Critical Care
Number of citations in Brain Injury, sixth most commonly citing journal: 193; RF = 5
Number of citations in Journal of Neurotrauma, eighth most commonly citing journal: 169; RF = 3
- Calculation of r-index for neurological critical care:
Interpretation
These results suggest that the topical key word “traumatic brain injury” has reverberated significantly more through the neurological critical care literature than it has the neurosurgical literature since 2010.
Example of r-Index Calculation for an Author
Author: Graham Teasdale
Indexed Publications: 229
Total number of citations (excluding self-citations): 16,591
Field of Neurological Surgery
Number of citations in Journal of Neurosurgery, first most commonly citing journal: 685; RF = 10
Number of citations in Neurosurgery, fifth most commonly citing journal: 441; RF = 6
- Calculation of r-index for neurological surgery:
Field of Neurological Critical Care
Brain Injury, third most commonly citing journal (482 citations); RF = 8
Journal of Neurotrauma, fourth most commonly citing journal (463 citations); RF = 7
- Calculation of r-index for neurological critical care:
Interpretation
These results suggest that the work of Graham Teasdale, when taken together, has reverberated somewhat more through the neurosurgical literature than it has through the neurocritical care literature.
Discussion
History of Bibliometrics
The conceptual introduction of what would ultimately develop into the field now known as bibliometrics occurred in the mid-1920s with 2 nearly simultaneous publications. The first of these involved the 1926 article by A. J. Lotka reporting a reproducible pattern in the frequency distribution of scientific productivity within academia.14 The second involved the 1927 article by Gross and Gross, who suggested that universities should select journals to purchase based on their citation patterns.10 Two seminal contributions followed—the introduction of the impact factor and Science Citation Index, both by Eugene Garfield, in the 1950s and 1960s, respectively.8,9 Shortly thereafter (in 1969), the term “bibliometrics” was coined by Alan Pritchard to define the field,17 which had heretofore been growing without a title. The term was meant to describe the “application of mathematical and statistical methods to books and other media….”17
A number of adaptive techniques were introduced shortly thereafter, in the 1970s, to accommodate the diversity of citation patterns within different fields of scientific research. The concept of “evaluative bibliometrics” was introduced by Francis Narin in 1976 with the intention of applying the practice of bibliometrics to the evaluation of scientific quality and performance.16 Further adaptive techniques continued to be introduced over the remaining decades of the 20th century. The most recent bibliometric novelty has been the h-index, which was introduced by J. E. Hirsch in 2005 and serves to quantify an individual's scientific research output based on the citation patterns of his or her work. Over the years since the introduction of the h-index a number of adaptive techniques have been proposed to enhance this metric's applicability to a diverse range of scientific fields.2,4–6,12,13
Limitations of Current Bibliometric Indexes
It is clear from even this briefest of overviews that with scientific subspecialization comes bibliometric subspecialization. This limits the application of any one metric across scientific fields of study, making quantification across scientific fields challenging at best and inaccurate at worst. Biomedical research has long been recognized as one of the most challenging realms of scientific inquiry in which to apply these performance evaluators, given the dramatic differences in citation patterns within the biomedical sciences.1,20 Therefore, in an era of multidisciplinary research, a single quantitative tool is necessary to assess scientific relevance across fields.
Reverberation Index
Given these noted shortcomings of currently available bibliometric indexes, the reverberation index (r-index) was created with the express purpose of assessing the impact of a particular component of the scientific process (citation, key word, or author) on either an individual field of study or providing a means of comparison of the impact on one field versus another. This quantification merely represents the degree to which that citation, concept, or author has “reverberated” within a single field or within one field compared with another. While often unsavory to consider, this quantification may then allow assessment of the impact of a single work or finding on the particular subspecialty field from which it was discovered compared with a parent field of study. Alternatively, comparisons may be made between 2 fields to which a single finding has dual relevance. Take for example the above noted differences in the r-indexes between neurosurgery and neurocritical care. The differences and similarities highlighted by quantifying the reverberation of the given search terms between the 2 fields identifies nuances that may otherwise go unappreciated. Additionally, in an era of collaborative research and tight funding budgets, funds appropriations between different academic departments or funding organizations may be considered based on r-index values for a particular investigator or discovery. Application of the r-index would also provide a means of effective quantitative and objective comparison between 2 researchers being considered for academic promotion within a single field of research. See Table 1 for additional potential applications of the r-index.
Sample applications of the r-index
| Application | Example |
|---|---|
| Calculation of r-index for an academic faculty member whose clinical & research activities cross departmental boundaries | A psychiatrist with a particular interest in depression & anxiety in patients undergoing epilepsy surgery establishes a weekly half-day clinic specifically to work with this patient population. Multiple collaborative research projects result from this multidisciplinary endeavor, which are largely published in the neurology & neurosurgery literature. The psychiatrist pursues grant funding to further investigate this patient population. This grant requires 30% protected research time. The psychiatry department will only support 10% protected time. Calculation of this investigator's r-index demonstrates more significant reverberation of the investigator's work in the fields of neurosurgery & neurology than psychiatry. With this information, the investigator is able to negotiate 10% support from both the neurology & neurosurgery departments to fund the remainder of the necessary protected time. |
| Calculation of r-index for a research topic to determine relative reverberation btwn similar fields of medicine for a given topic as a means by which to identify appropriate agencies for grant submission | An investigator is interested in studying the long-term impact of concussions sustained during childhood. A number of potential funding agencies are considered, including NINDS, NIMH, & the National Institute of Child Health & Human Development. Calculation of the r-index for the key word “concussion” identified a significantly higher reverberation of the key word “concussion” within the neurology, neurosurgery, & neuroscience literature than in the psychiatric or pediatric literature. With this information, the investigator prepares grant submissions to public & private funding agencies focusing on the neurosciences. |
| Calculation of r-index for a journal article that publishes findings upon which a new research project is based as a means by which to identify appropriate agencies for grant submission | A new gene is found to be associated with glioblastoma. An investigator is proposing a new study to investigate this gene's association with the clinical course of glioblastoma. Calculation of the r-index of the journal article detailing the initial relationship btwn this gene & glioblastoma identifies a significantly higher r-index within the oncology literature than in the neuroscience or neurosurgery literature. With this information, the researcher prepares grant submissions for public & foundation funding agencies particularly oriented toward neuro-oncology rather than neurosurgery or neuroscience. |
| Calculation of r-index for academic faculty members being considered for academic promotion | Calculation of the r-indexes for multiple academic faculty members being considered for academic promotion allows quantitative comparison of academic productivity & impact, which can then be combined with clinical productivity & academic stewardship, among other factors, to guide promotion of one faculty member versus another. This can be implemented at the department-, school-, or university-wide level given the r-index's flexibility for comparison across academic fields of study. |
| Calculation of r-indexes comparing 2 seminal articles to assess comparative reverberation within the neurosurgical literature | Calculation of the r-index for the seminal article highlighting the descriptive classification scale for AVMs (Spetzler & Martin, 198618) results in an r-index of 28.3. Comparison of this value to the r-index of 4.1 for the seminal randomized, controlled trial for the treatment of glioblastoma (Stupp et al., 200519) highlights the differential reverberation of these 2 contributions to the neurosurgical literature. This is not to say that one has more clinical impact than another but rather that descriptive classification scales shape our understanding of a disease process & its treatment more perhaps than do randomized controlled trials. |
NICHD = National Institute of Child Health and Human Development; NIMH = National Institute of Mental Health; NINDS = National Institute of Neurological Disorders and Stroke.
Conclusions
Effective comparison across scientific fields requires a novel bibliometric index which transcends the specific citation patterns of any single field. The reverberation index, or r-index, does just that. The flexibility of this novel metric allows effective comparison of the impact of a component of the scientific process on different aspects of the scientific community.
Acknowledgments
We would like to thank Cathy Sarli at Washington University's Becker Medical Library for her assistance in gathering preliminary resources used in the formulation of the r-index.
Disclosures
The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.
Author Contributions
Conception and design: both authors. Analysis and interpretation of data: both authors. Drafting the article: Bandt. Critically revising the article: both authors. Reviewed submitted version of manuscript: both authors. Approved the final version of the manuscript on behalf of both authors: Bandt. Administrative/technical/material support: Bandt. Study supervision: Dacey.
References
- 1↑
Aoun SG, Bendok BR, Rahme RJ, Dacey RG Jr, Batjer HH: Standardizing the evaluation of scientific and academic performance in neurosurgery—critical review of the “h” index and its variants. World Neurosurg 80:e85–e90, 2013
- 2↑
Barendse W: The strike rate index: a new index for journal quality based on journal size and the h-index of citations. Biomed Digit Libr 4:3, 2007
- 3↑
Bjork B, Roos A, Lauri M: Scientific journal publishing: yearly volume and open access availability. Inform Res 14:391, 2009
- 4
Bornmann L: Are there better indices for evaluation purposes than the h index?. J Am Soc Inf Sci 59:830–837, 2008
- 5
Egghe L: Theory and practise of the g-index. Scientometrics 69:131–152, 2006
- 6
Egghe L, Rousseau R: An h-index weighted by citation impact. Inf Process Manage 44:770–780, 2008
- 7↑
Garfield E: Citation frequency as a measure of research activity and performance. Essays of an Information Scientist Philadelphia, ISI Press, 1977. 1:406–408
- 8↑
Garfield E: Citation indexes for science; a new dimension in documentation through association of ideas. Science 122:108–111, 1955
- 11↑
Hirsch JE: An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 102:16569–16572, 2005
- 12↑
Jin B, Liang L, Rousseau R, Egghe L: The R- and AR-indices: complementing the h-index. Chin Sci Bull 52:855–863, 2007
- 13↑
Kosmulski M: A new Hirsch-type index saves time and works equally well as the original h-index. ISSI Newsletter 2:4–6, 2006
- 14↑
Lotka AJ: The frequency distribution of scientific productivity. J Wash Acad Sci 16:317–323, 1926
- 16↑
Narin F: Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity Washington, DC, Computer Horizons, Inc, 1976
- 18↑
Spetzler RF, Martin NA: A proposed grading system for arteriovenous malformations. J Neurosurg 5:476–483, 1986
- 19↑
Stupp R, Mason WP, van den Bent MJ, Weller M, Fisher B, Taphoorn MJB, et al.: Radiotherapy plus concomitant and adjuvant temozolomide for glioblastoma. N Engl J Med 352:987–996, 2005
- 20↑
van Eck NJ, Waltman L, van Raan AF, Klautz RJ, Peul WC: Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PLoS One 8:e62395, 2013
