Methodology and reporting of meta-analyses in the neurosurgical literature

A review

Full access

Object

Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology.

Methods

The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper?

Results

Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies.

Conclusions

Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.

Abbreviations used in this paper:AMSTAR = Assessment of Multiple Systematic Reviews; CONSORT = Consolidated Standards of Reporting Trials; JNS = Journal of Neurosurgery; MOOSE = Meta-Analysis of Observational Studies in Epidemiology; OQAQ = Overview of Quality Assesment Questionnaire; PRISMA = Preferred Reporting Items for Systematic reviews and Meta-analyses; QUORUM = Quality of Reporting of Meta-analyses.

Object

Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology.

Methods

The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper?

Results

Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies.

Conclusions

Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.

With the rapid proliferation of traditional printed and Internet-based literature, it is becoming increasingly difficult for neurosurgeons to stay up to date without having to spend hours searching and reading articles. Thus, review articles play an important role in continuing medical education.23,86 Of the 3 types of review articles, a narrative or journalistic review is a simple, broad review of a given topic, typically done by an expert in the field, often with no specified intervention or outcome(s) evaluated. A systematic review is a much more rigorous and labor-intensive undertaking that seeks to answer a clearly defined question through a multistep process that is carried out in accordance with explicit, transparent, and reproducible methods. Systematic reviews summarize findings from papers located using a thorough search strategy and meeting certain eligibility criteria. The purpose of conducting a systematic review in such a stepwise, thorough fashion is to limit the introduction of bias—any process that systematically and nonrandomly causes a deviation of results and inferences from the truth—thus making the conclusions of the review more reliable and accurate.45

A meta-analysis is a direct extension of and often included in a systematic review. It uses the same steps but integrates the data from the studies using statistical techniques to obtain a pooled-effect estimate with greater power to detect true differences (and, by definition, a reduced chance of false-negative results, or Type II error) and thus is less influenced by the findings of any one study. Meta-analyses have become increasingly popular in the medical literature but no less controversial.33 For example, there are cases in which the findings of meta-analyses that were based on small trials were subsequently disproven by a single large randomized trial.35,62 As with systematic reviews, strict adherence to methodology is critical with meta-analyses to make them valid and precise and to avoid the oft-stated “garbage-in, garbage-out” criticism.35

A major advance in evidence-based medicine has been the development of structured checklists for methodology and reporting of systematic reviews and meta-analyses. The first checklist specific to meta-analyses was the Quality of Reporting of Meta-analyses (QUOROM),68 which was published in 1999 and designed to address the suboptimal reporting of meta-analyses. QUOROM is similar to the Consolidated Standards of Reporting Trials (CONSORT, www.consort-statement.org) for reporting of randomized controlled trials, which was published in 1996.14,70 In 2009, QUOROM was revised to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA, www.prisma-statement.org) to encompass both systematic reviews and meta-analyses and address several conceptual and practical advances in the science of secondary research.69 Because of the increasing number of published meta-analyses using observational studies, the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) Group created a 6-section reporting cookbook: background, search strategy, methods, results, discussion, and conclusion.100 The Cochrane Collaboration provides a detailed, standardized handbook for systematic reviews that must be used by all submitting authors (http://www.cochrane-handbook.org/).

Adequate reporting does not necessarily ensure that the contents of the document are valid and precise. In other words, a systematic review or meta-analysis may be well written, but if the methods employed to derive a pooled estimate of the effect size are flawed, then the entire process is of questionable value. Indeed, the creators were clear that PRISMA could be useful for critical appraisal of published systematic reviews and meta-analyses but should not be used as an instrument to gauge their quality.69 The overall quality of a systematic review or meta-analysis, in our opinion, is a function of proper reporting but, much more importantly, of using the correct methodology to limit bias and ensure the internal validity of the findings. Similar to checklists for reporting, checklists for methodology have been created. Several well-known early questionnaires were the Overview of Quality Assessment Questionnaire (OQAQ),76,77 the Potsdam guidelines,22 and the Sacks instrument.88 In 2007, the Assessment of Multiple Systematic Reviews (AMSTAR) was developed by combining elements of the OQAQ and the Sacks instrument as well as other items based on methodological advances that had been made since OQAQ and Sacks had been introduced.92 It has been found to be valid, feasible, reliable, and to have good inter-rater agreement.91,93 Since their development, PRISMA and AMSTAR have become widely accepted by many journals as the tools to ensure proper reporting and methodology of systematic reviews and meta-analyses. In addition to these checklists, many textbooks have been published that thoroughly describe the various components and their function within a meta-analysis or systematic review.18,34,104,112

It is our impression that the term “meta-analysis” is being used with increasing frequency in the neurosurgical literature; however, is it being used appropriately? Is meta-analysis being performed too frequently? The word seems to be used to denote any attempt at combining data from two or more publications. Until very recently, academic neurosurgery has not mandated the use of the various checklists to ensure quality of reporting and methodology. The journal Neurosurgery now requires that all meta-analyses be reported as recommended by PRISMA or MOOSE guidelines, but does not require the use of a methodology questionnaire.11 We set out to apply the PRISMA and AMSTAR checklists to all meta-analyses in the neurosurgical literature to assess the quality of reporting and methodology.

Methods

Search

A search was conducted to identify all meta-analyses that had ever been published in the JNS (Journal of Neurosurgery) Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Spine, Journal of Neurosurgery: Pediatrics, and Neurosurgical Focus) or in Neurosurgery through 2012. The words “meta-analysis,” “metaanalysis,” “meta-analyses,” and “metaanalyses” were searched either in the title, abstract, or key words. The search was conducted on PubMed/MEDLINE, and the results were cross-referenced with searches performed on the web sites of the respective journals.

Definition

There are 6 components that we deemed necessary for a meta-analysis. We defined a meta-analysis as a:

  1. multistep process that

  2. seeks to answer a clearly defined question, or questions, by comparing the impact of one or more interventions, exposures, or possible risk or predictive factors (there must be an exposure group[s] and control group[s]) on prespecified outcome(s) by

  3. combining the results of previously published randomized controlled trials or observational studies that were found by a

  4. reproducible and comprehensive search strategy that met prespecified entry criteria.

  5. An overall or pooled effect size (for example, odds ratio, rate ratio) is calculated by

  6. using appropriate statistical computational methods (fixed- vs random-effects models, measures of heterogeneity or inconsistency).

Although important and preferred, we did not define subgroup or sensitivity analysis or an assessment of publication bias as required elements of a meta-analysis.

Meta-analyses were categorized as analytical, exploratory, or both, as defined by Anello and Fleiss.5 An analytical type was one in which the main goal was to determine the common or pooled estimate of the effect size. When the focus of the meta-analysis was to explain why the effect sizes vary (that is, to search of sources of variability such as subgroup or sensitivity analysis, meta-regression analysis), then it was considered exploratory.

A “pure” meta-analysis was defined as one in which the entire focus of the paper is the meta-analysis. Conversely, a meta-analysis where the authors first detail their own institution's experience and then secondarily add their results to the literature and perform a meta-analysis was considered “mixed.”

An observational study, as defined by Peipert and Phipps,80 is an etiological or effectiveness study using cross-sectional study, a case series, a case-control design, a design with historical controls, or a cohort design.

Data Collection

AMSTAR and PRISMA were applied to all meta-analyses found in our search. AMSTAR is an 11-item document used to assess quality of methodology for systematic reviews and meta-analyses (Appendix 1). Although the creators of AMSTAR had 4 answer options for each item—“yes,” “no,” “can't answer,” and “not applicable”—we judged that a “can't answer” response was equivalent to a “no.” To get a “yes” for an item, the meta-analysis must contain all the major components within that item, as detailed in Appendix 1.

PRISMA is a checklist for reporting that has 27 entries divided into 7 sections: title, abstract, introduction, methods, results, discussion, and funding (Appendix 2). The PRISMA authors have provided a detailed Explanation and Elaboration document that details the meaning and rationale for each checklist item.64 As with AMSTAR, we recorded one of 3 answers for each item: “yes,” “no,” “not applicable.” Some of the items in the checklist contain multiple components, so if at least half of them were met, then we gave that item a “yes.” If less than half of the components were met or a key component was lacking, the item was given a “no.” The PRISMA document also contains a 4-phase flow diagram that depicts the process of identifying, screening, determining eligibility, and presenting the reader with the final number of papers that were used in the systematic review or meta-analysis (Fig. 1). We recorded whether such a flow diagram was present in the meta-analysis or if the authors provided enough data to construct one.

Fig. 1.
Fig. 1.

Flow of information through the different phases of the literature search for a meta-analysis or systematic review.

Each meta-analysis was scrutinized using the AMSTAR and PRISMA checklists, and a score was assigned to the paper. The score was the proportion of items present (that is, the number of items that were given a “yes” response) divided by the number of items in the respective instrument. Each paper was reviewed on 3 separate occasions by the primary author (P.K.). Other data collected for each meta-analysis included the type of papers that were used in the meta-analysis (randomized controlled trials, observational studies, or both), the type of meta-analysis (pure vs mixed, analytical vs exploratory vs both), the topic of the meta-analysis (spine, trauma, vascular, infection prevention, functional/epilepsy, pain, oncology, peripheral nerve, pediatrics, and other), whether it satisfied our definition of a meta-analysis, whether there was one or more individuals identified as contributing to the paper who conceivably had additional training in meta-analysis techniques, such as a biostatistician, epidemiologist, or one with a master's degree in public health, and whether the authors reported using one or more established reporting or methodology questionnaires as mentioned in the Introduction.

Questions

We sought to answer the following 4 questions: Is the quality of reporting and methodology of meta-analyses in the neurosurgery literature

  1. Improving with time? Time was divided into the following 3 time periods: 1990–1999, 2000–2009, and 2010–2012. This corresponds with before and after the release of the QUOROM and PRISMA statements.

  2. Better if the authors or other acknowledged contributors include one or more individuals who might have additional expertise in conducting a meta-analysis?

  3. Different when comparing pure versus mixed meta-analysis?

  4. Different in those meta-analyses that met our definition compared with those that did not?

Statistical Analysis

For each paper, an AMSTAR and PRISMA “score” was calculated. The responses for each item from both checklists were collapsed into “yes” versus “not yes” categories, with the “yes” responses being summed then divided by the total number of items in the respective instrument. The number of items in the AMSTAR instrument—the denominator for the calculation of the score—is 11. The denominator used in the PRISMA score differed depending on whether the authors conducted any additional analyses (items 16 and 23 from the checklist). If they did not conduct any additional analyses (“not applicable”), then the denominator was reduced from 27 to 25 as a way to distinguish (and not penalize) those papers that did not include additional analyses from those that did. Both the AMSTAR and PRISMA scores are bounded between 0 and 1 (0% and 100%).

The statistical analysis in the present paper was largely descriptive. Frequencies were calculated presenting the distribution of papers by journal, time period (1990–1999, 2000–2009, 2010–2012), pure versus mixed analysis, presence of meta-analysis expert, paper types constituting the meta-analyses, topic, and our assessment of whether the paper qualifies as a meta-analysis (yes vs no). Tests of statistical significance relied on the Wilcoxon rank-sum test or Spearman test, and significance was reached if p < 0.05. All p values are 2-sided. Stata/SE 11.2 was used for all analyses.

Results

We identified 76 meta-analyses in the neurosurgery literature from 1990 through the end of 2012 using our search criteria (Table 1). Two articles were excluded from analysis because the meta-analysis performed was of such poor quality that AMSTAR and PRISMA could not be applied.4,59 The recent meta-analysis by van Leeuwen et al.110 was not analyzed because it is an individual patient data meta-analysis, and many of the items in both checklists were “N/A.” We also excluded our recent meta-analysis to avoid a conflict of interest.58 Thus, 72 papers were included in our analysis. The Journal of Neurosurgery and the other JNS Publishing Group journals published 65% of the meta-analyses, and the most common topics in all journals were vascular (21 meta-analyses, 29%). More than half of the meta-analyses (39, 54%) were based solely on observational studies and were either analytical (29, 40%) or analytical and exploratory (21, 29%). In 11 (15%) studies, the authors stated that they used one of the 3 available reporting checklists (MOOSE, QUOROM, PRISMA) in preparing their paper, but no one reported using a methodology checklist.

TABLE 1:

Characteristics of 72 meta-analyses evaluated

DescriptionNo. of Papers
journal
 JNS Publishing Group journals47
  Journal of Neurosurgery35
  Journal of Neurosurgery: Spine8
  Journal of Neurosurgery: Pediatrics2
  Neurosurgical Focus2
 Neurosurgery25
topic
 vascular3,6,10,16,19,21,25,26,47,50,57,59,60,63,65,67,73,82,95,99,10821
 spine1,12,24,28,38,43,46,53,74,83,85,94,11613
 oncology29,52,101–103,105,106,109,1159
 trauma2,17,42,51,72,79,97,988
 infection prevention7–9,48,84,96,1147
 pain901
 functional/epilepsy39–41,78,87,1116
 pediatrics32,372
 peripheral nerve541
 other13,49,75,894
types of studies
 observational39
 randomized controlled trial13
 both12
 unknown/neither/case series/case reports8
type of meta-analysis
 pure/mixed58/14
 analytic/exploratory/both/IPD*/NA29/1/21/2/19

Analytic meta-analysis with individual patient data.

Could not assign analytic or exploratory designation to the meta-analysis that did not meet our definition.

Figure 2 depicts the number of meta-analyses that were given a “yes” response to each of the 11 questions in AMSTAR. Fifty percent or more of the papers received a positive response for 3 items (Q1, Q6, Q9). Most papers clearly stated their research question and inclusion criteria (Q1), but there were very low responses for duplication of study selection and data extraction (Q2), use of “gray literature” (Q4), list of included and excluded studies (Q5), and assessment and analysis of study quality (Q7, Q8), and only one study documented source(s) of financial support for the meta-analysis and each of the included studies (Q11). Half of the papers used appropriate methods to combine findings (Q9). Publication bias was adequately assessed in 21 papers (29%, Q10).

Fig. 2.
Fig. 2.

The number of papers that fulfilled each item in the AMSTAR checklist. The dotted vertical line indicates 50% of analyzed papers (n = 36).

The results for PRISMA questions are shown in Fig. 3. More than 50% of the papers (more than 36) had positive responses for 16 (59%) of the 27 PRISMA items. Conversely, there were low responses for the remaining 11 questions—29 or fewer papers successfully satisfied these items. High responses were recorded for the items in PRISMA that required using “meta-analysis” in the title of the paper, having a structured abstract, describing rationale and objectives of the meta-analysis in the introduction, and providing inclusion/exclusion criteria and details of the search strategy, but few adequately reported the process for selecting studies and collecting data (Q9, Q10). No papers indicated that a protocol existed and/or was registered and accessible. As for AMSTAR, it was infrequent for a paper to assess bias within individual studies, that is, assigning some measure of quality to the included studies (Q12, Q19) and across studies, that is, publication bias (Q15, Q22). Almost all papers that adequately described methods for additional analyses, such as sensitivity analysis, subgroup analysis, or meta-regression, presented their results (Q16, Q23). The 3 items—summary of evidence, study limitations, and conclusions—for the Discussion (Q24–26) had positive responses in 44 or more of the papers. Funding, which unlike in the case of AMSTAR was specific only for the meta-analysis, was reported in 26 papers. Seventeen papers (24%) contained a search flow diagram or presented enough data to construct one.

Fig. 3.
Fig. 3.

The number of papers that fulfilled each item in the PRISMA checklist. The dotted vertical line indicates 50% of analyzed papers (n = 36).

Study Questions

The results of our 4 study questions are shown in Table 2. The average AMSTAR and PRISMA scores for all 72 papers were 31% and 55%, respectively. Both the AMSTAR and the PRISMA scores appreciably improved with time, although only the PRISMA score showed a statistically significant increase (p = 0.1093 and p = 0.0004, respectively) (Fig. 4). Both scores were much improved when a plausible expert was involved (p = 0.0003 and p = 0.0117, respectively), when the paper was a pure meta-analysis (p = 0.0034 and p = 0.0018, respectively), and when it satisfied our definition of a meta-analysis (p < 0.0001 for both). The proportions of papers that did not meet our definition based on time intervals were 5 (56%) of 9 from 1990 through 1999, 9 (30%) of 30 from 2000 through 2009, and 7 (21%) of 33 from 2010 through 2012.

TABLE 2:

AMSTAR and PRISMA results

DescriptionAMSTARPRISMA
overall mean score (range)31% (0–82%)53% (4–93%)
time period (no. of papers, mean score)
 1990–19999, 19%9, 35%
 2000–200930, 31%30, 52%
 2010–201233, 34%33, 63%
p = 0.1093p = 0.0004
meta-analysis expert (no. of papers, mean score)
 yes31, 40%31, 63%
 no41, 24%41, 49%
p = 0.0003p = 0.0117
pure vs mixed (no. of papers, mean score)
 pure58, 35%58, 59%
 mixed14, 16%14, 38%
p = 0.0034p = 0.0018
met our definition (no. of papers, mean score)
 yes51, 38%51, 63%
 no21, 13%21, 35%
p <0.0001p <0.0001
Fig. 4.
Fig. 4.

Proportion of items present by decade. Upper: AMSTAR checklist. Lower: PRISMA checklist.

Discussion

Definition and History

The term “meta-analysis” was first proposed by Glass in 1976 to mean the “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.”44 The purpose of a meta-analysis is to increase statistical power for primary end points and for subgroups, to resolve uncertainty when reports disagree, to improve the accuracy of effect size estimates, and to answer questions not posed at the start of individual trials.88 It is in itself an observational study and makes the findings of an original study replace the individual patient as the unit of analysis.

Our Results

Our study shows that the use of meta-analyses is becoming more frequent in the neurosurgery literature. Since 2010, there have been 26 reported in the JNS Publishing Group journals or Neurosurgery, whereas there were 39 in the previous 20 years. This proliferation of secondary research, which includes systematic reviews, occurring across the medical literature, reflects the need for clinicians to synthesize the vast amounts of existing information and provide their patients with the most up-to-date evidence-based treatment recommendations; however, the value of meta-analyses depends on the secondary research being conducted with sound methodology and being reported completely and with transparency. Reports that are biased or have flawed methodology can lead to inappropriate estimates of treatment effects.

We analyzed all qualifying meta-analyses published in the specified journals during the study period according to our search criteria, using AMSTAR and PRISMA for methodology and reporting, respectively. Overall, the mean “scores” for both checklists were less than ideal: 31% for AMSTAR and 53% for PRISMA. In other words, the average meta-analysis in the neurosurgery literature had satisfactorily fulfilled approximately 3 of 11 items in AMSTAR and 14 of 27 items in PRISMA. Thus, the average meta-analysis had better reporting than methodology. All but 2 of the domains in AMSTAR were fulfilled in less than half of the papers (Fig. 1). Only Items 1 (research question) and 6 (characteristics of included studies) were adequately completed. We found major deficiencies in the method of study selection and data extraction, the literature search strategy, and assessment of scientific quality, heterogeneity, and publication bias of the included studies. Only one paper1 described the sources of funding for the meta-analysis and all included papers (Q11 AMSTAR), whereas 26 papers indicated funding for the meta-analysis only (Q27 PRISMA). All meta-analyses, particularly those that deal with implantable devices or medications, should fully disclose financial support.

Reporting and methodology of meta-analyses in the neurosurgical literature vary dramatically, as evidenced by the wide range of scores (Table 2). Twenty-one papers (29%) did not meet our basic definition of a meta-analysis. The difference between the mean AMSTAR and PRISMA scores for those meta-analyses that met our definition of a meta-analysis and those that did not was statistically significant (Table 2). Even when we exclude studies that did not meet our definition of a meta-analysis, thereby removing arguably the worst meta-analyses, the ranges of scores for AMSTAR and PRISMA were 9%–82% and 20%–93%, respectively. There were numerous examples of meta-analyses at both ends of the overall quality spectrum.12,13,16,24,37,43,49,89,96,101,105 Surprisingly, experience in performing a meta-analysis also did not ensure that it was of higher overall quality. For example, authors from a single institution produced no less than 8 meta-analyses with deficiencies that were often repeated from one paper to the next.39–41,52,87,101–103

The meta-analysis by Lucas et al.65 on the treatment of intracranial dural arteriovenous malformations is an example of what we contend is an inaccurate use of the term “meta-analysis.” The authors reviewed existing literature but simply summed the percentage of cases that were successes and failures for a given treatment modality and applied a chi-square test, which is not a proper method.36 The following lengthy commentary by one of the paper's reviewers, an epidemiologist, summarizes precisely the problem with many meta-analyses in the neurosurgery literature (this commentary, along with others, was published along with the study):

We are concerned about describing this report as a “meta-analysis.” It is true that meta-analysis can be simply defined as a systematic review that uses statistical methods to combine and summarize the results of several trials, and any review of such as this might be argued to be a “meta-analysis.” However, epidemiologists have something different in mind when they discuss meta-analyses, or at least meta-analyses that are considered valid in the growing field of evidence-based medicine. The best meta-analyses compile information from controlled and/or randomized trials, but even when uncontrolled case series or observational studies (such as those presented in this review) are combined for statistical reanalysis, the reader has to have some reassurance that the patient populations receiving the different treatments are similar and homogeneous. In this review, for example, we really have no idea of what kind of patients are being compared, other than that their malformations were located in a certain anatomic region. No effort was made to present or adjust for patient characteristics that may have confounded the analysis, and it is not clear that the data collection was systematic or scored for quality. We are not sure that the conclusions of this combined analysis need to be considered any more valid than those of any one of the series included in its review.

Such “pseudo-meta-analyses” by Lucas et al. and others should be more precisely considered an attempt at conducting a systematic or narrative review. In our opinion, neurosurgeons, for the most part, have only a general understanding of meta-analyses and thus believe that they can be done with relative ease. This belief may be fostered by the easy availability and access to meta-analysis software such as RevMan (Cochrane Collaboration). To the contrary, however, Berman and Parker15 correctly state that a meta-analysis “…cannot be thought of as a quick and easy way to pull a lot of studies together and come up with a publication, but like any other study, requires an appreciable investment of time in planning and implementation.”

Although there is clearly overlap in the items between PRISMA and AMSTAR, PRISMA is only a guide on what to write in a meta-analysis paper; it does not provide the specifics on how to conduct the meta-analysis. More succinctly, adequacy of reporting does not ensure proper methodology. This is why the creators of PRISMA were clear in stating that it is “…not a quality assessment instrument to gauge the quality of a systematic review.”69 This point is exemplified by the meta-analysis by de Oliveira et al.26 The authors sought to answer whether clipping or coiling ruptured aneurysms had any effect on shunt dependency. This meta-analysis was fairly well reported, satisfying 14 of 25 PRISMA items (Q16 and Q23 were not applicable). The authors concluded that coiling was associated with a higher risk of shunt dependency than clipping, but they observed significant heterogeneity (as indicated from the test of heterogeneity), which should have prompted them to reconsider use of a fixed-effects model. A random-effects model, in addition to being a more appropriate method, also had the drastic effect of nullifying their conclusion: there was no longer a benefit to surgery compared with coiling.81

Despite our strong criticisms, we have demonstrated that meta-analyses in the neurosurgery literature do seem to be improving in both reporting and methodology with time. They are also of higher quality when the authors focused solely on designing and conducting a meta-analysis and when the authors included one or more persons that had specialized training in conducting a meta-analysis. It has been argued that the creation of a meta-analysis requires a team, which should include both statisticians and physicians, each bringing their own area of expertise to the task at hand.45,112

Literature Review

Although our study represents the first attempt, to our knowledge, to apply the PRISMA checklists to existing meta-analyses, retrospective assessment of methodology quality has been performed before. Delaney et al.27 used OQAQ to assess the quality of reports in the critical care literature before and after the publication of the QUOROM statement. They found that overall quality was poor (mean overall OQAQ score 3.3 of 7) and noted particular shortcomings in the literature search and avoidance of bias in the inclusion of studies, but they did find that the quality improved after the release of the QUOROM statement. Dixon et al.31 found that most studies in the general surgery literature, even after QUOROM, had major methodological flaws, with a median OQAQ score of 3.3. Factors associated with low overall scientific quality included the absence of any prior meta-analysis publications by the authors and meta-analyses produced by surgical department members without external collaboration, a finding that we observed as well. Dijkman et al.30 found that the quality of methodology of meta-analyses in the orthopedic literature did improve with time but that a substantial proportion continued to show extensive flaws, again similar to our results. Similar appraisals of quality of systematic reviews have been done in the anesthesia,20 general medical,66 and emergency medicine55 literature, as well as a cross-sectional sample of all systematic reviews entered into MEDLINE at a particular time and year.71 Turner et al.107 recently demonstrated that trials published in journals that endorse the CONSORT checklist are significantly more completely reported for several key CONSORT items than those published in nonendorsing journals or prior to endorsement in a given journal. Kiehna et al. found that the mean CONSORT scores were higher in 3 major medical journals (JAMA, Lancet, New England Journal of Medicine) that endorsed the CONSORT statement than in 5 neurosurgical journals, including Journal of Neurosurgery and Neurosurgery, that did not (41 vs 26.4).56

Is a Meta-Analysis Appropriate?

In the process of formulating the primary question for a meta-analysis, the researcher must also ask whether the question is clinically relevant and whether it is appropriate to compare groups of patients. For example, a recent well-conducted meta-analysis was designed to answer whether there was a difference in the improvement in leg pain and complication rates in patients undergoing a traditional open lumbar microdiscectomy compared with those who had a minimally invasive discectomy.24 There is little doubt that this constitutes a clinically relevant and valid comparison and is worthy of a meta-analysis; however, other meta-analyses have posed questions that are of questionable clinical relevance or inappropriate for a meta-analysis. For example, one could argue that most children who undergo a craniotomy for a craniopharyngioma have a larger and more complex tumor than those children who undergo a transsphenoidal operation, usually having a tumor confined within or just above the sella.37 Contrasting these 2 groups of patients is certainly clinically pertinent, but it is debatable whether a formal meta-analysis can or should be done. Other examples included evaluating the survival of patients with glioblastoma who underwent a biopsy only versus a resection,106 the timing of surgery for patient with aneurysmal subarachnoid hemorrhage,25 and comparing the risk of cranial neuropathy in patients who underwent radiosurgery compared with surgery for cavernous sinus meningiomas.101 In other papers, there was no comparison group at all.13,17,105,115 For these reports, the term “meta-analysis,” according to our definition, would not have been appropriate. Neurosurgery has recently stated that they will not accept noncomparative meta-analyses for publication because of the many undetectable and insurmountable biases.11 In summary, a meta-analysis should not be done if the question at hand is currently clinically irrelevant or outdated, if the treatment and control groups are so different as to preclude a meaningful comparison, or if there is no comparison group. In these situations, a systematic review of the literature may be the better choice.

Study Limitations

There are a number of limitations to the present study. Although we included a large number of meta-analyses in our evaluation, our search may not have identified all meta-analyses published in the specified neurosurgical journals. Some meta-analyses may also have been missed if they were contained within a systematic review and the authors did not identify it as a meta-analysis. Some of the items in the questionnaires, particularly PRISMA, introduce an element of subjectivity when there were multiple components within a single item (see Appendices 1 and 2). As stated in Methods, we were much more stringent in categorizing whether a paper fulfilled an item within AMSTAR than within PRISMA. Likewise, assigning an all-or-none score for an item may be misleading, falsely lowering the perceived quality of a paper. This has recently been addressed by Kung et al.,61 who created a revised AMSTAR (R-AMSTAR) in which each of the 11 items is subdivided into 4 components with a score ranging from 11 (where none of the items were fulfilled) to 44. This revised AMSTAR permits a detailed analysis of acceptable versus deficient aspects of each systematic review or meta-analysis and reliably quantifies the 11 domains; however, R-AMSTAR has yet to be widely tested or accepted. Our use of the AMSTAR and PRISMA checklists to calculate a “score” is also a valid criticism and limitation since neither instrument was intended for use in this way. Our score, however, is a simple proportion defined as the number of items fulfilled in the instrument divided by the total number of items contained in the instrument. The respective scores are straightforward and easily interpretable ways to present and quantify the quality of a meta-analysis that has, also, precedent.34 Finally, there were a few meta-analyses of individual patient data.46,51,110 These represent rare and unique meta-analyses in which the application of PRISMA and AMSTAR is challenging and should be conducted using established techniques.113

Our Recommendations

The results of our study underscore the need to standardize the way meta-analyses (and systematic reviews) are designed and reported. Applying PRISMA and AMSTAR to future meta-analyses should result in higher quality in both reporting and methodology and decrease the inappropriate and incorrect use of meta-analytical techniques (what we term “pseudo-meta-analyses”) and the word “meta-analysis.” We offer the following recommendations.

  1. A clear definition of a meta-analysis (and systematic review) needs to be adopted.

  2. If prospective authors decide to use the term “meta-analysis” to describe their study, both PRISMA and AMSTAR need to be applied. These checklists should be easily accessible through the website of the journal to which they wish to submit. If their study does not conform to the definition of a “meta-analysis” or “systematic review,” then other terms such as “narrative review” or “literature review” should be used instead, with the recognition that these studies will not meet the same criteria.

  3. Reviewers for the journal need to be familiar or “trained” in these checklists so that they can recognize whether the items have been fulfilled.

  4. Physicians should team up with one or more persons who have expertise in the statistical techniques of meta-analyses.

  5. We encourage journals to create a link on their website where all systematic reviews and meta-analyses that have been published by that journal are categorized and readily accessible to readers. Prospective authors can view what meta-analyses have already been done, and neurosurgeons can rapidly find topic reviews to assist them in the application of evidence-based medicine to their clinical practice.

Conclusions

Oxman et al.77 stated, “The fact that a review article is published in a peer-reviewed journal, even a prestigious one, is no guarantee of scientific quality.” Our study verifies this claim. We have shown that there is wide variability in the reporting and methodology of meta-analyses in the neurosurgery literature by applying the PRISMA and AMSTAR checklists. Many meta-analyses contained major deficiencies and some used the word “meta-analysis” inappropriately or incorrectly. The overall quality, however, is improving with time and is better when the authors collaborate with others who are familiar with meta-analysis techniques. The publishers of the Journal of Neurosurgery, JNS Publishing Group, should insist that reporting and methodology checklists be used for every submitted meta-analysis to elevate the quality of these important pieces of evidence-based documents to levels on par with other medical journals.

Acknowledgment

We thank Kristin Kraus, M.Sc., for editorial assistance with the paper.

Disclosure

The authors do not report any conflict of interest concerning the materials or methods used in this study or the findings specified in this paper.

Author contributions to the study and manuscript preparation include the following. Conception and design: all authors. Acquisition of data: Klimo. Analysis and interpretation of data: Klimo, Thompson, Ragel. Drafting the article: all authors. Critically revising the article: all authors. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Klimo.

References

  • 1

    Agarwal RWilliams KUmscheid CAWelch WC: Osteo-inductive bone graft substitutes for lumbar fusion: a systematic review. Clinical article. J Neurosurg Spine 11:7297402009

    • Search Google Scholar
    • Export Citation
  • 2

    Almenawer SBogza IYarascavitch BSne NFarrokhyar FMurty N: The value of scheduled repeat cranial computed tomography following mild head injury: single-center series and meta-analysis. Neurosurgery [epub ahead of print]2012

    • Search Google Scholar
    • Export Citation
  • 3

    Amin-Hanjani SOgilvy CSBarker FG II: Does intracisternal thrombolysis prevent vasospasm after aneurysmal subarachnoid hemorrhage? A meta-analysis. Neurosurgery 54:3263352004

    • Search Google Scholar
    • Export Citation
  • 4

    Andisheh BBrahme ABitaraf MAMavroidis PLind BK: Clinical and radiobiological advantages of single-dose stereotactic light-ion radiation therapy for large intracranial arteriovenous malformations. Technical note. J Neurosurg 111:9199262009

    • Search Google Scholar
    • Export Citation
  • 5

    Anello CFleiss JL: Exploratory or analytic meta-analysis: should we distinguish between them?. J Clin Epidemiol 48:1091181995

  • 6

    Awad IALittle JRAkarawi WPAhl J: Intracranial dural arteriovenous malformations: factors predisposing to an aggressive neurological course. J Neurosurg 72:8398501990

    • Search Google Scholar
    • Export Citation
  • 7

    Barker FG II: Efficacy of prophylactic antibiotic therapy in spinal surgery: a meta-analysis. Neurosurgery 51:3914012002

  • 8

    Barker FG II: Efficacy of prophylactic antibiotics against meningitis after craniotomy: a meta-analysis. Neurosurgery 60:8878942007

    • Search Google Scholar
    • Export Citation
  • 9

    Barker FG II: Efficacy of prophylactic antibiotics for craniotomy: a meta-analysis. Neurosurgery 35:4844921994

  • 10

    Barker FG IIOgilvy CS: Efficacy of prophylactic nimodipine for delayed ischemic deficit after subarachnoid hemorrhage: a metaanalysis. J Neurosurg 84:4054141996

    • Search Google Scholar
    • Export Citation
  • 11

    Barker FG IIOyesiku NM: Improving the quality of research reports in Neurosurgery®: the CONSORT, PRISMA, MOOSE, STARD, STROBE statements and the EQUATOR network. Neurosurgery 68:152011

    • Search Google Scholar
    • Export Citation
  • 12

    Bartels RHDonk RVerbeek AL: No justification for cervical disk prostheses in clinical practice: a meta-analysis of randomized controlled trials. Neurosurgery 66:115311602010

    • Search Google Scholar
    • Export Citation
  • 13

    Bauer DFRazdan SNBartolucci AAMarkert JM: Meta-analysis of hemorrhagic complications from ventriculostomy placement by neurosurgeons. Neurosurgery 69:2552602011

    • Search Google Scholar
    • Export Citation
  • 14

    Begg CCho MEastwood SHorton RMoher DOlkin I: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276:6376391996

    • Search Google Scholar
    • Export Citation
  • 15

    Berman NGParker RA: Meta-analysis: neither quick nor easy. BMC Med Res Methodol 2:102002

  • 16

    Boogaarts HDMenovsky Tde Vries JVerbeek ALLenders JWGrotenhuis JA: Primary hypertension and neurovascular compression: a meta-analysis of magnetic resonance imaging studies. A review. J Neurosurg 116:1471562012

    • Search Google Scholar
    • Export Citation
  • 17

    Bor-Seng-Shu EFigueiredo EGAmorim RLTeixeira MJValbuza JSde Oliveira MM: Decompressive craniectomy: a meta-analysis of influences on intracranial pressure and cerebral perfusion pressure in the treatment of traumatic brain injury. A review. J Neurosurg 117:5895962012

    • Search Google Scholar
    • Export Citation
  • 18

    Borenstein MHedges LVHiggins JPTRothstein HR: Introduction to Meta-Analysis West SussexJohn Wiley & Sons2009

  • 19

    Chappell ETMoure FCGood MC: Comparison of computed tomographic angiography with digital subtraction angiography in the diagnosis of cerebral aneurysms: a meta-analysis. Neurosurgery 52:6246312003

    • Search Google Scholar
    • Export Citation
  • 20

    Choi PTHalpern SHMalik NJadad ARTramèr MRWalder B: Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews. Anesth Analg 92:7007092001

    • Search Google Scholar
    • Export Citation
  • 21

    Cloft HJKallmes DFKallmes MHGoldstein JHJensen MEDion JE: Prevalence of cerebral aneurysms in patients with fibromuscular dysplasia: a reassessment. J Neurosurg 88:4364401998

    • Search Google Scholar
    • Export Citation
  • 22

    Cook DJSackett DLSpitzer WO: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. J Clin Epidemiol 48:1671711995

    • Search Google Scholar
    • Export Citation
  • 23

    Crowther MLim WCrowther MA: Systematic review and meta-analysis methodology. Blood 116:314031462010

  • 24

    Dasenbrock HHJuraschek SPSchultz LRWitham TFSciubba DMWolinsky JP: The efficacy of minimally invasive discectomy compared with open discectomy: a meta-analysis of prospective randomized controlled trials. Clinical article. J Neurosurg Spine 16:4524622012

    • Search Google Scholar
    • Export Citation
  • 25

    de Gans KNieuwkamp DJRinkel GJAlgra A: Timing of aneurysm surgery in subarachnoid hemorrhage: a systematic review of the literature. Neurosurgery 50:3363422002

    • Search Google Scholar
    • Export Citation
  • 26

    de Oliveira JGBeck JSetzer MGerlach RVatter HSeifert V: Risk of shunt-dependent hydrocephalus after occlusion of ruptured intracranial aneurysms by surgical clipping or endovascular coiling: a single-institution series and meta-analysis. Neurosurgery 61:9249342007

    • Search Google Scholar
    • Export Citation
  • 27

    Delaney ABagshaw SMFerland AManns BLaupland KBDoig CJ: A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9:R575R5822005

    • Search Google Scholar
    • Export Citation
  • 28

    DeLong WBPolissar NNeradilek B: Timing of surgery in cauda equina syndrome with urinary retention: meta-analysis of observational studies. J Neurosurg Spine 8:3053202008

    • Search Google Scholar
    • Export Citation
  • 29

    Di Maio STemkin NRamanathan DSekhar LN: Current comprehensive management of cranial base chordomas: 10-year meta-analysis of observational studies. Clinical article. J Neurosurg 115:109411052011

    • Search Google Scholar
    • Export Citation
  • 30

    Dijkman BGAbouali JAKooistra BWConter HJPoolman RWKulkarni AV: Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?. J Bone Joint Surg Am 92:48572010

    • Search Google Scholar
    • Export Citation
  • 31

    Dixon EHameed MSutherland FCook DJDoig C: Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg 241:4504592005

    • Search Google Scholar
    • Export Citation
  • 32

    Durham SRFjeld-Olenec K: Comparison of posterior fossa decompression with and without duraplasty for the surgical treatment of Chiari malformation Type I in pediatric patients: a meta-analysis. J Neurosurg Pediatr 2:42492008

    • Search Google Scholar
    • Export Citation
  • 33

    Egger MSmith GD: Meta-analysis. Potentials and promise. BMJ 315:137113741997

  • 34

    Egger MSmith GDAltman DG: Systematic Reviews in Health Care: Meta-Analysis in Context ed 2LondonBMJ Publishing Group2009

  • 35

    Egger MSmith GDSterne JA: Uses and abuses of meta-analysis. Clin Med 1:4784842001

  • 36

    Elashoff JD: Combining results of clinical trials. Gastroenterology 75:117011721978

  • 37

    Elliott REJane JA JrWisoff JH: Surgical management of craniopharyngiomas in children: meta-analysis and comparison of transcranial and transsphenoidal approaches. Neurosurgery 69:6306432011

    • Search Google Scholar
    • Export Citation
  • 38

    Elliott RETanweer OBoah ASmith MLFrempong-Boadu A: Comparison of safety and stability of C-2 pars and pedicle screws for atlantoaxial fusion: meta-analysis and review of the literature. A systematic review. J Neurosurg Spine 17:5775932012

    • Search Google Scholar
    • Export Citation
  • 39

    Englot DJChang EFAuguste KI: Vagus nerve stimulation for epilepsy: a meta-analysis of efficacy and predictors of response. A review. J Neurosurg 115:124812552011

    • Search Google Scholar
    • Export Citation
  • 40

    Englot DJHan SJLawton MTChang EF: Predictors of seizure freedom in the surgical treatment of supratentorial cavernous malformations. Clinical article. J Neurosurg 115:116911742011

    • Search Google Scholar
    • Export Citation
  • 41

    Englot DJWang DDRolston JDShih TTChang EF: Rates and predictors of long-term seizure freedom after frontal lobe epilepsy surgery: a systematic review and meta-analysis. Clinical article. J Neurosurg 116:104210482012

    • Search Google Scholar
    • Export Citation
  • 42

    Farace EAlves WM: Do women fare worse: a metaanalysis of gender differences in traumatic brain injury outcome. J Neurosurg 93:5395452000

    • Search Google Scholar
    • Export Citation
  • 43

    Fraser JFHärtl R: Anterior approaches to fusion of the cervical spine: a metaanalysis of fusion rates. J Neurosurg Spine 6:2983032007

    • Search Google Scholar
    • Export Citation
  • 44

    Glass GV: Primary, secondary and meta-analysis of research. Educ Res 5:381976

  • 45

    Greenhalgh T: Papers that summarise other papers (systematic reviews and meta-analyses). How To Read a Paper: The Basics of Evidence-Based Medicine ed 4West SussexJohn Wiley & Sons2010. 113131

    • Search Google Scholar
    • Export Citation
  • 46

    Groen RJMiddel BMeilof JFde Vos-van de Biezenbos JBEnting RHCoppes MH: Operative treatment of anterior thoracic spinal cord herniation: three new cases and an individual patient data meta-analysis of 126 case reports. Neurosurgery 64:3 Supplons145ons1602009

    • Search Google Scholar
    • Export Citation
  • 47

    Gross BADu R: The natural history of cerebral dural arteriovenous fistulae. Neurosurgery 71:5946032012

  • 48

    Haines SJWalters BC: Antibiotic prophylaxis for cerebrospinal fluid shunts: a metanalysis. Neurosurgery 34:87921994

  • 49

    Hamilton MGYee WHHull RDGhali WA: Venous thromboembolism prophylaxis in patients undergoing cranial neurosurgery: a systematic review and meta-analysis. Neurosurgery 68:5715812011

    • Search Google Scholar
    • Export Citation
  • 50

    Huang Jvan Gelder JM: The probability of sudden death from rupture of intracranial aneurysms: a meta-analysis. Neurosurgery 51:110111072002

    • Search Google Scholar
    • Export Citation
  • 51

    Hukkelhoven CWSteyerberg EWRampen AJFarace EHabbema JDMarshall LF: Patient age and outcome following severe traumatic brain injury: an analysis of 5600 patients. J Neurosurg 99:6666732003

    • Search Google Scholar
    • Export Citation
  • 52

    Ivan MESughrue MEClark AJKane AJAranda DBarani IJ: A meta-analysis of tumor control rates and treatment-related morbidity for patients with glomus jugulare tumors. Clinical article. J Neurosurg 114:129913052011

    • Search Google Scholar
    • Export Citation
  • 53

    Jirarattanaphochai KJung S: Nonsteroidal antiinflammatory drugs for postoperative pain management after lumbar spine surgery: a meta-analysis of randomized controlled trials. J Neurosurg Spine 9:22312008

    • Search Google Scholar
    • Export Citation
  • 54

    Joist AJoosten UWetterkamp DNeuber MProbst ARieger H: Anterior interosseous nerve compression after supracondylar fracture of the humerus: a metaanalysis. J Neurosurg 90:105310561999

    • Search Google Scholar
    • Export Citation
  • 55

    Kelly KDTravers ADorgan MSlater LRowe BH: Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med 38:5185262001

    • Search Google Scholar
    • Export Citation
  • 56

    Kiehna ENStarke RMPouratian NDumont AS: Standards for reporting randomized controlled trials in neurosurgery. A review. J Neurosurg 114:2802852011

    • Search Google Scholar
    • Export Citation
  • 57

    King JT JrBerlin JAFlamm ES: Morbidity and mortality from elective surgery for asymptomatic, unruptured, intracranial aneurysms: a meta-analysis. J Neurosurg 81:8378421994

    • Search Google Scholar
    • Export Citation
  • 58

    Klimo P JrThompson CJRagel BTBoop FA: Antibiotic-impregnated shunt systems versus standard shunt systems: a meta- and cost-savings analysis. Clinical article. J Neurosurg Pediatr 8:6006122011

    • Search Google Scholar
    • Export Citation
  • 59

    Komotar RJStarke RMOtten MLMerkow MBGarrett MCMarshall RS: The role of indirect extracranial-intracranial bypass in the treatment of symptomatic intracranial atheroocclusive disease. Clinical article. J Neurosurg 110:8969042009

    • Search Google Scholar
    • Export Citation
  • 60

    Krishna VKim DH: Ethnic differences in risk factors for subarachnoid hemorrhage. J Neurosurg 107:5225292007

  • 61

    Kung JChiappelli FCajulis OOAvezova RKossan GChew L: From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 4:84912010

    • Search Google Scholar
    • Export Citation
  • 62

    LeLorier JGrégoire GBenhaddad ALapierre JDerderian F: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 337:5365421997

    • Search Google Scholar
    • Export Citation
  • 63

    Levy EIFirlik ADWisniewski SRubin GJungreis CAWechsler LR: Factors affecting survival rates for acute vertebrobasilar artery occlusions treated with intraarterial thrombolytic therapy: a meta-analytical approach. Neurosurgery 45:5395481999

    • Search Google Scholar
    • Export Citation
  • 64

    Liberati AAltman DGTetzlaff JMulrow CGøtzsche PCIoannidis JP: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339:b27002009

    • Search Google Scholar
    • Export Citation
  • 65

    Lucas CPZabramski JMSpetzler RFJacobowitz R: Treatment for intracranial dural arteriovenous malformations: a meta-analysis from the English language literature. Neurosurgery 40:111911321997

    • Search Google Scholar
    • Export Citation
  • 66

    McAlister FAClark HDvan Walraven CStraus SELawson FMMoher D: The medical review article revisited: has the science improved?. Ann Intern Med 131:9479511999

    • Search Google Scholar
    • Export Citation
  • 67

    McColgan PThant KZSharma P: The genetics of sporadic ruptured and unruptured intracranial aneurysms: a genetic meta-analysis of 8 genes and 13 polymorphisms in approximately 20,000 individuals. Clinical article. J Neurosurg 112:7147212010

    • Search Google Scholar
    • Export Citation
  • 68

    Moher DCook DJEastwood SOlkin IRennie DStroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 354:189619001999

    • Search Google Scholar
    • Export Citation
  • 69

    Moher DLiberati ATetzlaff JAltman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:b25352009

    • Search Google Scholar
    • Export Citation
  • 70

    Moher DSchulz KFAltman D: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285:198719912001

    • Search Google Scholar
    • Export Citation
  • 71

    Moher DTetzlaff JTricco ACSampson MAltman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4:e782007

    • Search Google Scholar
    • Export Citation
  • 72

    Mortazavi MMRomeo AKDeep AGriessenauer CJShoja MMTubbs RS: Hypertonic saline for treating raised intracranial pressure: literature review with meta-analysis. A review. J Neurosurg 116:2102212012

    • Search Google Scholar
    • Export Citation
  • 73

    Noble AJSchenk T: Which variables help explain the poor health-related quality of life after subarachnoid hemorrhage? A meta-analysis. Neurosurgery 66:7727832010

    • Search Google Scholar
    • Export Citation
  • 74

    Nourbakhsh AShi RVannemreddy PNanda A: Operative versus nonoperative management of acute odontoid Type II fractures: a meta-analysis. Clinical article. J Neurosurg Spine 11:6516582009

    • Search Google Scholar
    • Export Citation
  • 75

    Ooi YCDagi TFMaltenfort MRincon FVibbert MJabbour P: Tight glycemic control reduces infection and improves neurological outcome in critically ill neurosurgical and neurological patients. Neurosurgery 71:6927022012

    • Search Google Scholar
    • Export Citation
  • 76

    Oxman ADGuyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 44:127112781991

  • 77

    Oxman ADGuyatt GHSinger JGoldsmith CHHutchison BGMilner RA: Agreement among reviewers of review articles. J Clin Epidemiol 44:91981991

    • Search Google Scholar
    • Export Citation
  • 78

    Palur RSBerk CSchulzer MHoney CR: A metaanalysis comparing the results of pallidotomy performed using microelectrode recording or macroelectrode stimulation. J Neurosurg 96:105810622002

    • Search Google Scholar
    • Export Citation
  • 79

    Panczykowski DMTomycz NDOkonkwo DO: Comparative effectiveness of using computed tomography alone to exclude cervical spine injuries in obtunded or intubated patients: meta-analysis of 14,327 patients with blunt trauma. A review. J Neurosurg 115:5415492011

    • Search Google Scholar
    • Export Citation
  • 80

    Peipert JFPhipps MG: Observational studies. Clin Obstet Gynecol 41:2352441998

  • 81

    Quigley M: Risk of shunt-dependent hydrocephalus after occlusion of ruptured intracranial aneurysms by surgical clipping or endovascular coiling: a single-institution series and meta-analysis. Neurosurgery 63:E12092008. (Letter)

    • Search Google Scholar
    • Export Citation
  • 82

    Qureshi AIKirmani JFDivani AAHobson RW II: Carotid angioplasty with or without stent placement versus carotid endarterectomy for treatment of carotid stenosis: a meta-analysis. Neurosurgery 56:117111812005

    • Search Google Scholar
    • Export Citation
  • 83

    Ranguis SCLi DWebster AC: Perioperative epidural steroids for lumbar spine surgery in degenerative spinal disease. A review. J Neurosurg Spine 13:7457572010

    • Search Google Scholar
    • Export Citation
  • 84

    Ratilal BCosta JSampaio C: Antibiotic prophylaxis for surgical introduction of intracranial ventricular shunts: a systematic review. J Neurosurg Pediatr 1:48562008

    • Search Google Scholar
    • Export Citation
  • 85

    Ratliff JKCooper PR: Cervical laminoplasty: a critical review. J Neurosurg 98:3 Suppl2302382003

  • 86

    Ressing MBlettner MKlug SJ: Systematic literature reviews and meta-analyses: part 6 of a series on evaluation of scientific publications. Dtsch Arztebl Int 106:4564632009

    • Search Google Scholar
    • Export Citation
  • 87

    Rowland NCEnglot DJCage TASughrue MEBarbaro NMChang EF: A meta-analysis of predictors of seizure freedom in the surgical management of focal cortical dysplasia. Clinical article. J Neurosurg 116:103510412012

    • Search Google Scholar
    • Export Citation
  • 88

    Sacks HSBerrier JReitman DAncona-Berk VAChalmers TC: Meta-analyses of randomized controlled trials. N Engl J Med 316:4504551987

    • Search Google Scholar
    • Export Citation
  • 89

    Sekula RF JrBhatia SFrederickson AMJannetta PJQuigley MRSmall GA: Utility of intraoperative electromyography in microvascular decompression for hemifacial spasm: a meta-analysis. Neurosurg Focus 27:4E102009

    • Search Google Scholar
    • Export Citation
  • 90

    Sekula RF JrFrederickson AMJannetta PJQuigley MRAziz KMArnone GD: Microvascular decompression for elderly patients with trigeminal neuralgia: a prospective study and systematic review with meta-analysis. Clinical article. J Neurosurg 114:1721792011

    • Search Google Scholar
    • Export Citation
  • 91

    Shea BJBouter LMPeterson JBoers MAndersson NOrtiz Z: External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2:e13502007

    • Search Google Scholar
    • Export Citation
  • 92

    Shea BJGrimshaw JMWells GABoers MAndersson NHamel C: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:102007

    • Search Google Scholar
    • Export Citation
  • 93

    Shea BJHamel CWells GABouter LMKristjansson EGrimshaw J: AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62:101310202009

    • Search Google Scholar
    • Export Citation
  • 94

    Shin BJJames ARNjoku IUHärtl R: Pedicle screw navigation: a systematic review and meta-analysis of perforation risk for computer-navigated versus freehand insertion. A review. J Neurosurg Spine 17:1131222012

    • Search Google Scholar
    • Export Citation
  • 95

    Siddiq FMemon MZVazquez GSafdar AQureshi AI: Comparison between primary angioplasty and stent placement for symptomatic intracranial atherosclerotic disease: meta-analysis of case series. Neurosurgery 65:102410342009

    • Search Google Scholar
    • Export Citation
  • 96

    Sonabend AMKorenfeld YCrisman CBadjatia NMayer SAConnolly ES Jr: Prevention of ventriculostomy-related infections with prophylactic antibiotics and antibiotic-coated external ventricular drains: a systematic review. Neurosurgery 68:99610052011

    • Search Google Scholar
    • Export Citation
  • 97

    Sorani MDManley GT: Dose-response relationship of mannitol and intracranial pressure: a metaanalysis. J Neurosurg 108:80872008

  • 98

    Stein SCGeorgoff PMeghan SMirza KLEl Falaky OM: Relationship of aggressive monitoring and treatment to improved outcomes in severe traumatic brain injury. Clinical article. J Neurosurg 112:110511122010

    • Search Google Scholar
    • Export Citation
  • 99

    Steinmetz MPChow MMKrishnaney AAAndrews-Hinders DBenzel ECMasaryk TJ: Outcome after the treatment of spinal dural arteriovenous fistulae: a contemporary single-institution series and meta-analysis. Neurosurgery 55:77882004

    • Search Google Scholar
    • Export Citation
  • 100

    Stroup DFBerlin JAMorton SCOlkin IWilliamson GDRennie D: Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA 283:200820122000

    • Search Google Scholar
    • Export Citation
  • 101

    Sughrue MERutkowski MJAranda DBarani IJMcDermott MWParsa AT: Factors affecting outcome following treatment of patients with cavernous sinus meningiomas. Clinical article. J Neurosurg 113:108710922010

    • Search Google Scholar
    • Export Citation
  • 102

    Sughrue MERutkowski MJAranda DBarani IJMcDermott MWParsa AT: Treatment decision making based on the published natural history and growth rate of small meningiomas. A review and meta-analysis. J Neurosurg 113:103610422010

    • Search Google Scholar
    • Export Citation
  • 103

    Sughrue MEShah JKDevin JKKunwar SBlevins LS Jr: Utility of the immediate postoperative cortisol concentrations in patients with Cushing's disease. Neurosurgery 67:6886952010

    • Search Google Scholar
    • Export Citation
  • 104

    Sutton AJAbrams KRJones DRSheldon TASong F: Methods for Meta-Analysis in Medical Research West SussexJohn Wiley & Sons2000

  • 105

    Tabaee AAnand VKBarrón YHiltzik DHBrown SMKacker A: Endoscopic pituitary surgery: a systematic review and meta-analysis. Clinical article. J Neurosurg 111:5455542009

    • Search Google Scholar
    • Export Citation
  • 106

    Tsitlakidis AForoglou NVenetis CAPatsalas IHatzisotiriou ASelviaridis P: Biopsy versus resection in the management of malignant gliomas: a systematic review and meta-analysis. A review. J Neurosurg 112:102010322010

    • Search Google Scholar
    • Export Citation
  • 107

    Turner LShamseer LAltman DGWeeks LPeters JKober T: Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 11:MR0000302012

    • Search Google Scholar
    • Export Citation
  • 108

    van Dijk JMTerBrugge KGVan der Meer FJWallace MCRosendaal FR: Thrombophilic factors and the formation of dural arteriovenous fistulas. J Neurosurg 107:56592007

    • Search Google Scholar
    • Export Citation
  • 109

    Van Gompel JJFrank GPasquini EZoli MHoover JLanzino G: Expanded endonasal endoscopic resection of anterior fossa meningiomas: report of 13 cases and meta-analysis of the literature. Neurosurg Focus 30:5E152011

    • Search Google Scholar
    • Export Citation
  • 110

    van Leeuwen NLingsma HFPerel PLecky FRoozenbeek BLu J: Prognostic value of major extracranial injury in traumatic brain injury: an individual patient data meta-analysis in 39,274 patients. Neurosurgery 70:8118182012

    • Search Google Scholar
    • Export Citation
  • 111

    Weaver FFollett KHur KIppolito DStern M: Deep brain stimulation in Parkinson disease: a metaanalysis of patient outcomes. J Neurosurg 103:9569672005

    • Search Google Scholar
    • Export Citation
  • 112

    Whitehead A: Meta-Analysis of Controlled Clinical Trials West SussexJohn Wiley & Sons2002

  • 113

    Whitehead A: Meta-analysis using individual patient data. Meta-Analysis of Controlled Clinical Trials West SussexJohn Wiley & Sons2002. 99150

    • Search Google Scholar
    • Export Citation
  • 114

    Yadla SCampbell PGChitale RMaltenfort MGJabbour PSharan AD: Effect of early surgery, material, and method of flap preservation on cranioplasty infections: a systematic review. Neurosurgery 68:112411302011

    • Search Google Scholar
    • Export Citation
  • 115

    Yoshimoto Y: Systematic review of the natural history of vestibular schwannoma. J Neurosurg 103:59632005

  • 116

    Zhou ZJZhao FDFang XQZhao XFan SW: Meta-analysis of instrumented posterior interbody fusion versus instrumented posterolateral fusion in the lumbar spine. A review. J Neurosurg Spine 15:2953102011

    • Search Google Scholar
    • Export Citation
Appendix 1

AMSTAR: a measurement tool to assess the methodological quality of systematic reviews*

1. Was an “a priori” design provided?

The research question and inclusion criteria should be established before the conduct of the review.

Note: Need to refer to a protocol, ethics approval, or pre-determined/a priori published research objectives to score a “yes.”
2. Was there duplicate study selection and data extraction?

There should be at least 2 independent data extractors and a consensus procedure for disagreements should be in place.

Note: 2 people do study selection, 2 people do data extraction, consensus process or 1 person checks the other's work.
3. Was a comprehensive literature search performed?

At least 2 electronic sources should be searched. The report must include years and databases used (e.g., Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.

Note: If at least 2 sources + 1 supplementary strategy used, select “yes” (Cochrane register/Central counts as 2 sources; a gray literature search counts as supplementary).
4. Was the status of publication (i.e., gray literature) used as an inclusion criterion?

The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language etc.

Note: If review indicates that there was a search for “gray literature” or “unpublished literature,” indicate “yes.” SIGLE database, dissertations, conference proceedings, and trial registries are all considered gray for this purpose. If searching a source that contains both gray and non-gray, must specify that they were searching for gray/unpublished lit.
5. Was a list of studies (included and excluded) provided?

A list of included and excluded studies should be provided.

Note: Acceptable if the excluded studies are referenced. If there is an electronic link to the list but the link is dead, select “no.”
6. Were the characteristics of the included studies provided?

In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed (e.g., age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases) should be reported.

Note: Acceptable if not in table format as long as they are described as above.
7. Was the scientific quality of the included studies assessed and documented?

“A priori” methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.

Note: Can include use of a quality scoring tool or checklist, e.g., Jadad scale, risk of bias, sensitivity analysis, etc., or a description of quality items, with some kind of result for EACH study (“low” or “high” is fine, as long as it is clear which studies scored “low” and which scored “high;” a summary score/range for all studies is not acceptable).
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?

The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations.

Note: Might say something such as “the results should be interpreted with caution due to poor quality of included studies.” Cannot score “yes” for this question if scored “no” for Question 7.
9. Were the methods used to combine the findings of studies appropriate?

For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e., chi-squared test for homogeneity, I2). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e., is it sensible to combine?).

Note: Indicate “yes” if they mention or describe heterogeneity, i.e., if they explain that they cannot pool because of heterogeneity/variability between interventions.
10. Was the likelihood of publication bias assessed?

An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test, Hedges-Olkin).

Note: If no test values or funnel plot included, score “no.” Score “yes” if mentions that publication bias could not be assessed because there were fewer than 10 included studies.
11. Was the conflict of interest included?

Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.

Note: To get a “yes,” must indicate source of funding or support for the systematic review and for each of the included studies.

Modified from Shea et al. BMC Medical Research Methodology 7:10, 2007. Creative Commons Attribution License.

Appendix 2

PRISMA—checklist of items to include when reporting a systematic review or meta-analysis*

Section/TopicItem No.Checklist Item
Title
Title1Identify the report as a systematic review, meta-analysis, or both
Abstract
Structured summary2Provide a structured summary including, as applicable, background, objectives, data sources, study eligibility criteria, participants, interventions, study appraisal and synthesis methods, results, limitations, conclusions and implications of key findings, systematic review registration number
Introduction
Rationale3Describe the rationale for the review in the context of what is already known
Objectives4Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS)
Methods
Protocol and registration5Indicate if a review protocol exists, if and where it can be accessed (such as web address), and, if available, provide registration information including registration number
Eligibility criteria6Specify study characteristics (such as PICOS, length of follow-up) and report characteristics (such as years considered, language, publication status) used as criteria for eligibility, giving rationale
Information sources7Describe all information sources (such as databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched
Search8Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated
Study selection9State the process for selecting studies (that is, screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis)
Data collection process10Describe method of data extraction from reports (such as piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators
Data items11List and define all variables for which data were sought (such as PICOS, funding sources) and any assumptions and simplifications made
Risk of bias in individual studies12Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis
Summary measures13State the principal summary measures (such as risk ratio, difference in means).
Synthesis of results14Describe the methods of handling data and combining results of studies, if done, including measures of consistency (such as I2 statistic) for each meta-analysis
Risk of bias across studies15Specify any assessment of risk of bias that may affect the cumulative evidence (such as publication bias, selective reporting within studies)
Additional analyses16Describe methods of additional analyses (such as sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified
Results
Study selection17Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram
Study characteristics18For each study, present characteristics for which data were extracted (such as study size, PICOS, follow-up period) and provide the citations
Risk of bias within studies19Present data on risk of bias of each study and, if available, any outcome-level assessment (see item 12).
Results of individual studies20For all outcomes considered (benefits or harms), present for each study (a) simple summary data for each intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot
Synthesis of results21Present results of each meta-analysis done, including confidence intervals and measures of consistency
Risk of bias across studies22Present results of any assessment of risk of bias across studies (see item 15)
Additional analysis23Give results of additional analyses, if done (such as sensitivity or subgroup analyses, meta-regression) (see item 16)
Discussion
Summary of evidence24Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (such as health care providers, users, and policy makers)
Limitations25Discuss limitations at study and outcome level (such as risk of bias), and at review level (such as incomplete retrieval of identified research, reporting bias)
Conclusions26Provide a general interpretation of the results in the context of other evidence, and implications for future research
Funding
Funding27Describe sources of funding for the systematic review and other support (such as supply of data) and role of funders for the systematic review

Moher D, et al: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:b2535, 2009. Creative Commons Attribution Non-Commercial License.

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

Address correspondence to: Paul Klimo Jr., M.D., M.P.H., 6325 Humphreys Blvd., Memphis, TN 38120. email: pklimo@semmesmurphey.com.

Please include this information when citing this paper: published online January 24, 2014; DOI: 10.3171/2013.11.JNS13195.

© AANS, except where prohibited by US copyright law.

Headings

Figures

  • View in gallery

    Flow of information through the different phases of the literature search for a meta-analysis or systematic review.

  • View in gallery

    The number of papers that fulfilled each item in the AMSTAR checklist. The dotted vertical line indicates 50% of analyzed papers (n = 36).

  • View in gallery

    The number of papers that fulfilled each item in the PRISMA checklist. The dotted vertical line indicates 50% of analyzed papers (n = 36).

  • View in gallery

    Proportion of items present by decade. Upper: AMSTAR checklist. Lower: PRISMA checklist.

References

  • 1

    Agarwal RWilliams KUmscheid CAWelch WC: Osteo-inductive bone graft substitutes for lumbar fusion: a systematic review. Clinical article. J Neurosurg Spine 11:7297402009

    • Search Google Scholar
    • Export Citation
  • 2

    Almenawer SBogza IYarascavitch BSne NFarrokhyar FMurty N: The value of scheduled repeat cranial computed tomography following mild head injury: single-center series and meta-analysis. Neurosurgery [epub ahead of print]2012

    • Search Google Scholar
    • Export Citation
  • 3

    Amin-Hanjani SOgilvy CSBarker FG II: Does intracisternal thrombolysis prevent vasospasm after aneurysmal subarachnoid hemorrhage? A meta-analysis. Neurosurgery 54:3263352004

    • Search Google Scholar
    • Export Citation
  • 4

    Andisheh BBrahme ABitaraf MAMavroidis PLind BK: Clinical and radiobiological advantages of single-dose stereotactic light-ion radiation therapy for large intracranial arteriovenous malformations. Technical note. J Neurosurg 111:9199262009

    • Search Google Scholar
    • Export Citation
  • 5

    Anello CFleiss JL: Exploratory or analytic meta-analysis: should we distinguish between them?. J Clin Epidemiol 48:1091181995

  • 6

    Awad IALittle JRAkarawi WPAhl J: Intracranial dural arteriovenous malformations: factors predisposing to an aggressive neurological course. J Neurosurg 72:8398501990

    • Search Google Scholar
    • Export Citation
  • 7

    Barker FG II: Efficacy of prophylactic antibiotic therapy in spinal surgery: a meta-analysis. Neurosurgery 51:3914012002

  • 8

    Barker FG II: Efficacy of prophylactic antibiotics against meningitis after craniotomy: a meta-analysis. Neurosurgery 60:8878942007

    • Search Google Scholar
    • Export Citation
  • 9

    Barker FG II: Efficacy of prophylactic antibiotics for craniotomy: a meta-analysis. Neurosurgery 35:4844921994

  • 10

    Barker FG IIOgilvy CS: Efficacy of prophylactic nimodipine for delayed ischemic deficit after subarachnoid hemorrhage: a metaanalysis. J Neurosurg 84:4054141996

    • Search Google Scholar
    • Export Citation
  • 11

    Barker FG IIOyesiku NM: Improving the quality of research reports in Neurosurgery®: the CONSORT, PRISMA, MOOSE, STARD, STROBE statements and the EQUATOR network. Neurosurgery 68:152011

    • Search Google Scholar
    • Export Citation
  • 12

    Bartels RHDonk RVerbeek AL: No justification for cervical disk prostheses in clinical practice: a meta-analysis of randomized controlled trials. Neurosurgery 66:115311602010

    • Search Google Scholar
    • Export Citation
  • 13

    Bauer DFRazdan SNBartolucci AAMarkert JM: Meta-analysis of hemorrhagic complications from ventriculostomy placement by neurosurgeons. Neurosurgery 69:2552602011

    • Search Google Scholar
    • Export Citation
  • 14

    Begg CCho MEastwood SHorton RMoher DOlkin I: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276:6376391996

    • Search Google Scholar
    • Export Citation
  • 15

    Berman NGParker RA: Meta-analysis: neither quick nor easy. BMC Med Res Methodol 2:102002

  • 16

    Boogaarts HDMenovsky Tde Vries JVerbeek ALLenders JWGrotenhuis JA: Primary hypertension and neurovascular compression: a meta-analysis of magnetic resonance imaging studies. A review. J Neurosurg 116:1471562012

    • Search Google Scholar
    • Export Citation
  • 17

    Bor-Seng-Shu EFigueiredo EGAmorim RLTeixeira MJValbuza JSde Oliveira MM: Decompressive craniectomy: a meta-analysis of influences on intracranial pressure and cerebral perfusion pressure in the treatment of traumatic brain injury. A review. J Neurosurg 117:5895962012

    • Search Google Scholar
    • Export Citation
  • 18

    Borenstein MHedges LVHiggins JPTRothstein HR: Introduction to Meta-Analysis West SussexJohn Wiley & Sons2009

  • 19

    Chappell ETMoure FCGood MC: Comparison of computed tomographic angiography with digital subtraction angiography in the diagnosis of cerebral aneurysms: a meta-analysis. Neurosurgery 52:6246312003

    • Search Google Scholar
    • Export Citation
  • 20

    Choi PTHalpern SHMalik NJadad ARTramèr MRWalder B: Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews. Anesth Analg 92:7007092001

    • Search Google Scholar
    • Export Citation
  • 21

    Cloft HJKallmes DFKallmes MHGoldstein JHJensen MEDion JE: Prevalence of cerebral aneurysms in patients with fibromuscular dysplasia: a reassessment. J Neurosurg 88:4364401998

    • Search Google Scholar
    • Export Citation
  • 22

    Cook DJSackett DLSpitzer WO: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. J Clin Epidemiol 48:1671711995

    • Search Google Scholar
    • Export Citation
  • 23

    Crowther MLim WCrowther MA: Systematic review and meta-analysis methodology. Blood 116:314031462010

  • 24

    Dasenbrock HHJuraschek SPSchultz LRWitham TFSciubba DMWolinsky JP: The efficacy of minimally invasive discectomy compared with open discectomy: a meta-analysis of prospective randomized controlled trials. Clinical article. J Neurosurg Spine 16:4524622012

    • Search Google Scholar
    • Export Citation
  • 25

    de Gans KNieuwkamp DJRinkel GJAlgra A: Timing of aneurysm surgery in subarachnoid hemorrhage: a systematic review of the literature. Neurosurgery 50:3363422002

    • Search Google Scholar
    • Export Citation
  • 26

    de Oliveira JGBeck JSetzer MGerlach RVatter HSeifert V: Risk of shunt-dependent hydrocephalus after occlusion of ruptured intracranial aneurysms by surgical clipping or endovascular coiling: a single-institution series and meta-analysis. Neurosurgery 61:9249342007

    • Search Google Scholar
    • Export Citation
  • 27

    Delaney ABagshaw SMFerland AManns BLaupland KBDoig CJ: A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9:R575R5822005

    • Search Google Scholar
    • Export Citation
  • 28

    DeLong WBPolissar NNeradilek B: Timing of surgery in cauda equina syndrome with urinary retention: meta-analysis of observational studies. J Neurosurg Spine 8:3053202008

    • Search Google Scholar
    • Export Citation
  • 29

    Di Maio STemkin NRamanathan DSekhar LN: Current comprehensive management of cranial base chordomas: 10-year meta-analysis of observational studies. Clinical article. J Neurosurg 115:109411052011

    • Search Google Scholar
    • Export Citation
  • 30

    Dijkman BGAbouali JAKooistra BWConter HJPoolman RWKulkarni AV: Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?. J Bone Joint Surg Am 92:48572010

    • Search Google Scholar
    • Export Citation
  • 31

    Dixon EHameed MSutherland FCook DJDoig C: Evaluating meta-analyses in the general surgical literature: a critical appraisal. Ann Surg 241:4504592005

    • Search Google Scholar
    • Export Citation
  • 32

    Durham SRFjeld-Olenec K: Comparison of posterior fossa decompression with and without duraplasty for the surgical treatment of Chiari malformation Type I in pediatric patients: a meta-analysis. J Neurosurg Pediatr 2:42492008

    • Search Google Scholar
    • Export Citation
  • 33

    Egger MSmith GD: Meta-analysis. Potentials and promise. BMJ 315:137113741997

  • 34

    Egger MSmith GDAltman DG: Systematic Reviews in Health Care: Meta-Analysis in Context ed 2LondonBMJ Publishing Group2009

  • 35

    Egger MSmith GDSterne JA: Uses and abuses of meta-analysis. Clin Med 1:4784842001

  • 36

    Elashoff JD: Combining results of clinical trials. Gastroenterology 75:117011721978

  • 37

    Elliott REJane JA JrWisoff JH: Surgical management of craniopharyngiomas in children: meta-analysis and comparison of transcranial and transsphenoidal approaches. Neurosurgery 69:6306432011

    • Search Google Scholar
    • Export Citation
  • 38

    Elliott RETanweer OBoah ASmith MLFrempong-Boadu A: Comparison of safety and stability of C-2 pars and pedicle screws for atlantoaxial fusion: meta-analysis and review of the literature. A systematic review. J Neurosurg Spine 17:5775932012

    • Search Google Scholar
    • Export Citation
  • 39

    Englot DJChang EFAuguste KI: Vagus nerve stimulation for epilepsy: a meta-analysis of efficacy and predictors of response. A review. J Neurosurg 115:124812552011

    • Search Google Scholar
    • Export Citation
  • 40

    Englot DJHan SJLawton MTChang EF: Predictors of seizure freedom in the surgical treatment of supratentorial cavernous malformations. Clinical article. J Neurosurg 115:116911742011

    • Search Google Scholar
    • Export Citation
  • 41

    Englot DJWang DDRolston JDShih TTChang EF: Rates and predictors of long-term seizure freedom after frontal lobe epilepsy surgery: a systematic review and meta-analysis. Clinical article. J Neurosurg 116:104210482012

    • Search Google Scholar
    • Export Citation
  • 42

    Farace EAlves WM: Do women fare worse: a metaanalysis of gender differences in traumatic brain injury outcome. J Neurosurg 93:5395452000

    • Search Google Scholar
    • Export Citation
  • 43

    Fraser JFHärtl R: Anterior approaches to fusion of the cervical spine: a metaanalysis of fusion rates. J Neurosurg Spine 6:2983032007

    • Search Google Scholar
    • Export Citation
  • 44

    Glass GV: Primary, secondary and meta-analysis of research. Educ Res 5:381976

  • 45

    Greenhalgh T: Papers that summarise other papers (systematic reviews and meta-analyses). How To Read a Paper: The Basics of Evidence-Based Medicine ed 4West SussexJohn Wiley & Sons2010. 113131

    • Search Google Scholar
    • Export Citation
  • 46

    Groen RJMiddel BMeilof JFde Vos-van de Biezenbos JBEnting RHCoppes MH: Operative treatment of anterior thoracic spinal cord herniation: three new cases and an individual patient data meta-analysis of 126 case reports. Neurosurgery 64:3 Supplons145ons1602009

    • Search Google Scholar
    • Export Citation
  • 47

    Gross BADu R: The natural history of cerebral dural arteriovenous fistulae. Neurosurgery 71:5946032012

  • 48

    Haines SJWalters BC: Antibiotic prophylaxis for cerebrospinal fluid shunts: a metanalysis. Neurosurgery 34:87921994

  • 49

    Hamilton MGYee WHHull RDGhali WA: Venous thromboembolism prophylaxis in patients undergoing cranial neurosurgery: a systematic review and meta-analysis. Neurosurgery 68:5715812011

    • Search Google Scholar
    • Export Citation
  • 50

    Huang Jvan Gelder JM: The probability of sudden death from rupture of intracranial aneurysms: a meta-analysis. Neurosurgery 51:110111072002

    • Search Google Scholar
    • Export Citation
  • 51

    Hukkelhoven CWSteyerberg EWRampen AJFarace EHabbema JDMarshall LF: Patient age and outcome following severe traumatic brain injury: an analysis of 5600 patients. J Neurosurg 99:6666732003

    • Search Google Scholar
    • Export Citation
  • 52

    Ivan MESughrue MEClark AJKane AJAranda DBarani IJ: A meta-analysis of tumor control rates and treatment-related morbidity for patients with glomus jugulare tumors. Clinical article. J Neurosurg 114:129913052011

    • Search Google Scholar
    • Export Citation
  • 53

    Jirarattanaphochai KJung S: Nonsteroidal antiinflammatory drugs for postoperative pain management after lumbar spine surgery: a meta-analysis of randomized controlled trials. J Neurosurg Spine 9:22312008

    • Search Google Scholar
    • Export Citation
  • 54

    Joist AJoosten UWetterkamp DNeuber MProbst ARieger H: Anterior interosseous nerve compression after supracondylar fracture of the humerus: a metaanalysis. J Neurosurg 90:105310561999

    • Search Google Scholar
    • Export Citation
  • 55

    Kelly KDTravers ADorgan MSlater LRowe BH: Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med 38:5185262001

    • Search Google Scholar
    • Export Citation
  • 56

    Kiehna ENStarke RMPouratian NDumont AS: Standards for reporting randomized controlled trials in neurosurgery. A review. J Neurosurg 114:2802852011

    • Search Google Scholar
    • Export Citation
  • 57

    King JT JrBerlin JAFlamm ES: Morbidity and mortality from elective surgery for asymptomatic, unruptured, intracranial aneurysms: a meta-analysis. J Neurosurg 81:8378421994

    • Search Google Scholar
    • Export Citation
  • 58

    Klimo P JrThompson CJRagel BTBoop FA: Antibiotic-impregnated shunt systems versus standard shunt systems: a meta- and cost-savings analysis. Clinical article. J Neurosurg Pediatr 8:6006122011

    • Search Google Scholar
    • Export Citation
  • 59

    Komotar RJStarke RMOtten MLMerkow MBGarrett MCMarshall RS: The role of indirect extracranial-intracranial bypass in the treatment of symptomatic intracranial atheroocclusive disease. Clinical article. J Neurosurg 110:8969042009

    • Search Google Scholar
    • Export Citation
  • 60

    Krishna VKim DH: Ethnic differences in risk factors for subarachnoid hemorrhage. J Neurosurg 107:5225292007

  • 61

    Kung JChiappelli FCajulis OOAvezova RKossan GChew L: From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 4:84912010

    • Search Google Scholar
    • Export Citation
  • 62

    LeLorier JGrégoire GBenhaddad ALapierre JDerderian F: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 337:5365421997

    • Search Google Scholar
    • Export Citation
  • 63

    Levy EIFirlik ADWisniewski SRubin GJungreis CAWechsler LR: Factors affecting survival rates for acute vertebrobasilar artery occlusions treated with intraarterial thrombolytic therapy: a meta-analytical approach. Neurosurgery 45:5395481999

    • Search Google Scholar
    • Export Citation
  • 64

    Liberati AAltman DGTetzlaff JMulrow CGøtzsche PCIoannidis JP: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339:b27002009

    • Search Google Scholar
    • Export Citation
  • 65

    Lucas CPZabramski JMSpetzler RFJacobowitz R: Treatment for intracranial dural arteriovenous malformations: a meta-analysis from the English language literature. Neurosurgery 40:111911321997

    • Search Google Scholar
    • Export Citation
  • 66

    McAlister FAClark HDvan Walraven CStraus SELawson FMMoher D: The medical review article revisited: has the science improved?. Ann Intern Med 131:9479511999

    • Search Google Scholar
    • Export Citation
  • 67

    McColgan PThant KZSharma P: The genetics of sporadic ruptured and unruptured intracranial aneurysms: a genetic meta-analysis of 8 genes and 13 polymorphisms in approximately 20,000 individuals. Clinical article. J Neurosurg 112:7147212010

    • Search Google Scholar
    • Export Citation
  • 68

    Moher DCook DJEastwood SOlkin IRennie DStroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet 354:189619001999

    • Search Google Scholar
    • Export Citation
  • 69

    Moher DLiberati ATetzlaff JAltman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:b25352009

    • Search Google Scholar
    • Export Citation
  • 70

    Moher DSchulz KFAltman D: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285:198719912001

    • Search Google Scholar
    • Export Citation
  • 71

    Moher DTetzlaff JTricco ACSampson MAltman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4:e782007

    • Search Google Scholar
    • Export Citation
  • 72

    Mortazavi MMRomeo AKDeep AGriessenauer CJShoja MMTubbs RS: Hypertonic saline for treating raised intracranial pressure: literature review with meta-analysis. A review. J Neurosurg 116:2102212012

    • Search Google Scholar
    • Export Citation
  • 73

    Noble AJSchenk T: Which variables help explain the poor health-related quality of life after subarachnoid hemorrhage? A meta-analysis. Neurosurgery 66:7727832010

    • Search Google Scholar
    • Export Citation
  • 74

    Nourbakhsh AShi RVannemreddy PNanda A: Operative versus nonoperative management of acute odontoid Type II fractures: a meta-analysis. Clinical article. J Neurosurg Spine 11:6516582009

    • Search Google Scholar
    • Export Citation
  • 75

    Ooi YCDagi TFMaltenfort MRincon FVibbert MJabbour P: Tight glycemic control reduces infection and improves neurological outcome in critically ill neurosurgical and neurological patients. Neurosurgery 71:6927022012

    • Search Google Scholar
    • Export Citation
  • 76

    Oxman ADGuyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 44:127112781991

  • 77

    Oxman ADGuyatt GHSinger JGoldsmith CHHutchison BGMilner RA: Agreement among reviewers of review articles. J Clin Epidemiol 44:91981991

    • Search Google Scholar
    • Export Citation
  • 78

    Palur RSBerk CSchulzer MHoney CR: A metaanalysis comparing the results of pallidotomy performed using microelectrode recording or macroelectrode stimulation. J Neurosurg 96:105810622002

    • Search Google Scholar
    • Export Citation
  • 79

    Panczykowski DMTomycz NDOkonkwo DO: Comparative effectiveness of using computed tomography alone to exclude cervical spine injuries in obtunded or intubated patients: meta-analysis of 14,327 patients with blunt trauma. A review. J Neurosurg 115:5415492011

    • Search Google Scholar
    • Export Citation
  • 80

    Peipert JFPhipps MG: Observational studies. Clin Obstet Gynecol 41:2352441998

  • 81

    Quigley M: Risk of shunt-dependent hydrocephalus after occlusion of ruptured intracranial aneurysms by surgical clipping or endovascular coiling: a single-institution series and meta-analysis. Neurosurgery 63:E12092008. (Letter)

    • Search Google Scholar
    • Export Citation
  • 82

    Qureshi AIKirmani JFDivani AAHobson RW II: Carotid angioplasty with or without stent placement versus carotid endarterectomy for treatment of carotid stenosis: a meta-analysis. Neurosurgery 56:117111812005

    • Search Google Scholar
    • Export Citation
  • 83

    Ranguis SCLi DWebster AC: Perioperative epidural steroids for lumbar spine surgery in degenerative spinal disease. A review. J Neurosurg Spine