Letter to the Editor: Regarding Pediatric Hydrocephalus Guidelines

Free access

TO THE EDITOR: I was intrigued by the counterintuitive result reported by Klimo et al.5 (Klimo P Jr, Van Poppel M, Thompson CJ, et al: Pediatric hydrocephalus: systematic literature review and evidence-based guidelines. Part 6: Preoperative antibiotics for shunt surgery in children with hydrocephalus: a systematic review and meta-analysis. JNeurosurg Pediatr (Suppl) 14:44-52, November 2014) in the Pediatric Hydrocephalus Guideline paper addressing preoperative antibiotic use and its apparent lack of effect in decreasing shunt infection rate. The authors are to be congratulated for completing the daunting task of evaluating 184 abstracts spanning years of neurosurgical literature,

TO THE EDITOR: I was intrigued by the counterintuitive result reported by Klimo et al.5 (Klimo P Jr, Van Poppel M, Thompson CJ, et al: Pediatric hydrocephalus: systematic literature review and evidence-based guidelines. Part 6: Preoperative antibiotics for shunt surgery in children with hydrocephalus: a systematic review and meta-analysis. JNeurosurg Pediatr (Suppl) 14:44-52, November 2014) in the Pediatric Hydrocephalus Guideline paper addressing preoperative antibiotic use and its apparent lack of effect in decreasing shunt infection rate. The authors are to be congratulated for completing the daunting task of evaluating 184 abstracts spanning years of neurosurgical literature, and for very thoughtfully selecting 7 studies of the highest quality on which the evidence basis for this important practice issue may be best founded. The conclusions, unfortunately, may be premature or, worse, actively misleading.

I am not an expert in statistical data analysis; my viewpoint mirrors that found in standard basic references on statistics (Cohen’s book3 and the following online source: http://web.stanford.edu/~kcobb/hrp259/lecture11.ppt). However, I was struck at first glance by the extremely low number of patients included in the data sets, best illustrated but certainly not unique to this particular guideline paper. The issue regards statistical power.

I agree with the assertion that most neurosurgeons choose to use antibiotics during shunt operations, with a variably reported shunt infection risk around 5%.1,2,6 A rough calculation suggests that to detect an absolute difference of 10% in shunt infection rate with 80% power if antibiotics are not used, it would be necessary to have 745 patients each in the treatment and control arms. Yet the Level 1 recommendation from the preoperative antibiotic chapter is based on 280 patients in each arm.

With n = 280, a 10% difference in infection rate would be detectable with 80% power assuming that the baseline shunt infection rate was about 15%, which is higher than typically reported and certainly well outside the institutional experience everywhere I have trained or practiced. Smaller effects of treatment or lower baseline shunt infection rates would require substantially larger numbers of data points, potentially by an order of magnitude or more.

More generally, it is a straightforward algebraic exercise to calculate how large a sample size is needed to detect a given effect (for example, a 10% reduction in shunt infections). Alternatively, one could start with a given sample size and calculate the minimum treatment effect discernible. I think the omission of this information is very serious and has a direct bearing on the conclusions that can be drawn from the data.

Qualitative consideration suggests that a very small treatment effect with perioperative antibiotic use would have a significant impact. If one assumes, conservatively, that the cost of administering one dose of antibiotic prophylaxis is about 1% of the total costs associated with treatment of a shunt infection, then there is clearly a large benefit associated with a reduction in shunt infections on the order of 1%, based exclusively on cost analysis and not even considering associated patient morbidity. The aggregation of 35 years’ worth of published data is apparently inadequate to achieve this level of precision.

The statistical power issue receives minimal recognition; the possibility of “failing to reject a false null hypothesis” is mentioned in passing5 but is not satisfactorily quantified, which would be straightforward to do. The pervasive phenomenon of underpowered yet “statistically significant” findings in the medical literature has been previously described in detail.4 To clarify: there are simply not enough patients in the shunt infection meta-analysis to detect a small but clinically very significant difference between the antibiotic and no-antibiotic groups.

The authors may have succumbed to the temptation to conflate absence of evidence with evidence of absence, but this reflects a more fundamental problem: the breakdown of the evidence-based medicine paradigm itself. Insufficient evidence, not recognized as such, is used to draw an invalid conclusion, which is characterized as being of the highest degree of certainty and published as a guideline sanctioned by the American Association of Neurological Surgeons and the American Society of Pediatric Neurosurgeons. After 35 years and hundreds of publications on the subject, this is exactly where we find ourselves with the antibiotics question for shunt surgery. Evidence-based medicine does not work if we don’t know what evidence is.

I think that incomplete application of statistical methods to form an evidence basis is actively counterproductive. Mathematically rigorous adherence to evidence-based medicine standards in this context appears frankly unrealistic given the high numbers of patients needed to study, the wide array of independent potential confounding factors for which to control, and the relative scarcity of pediatric neurosurgical disease processes. To answer relatively basic questions, an unrealistic length of time would be necessary for patient accrual, while treatment practices remain frozen in time.

At the root of this issue is perhaps a philosophical question: is it better to have some information that is flawed, or none at all? An absolute answer may not exist. My bias is towards the latter, but if the choice is otherwise, it must be made with as clear an understanding of the limitations as possible.

References

  • 1

    Ajir FLevin ABDuff TA: Effect of prophylactic methicillin on cerebrospinal fluid shunt infections in children. Neurosurgery 9:681981

  • 2

    Blum JSchwartz MVoth D: Antibiotic single-dose prophylaxis of shunt infections. Neurosurg rev 12:2392441989

  • 3

    Cohen J: Statistical Power analysis for the Behavioral sciences ed 2Mahwah, NJLawrence Erlbaum Associates1988

  • 4

    Ioannidis JPA: Why most published research findings are false. Plos Med 2:e1242005

  • 5

    Klimo P JrVan Poppel MThompson CJBaird LCDuhaime ACFlannery AM: Pediatric hydrocephalus: systematic literature review and evidence-based guidelines. Part 6: Preoperative antibiotics for shunt surgery in children with hydrocephalus: a systematic review and meta-analysis. J Neurosurg Pediatr (suppl) 14:44522014

  • 6

    McCullough DCKane JGHarleman GWells M: Antibiotic prophylaxis in ventricular shunt surgery. II. Antibiotic considerations in cerebrospinal fluid. Childs Brain 7:1901941980

Response

We thank Dr. Gump for his attentive reading of our paper and for taking the time to write his letter, but much of what he states is probably moot given our revised results and conclusion (please see our erratum notice and corresponding revised paper).

We would like to respond to several of Dr. Gump’s points. Dr. Gump’s quantitative speculation to explain results that he does not agree with has no role in a systematic review or meta-analysis. We, too, would have liked to include all studies or our own data; however, this paper strictly focused on evidence-based literature derived from a search that met described criteria. For that reason, experience or generally known facts cannot be included. Our analysis was limited to those papers that qualified for publication given the search criteria.

Dr. Gump’s sample size calculation seems correct, although one could argue that the presumption of a shunt infection rate of 5% during the 1980s and early 1990s— the time period when the 9 clinical trials included in our review were published—is questionable. A shunt infection rate of 5% today is acceptable with modern operating rooms and the use of other infection prevention strategies such as antibiotic-impregnated shunt systems or intrathecal antibiotics. Furthermore, a relative risk reduction of 10%, not the “absolute difference,” would mean reducing the infection rate from 5% to 4.5%. Is that a “small but clinically very significant” difference? That would be up to the surgeon to decide.

The only statistical manipulation that is allowed for with a systematic review is a meta-analysis. We know that our data were limited by what was available for analysis. The use of the meta-analyses and forest plots was an effort to aggregate eligible data into a group that could be analyzed to look for statistical significance, if it was present.

We appropriately raised the concern for lack of power among the various trials, contrary to the “minimal recognition” as stated by Dr. Gump. The statement, “incomplete application of statistical methods to form an evidence basis is actively counterproductive” is simply false. His assertion would allow anyone to put forth infinite numerical iterations—“what ifs”—to further an argument for or against prophylactic antibiotics or, for that matter, any clinical question. Dr. Gump states that there were insufficient patients to detect a small difference between the groups. This may be true, as our revised analysis suggests, but this is far from concrete proof.

We are constrained to what is available in the literature—no more, no less. Dr. Gump concludes his letter by maintaining that he prefers no information to information that is flawed or contains counterintuitive results. We respect his opinion, but we would like to offer an alternative viewpoint. As researchers and clinicians, we should metaphorically open doors and seek answers to our questions, even ones to which we think we know the answers. There can be no progress if such doors remain closed. The existing literature on any topic should be periodically reviewed, critiqued, and summarized, so all can formulate their own opinions.

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

DISCLOSURE The author reports no conflict of interest.

INCLUDE WHEN CITING Published online May 8, 2015; DOI: 10.3171/2014.11.PEDS14642.

© AANS, except where prohibited by US copyright law.

Headings

References

1

Ajir FLevin ABDuff TA: Effect of prophylactic methicillin on cerebrospinal fluid shunt infections in children. Neurosurgery 9:681981

2

Blum JSchwartz MVoth D: Antibiotic single-dose prophylaxis of shunt infections. Neurosurg rev 12:2392441989

3

Cohen J: Statistical Power analysis for the Behavioral sciences ed 2Mahwah, NJLawrence Erlbaum Associates1988

4

Ioannidis JPA: Why most published research findings are false. Plos Med 2:e1242005

5

Klimo P JrVan Poppel MThompson CJBaird LCDuhaime ACFlannery AM: Pediatric hydrocephalus: systematic literature review and evidence-based guidelines. Part 6: Preoperative antibiotics for shunt surgery in children with hydrocephalus: a systematic review and meta-analysis. J Neurosurg Pediatr (suppl) 14:44522014

6

McCullough DCKane JGHarleman GWells M: Antibiotic prophylaxis in ventricular shunt surgery. II. Antibiotic considerations in cerebrospinal fluid. Childs Brain 7:1901941980

TrendMD

Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 67 67 6
PDF Downloads 116 116 8
EPUB Downloads 0 0 0

PubMed

Google Scholar