Letter to the Editor. Errors in the meta-analysis of outcomes and complications of MRgFUS

Free access

If the inline PDF is not rendering correctly, you can download the PDF file here.

TO THE EDITOR: We read with interest the meta-analysis by Mohammed et al.2 on magnetic resonance–guided focused ultrasound (MRgFUS) ablations for treating tremor (Mohammed N, Patra D, Nanda A: A meta-analysis of outcomes and complications of magnetic resonance–guided focused ultrasound in the treatment of essential tremor. Neurosurg Focus 44(2):E4, February 2018). We are delighted to see such work, as the subject is of course of interest and significance, and its timing germane now that a number of relevant studies have been published.

However, we found several relevant factual errors in the representation of data from one of our publications3 cited in this work,2 which unfortunately casts a shadow on the overall analysis: 1) Our study was cited as retrospective when in fact it had been performed in a prospective manner. 2) In-procedure complications were reported for 5 patients (Table 3), although our publication reported them in only 4 patients. 3) Figure 3B suggests the occurrence of in-procedure nausea/vomiting and Fig. 3C and E suggest the occurrence of paresthesia (event rates stated as 0.071), although we did not report this. 4) Fig. 3D and F suggest that ataxia persisted at 3 and 12 months (event rates stated as 0.333 and 0.071, respectively), although our study stated that signs of ataxia “resolved within 3 months.” 5) Relatedly, we only reported a maximum follow-up of 6 months. 6) Finally, “persisting limb weakness” at 3 months after the intervention was reported in Table 4, although we did not report any paresis. We are also uncertain why our quality of life data, as measured by the standard Quality of Life in Essential Tremor (QUEST) tool and stated in the abstract, were omitted from the analysis. Additional minor inaccuracies—for example, in stating the age of our study participants inconsistently and incorrectly twice (Table 1 and text)—add to these concerns.

Even without cross-checking the accuracy of data points derived from all other included publications, one wonders about the validity of the authors’ results, although overall their conclusions might be similar to those reported in other publications.4

We would also like to express our concerns with the use of a funnel plot for the evaluation of publication bias when only 8 studies were included; in order for such a test to give valid results, it has been argued that at least 30 studies of sufficient power would be necessary, and its interpretation should therefore be done with caution.1

Meta-analyses are powerful scientific tools used to condense findings from individual studies in an accessible way, and their popularity is hence increasing steadily. However, as with other scientific endeavors, their validity depends on the accurate and meticulous execution of all necessary steps involved, and thus, the following of publication standards and guidelines is nice but ultimately pointless if primary data are not handled adequately.

References

  • 1

    Lau JIoannidis JPATerrin NSchmid CHOlkin I: The case of the misleading funnel plot. BMJ 333:5976002006

  • 2

    Mohammed NPatra DNanda A: A meta-analysis of outcomes and complications of magnetic resonance-guided focused ultrasound in the treatment of essential tremor. Neurosurg Focus 44(2):E42018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Schreglmann SRBauer RHägele-Link SBhatia KPNatchev PWegener N: Unilateral cerebellothalamic tract ablation in essential tremor by MRI-guided focused ultrasound. Neurology 88:132913332017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Schreglmann SRKrauss JKChang JWBhatia KPKägi G: Functional lesional neurosurgery for tremor: a systematic review and meta-analysis. J Neurol Neurosurg Psychiatr [epub ahead of print] 2018

    • Search Google Scholar
    • Export Citation

Response

We appreciate the interest shown by Schreglmann et al. in our article.

First, we would like to address point 3. The questions in point 3 relate to the event rates for in-procedure nausea/vomiting (Fig. 3B) and paresthesias (Fig. 3C and E), which were indicated to be 0.071, when the study did not report any cases of in-procedure nausea/vomiting or paresthesia. Yet the event rate quoted against this study in Fig. 3B, C, E, and F is 0.071. We understand the concerns raised due to this.

A meta-analysis aims at systematically distilling and connecting data from different studies in order to formulate an overall opinion regarding a given topic. A simple arithmetic average of the results from all the included studies would be erroneous. The results from studies with a small number of subjects carry a lesser significance than do those from studies with a larger number of patients. Methods for meta-analysis use a weighted average of the results, in which the larger trials have more influence than the smaller ones.1 A meta-analysis considers not only the number of events, or evident event rates, but also the number of subjects in the study to evaluate the “weight” and effect size. The raw metric data are standardized across all the studies and then subjected to pooling. A zero event rate is also given value in the meta-analysis calculation.

The event rate of 0.071 mentioned in Fig. 3 is not the raw metric event rate (i.e., number of cases/total number of cases). This event rate is calculated with the following formula.

equ1
where e = 2.718281828.

The statistical software that we used for the analysis was Comprehensive Meta-Analysis software (version 3.3.070). The software also gives the methodology for the calculation of the result. The following calculation relates to how the final effect size result of 0.071, displayed in Fig. 3B, C, E, and F, was arrived at despite having a raw event rate of 0%.

equ2
equ3
equ4
equ5
where e = 2.718281828.

The event rate was equal to 0, so 0.5 was added to event and nonevent values for computation of the logit event rate and its variance.

equ6
equ7
equ8
equ9

Similarly, for the study by Elias et al.,2 with a raw event rate of 93%, applying the formula yields the effective event rate of 0.93, which in this case is the same as the raw event rate.

equ10
equ11
equ12
equ13
where e = 2.718281828.
equ14
equ15
equ16
equ17

We do understand that, on observing the forest plot, the authors were alarmed to see an event rate more than zero for something that was not reported. However, we want to assure them that the event rate was correctly entered as zero for the study by Schreglmann et al.4 and that the event rate shown in the graph is not the raw event rate. Inclusion of zero total event trials would encompass all the studies involved and provide a more generalizable estimate of treatment effect.3

The event rate of 0.071 shown in Fig. 3F for ataxia at 12 months and for paresthesias occurring between 0 and 3 months and at 12 months (Fig. 3C and E) is also explained by the calculations shown above. The nature of the study by Schreglmann et al. is prospective and not retrospective, as they state in point 1.

Schreglmann et al. also express concerns regarding the non-inclusion of quality of life data from their study in our analysis. We attempted to analyze the quality of life outcomes from studies that used the QUEST questionnaire. The Methods section of the paper by Schreglmann et al.4 states, “At study visits before and 48 hours and 1, 3, and 6 months after intervention, full neurologic examination, Clinical Rating Scale for Tremors, Quality of Life in Essential Tremor, and manual dexterity (9-hole peg test) were documented.” Although the authors mention assessment of quality of life in essential tremor, the methodology for assessment and the scale used were not clear, and hence this aspect of the study was not included in our analysis on the quality of life in essential tremor. However, now that we are aware that the QUEST (Quality of Life in Essential Tremor) questionnaire was used, we included the study in the analysis and found the pooled improvement in the QUEST score to be 50.7% (random-effects model). The use of a funnel plot for the analysis of bias has its own limitations. We agree that this is a meta-analysis of a small number of studies, and it will have its inherent bias for this reason. This is a limitation of this study.

In our study, we grouped the complications into 3 time frames: in-procedure complications, complications occurring between 0 and 3 months postoperatively, and late complications that occur between 6 and 12 months after the procedure. Some studies reported outcomes at 6 months, while others have reported them at 12 months. We grouped the complications occurring at 6 months and at 12 months together. The subsection describing complications at 3 months includes the complications occurring between 0 and 3 months postprocedure. Regarding points 4 and 5 raised by Schregelmann et al., Fig 3D includes an event rated of 0.333 because of their case in which the patient developed ataxia in the time frame of 0–3 months and in whom the ataxia resolved. We apologize for the confusion due to labeling of the figure. The event rate of 0.071 shown in Fig. 3F for ataxia at 12 months can be explained with the same explanation given for point 3. The study by Schreglmann et al. does not report any case of ataxia at 12 months. The event rate of 0.071 is not the raw event rate but a logit transformed event rate.

Regarding point 5 in the letter from Schreglmann et al., they reported in their study4 that one patient developed hand clumsiness that resolved within 3 months. This case was included in the “Limb Weakness” column of our Table 4. We apologize for the confusion in the title of this table, which should have been: “Complications occurring between 0 and 3 months (resolved and persisting).”

In addition, the mean age of patients included in the study by Schreglmann et al. was incorrectly cited in Table 1 as 70 ± 8.5 years and in the Results section (page 3, line 4) as 71 ± 8.3 years. In both places it should have been listed as 70.7 ± 8.5 years. We thank Schreglmann et al. for bringing to our attention in point 2 that the in-procedure complications listed in Table 3 should be 4 and not 5.4 The revised pooled estimate of in-procedure dizziness is 43.4% and does not alter the conclusions of the study.

We thank the editor for giving us an opportunity to clarify the questions that were raised.

References

  • 1

    Egger MSmith GDPhillips AN: Meta-analysis: principles and procedures. BMJ 315:153315371997

  • 2

    Elias WJHuss DVoss TLoomba JKhaled MZadicario E: A pilot study of focused ultrasound thalamotomy for essential tremor. N Engl J Med 369:6406482013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Friedrich JOAdhikari NKBeyene J: Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol 7:52007

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Schreglmann SRBauer RHagele-Link SBhatia KPNatchev PWegener N: Unilateral cerebellothalamic tract ablation in essential tremor by MRI-guided focused ultrasound. Neurology 88:132913332017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

INCLUDE WHEN CITING DOI: 10.3171/2018.3.FOCUS1898.

Disclosures The authors report no conflict of interest.

© AANS, except where prohibited by US copyright law.

Headings

References

  • 1

    Lau JIoannidis JPATerrin NSchmid CHOlkin I: The case of the misleading funnel plot. BMJ 333:5976002006

  • 2

    Mohammed NPatra DNanda A: A meta-analysis of outcomes and complications of magnetic resonance-guided focused ultrasound in the treatment of essential tremor. Neurosurg Focus 44(2):E42018

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Schreglmann SRBauer RHägele-Link SBhatia KPNatchev PWegener N: Unilateral cerebellothalamic tract ablation in essential tremor by MRI-guided focused ultrasound. Neurology 88:132913332017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 4

    Schreglmann SRKrauss JKChang JWBhatia KPKägi G: Functional lesional neurosurgery for tremor: a systematic review and meta-analysis. J Neurol Neurosurg Psychiatr [epub ahead of print] 2018

    • Search Google Scholar
    • Export Citation
  • 1

    Egger MSmith GDPhillips AN: Meta-analysis: principles and procedures. BMJ 315:153315371997

  • 2

    Elias WJHuss DVoss TLoomba JKhaled MZadicario E: A pilot study of focused ultrasound thalamotomy for essential tremor. N Engl J Med 369:6406482013

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation
  • 3

    Friedrich JOAdhikari NKBeyene J: Inclusion of zero total event trials in meta-analyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol 7:52007

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 4

    Schreglmann SRBauer RHagele-Link SBhatia KPNatchev PWegener N: Unilateral cerebellothalamic tract ablation in essential tremor by MRI-guided focused ultrasound. Neurology 88:132913332017

    • Crossref
    • PubMed
    • Search Google Scholar
    • Export Citation

TrendMD

Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 175 175 13
PDF Downloads 151 151 12
EPUB Downloads 0 0 0

PubMed

Google Scholar