Editorial: Patient validation of retrospective data

Full access

If the inline PDF is not rendering correctly, you can download the PDF file here.

In this issue of the Journal of Neurosurgery, Drewes and colleagues from Norway compare outcomes assessments from retrospective reviews of hospital records to structured phone interviews conducted 30 days postoperatively in almost 200 patients who underwent resection of intracranial tumors.4 This is an important paper because outcomes reports in the neurosurgical literature frequently rely on retrospective review of hospital records. Unfortunately, how accurate these are and how representative they are of the patient experience is not clear. While these authors found that the specificity of record review was high, suggesting that new deficits were unlikely to be present in the hospital records when they were not acknowledged by the patient, the sensitivity was disappointingly low, ranging from 0.07 to 0.52 across different neurological domains. This lack of sensitivity to neurological deficits resulting from neurosurgical treatment should not be unexpected. It has been well shown that adverse events are typically higher in randomized controlled trials (RCTs), where specific efforts are made to document outcomes, than in retrospective case series.3,6 This study demonstrates the unreliability of such retrospective reviews and calls for validated measures of neurosurgical morbidity.

The primary use of retrospective studies as evidence for procedural efficacy and patient outcomes has been a major limitation of the neurosurgical literature. These studies, which provide only Level III, IV, or V evidence account for 89.8% of all studies in top neurosurgery journals.8 Such studies, while easy to perform, suffer from numerous limitations, including selection bias, missing data, and reliance on the accuracy of clinical records. As a result, these studies have been shown to be significantly flawed with respect to the reporting of clinical diagnoses, procedure types and frequency, and complication rates.2,7 With increasing awareness of the limitations of retrospective reviews, steps towards improving the quality of neurosurgery research have been made through the establishment of prospective registries such as the National Neurosurgery Quality and Outcomes Database (N2QOD). Although data from such registries are often reported as retrospective reviews of prospectively collected data, they offer an improvement over traditional retrospective studies. Still, these studies should rarely be used to compare outcomes from different treatments unless they can demonstrate that the patient cohorts being compared were equally likely to be assigned to the different treatment arms—this is absolutely key.

An important point noted by Drewes et al. is the utility of patient-reported outcomes (PROs). These measures are reports of a patient’s health status that come from the patient without further interpretation by a clinician. PROs have proven valuable in a variety of aspects of clinical research, such as adverse event monitoring and evaluation of patient quality of life. As a result, nearly a quarter of US drug products approved from 2006 to 2010 were granted treatment benefit claims based on PROs.5 Due to the importance of such data, various initiatives have also been started by the National Institutes of Health and the Food and Drug Administration to improve the reporting of PROs and identify and validate existing PRO instruments. Currently, the quality of reporting and analysis of PROs in RCTs remains variable, leading to a CONSORT (Consolidated Standards of Reporting Trials) Statement extension outlining 5 items that are recommended for RCTs in which PROs are primary or secondary end points.1

The use of PROs in neurosurgical studies has thus far been limited and primarily exists in literature reporting patient outcomes following spine surgery. They have uncommonly been used in retrospective cohort studies to report adverse effects, resulting in disparities between the complication rates noted in these studies and large prospective trials, as the authors mention. Furthermore, PROs have rarely been reported in cerebrovascular and neurooncology studies, because the primary focus has typically been on variables such as perioperative complications, recurrence or rebleeding rates, need for reoperation, and overall survival, among others. While important, these data points do not allow for the evaluation of the impact of our surgical and medical treatments on a patient’s quality of life. With the increasing emphasis by regulators, policymakers, and insurance companies on the need to incorporate PROs in therapy risk-benefit profiles, it is critical that we begin to evaluate these outcomes in future studies to not only ensure the continued reimbursement of our care, but also to assist with selecting the optimal treatment modality for patients with neurosurgical diseases.

References

  • 1

    Calvert MBlazeby JAltman DGRevicki DAMoher DBrundage MD: Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA 309:8148222013

    • Search Google Scholar
    • Export Citation
  • 2

    Campbell PGMalone JYadla SChitale RNasser RMaltenfort MG: Comparison of ICD-9-based, retrospective, and prospective assessments of perioperative complications: assessment of accuracy in reporting. J Neurosurg Spine 14:16222011

    • Search Google Scholar
    • Export Citation
  • 3

    De Witt Hamer PCRobles SGZwinderman AHDuffau HBerger MS: Impact of intraoperative stimulation brain mapping on glioma surgery outcome: a meta-analysis. J Clin Oncol 30:255925652012

    • Search Google Scholar
    • Export Citation
  • 4

    Drewes CSagberg LMJakola ASGulati SSolheim O: Morbidity after intracranial tumor surgery: sensitivity and specificity of retrospective review of medical records compared to patient-reported outcomes at 30 days. J Neurosurg epub ahead of printAugust72015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5

    Gnanasakthy AMordin MClark MDeMuro CFehnel SCopley-Merriman C: A review of patient-reported outcome labels in the United States: 2006 to 2010. Value Health 15:4374422012

    • Search Google Scholar
    • Export Citation
  • 6

    Solheim OJakola ASUnsgård G: Scientific alchemy and proposed gold standards of care. World Neurosurg 82:e566e5672014

  • 7

    Woodworth GFBaird CJGarces-Ambrossi GTonascia JTamargo RJ: Inaccuracy of the administrative database: comparative analysis of two databases for the diagnosis and treatment of intracranial aneurysms. Neurosurgery 65:2512572009

    • Search Google Scholar
    • Export Citation
  • 8

    Yarascavitch BAChuback JEAlmenawer SAReddy KBhandari M: Levels of evidence in the neurosurgical literature: more tribulations than trials. Neurosurgery 71:113111382012

    • Search Google Scholar
    • Export Citation

Response

We would like to thank Dr. Babu and Dr. Sampson for their insightful editorial.

We very much agree that the low validity of retrospective review of medical records calls for better and validated measures of treatment-related morbidity in neurosurgery. Although we are throwing stones in our own glass house, we believe that it is time to address validity issues for common neurosurgical outcome variables. Lord Kelvin said: “If you cannot measure it, you cannot improve it.” Comparisons across neurosurgical outcome studies are usually invalid due to differences in assessment. The evidence is not improved by pooling the weaknesses in systematic reviews of heterogeneous studies of low quality and with ill-defined end points, as we have pointed out earlier.14

As neurosurgery has evolved into a more mature field, therapeutic quantum leaps are less frequent. Surgical results may nevertheless slowly improve over time due to adjustments of techniques, new technology, and improved patient selection. Because we can no longer expect to encounter enormous differences in treatment effects, sensitive and valid outcome measures are increasingly needed.

The practical, complex, and empirical nature of neurosurgery and the obstacles and limitations associated with RCTs still leave a major role for observational studies. Case series or cohort studies may serve as valuable audits for the conducting surgeons and institutions, equivalent to Phase IV drug trials (i.e., “post-market” surveillance of the procedures or tools). However, selection bias, assessment bias, and publication bias are common.16,17 RCTs account for only 3% of neurosurgical studies found in MEDLINE, and only about half of such trials are of good quality.13 While randomized controlled trials can assess efficacy (how interventions work under ideal settings in carefully selected patients), observational studies in unselected patients can assess effectiveness (how interventions work in non-ideal everyday settings).3 However, in stark contrast to medical drug studies, effectiveness studies in surgery often report much better results than stringent efficacy studies. Thus, while RCTs in medicine are criticized for boosting treatment effects through careful patient selection and close follow-up, surgical RCTs may instead be accused of unacceptable poor treatment results of little relevance to expert surgeons, proving their excellence in retrospective case series (the classic “not in my hands” argument).

Given the steady predominance of case series in neurosurgery, a priority should be to develop and embrace standards of reporting in case series and other observational studies. While harmonization of assessments with, for example, the Good Clinical Practice Guidelines (http://www.ich.org/home.html) is enforced in drug studies, attempts at classifying common measures such as complications,7 are not much embraced in neurosurgery. Validated neurological functional scales could allow for more valid and sensitive comparisons across studies than the more arbitrary or insensitive measures used today. Patient-reported outcomes at specified time points could be another step toward reducing assessment bias and emphasizing outcomes that are relevant to the patients themselves.6,12 Assessing quality of life or quality of survival in brain operations that are essentially palliative in nature may, for example, be more relevant than extent of resection. Still, patient-reported outcomes are more demanding and call for standardized follow-up of patients instead of retrospective review of medical records. Interestingly, Harvey Cush-ing already understood the value of patient-reported data, as he retained careful follow-up records by asking his patients to record their status in a letter on the anniversary of their operation.1 Disease specific questionnaires should be developed and embraced for different intracranial diseases, similar to the Oswestry Disability Index questionnaire often used for degenerative lumbar spine surgery.2 The cancer-specific health-related quality-of-life instrument EORTC QLQC-30 has a brain tumor module (BN-20)9 developed for patients with brain tumors, but unfortunately it contains mainly questions relevant for assessing adverse reactions to chemotherapy and radiotherapy. Quality-of-life questionnaires could also pave the way for more cost-effectiveness analyses since QALYs (quality-adjusted life years) and cost per QALY can be calculated.

Community-based prospective registration of all patients who underwent surgery, such as in treatment registries with validated outcome measures, can enable pragmatic comparative effectiveness studies without randomization.8,10,11,15 To overcome problems with imbalance between study groups, propensity matching4 can be carried out instead of the often flawed post hoc regression analyses commonly seen today. Compulsory registration of protocols prior to data release from treatment registries could further reduce data-driven fishing trips suspected in some observational studies.

In a famous editorial in The Lancet several years ago, Richard Horton compared surgical research to the comic opera, with many questions and few answers, and claimed that surgical research is often of questionable value due to the poor methodological quality.5 Editorial boards and reviewers have a responsibility to improve the value of neurosurgical research by raising standards and embracing new higher standards to ensure that the comic opera has a happy ending.

References

  • 1

    Cohen-Gadol AALaws ERSpencer DDDe Salles AA: The evolution of Harvey Cushing’s surgical approach to pituitary tumors from transsphenoidal to transfrontal. J Neurosurg 103:3723772005

    • Search Google Scholar
    • Export Citation
  • 2

    Fairbank JCPynsent PB: The Oswestry Disability Index. Spine (Phila Pa 1976) 25:294029522000

  • 3

    Haynes B: Can it work? Does it work? Is it worth it? The testing of healthcareinterventions is evolving. BMJ 319:6526531999

  • 4

    Hemmila MRBirkmeyer NJArbabi SOsborne NHWahl WLDimick JB: Introduction to propensity scores: A case study on the comparative effectiveness of laparoscopic vs open appendectomy. Arch Surg 145:9399452010

    • Search Google Scholar
    • Export Citation
  • 5

    Horton R: Surgical research or comic opera: questions, but few answers. Lancet 347:9849851996

  • 6

    Jakola ASGulati SWeber CUnsgard GSolheim O: Postoperative deterioration in health related quality of life as predictor for survival in patients with glioblastoma: a prospective study. PLoS One 6:e285922011

    • Search Google Scholar
    • Export Citation
  • 7

    Landriel Ibanez FAHem SAjler PVecchi ECiraolo CBaccanelli M: A new classification of complications in neurosurgery. World Neurosurg 75:7097152011

    • Search Google Scholar
    • Export Citation
  • 8

    Magge DGooding WChoudry HSteve JSteel JZureikat A: Comparative effectiveness of minimally invasive and open distal pancreatectomy for ductal adenocarcinoma. JAMA Surg 148:5255312013

    • Search Google Scholar
    • Export Citation
  • 9

    Mauer MStupp RTaphoorn MJCoens COsoba DMarosi C: The prognostic value of health-related quality-of-life data in predicting survival in glioblastoma cancer patients: results from an international randomised phase III EORTC Brain Tumour and Radiation Oncology Groups, and NCIC Clinical Trials Group study. Br J Cancer 97:3023072007

    • Search Google Scholar
    • Export Citation
  • 10

    Nerland UJakola ASolheim OWeber CRao VLønne G: Minimally invasive decompression versus open laminectomy for central stenosis of the lumbar spine: pragmatic comparative effectiveness study. BMJ 350:h16032015

    • Search Google Scholar
    • Export Citation
  • 11

    Nerland USJakola ASSolheim OWeber CRao VLønne G: Comparative effectiveness of microdecompression and laminectomy for central lumbar spinal stenosis: study protocol for an observational study. BMJ Open 4:e0046512014

    • Search Google Scholar
    • Export Citation
  • 12

    Sagberg LMJakola ASSolheim O: Quality of life assessed with EQ-5D in patients undergoing glioma surgery: what is the responsiveness and minimal clinically important difference?. Qual Life Res 23:142714342014

    • Search Google Scholar
    • Export Citation
  • 13

    Scholler KLicht STonn JCUhl E: Randomized controlled trials in neurosurgery—how good are we?. Acta Neurochir (Wien) 151:5195272009

    • Search Google Scholar
    • Export Citation
  • 14

    Solheim OJakola ASUnsgård G: Scientific alchemy and proposed gold standards of care. World Neurosurg 82:e566e5672014

  • 15

    Suri RMVanoverschelde JLGrigioni FSchaff HVTribouilloy CAvierinos JF: Association between early surgical intervention vs watchful waiting and outcomes for mitral regurgitation due to flail mitral valve leaflets. JAMA 310:6096162013

    • Search Google Scholar
    • Export Citation
  • 16

    Syin DWoreta TChang DCCameron JLPronovost PJMakary MA: Publication bias in surgery: implications for informed consent. J Surg Res 143:88932007

    • Search Google Scholar
    • Export Citation
  • 17

    Yoshimoto Y: Publication bias in neurosurgery: lessons from series of unruptured aneurysms. Acta Neurochir (Wien) 145:45482003

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

ACCOMPANYING ARTICLE DOI: 10.3171/2014.12.JNS142206.

INCLUDE WHEN CITING Published online August 7, 2015; DOI: 10.3171/2015.3.JNS142959.

DISCLOSURE The authors report no conflict of interest.

© AANS, except where prohibited by US copyright law.

Headings

References

  • 1

    Calvert MBlazeby JAltman DGRevicki DAMoher DBrundage MD: Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA 309:8148222013

    • Search Google Scholar
    • Export Citation
  • 2

    Campbell PGMalone JYadla SChitale RNasser RMaltenfort MG: Comparison of ICD-9-based, retrospective, and prospective assessments of perioperative complications: assessment of accuracy in reporting. J Neurosurg Spine 14:16222011

    • Search Google Scholar
    • Export Citation
  • 3

    De Witt Hamer PCRobles SGZwinderman AHDuffau HBerger MS: Impact of intraoperative stimulation brain mapping on glioma surgery outcome: a meta-analysis. J Clin Oncol 30:255925652012

    • Search Google Scholar
    • Export Citation
  • 4

    Drewes CSagberg LMJakola ASGulati SSolheim O: Morbidity after intracranial tumor surgery: sensitivity and specificity of retrospective review of medical records compared to patient-reported outcomes at 30 days. J Neurosurg epub ahead of printAugust72015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 5

    Gnanasakthy AMordin MClark MDeMuro CFehnel SCopley-Merriman C: A review of patient-reported outcome labels in the United States: 2006 to 2010. Value Health 15:4374422012

    • Search Google Scholar
    • Export Citation
  • 6

    Solheim OJakola ASUnsgård G: Scientific alchemy and proposed gold standards of care. World Neurosurg 82:e566e5672014

  • 7

    Woodworth GFBaird CJGarces-Ambrossi GTonascia JTamargo RJ: Inaccuracy of the administrative database: comparative analysis of two databases for the diagnosis and treatment of intracranial aneurysms. Neurosurgery 65:2512572009

    • Search Google Scholar
    • Export Citation
  • 8

    Yarascavitch BAChuback JEAlmenawer SAReddy KBhandari M: Levels of evidence in the neurosurgical literature: more tribulations than trials. Neurosurgery 71:113111382012

    • Search Google Scholar
    • Export Citation
  • 1

    Cohen-Gadol AALaws ERSpencer DDDe Salles AA: The evolution of Harvey Cushing’s surgical approach to pituitary tumors from transsphenoidal to transfrontal. J Neurosurg 103:3723772005

    • Search Google Scholar
    • Export Citation
  • 2

    Fairbank JCPynsent PB: The Oswestry Disability Index. Spine (Phila Pa 1976) 25:294029522000

  • 3

    Haynes B: Can it work? Does it work? Is it worth it? The testing of healthcareinterventions is evolving. BMJ 319:6526531999

  • 4

    Hemmila MRBirkmeyer NJArbabi SOsborne NHWahl WLDimick JB: Introduction to propensity scores: A case study on the comparative effectiveness of laparoscopic vs open appendectomy. Arch Surg 145:9399452010

    • Search Google Scholar
    • Export Citation
  • 5

    Horton R: Surgical research or comic opera: questions, but few answers. Lancet 347:9849851996

  • 6

    Jakola ASGulati SWeber CUnsgard GSolheim O: Postoperative deterioration in health related quality of life as predictor for survival in patients with glioblastoma: a prospective study. PLoS One 6:e285922011

    • Search Google Scholar
    • Export Citation
  • 7

    Landriel Ibanez FAHem SAjler PVecchi ECiraolo CBaccanelli M: A new classification of complications in neurosurgery. World Neurosurg 75:7097152011

    • Search Google Scholar
    • Export Citation
  • 8

    Magge DGooding WChoudry HSteve JSteel JZureikat A: Comparative effectiveness of minimally invasive and open distal pancreatectomy for ductal adenocarcinoma. JAMA Surg 148:5255312013

    • Search Google Scholar
    • Export Citation
  • 9

    Mauer MStupp RTaphoorn MJCoens COsoba DMarosi C: The prognostic value of health-related quality-of-life data in predicting survival in glioblastoma cancer patients: results from an international randomised phase III EORTC Brain Tumour and Radiation Oncology Groups, and NCIC Clinical Trials Group study. Br J Cancer 97:3023072007

    • Search Google Scholar
    • Export Citation
  • 10

    Nerland UJakola ASolheim OWeber CRao VLønne G: Minimally invasive decompression versus open laminectomy for central stenosis of the lumbar spine: pragmatic comparative effectiveness study. BMJ 350:h16032015

    • Search Google Scholar
    • Export Citation
  • 11

    Nerland USJakola ASSolheim OWeber CRao VLønne G: Comparative effectiveness of microdecompression and laminectomy for central lumbar spinal stenosis: study protocol for an observational study. BMJ Open 4:e0046512014

    • Search Google Scholar
    • Export Citation
  • 12

    Sagberg LMJakola ASSolheim O: Quality of life assessed with EQ-5D in patients undergoing glioma surgery: what is the responsiveness and minimal clinically important difference?. Qual Life Res 23:142714342014

    • Search Google Scholar
    • Export Citation
  • 13

    Scholler KLicht STonn JCUhl E: Randomized controlled trials in neurosurgery—how good are we?. Acta Neurochir (Wien) 151:5195272009

    • Search Google Scholar
    • Export Citation
  • 14

    Solheim OJakola ASUnsgård G: Scientific alchemy and proposed gold standards of care. World Neurosurg 82:e566e5672014

  • 15

    Suri RMVanoverschelde JLGrigioni FSchaff HVTribouilloy CAvierinos JF: Association between early surgical intervention vs watchful waiting and outcomes for mitral regurgitation due to flail mitral valve leaflets. JAMA 310:6096162013

    • Search Google Scholar
    • Export Citation
  • 16

    Syin DWoreta TChang DCCameron JLPronovost PJMakary MA: Publication bias in surgery: implications for informed consent. J Surg Res 143:88932007

    • Search Google Scholar
    • Export Citation
  • 17

    Yoshimoto Y: Publication bias in neurosurgery: lessons from series of unruptured aneurysms. Acta Neurochir (Wien) 145:45482003

TrendMD

Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 172 172 7
PDF Downloads 183 183 9
EPUB Downloads 0 0 0

PubMed

Google Scholar