Building an electronic health record integrated quality of life outcomes registry for spine surgery

Full access

OBJECT

Demonstrating the value of spine care requires adequate outcomes assessment. Long-term outcomes are best measured as overall improvement in quality of life (QOL) after surgical intervention. Present registries often require parallel data entry, introducing inefficiencies and limiting compliance. The authors detail the methodology of constructing an integrated electronic health record (EHR) system to collect QOL metrics and demonstrate the effect of data collection on routine clinical workflow. A streamlined approach to collecting QOL data can capture patient data without requiring dual data entry and without increasing clinic visit times.

METHODS

Through extensive literature review, a combination of QOL assessments was selected, consisting of the Patient Health Questionnaire-2 and -9, Oswestry Disability Index, Neck Disability Index, and visual analog scale for pain. These metrics were used to provide assessment of QOL following spine surgery and were incorporated into standard clinic workflow by a multidisciplinary team of surgeons, advanced practice providers, and health care information technology specialists. A clinical dashboard tracking more than 25 patient variables was developed. Clinic flow was assessed and opportunities for improvement reviewed. Duration of clinic visits before and after initiation of QOL measure capture was recorded, with assessment of mean clinic visit times for the 12 months before and the 12 months after implementation.

RESULTS

The integrated QOL capture was instituted for 3 spine surgeons in a tertiary care academic center. In the 12-month period prior to initiating collection of QOL data, 806 new patient visits were completed with an average visit time of 127.9 ± 51.5 minutes. In the 12 months after implementation, 1013 new patient visits were recorded, with 791 providing QOL measures with an average visit time of 117.0 ± 45.7 minutes. Initially the primary means of collecting patient outcome data was via paper form, with gradual transition to collection via entry into the electronic medical records system. To improve electronic data capture, paper forms were eliminated and an online portal used as part of the patient rooming process. This improved electronic capture to nearly 98% without decreasing the number of patients enrolled in the process.

CONCLUSIONS

A systematic approach to collecting spine-related QOL data within an EHR system is feasible and offers distinct advantages over registries that require dual data entry. The process of data collection does not impact patients’ clinical visit or providers’ clinical workflow. This approach is scalable, and may form the foundation for a decentralized outcomes registry network.

ABBREVIATIONSAPP = advanced practice provider; CPT = Current Procedural Terminology; DRG = diagnosis-related group; DSM-IV = Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; EDW = enterprise data warehouse; EHR = electronic health record; ICD-9-CM = Classification of Diseases, Ninth Revision, Clinical Modification; MA = medical assistant; MS-DRG = Medicare Severity-DRG; NDI = Neck Disability Index; NSQIP = National Surgical Quality Improvement Program; N2QOD = National Neurosurgery Quality and Outcomes Database; ODI = Oswestry Disability Index; PHQ = Patient Health Questionnaire; QOL = quality of life; VAS = visual analog scale for pain.

OBJECT

Demonstrating the value of spine care requires adequate outcomes assessment. Long-term outcomes are best measured as overall improvement in quality of life (QOL) after surgical intervention. Present registries often require parallel data entry, introducing inefficiencies and limiting compliance. The authors detail the methodology of constructing an integrated electronic health record (EHR) system to collect QOL metrics and demonstrate the effect of data collection on routine clinical workflow. A streamlined approach to collecting QOL data can capture patient data without requiring dual data entry and without increasing clinic visit times.

METHODS

Through extensive literature review, a combination of QOL assessments was selected, consisting of the Patient Health Questionnaire-2 and -9, Oswestry Disability Index, Neck Disability Index, and visual analog scale for pain. These metrics were used to provide assessment of QOL following spine surgery and were incorporated into standard clinic workflow by a multidisciplinary team of surgeons, advanced practice providers, and health care information technology specialists. A clinical dashboard tracking more than 25 patient variables was developed. Clinic flow was assessed and opportunities for improvement reviewed. Duration of clinic visits before and after initiation of QOL measure capture was recorded, with assessment of mean clinic visit times for the 12 months before and the 12 months after implementation.

RESULTS

The integrated QOL capture was instituted for 3 spine surgeons in a tertiary care academic center. In the 12-month period prior to initiating collection of QOL data, 806 new patient visits were completed with an average visit time of 127.9 ± 51.5 minutes. In the 12 months after implementation, 1013 new patient visits were recorded, with 791 providing QOL measures with an average visit time of 117.0 ± 45.7 minutes. Initially the primary means of collecting patient outcome data was via paper form, with gradual transition to collection via entry into the electronic medical records system. To improve electronic data capture, paper forms were eliminated and an online portal used as part of the patient rooming process. This improved electronic capture to nearly 98% without decreasing the number of patients enrolled in the process.

CONCLUSIONS

A systematic approach to collecting spine-related QOL data within an EHR system is feasible and offers distinct advantages over registries that require dual data entry. The process of data collection does not impact patients’ clinical visit or providers’ clinical workflow. This approach is scalable, and may form the foundation for a decentralized outcomes registry network.

ABBREVIATIONSAPP = advanced practice provider; CPT = Current Procedural Terminology; DRG = diagnosis-related group; DSM-IV = Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; EDW = enterprise data warehouse; EHR = electronic health record; ICD-9-CM = Classification of Diseases, Ninth Revision, Clinical Modification; MA = medical assistant; MS-DRG = Medicare Severity-DRG; NDI = Neck Disability Index; NSQIP = National Surgical Quality Improvement Program; N2QOD = National Neurosurgery Quality and Outcomes Database; ODI = Oswestry Disability Index; PHQ = Patient Health Questionnaire; QOL = quality of life; VAS = visual analog scale for pain.

Degenerative spine disease is a common and debilitating condition with significant economic costs. Direct medical costs due to low-back pain doubled to $102 billion between 1997 and 2005,14 and the number of lumbar fusion procedures for degenerative lumbar disease has quadrupled in the past 20 years, resulting in increased health care spending. Some authors note that these increases in surgical procedures have not been matched by improved overall patient outcomes.6,10 Although spine surgery has been shown to provide greater value for certain conditions,25,26 there is a significant need to develop informative outcomes assessments to better define its value.

Functional outcomes and quality of life (QOL) measures are vital to defining patient outcomes in operative care.19 Elective spinal surgery is preference sensitive; reported postoperative QOL and functional status measures are excellent end points to determine value of surgical care provided. Major payers for health care, such as the Centers for Medicare & Medicaid Services, are becoming increasingly focused upon assessing the value of care, studying outcomes, and correlating dollars spent to expected outcomes as assessed by longitudinal data.

Patient registries are a valuable approach to demonstrating the value of spine care. Present registries range from single-institution to large health system and even national level data.4,5 Registry data have been used to demonstrate the value of spine care.2,4,16 However, present registry efforts require redundant data entry. Clinical data used for patient care are either recorded on a paper patient chart or in an electronic health record (EHR), and registry data are separately recorded through an online portal or paper form. This requires redundant entry of the same elements into different data environments, introduces inefficiencies that may impair compliance, presents potential for data entry errors, and may limit implementation of registries through poor clinician compliance.

Parallel data entry is the primary method used to collect clinical measures in spine patient registry efforts. Routinely, questionnaires are administered to patients and results are manually entered into a registry to be combined with other outcomes data. Patient demographics, comorbidities, surgical procedure, and other clinical data must be reentered into the registry as well. Quality of care and outcomes data are recorded in parallel with patient care.

A system for capturing outcomes data directly into the same EHR used for patient care has not been developed; the challenges of working within a given EHR’s limitations have been considered too great to overcome. This unfortunately relegates capture of outcomes data to a second order status, with direct patient care the primary focus of clinic processes.

We explore the viability of capturing spine surgery outcomes data as part of clinic routine, with direct entry into a clinical EHR used for patient care. In this system, the QOL measures captured are immediately available for clinician review. Abstraction and assessment are performed on data available in the existing EHR utilizing an enterprise server clinical dashboard developed in parallel to the QOL capture process. All of the data recorded in the EHR are potentially available for extraction into the enterprise server, as opposed to a select set of variables chosen for registry recording.

To aid other institutions considering a similar approach to capturing patient QOL data through an integrated EHR process, we review our development process, focusing upon the individual steps required for successful implementation. Challenges faced by our team are reviewed. Two primary goals of our effort were to not adversely impact clinic times with the addition of QOL capture and to move to a purely electronic system. We review the results of process changes implemented to achieve these goals.

Methods

Choice of QOL Measures for Capture

Through a literature review and an assessment of patient opinion of which outcome metrics are most valuable, we identified validated instruments to measure and determine QOL associated with spinal disease. We concluded that a combination of instruments would provide the best understanding of postoperative QOL and selected the Patient Health Questionnaire-9 (PHQ-9), Oswestry Disability Index (ODI), Neck Disability Index (NDI), and visual analog scale for pain (VAS). The methodology of our choice of QOL measures has been previously reported.13

The PHQ-9 is a self-administered screening tool for depression in patients that evaluates the 9 criteria for major depressive disorder, as outlined in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). Each question is scored from 0 (“not at all”) to 3 (“nearly every day”), allowing for a total score ranging from 0 to 27. A higher score indicates a greater degree of depression. The diagnostic validity of the PHQ-9 has been established in large multicenter analyses.22 The ODI is a well-established measure of condition-specific outcomes in spinal disorders. Although multiple versions exist, the ODI continues to provide a valid and rigorous assessment of disability. The NDI is a 10-item self-administered questionnaire that measures disability in patients with neck pain.7 Each item is scored from 0 to 5, producing a total score ranging from 0 to 50. As the score increases, so does the patient’s disability. The 10 items include pain intensity, personal care, lifting, reading, headaches, concentration, work, driving, sleeping, and recreation.23,24 The VAS assesses pain as a one-dimensional phenomenon that varies only in intensity and is commonly used to assess pain. It allows for a score range from 1–10, with 1 representing no pain and 10 indicating unbearable pain.

Establishing a Multidisciplinary Team

A multidisciplinary team consisting of a surgeon-champion, neurosurgical advanced practice providers (APPs), hospital quality improvement and clinical informatics professionals, clinic administrators, medical assistants (MAs), and new patient coordinators facilitated the implementation of this project (Table 1). Inclusion of hospital administration, quality assessment personnel, clinicians, and programmers was necessary. Incorporating stakeholders from disparate aspects of the hospital fostered idea generation, encouraged iterative improvements, and developed smooth end-to-end workflow and effective problem solving.

TABLE 1.

Team members and roles required for developing an integrated QOL capture system

Team MemberRole
Neurosurgeon-championIdentify problem & idea; provide leadership
EHR programmerDevelop & implement EHR-based modules
Neurosurgical nursing staffOversee & guide clinic implementation
Hospital administratorProvide authority & approval for implementation
Quality improvement specialistResponsible for day-to-day management; domain knowledge of quality improvement strategies
Information & data technology specialistCreate the data structures necessary to retain collected data; build clinician dashboard

Initially, a surgeon-champion partnered with a senior administrator with the shared vision of redefining the process of capturing clinical quality metrics and demonstrating the value of spine care to individual patients through an integrated data entry effort. Addition of the patient QOL metrics required expertise of an EHR programmer. Clinical staff, including both APPs and MAs, were involved from the outset of the project to guide implementation.

Developing effective data extraction methodology was one of the initial challenges faced by the team. Inability to extract and benchmark QOL data captured into the EHR was an early concern; challenges with aggregating EHR data have been described.12 To address these issues, information technology team members worked in parallel to develop a clinical dashboard that would allow for easy extraction of aggregate information from patient populations.

EHR Programming

We built a patient-input QOL questionnaire using a proprietary EHR questionnaire editor, working with the Epic MyHealth platform (Epic Systems). EHR programming allowed inclusion of any new patient visit to the spine clinic, determined as a visit with an International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) code denoting spine pathology and a visit to any of the 3 clinicians involved in the spine division, resulting in that patient’s addition to a patient set within the EHR. New patient visits, when scheduled by a new patient coordinator, triggered a QOL questionnaire to be sent to each patient 30 days prior to the clinic visit via the MyHealth patient portal (Fig. 1). Initially, for patients who did not complete the QOL baseline assessment via the MyHealth patient portal, the QOL questionnaire was completed by electronic tablet or was manually entered by clinic staff. The QOL questionnaires are, at present, only available in English.

FIG. 1.
FIG. 1.

This secure patient portal allows patients to complete baseline and follow-up questionnaires remotely. Copyright 2014 Epic Systems Corporation. Used with permission.

The surgical intervention was defined as an anchor event and the EHR was programmed to deliver the QOL questionnaire at discrete time points (3, 6, and 12 months postoperatively). Patients had a 60-day window to complete the questionnaire. The baseline QOL metrics were immediately available for clinician review as part of the Epic EHR display.

Incorporating QOL Capture into Clinic Flow

New patient coordinators requested that all new spine patients complete questionnaires via the digital patient portal prior to their clinic visit. We designed an intuitive and patient-friendly system within the online portal to accommodate a diverse patient population. We met with a neurosurgery Patient Advisory Council (PAC) to review the user friendliness of the questionnaires. Feedback from the PAC was integrated into the formatting of the questionnaires in the MyHealth portal as well as into the paper copies initially used in clinic. The members of the PAC agreed that patients would have an incentive to complete the QOLs online if they knew that participation would lead to improved patient care and outcomes in the future.

A check-in process was established that allowed MAs to rapidly determine if questionnaires had been completed. A prompt would present to the front desk staff if a QOL questionnaire was required for a given patient, with a notation indicating whether the questionnaire had been completed. If the questionnaire was not completed at the time of the visit, clinic staff assisted the patient in completion of the questionnaire. Electronic entry using the digital tablet was offered first to all patients, but manual completion and data entry of the questionnaire were also available.

When initial implementation yielded overreliance upon paper charting, the clinic pathway was altered to incorporate direct patient data entry of QOL elements into the Epic environment via clinic exam room computer. A shell EHR questionnaire was developed with limited access to EHR protected health information. The downtime while patients were waiting for a provider was used for QOL capture.

Evaluation of Clinic Flow

To accomplish the goal of capturing these clinical metrics without increasing the amount of time required for a new patient visit, we examined the clinic intake, evaluation, and discharge processes. We developed a list of items required as part of the patient rooming process by the MAs: importing of outside radiology CDs into the internal viewing system, verification of primary care physician and pharmacy data, registration for MyHealth, completion of new patient paperwork, and recording of vitals.

While all of these elements were eventually completed on all new patients seen, there was no standard workflow and no routine for the patient intake process. Sometimes patients would be roomed without their outside radiology CDs being imported, creating delays for the APPs and physicians. Individual elements of the intake process could be omitted, and MAs who were not familiar with clinic processes would often not complete all elements of the rooming process.

We sought to improve the efficiency of the patient rooming process by creating a standardized workflow, summarized by a checklist that had to be completed with all new patient visits (Fig. 2). By decreasing variation and standardizing processes, we felt enough time savings from a new patient visit would be achieved to offset the additional time required by QOL capture.

FIG. 2.
FIG. 2.

A patient intake checklist was developed to standardize work flow during the patient rooming process.

Building a Clinician-Facing Dashboard

An institution-specific enterprise data warehouse (EDW) that directly extracts data from the Epic EHR was developed in parallel to the QOL capture process. There was considerable discussion at the outset of our project development that the effort could yield inputting of considerable volumes of additional QOL data into an EHR, while not allowing aggregation and access to this data in the future.

The source data within the EDW were aggregated and processed into a subject area mart (SAM) focused on patient-reported outcomes of our spine patient population. Essentially all unique fields within the Epic environment are eligible for inclusion in the EDW. The output of the EDW and SAM is a multifaceted clinical dashboard (Fig. 3).

FIG. 3.
FIG. 3.

Clinician-facing dashboard provides the ability to monitor spine patient population in real time with a variety of filters including demographic criteria, procedure performed (by CPT or MS-DRG code), and operating surgeon.

To structure the output of the EDW dashboard into a frame relevant to clinicians, the dashboard was developed to permit searches of patient records based upon Common Procedural Terminology (CPT) code, patient name or medical record number, and provider. Output of the database was divided into individual and aggregate QOL metrics, complication occurrence, follow-up, length of stay, and patient satisfaction based upon a Press Ganey survey. We chose approximately 25 common patient and procedure variables, including patient comorbidities, body mass index, procedure, and other variables (Table 2). The dashboard allows observation of granular data, whether from a single patient, an individual surgeon’s patients, or a specific procedure.

TABLE 2.

Limited list of variables available for query of registry

CPT Code (any combination using OR/AND to combine)
Procedure
 MS-DRG
 Admit type
 Discharge department
 Degenerative diagnosis
 Cervical diagnosis
 Thoracic diagnosis
 Trauma flag
 Payer class
Patient
 Name
 Patient number
 Spine procedure flag
 Age group
 Sex
 10-year expected survival rate (Charlson method)
 Comorbidities
 Tobacco use
 Alcohol use
 Illicit drug use
 BMI group
Provider name
BMI = body mass index.

Both CPT and diagnosis-related groups (DRGs) were available for procedure, but we believed that the DRG designations were inadequate and did not provide adequate granularity. The CPT search function permitted the combination of different search strings such that an individual search could obtain baseline and follow-up QOL metrics on either a simple lumbar laminectomy (63047), a laminectomy with a posterolateral fusion (63047 and 22612), or a laminectomy with an instrumented posterolateral fusion (63047, 22612, and 22840). DRGs were obtained from hospital billing records recorded in the EHR, and CPT data were obtained from surgeon billing data, also recorded in Epic. By using the common parlance of CPT coding, the output of the dashboard is structured in an easily understandable format. Combination of filters is possible, with the addition of provider and procedure filters to the dataset. The dashboard was used for aggregating data from a variety of practitioners and patients, and was not used for direct patient care.

Choice of Patient Population

To limit potential confounding of data collection by patient presentation and focus upon elective patients, where there was the greatest potential for process improvements, we chose to restrict our population to patients presenting for evaluation for elective surgery for degenerative disease. We excluded patients with diagnoses of neoplasm, preoperative infection (osteomyelitis, discitis, epidural abscess), and trauma. All new patients presenting for evaluation were included in the assessment.

While our capture included both operative and nonoperative new patients, follow-up information is only captured for patients who have undergone surgery at 3, 6, and 12 months postoperatively. Patients with nonspine pathology (e.g., epilepsy, peripheral nerve, tumor) were excluded from the assessment.

Maintenance of an Iterative Improvement Process

By establishing a regular meeting time for the interdisciplinary team prior to, during, and following clinic implementation of the single-stream spine QOL registry, we were able to rapidly identify and resolve challenges that arose. Frequent reevaluation of the process coupled with the real-time data on patient participation in this initiative provided the team with the information required to continue improving this model.

Capture of patient QOL metrics, number of QOL questionnaires completed, and goals for implementation were incorporated into a Visibility Board in the clinic, allowing for immediate feedback to clinic MAs and clinicians and providing an opportunity for positive reinforcement of successful QOL metric capture over time.

Assessing the Impact of Implementation on Clinic Flow

To track the impact of implementation of our standardized rooming checklist and our QOL capture method on clinic flow and time for individual new patient clinic visits, we measured visit times recorded in the Epic environment. Time of patient check-in was directly compared with time of patient check-out, and the duration of a new patient visit was assessed. Patients for whom there was no clear check-in or check-out time were eliminated from this assessment.

We assessed all new patients seen by the involved 3 clinicians in the 12 months before and after implementation. The assessment necessarily captured operative and nonoperative new patient evaluations. Baseline data similarly captured all new patients. We included nonspine patients in the assessment as well because there is no mechanism within our EHR to extract clinic times limiting by ICD-9 diagnosis coding.

Tracking Use of Electronic Data Entry

To move toward a purely electronic means of data entry, we initially used the MyHealth portal and encouraged patients to obtain an account as part of their new patient intake process. When this led to limited results, we began using an in-clinic computer-based QOL capture, using the computers in the individual exam rooms and a secure EHR access session monitored by clinic staff. We tracked overall completion of QOL questionnaires and means of completion—paper, electronic prior to clinic visit, and electronic during clinic visit. Rates were tracked over time and captured as part of the dashboard system.

Results

Shared Data Environment

Here we depict a secure online interface for patient entry of questionnaire content that was accessible by computer or tablet (Fig. 1). QOL questionnaires were successfully built into the existing MyHealth patient portal. This portal linked directly with the hospital EHR and facilitated real-time analysis of patient QOL. The clinical dashboard was successfully completed and acts as a means of easy extraction of aggregated data from groups of clinic patients (Fig. 3).

Status of Patient Data Collection Since Registry Implementation

In the first 12 months after beginning QOL capture, there were 791 patients seen in spine clinic who provided QOL measures. Mean age was 57.8 years, and 372 patients (47%) were male. The capture rate was 78% (791 of 1013 patients). During the first 12 months of the effort, the primary means of capturing new patient outcome metrics was via paper form (86%), with limited use of tablet and email electronic data capture. We set a goal of 65% capture of our overall spine practice, allowing for the patient exclusions reviewed earlier.

With institution of exam room survey completion, we saw a substantial increase in electronic completion of surveys, with 98% completion via computer survey. We achieved complete elimination of paper surveys in Month 14 after beginning the QOL assessment process (Fig. 4).

FIG. 4.
FIG. 4.

Screenshot of dashboard showing detailed view of rates of survey completion. Adoption of new techniques in clinic led to elimination of paper-based surveys and transition to a purely EHR-based outcomes capture.

Impact on Clinic Workflow

Addition of patient QOL reporting did not increase clinic visit times. We anticipated that capture of QOL information would add approximately 5–7 minutes to each clinic visit, based upon team members completing the check-in and QOL capture process. By standardizing clinic work flow, we anticipated offsetting this additional time and not adding to the overall time required for a new patient visit.

To track the impact of process standardization and QOL capture on clinic workflow, we assessed check-in and check-out data for new patient visits for the 3 surgeons participating. Data were obtained from our Epic EHR for the 12 months prior to implementation (9/23/12 to 9/22/13) and for the 12 months after implementation (9/23/13 to 9/22/14). For the period prior to implementation, 806 new patient visits were recorded for the 3 surgeons. Average time from clinic check-in to check-out was 127.9 minutes (SD ± 51.5 minutes). To standardize data, we eliminated patients for whom there were no clear check-in and checkout values recorded in the EHR. For the 12 months after implementation, 1013 new patients were seen, with average clinic visit times of 115.5 minutes (SD ± 46.1 minutes). Of the 791 patients for whom QOL capture was completed, the average times for paper and electronic capture were similar (117.0; SD ± 45.7 minutes and 117.2; SD ± 42.1 minutes, respectively).

With implementation of standardized workflow and QOL data capture, we shortened average clinic visit time by approximately 10 minutes, a statistically significant decrease in clinic time (10.1%, p < 0.0001, 95% CI 9.923–25.07). During the period assessing clinic new patient visit times, there was no significant difference between patients with paper or electronic outcomes capture versus patients for whom QOLs were not recorded (p = 0.0635, 95% CI −0.378–13.578 and p = 0.3191, 95% CI −6.610–20.210, respectively).

Incorporation of Electronic Capture of QOL Data

We initially encouraged electronic capture of QOL data through the MyHealth web portal in the Epic EHR system. The sign in and account creation processes for MyHealth were limiting factors. While we internally set a goal of 75% MyHealth activation for all spine patients, we were unable to achieve sustained activation greater than 50% throughout the initial 18 months after implementation. This initially impaired our ability to convert to a purely electronic means of data capture, with most QOL questionnaires being completed on paper and then manually entered later in the day by clinic staff (Fig. 5).

FIG. 5.
FIG. 5.

Screenshot of dashboard showing detailed view of MyHealth activation rates.

This increased the opportunity for errors in data entry, slowed the clinic process by adding additional work at the end of a given clinic, and prevented individual QOL elements from being immediately available to a clinician during the patient evaluation process.

With incorporation of questionnaire completion via exam room computer facilitated by MA prompts for roomed patients, electronic completion rates increased to 98%, while capture rates remained stable (Fig. 4). Most importantly, patient QOL metrics entered electronically into the system were immediately available for clinician review. Hence, the QOL capture process provided information that could immediately influence care decisions, while also providing data for later extraction.

Utility of Clinical Dashboard

We created a clinical dashboard tracking more than 25 patient and procedure variables (Fig. 3). The dashboard allows near real time monitoring at the level of individual patients as well as specific cohorts. The dashboard is built to allow several different levels of analysis ranging from physician specific to procedure specific.

The dashboard was able to successfully extract data based upon CPT coding, DRG coding, operating surgeon, and individual patient.

Discussion

We present the first EHR-based patient outcomes registry and enterprise dashboard for spine surgery patients. Previous registry efforts have focused on establishing parallel duplicate research databases.

We developed a multidisciplinary clinical team with the goal of realizing EHR-based data collection of QOL assessments after spine surgery. Increased pressure to document the value of care in multiple clinical arenas adds impetus to this effort.

Our group used an EHR-based approach to quality assessment by embedding measures that enable prospective analysis of both clinical and patient-reported QOL outcomes. Streamlining data collection in this manner directly enhances the ability to analyze and interpret this data to improve patient care. We collected patient-reported QOL outcomes pre- and postoperatively without significantly impacting the ongoing clinic workflow.

Impetus to Capturing Quality Data

Multiple drivers are pushing greater capture of clinical outcomes measures after a variety of interventions. The paucity of outcome measures and the lack of consensus on what outcome measures should be followed in clinical practice have been emphasized.20 The International Consortium for Health Outcomes Measurement has developed multiple sets of outcome metrics for different conditions, including treatment of lumbar spine disease (see http://www.ichom.org/medical-conditions/low-back-pain/).

Present definitions of “quality” in health care often focus upon process-of-care measures or other factors that may be extracted from administrative data. It is arguable that present efforts at defining “quality” of care in spine surgery are not accurate.21

Financial penalties may be applied to institutions that do not comply with value-of-care reporting. Similarly, present systems mandate decreased reimbursement for facilities that are low quality or high cost. There is, at present, no additional reimbursement for collecting and reporting QOL data. Hence a mechanism for capture QOL data that does not require addition practice expense would be optimal.

Patient Registries

Efforts by the Society of Thoracic Surgeons (STS) demonstrate that despite the significant financial and logistical barriers to their establishment, registries present a clear opportunity to better define value in surgical care. Establishment of the STS database enabled patient risk stratification and risk prediction at the institutional and national levels, improved understanding of resource utilization, and provided data to support health policy development.27

The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) collects robust data aimed at improving surgical quality. With implementation of NSQIP, 82% of hospitals improved their morbidity rates and 66% saw improvement in mortality rates.11 NSQIP enables hospitals to use institution-specific clinical data to identify areas of improvement for the specific institution seeking to implement a quality improvement initiative. By directly collecting the data from patient records, the program avoids less reliable claims-based administrative data and prevents redundancy in collecting data in a parallel storage source.9 However, the NSQIP process does not offer capture of patient QOL metrics. The National Neurosurgery Quality and Outcomes Database (N2QOD) is a well-known, prospective clinical registry for neurosurgical patient care.3 In a recent evaluation of the efficacy of N2QOD to study outcomes after 2 common lumbar operations, the registry collected high-quality outcomes data with many patients participating in the 1-year follow-up.10,18 The N2QOD database will likely provide many additional insights into the benefits of operative care of degenerative spinal pathology.

Utilization of EHR to improve quality of spine surgery is a central feature of the Spine Carepath program of the Cleveland Clinic. Recognizing the ineffective adoption of back pain clinical guidelines by providers, the Cleveland Clinic pursued an evidence-based strategy for back care that leverages their EHR. Using structured clinical notes, providers capture key elements of treatment to provide the most effective care. Both diagnostic and therapeutic documentation are recorded and are accessible for retrospective review.15

Choice of QOL Measures

Definition of QOL for spine care following surgical intervention remains difficult due to various independent metrics with limited validation of their effectiveness when paired and combined. We employed 4 distinct metrics to capture patient QOL. The PHQ-9 screens for depression in patients on the basis of DSM-IV criteria. This tool is a well-established metric of depression, and Choi et al.5 recently reported that PHQ-9 offers a ready, patient-administered analysis of major depressive disorder (MDD) in patients with chronic spinal disorder.22 Receiver-operator curve analysis revealed an area under the curve of 0.768 for PHQ-9, consistent with 2 other screens of MDD.5 Extent of preoperative depression has been shown to influence postoperative patient satisfaction and patient-reported quality of life.1,17

The ODI and NDI provide patient-reported information on disability. The ODI is a tool for measuring disability in patients with back pain and has long been validated in determining patient outcome following spine surgery.7 The NDI is a comparable measure but assesses functional outcomes of the neck.23,24 Together, the ODI and NDI provide a comprehensive assessment of patient functional status following surgery and are an important metric when assessing QOL. Incorporation of the VAS enables assessment of pain and has demonstrated its usefulness in spine patients28

Benefits of Integrated Data Entry

Directly extracting the data from the EHR provides a single source for data collection and storage. The process of re-inputting the data into a second database is inefficient and consumes valuable resources. Furthermore, transfer of data from the EHR to a third-party registry is limited due to security issues.

This effort demonstrates that patient-centered QOL measures can be captured in a time-effective fashion without harming clinic efficiency. Streamlining our clinic rooming process through institution of standardized work flow, combined with electronic data entry of patient QOLs, decreased the time required for new visits by 10 minutes.

Present registry reporting involves capture of patient-specific data elements and QOL measures that are forwarded to central registry repositories. The registry data are then extracted and aggregated. Present registry reporting may not provide real-time access to patient QOL assessments. The importance of depression in the treatment of degenerative spinal pathology is well documented; however, if depression screening results are not available to clinicians, appropriate change in clinical pathways may not be feasible.3

Utility of a Dashboard for Data Review

EHRs have the potential to be a powerful technology that allows for effective communication, ease of data collection, and rapid data mining. Unfortunately, few of these potential benefits of EHRs have been realized. Nonetheless, increased payer pressure has led to widespread adoption of EHRs into clinical practice.8 The difficulty of extracting data from EHRs led to the parallel development of an enterprise data server that could search through the EHR database and develop reports in an efficient fashion. While present EHRs allow for capture of tremendous amounts of patient data, transforming that data into a format that is clinically useful requires additional support.

The dashboard used at our facility was developed in parallel with the implementation of EHR-based outcomes capture. The dashboard allows for easy extraction of clinical elements in a framework understood by clinicians, with procedures defined by CPT coding and search functions that allow assessment by surgeon, by procedure, or by patient. Development of a clinical dashboard insures that the QOL data entered into the EHR can be extracted and used for benchmark generation and quality assessments. The impact of clinical process changes may be followed with the QOL data extracted by the enterprise server dashboard.

Limitations

At present, our dashboard only reviews clinical data from patients in whom operative procedures have been performed. Although we have baseline data on 791 patients who have been evaluated in our spine clinic, we only track 263 patients who have had or who are scheduled to have surgery. Hence, most of the information captured in the effort to date is not presently available in our dashboard environment; this focus upon operatively treated patients is a potential source of bias. One opportunity for scaling this project is to incorporate follow-up with Physiatry or Pain Management teams at our facility, where patients undergoing conservative therapy may have similar QOL tracking.

We demonstrate that QOLs may be captured without increasing clinic visit times. However, generalizing this finding assumes that our degree of clinic efficiency is standard and representative of other spine clinics. More efficient clinics might find less benefit from the standardized workflow that was used in this effort to “buy” time for QOL capture.

Long-term capture of patient outcome metrics will be a challenge for our effort, as in any registry endeavor. Our initial approach is to evaluate the degree of postoperative capture that can be obtained with a purely automated system based only upon EHR prompts to patients. With further follow-up, it may become evident that greater focus upon capturing postoperative patients is necessary. This may require resources directed at encouraging completion of postoperative surveys by our patients.

The system is presently only available in an Epic environment. This limits the potential for scaling the effort, as the code and infrastructure are, at present, only available on a single EHR system. The challenges of integrating across systems may pose problems in future efforts. While it is likely that other EHRs have similar functionalities, this has not been explored by the authors.

While potentially improving the efficiency of spine QOL outcomes capture, this approach is, at present, only available at a single institution. This limits opportunities for data aggregation and restricts the potential growth of the project.

Future Opportunities

The next step in our development process will be to streamline the capture of postoperative follow-up information. At present, this capture is maximally efficient when a patient presents for follow-up in clinic. Due to the geographic breadth of our referral base, this is not always feasible. Our next development effort will be to remove barriers for patients to remotely complete QOL metrics, likely by developing an application that might be used by smartphone to facilitate QOL capture.

One of the benefits of a less integrated approach to develop an outcomes registry is the scalability of the platform. The MyHealth and Epic code for developing questionnaires is easily sharable and may be used at other centers to capture the same outcome metrics. One opportunity for further development would be to have multiple discrete centers capturing outcome metrics using this platform. A dashboard running at each center would allow local review of quality and outcomes data. A separate dashboard could capture de-identified data from each center involved in the project and develop aggregates of the entire patient population. The goal would be an integrated QOL registry capture with a decentralized approach, with individual centers contributing to a central repository.

The benefits of this strategy are significant. Each center would have access to its own data and could develop its own best practices around QOL capture. Variation in the elements captured, with additional practice-specific outcomes variables, is fostered. Outcomes data is immediately available and locally accessible. A centralized system for aggregating data would provide the opportunity for assessment of wider varieties of patients and quality assessments over a range of providers.

Conclusions

Evolving payment structures and the demand for more clearly defined value in clinical interventions has led to the development of patient registries to collect and analyze outcomes. We developed and implemented an EHR-based spine patient registry to collect clinical and QOL metrics. The combination of QOL capture and a standardization of clinic throughput led to an overall decrease in new patient clinic visit length with addition of QOL questionnaire recording.

Relative to parallel data entry methods, this approach has the potential to lower data entry errors, minimize additional labor, and improve ease and compliance in data collection. An EHR-based approach eliminates the disconnect between recording of clinical information used in patient care and the capture of outcomes data used for registry reporting or ongoing quality improvement efforts. This registry is subject to the same challenges in follow-up data collection as other efforts. However, a streamlined approach may be the next step in improving collection of outcomes data and defining value in spine care.

Author Contributions

Conception and design: Ratliff, Kearney, Lee, Chen, Berroya, Eisenberg, Shuer, Kerr. Acquisition of data: Azad, Wolf, Kearney, Lee, Flannery, Chen, Park. Analysis and interpretation of data: Ratliff, Azad, Kalani, Wolf, Kearney, Lee, Flannery, Berroya. Drafting the article: Ratliff, Azad, Kalani, Kearney. Critically revising the article: Ratliff, Azad, Kalani, Chen, Park. Reviewed submitted version of manuscript: Ratliff, Azad, Kalani. Approved the final version of the manuscript on behalf of all authors: Ratliff. Statistical analysis: Flannery. Administrative/technical/material support: Wolf, Lee, Flannery, Berroya, Eisenberg, Shuer, Kerr. Study supervision: Ratliff, Park, Shuer, Kerr.

References

  • 1

    Alvin MDMiller JASundar SLockwood MLubelski DNowacki AS: The impact of preoperative depression on quality of life outcomes after posterior cervical fusion. Spine J 15:79852015

    • Search Google Scholar
    • Export Citation
  • 2

    Asher ALDevin CJMroz TFehlings MParker SLMcGirt MJ: Clinical registries and evidence-based care pathways: raising the bar for meaningful measurement and delivery of value-based care. Spine (Phila Pa 1976) 39:22 Suppl 1S136S1382014

    • Search Google Scholar
    • Export Citation
  • 3

    Asher ALMcCormick PCSelden NRGhogawala ZMcGirt MJ: The National Neurosurgery Quality and Outcomes Database and NeuroPoint Alliance: rationale, development, and implementation. Neurosurg Focus 34:1E22013

    • Search Google Scholar
    • Export Citation
  • 4

    Asher ALSperoff TDittus RSParker SLDavies JMSelden N: The National Neurosurgery Quality and Outcomes Database (N2QOD): a collaborative North American outcomes registry to advance value-based spine care. Spine (Phila Pa 1976) 39:22 Suppl 1S106S1162014

    • Search Google Scholar
    • Export Citation
  • 5

    Choi YMayer TGWilliams MJGatchel RJ: What is the best screening test for depression in chronic spinal pain patients?. Spine J 14:117511822014

    • Search Google Scholar
    • Export Citation
  • 6

    Deyo RAGray DTKreuter WMirza SMartin BI: United States trends in lumbar fusion surgery for degenerative conditions. Spine (Phila Pa 1976) 30:144114472005

    • Search Google Scholar
    • Export Citation
  • 7

    Fairbank JCPynsent PB: The Oswestry Disability Index. Spine (Phila Pa 1976) 25:294029522000

  • 8

    Furukawa MFKing JPatel VHsiao CJAdler-Milstein JJha AK: Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Aff (Millwood) 33:167216792014

    • Search Google Scholar
    • Export Citation
  • 9

    Ghaferi AABirkmeyer JDDimick JB: Variation in hospital mortality associated with inpatient surgery. N Engl J Med 361:136813752009

    • Search Google Scholar
    • Export Citation
  • 10

    Ghogawala ZShaffrey CIAsher ALHeary RFLogvinenko TMalhotra NR: The efficacy of lumbar discectomy and single-level fusion for spondylolisthesis: results from the NeuroPoint-SD registry: clinical article. J Neurosurg Spine 19:5555632013

    • Search Google Scholar
    • Export Citation
  • 11

    Hall BLHamilton BHRichards KBilimoria KYCohen MEKo CY: Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 250:3633762009

    • Search Google Scholar
    • Export Citation
  • 12

    Liaw STTaggart JYu Hde Lusignan SKuziemsky CHayen A: Integrating electronic health record information to support integrated care: practical application of ontologies to improve the accuracy of diabetes disease registers. J Biomed Inform 52:3643722014

    • Search Google Scholar
    • Export Citation
  • 13

    Ludwig CVeeravagu AZhang MMaltenfort MGDowzicky PRatliff JK: From the patient’s point of view: patient-centered outcomes in spine surgery. Cureus 5:e1142013

    • Search Google Scholar
    • Export Citation
  • 14

    Martin BIDeyo RAMirza SKTurner JAComstock BAHollingworth W: Expenditures and health status among adults with back and neck problems. JAMA 299:6566642008

    • Search Google Scholar
    • Export Citation
  • 15

    Mazanec D: Much more than a guideline: the Spine Carepath. Cleveland Clinic Spinal Column 2012–2013:452012

  • 16

    McGirt MJParker SLAsher ALNorvell DSherry NDevin CJ: Role of prospective registries in defining the value and effectiveness of spine care. Spine (Phila Pa 1976) 39:22 Suppl 1S117S1282014

    • Search Google Scholar
    • Export Citation
  • 17

    Miller JADerakhshan ALubelski DAlvin MDMcGirt MJBenzel EC: The impact of preoperative depression on quality of life outcomes after lumbar surgery. Spine J 15:58642015

    • Search Google Scholar
    • Export Citation
  • 18

    Mummaneni PVWhitmore RGCurran JNZiewacz JEWadhwa RShaffrey CI: Cost-effectiveness of lumbar discectomy and single-level fusion for spondylolisthesis: experience with the NeuroPoint-SD registry. Neurosurg Focus 36:6E32014

    • Search Google Scholar
    • Export Citation
  • 19

    Porter ME: What is value in health care?. N Engl J Med 363:247724812010

  • 20

    Porter METeisberg EO: Redefining Health Care: Creating Value-Based Competition on Results BostonHarvard Business School Press2006

  • 21

    Ratliff JK: The perils of comparative effectiveness, cost-effectiveness, and value of care research: lessons learned from Washington State. Neurosurgery 61:Suppl 112152014

    • Search Google Scholar
    • Export Citation
  • 22

    Spitzer RLKroenke KWilliams JB: Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders Patient Health Questionnaire. JAMA 282:173717441999

    • Search Google Scholar
    • Export Citation
  • 23

    Vernon H: The Neck Disability Index: state-of-the-art, 1991–2008. J Manipulative Physiol Ther 31:4915022008

  • 24

    Vernon HMior S: The Neck Disability Index: a study of reliability and validity. J Manipulative Physiol Ther 14:4094151991

  • 25

    Weinstein JNLurie JDTosteson TDHanscom BTosteson ANBlood EA: Surgical versus nonsurgical treatment for lumbar degenerative spondylolisthesis. N Engl J Med 356:225722702007

    • Search Google Scholar
    • Export Citation
  • 26

    Weinstein JNTosteson TDLurie JDTosteson ANBlood EHanscom B: Surgical versus nonsurgical therapy for lumbar spinal stenosis. N Engl J Med 358:7948102008

    • Search Google Scholar
    • Export Citation
  • 27

    Wyse RKTaylor KM: Using the STS and multinational cardiac surgical databases to establish risk-adjusted benchmarks for clinical outcomes. Heart Surg Forum 5:2582642002

    • Search Google Scholar
    • Export Citation
  • 28

    Zanoli GStrömqvist BJönsson B: Visual analog scales for interpretation of back and leg pain intensity in patients operated for degenerative lumbar spine disorders. Spine (Phila Pa 1976) 26:237523802001

    • Search Google Scholar
    • Export Citation

If the inline PDF is not rendering correctly, you can download the PDF file here.

Article Information

Contributor Notes

Correspondence John Ratliff, Department of Neurosurgery, Stanford University School of Medicine, 300 Pasteur Dr., Stanford, CA 94305. email: jratliff@stanford.edu.INCLUDE WHEN CITING Published online October 2, 2015; DOI: 10.3171/2015.3.SPINE141127Disclosure The authors have no financial interest in the registry or methodologies reviewed in this article. CPT codes are a copyright and registered trademark of the American Medical Association, 2014.
Headings
Figures
  • View in gallery

    This secure patient portal allows patients to complete baseline and follow-up questionnaires remotely. Copyright 2014 Epic Systems Corporation. Used with permission.

  • View in gallery

    A patient intake checklist was developed to standardize work flow during the patient rooming process.

  • View in gallery

    Clinician-facing dashboard provides the ability to monitor spine patient population in real time with a variety of filters including demographic criteria, procedure performed (by CPT or MS-DRG code), and operating surgeon.

  • View in gallery

    Screenshot of dashboard showing detailed view of rates of survey completion. Adoption of new techniques in clinic led to elimination of paper-based surveys and transition to a purely EHR-based outcomes capture.

  • View in gallery

    Screenshot of dashboard showing detailed view of MyHealth activation rates.

References
  • 1

    Alvin MDMiller JASundar SLockwood MLubelski DNowacki AS: The impact of preoperative depression on quality of life outcomes after posterior cervical fusion. Spine J 15:79852015

    • Search Google Scholar
    • Export Citation
  • 2

    Asher ALDevin CJMroz TFehlings MParker SLMcGirt MJ: Clinical registries and evidence-based care pathways: raising the bar for meaningful measurement and delivery of value-based care. Spine (Phila Pa 1976) 39:22 Suppl 1S136S1382014

    • Search Google Scholar
    • Export Citation
  • 3

    Asher ALMcCormick PCSelden NRGhogawala ZMcGirt MJ: The National Neurosurgery Quality and Outcomes Database and NeuroPoint Alliance: rationale, development, and implementation. Neurosurg Focus 34:1E22013

    • Search Google Scholar
    • Export Citation
  • 4

    Asher ALSperoff TDittus RSParker SLDavies JMSelden N: The National Neurosurgery Quality and Outcomes Database (N2QOD): a collaborative North American outcomes registry to advance value-based spine care. Spine (Phila Pa 1976) 39:22 Suppl 1S106S1162014

    • Search Google Scholar
    • Export Citation
  • 5

    Choi YMayer TGWilliams MJGatchel RJ: What is the best screening test for depression in chronic spinal pain patients?. Spine J 14:117511822014

    • Search Google Scholar
    • Export Citation
  • 6

    Deyo RAGray DTKreuter WMirza SMartin BI: United States trends in lumbar fusion surgery for degenerative conditions. Spine (Phila Pa 1976) 30:144114472005

    • Search Google Scholar
    • Export Citation
  • 7

    Fairbank JCPynsent PB: The Oswestry Disability Index. Spine (Phila Pa 1976) 25:294029522000

  • 8

    Furukawa MFKing JPatel VHsiao CJAdler-Milstein JJha AK: Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Aff (Millwood) 33:167216792014

    • Search Google Scholar
    • Export Citation
  • 9

    Ghaferi AABirkmeyer JDDimick JB: Variation in hospital mortality associated with inpatient surgery. N Engl J Med 361:136813752009

    • Search Google Scholar
    • Export Citation
  • 10

    Ghogawala ZShaffrey CIAsher ALHeary RFLogvinenko TMalhotra NR: The efficacy of lumbar discectomy and single-level fusion for spondylolisthesis: results from the NeuroPoint-SD registry: clinical article. J Neurosurg Spine 19:5555632013

    • Search Google Scholar
    • Export Citation
  • 11

    Hall BLHamilton BHRichards KBilimoria KYCohen MEKo CY: Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 250:3633762009

    • Search Google Scholar
    • Export Citation
  • 12

    Liaw STTaggart JYu Hde Lusignan SKuziemsky CHayen A: Integrating electronic health record information to support integrated care: practical application of ontologies to improve the accuracy of diabetes disease registers. J Biomed Inform 52:3643722014

    • Search Google Scholar
    • Export Citation
  • 13

    Ludwig CVeeravagu AZhang MMaltenfort MGDowzicky PRatliff JK: From the patient’s point of view: patient-centered outcomes in spine surgery. Cureus 5:e1142013

    • Search Google Scholar
    • Export Citation
  • 14

    Martin BIDeyo RAMirza SKTurner JAComstock BAHollingworth W: Expenditures and health status among adults with back and neck problems. JAMA 299:6566642008

    • Search Google Scholar
    • Export Citation
  • 15

    Mazanec D: Much more than a guideline: the Spine Carepath. Cleveland Clinic Spinal Column 2012–2013:452012

  • 16

    McGirt MJParker SLAsher ALNorvell DSherry NDevin CJ: Role of prospective registries in defining the value and effectiveness of spine care. Spine (Phila Pa 1976) 39:22 Suppl 1S117S1282014

    • Search Google Scholar
    • Export Citation
  • 17

    Miller JADerakhshan ALubelski DAlvin MDMcGirt MJBenzel EC: The impact of preoperative depression on quality of life outcomes after lumbar surgery. Spine J 15:58642015

    • Search Google Scholar
    • Export Citation
  • 18

    Mummaneni PVWhitmore RGCurran JNZiewacz JEWadhwa RShaffrey CI: Cost-effectiveness of lumbar discectomy and single-level fusion for spondylolisthesis: experience with the NeuroPoint-SD registry. Neurosurg Focus 36:6E32014

    • Search Google Scholar
    • Export Citation
  • 19

    Porter ME: What is value in health care?. N Engl J Med 363:247724812010

  • 20

    Porter METeisberg EO: Redefining Health Care: Creating Value-Based Competition on Results BostonHarvard Business School Press2006

  • 21

    Ratliff JK: The perils of comparative effectiveness, cost-effectiveness, and value of care research: lessons learned from Washington State. Neurosurgery 61:Suppl 112152014

    • Search Google Scholar
    • Export Citation
  • 22

    Spitzer RLKroenke KWilliams JB: Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders Patient Health Questionnaire. JAMA 282:173717441999

    • Search Google Scholar
    • Export Citation
  • 23

    Vernon H: The Neck Disability Index: state-of-the-art, 1991–2008. J Manipulative Physiol Ther 31:4915022008

  • 24

    Vernon HMior S: The Neck Disability Index: a study of reliability and validity. J Manipulative Physiol Ther 14:4094151991

  • 25

    Weinstein JNLurie JDTosteson TDHanscom BTosteson ANBlood EA: Surgical versus nonsurgical treatment for lumbar degenerative spondylolisthesis. N Engl J Med 356:225722702007

    • Search Google Scholar
    • Export Citation
  • 26

    Weinstein JNTosteson TDLurie JDTosteson ANBlood EHanscom B: Surgical versus nonsurgical therapy for lumbar spinal stenosis. N Engl J Med 358:7948102008

    • Search Google Scholar
    • Export Citation
  • 27

    Wyse RKTaylor KM: Using the STS and multinational cardiac surgical databases to establish risk-adjusted benchmarks for clinical outcomes. Heart Surg Forum 5:2582642002

    • Search Google Scholar
    • Export Citation
  • 28

    Zanoli GStrömqvist BJönsson B: Visual analog scales for interpretation of back and leg pain intensity in patients operated for degenerative lumbar spine disorders. Spine (Phila Pa 1976) 26:237523802001

    • Search Google Scholar
    • Export Citation
TrendMD
Metrics

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 506 319 15
PDF Downloads 345 142 5
EPUB Downloads 0 0 0
PubMed
Google Scholar