TY - JOUR
T1 - Adherence to methodological standards in research using the National Inpatient Sample
AU - Khera, Rohan
AU - Angraal, Suveen
AU - Couch, Tyler
AU - Welsh, John W.
AU - Nallamothu, Brahmajee K.
AU - Girotra, Saket
AU - Chan, Paul S.
AU - Krumholz, Harlan M.
N1 - Funding Information:
Dr Khera is supported by the National Heart, Lung, and Blood Institute (NHLBI) (5T32HL125247-02) and the National Center for Advancing Translational Sciences (UL1TR001105) of the National Institutes of Health. Dr Girotra (K08 HL122527) and Dr Chan (1R01HL123980) are supported by funding from the NHLBI.
Funding Information:
Funding/Support: Dr Khera is supported by the
Publisher Copyright:
© 2017 American Medical Association. All rights reserved.
PY - 2017/11/28
Y1 - 2017/11/28
N2 - IMPORTANCE: Publicly available data sets hold much potential, but their unique design may require specific analytic approaches. OBJECTIVE: To determine adherence to appropriate research practices for a frequently used large public database, the National Inpatient Sample (NIS) of the Agency for Healthcare Research and Quality (AHRQ). DESIGN, SETTING, AND PARTICIPANTS: In this observational study of the 1082 studies published using the NIS from January 2015 through December 2016, a representative sample of 120 studies was systematically evaluated for adherence to practices required by AHRQ for the design and conduct of research using the NIS. EXPOSURES: None. MAIN OUTCOMES AND MEASURES: All studies were evaluated on 7 required research practices based on AHRQ’s recommendations and compiled under 3 domains: (1) data interpretation (interpreting data as hospitalization records rather than unique patients); (2) research design (avoiding use in performing state-, hospital-, and physician-level assessments where inappropriate; not using nonspecific administrative secondary diagnosis codes to study in-hospital events); and (3) data analysis (accounting for complex survey design of the NIS and changes in data structure over time). RESULTS: Of 120 published studies, 85% (n = 102) did not adhere to 1 or more required practices and 62% (n = 74) did not adhere to 2 or more required practices. An estimated 925 (95% CI, 852-998) NIS publications did not adhere to 1 or more required practices and 696 (95% CI, 596-796) NIS publications did not adhere to 2 or more required practices. A total of 79 sampled studies (68.3% [95% CI, 59.3%-77.3%]) among the 1082 NIS studies screened for eligibility did not account for the effects of sampling error, clustering, and stratification; 62 (54.4% [95% CI, 44.7%-64.0%]) extrapolated nonspecific secondary diagnoses to infer in-hospital events; 45 (40.4% [95% CI, 30.9%-50.0%]) miscategorized hospitalizations as individual patients; 10 (7.1% [95% CI, 2.1%-12.1%]) performed state-level analyses; and 3 (2.9% [95% CI, 0.0%-6.2%]) reported physician-level volume estimates. Of 27 studies (weighted; 218 studies [95% CI, 134-303]) spanning periods of major changes in the data structure of the NIS, 21 (79.7% [95% CI, 62.5%-97.0%]) did not account for the changes. Among the 24 studies published in journals with an impact factor of 10 or greater, 16 (67%) did not adhere to 1 or more practices, and 9 (38%) did not adhere to 2 or more practices. CONCLUSIONS AND RELEVANCE: In this study of 120 recent publications that used data from the NIS, the majority did not adhere to required practices. Further research is needed to identify strategies to improve the quality of research using the NIS and assess whether there are similar problems with use of other publicly available data sets.
AB - IMPORTANCE: Publicly available data sets hold much potential, but their unique design may require specific analytic approaches. OBJECTIVE: To determine adherence to appropriate research practices for a frequently used large public database, the National Inpatient Sample (NIS) of the Agency for Healthcare Research and Quality (AHRQ). DESIGN, SETTING, AND PARTICIPANTS: In this observational study of the 1082 studies published using the NIS from January 2015 through December 2016, a representative sample of 120 studies was systematically evaluated for adherence to practices required by AHRQ for the design and conduct of research using the NIS. EXPOSURES: None. MAIN OUTCOMES AND MEASURES: All studies were evaluated on 7 required research practices based on AHRQ’s recommendations and compiled under 3 domains: (1) data interpretation (interpreting data as hospitalization records rather than unique patients); (2) research design (avoiding use in performing state-, hospital-, and physician-level assessments where inappropriate; not using nonspecific administrative secondary diagnosis codes to study in-hospital events); and (3) data analysis (accounting for complex survey design of the NIS and changes in data structure over time). RESULTS: Of 120 published studies, 85% (n = 102) did not adhere to 1 or more required practices and 62% (n = 74) did not adhere to 2 or more required practices. An estimated 925 (95% CI, 852-998) NIS publications did not adhere to 1 or more required practices and 696 (95% CI, 596-796) NIS publications did not adhere to 2 or more required practices. A total of 79 sampled studies (68.3% [95% CI, 59.3%-77.3%]) among the 1082 NIS studies screened for eligibility did not account for the effects of sampling error, clustering, and stratification; 62 (54.4% [95% CI, 44.7%-64.0%]) extrapolated nonspecific secondary diagnoses to infer in-hospital events; 45 (40.4% [95% CI, 30.9%-50.0%]) miscategorized hospitalizations as individual patients; 10 (7.1% [95% CI, 2.1%-12.1%]) performed state-level analyses; and 3 (2.9% [95% CI, 0.0%-6.2%]) reported physician-level volume estimates. Of 27 studies (weighted; 218 studies [95% CI, 134-303]) spanning periods of major changes in the data structure of the NIS, 21 (79.7% [95% CI, 62.5%-97.0%]) did not account for the changes. Among the 24 studies published in journals with an impact factor of 10 or greater, 16 (67%) did not adhere to 1 or more practices, and 9 (38%) did not adhere to 2 or more practices. CONCLUSIONS AND RELEVANCE: In this study of 120 recent publications that used data from the NIS, the majority did not adhere to required practices. Further research is needed to identify strategies to improve the quality of research using the NIS and assess whether there are similar problems with use of other publicly available data sets.
UR - http://www.scopus.com/inward/record.url?scp=85037054347&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85037054347&partnerID=8YFLogxK
U2 - 10.1001/jama.2017.17653
DO - 10.1001/jama.2017.17653
M3 - Article
C2 - 29183077
AN - SCOPUS:85037054347
SN - 0098-7484
VL - 318
SP - 2011
EP - 2018
JO - JAMA - Journal of the American Medical Association
JF - JAMA - Journal of the American Medical Association
IS - 20
ER -