Chat with us, powered by LiveChat

CMS Hospital Star Ratings Based on Patient Experiences Shown to be a Poor Proxy for Clinical Quality

(Milpitas, CA – August 15, 2016) Quantros, a nationally recognized leader in quality and patient safety, released a new research study today which challenges the notion that CMS hospital star ratings based solely on patient experiences can serve as a reliable indicator for clinical quality. This study follows a wake of industry criticism regarding a recent research letter published in JAMA Internal Medicine based on a study approved by the institutional review board at the Harvard T.H. Chan School of Public Health. This highly controversial study suggested that consumers could choose 5-star hospitals without a concern for their clinical quality and a 5-star rating may in fact guide patients to better institutions1. According to research conducted by Quantros, this conclusion is misleading and may actually steer patients to hospitals with poor clinical outcomes.

Initially, Quantros evaluated the performance of all CMS star-rated hospitals using a composite outcomes score inclusive of mortality rates, complications rates, and AHRQ patient safety indicators for all clinical conditions at risk for an adverse event. This resulted in the assessment of more than 300 conditions commonly treated by general, acute hospitals. Binary logistic regression models were used to risk-adjust hospital outcomes for differences in patient characteristics (age, gender, MS-DRG cluster, number of major chronic conditions, and number of other significant comorbidities)2. Results were than summarized using a derivation of the company’s patent pending CareChex® composite percentile scoring methodology which incorporates a comprehensive analysis of the relative statistical significance of each quality measure for each hospital. The study linked CMS hospital star ratings from Hospital Compare based on federal fiscal year 2014 HCAHPS surveys with risk-adjusted outcomes data for the same period using the CMS Medicare Provider Analysis and Review File. As shown in Figure 1, the analysis revealed that hospitals with star ratings of 2, 3, or 4 had over three times the percentage of hospitals in the top 10% in the nation than 5-star hospitals. Equally, if not more surprisingly, 1-star hospitals actually outperformed 5-star hospitals with 6% experiencing composite outcome scores in the top 10% of the nation (≥ 90th percentile) compared to only 4% of 5-star hospitals.

Figure 1. Percentage of Hospitals with Composite Outcome Scores in the Top 10% in the Nation (≥ 90th Percentile) by CMS Star Rating

Figure 1. Percentage of Hospitals with Composite Outcome Scores in the Top 10% in the Nation (≥ 90th Percentile) by CMS Star Rating

Moreover, Figure 2 identifies that nearly half (47%) of 5-star hospitals were found to have composite outcome scores below the national average (< 50th percentile) with 2, 3, and 4-star hospitals having the lowest percentage of poor performing hospitals.

Figure 2. Percentage of Hospitals with Composite Outcome Scores Below the National Average (< 50th Percentile) by CMS Star Rating

Figure 2. Percentage of Hospitals with Composite Outcome Scores Below the National Average (< 50th Percentile) by CMS Star Rating

Dr. Frank Mazza, chief medical officer at Quantros indicated, “These findings clearly confirm that consumers cannot safely assume that hospitals with a CMS 5-star rating will provide better clinical quality than other star-rated hospitals. In fact, reliance on 5-star ratings will place them at substantial risk of choosing a hospital that provides sub-standard care.”

The study also investigated whether a positive correlation existed between actual patient experience scores and composite outcome scores across 3,456 star-rated hospitals based on their percentile ranking. As displayed in Table 1, the study found there was no positive correlation between patient experiences and composite outcomes. Instead, a slightly negative correlation was observed, but the results were not statistically significant. After evaluating each individual outcome measure, a significant negative correlation was found between mortality and patient experiences although the association was weak and a non-significant negative correlation existed between complications and patient experiences. Only patient safety was found to have a positive correlation, however, the association was very weak.

Table 1. Summary of Correlations between Patient Experiences and Clinical Outcomes

* Statistically significant if p-value < .01 ** 1.000 = perfectly positive correlation; -1.000 = perfectly negative correlation

* Statistically significant if p-value < .01
** 1.000 = perfectly positive correlation; -1.000 = perfectly negative correlation


Annie Callanan, CEO of Quantros commented, “While we commend hospitals with positive patient experiences, we believe the industry needs to recognize that CMS star ratings are a poor proxy for guiding patients to safe, high quality care. There is simply no substitute for measuring the actual safety and quality of care delivered. As a company, we are committed to providing consumers and purchasers with reliable tools for identifying high value healthcare.”

The company offered reasons why findings from the study performed at Harvard’s School of Public Health differed from their own. The study:

  1. Linked HCAHPS patient experience data from second quarter 2014 through first quarter 2015 with 2013 patient outcomes data from the 100% Medicare Inpatient File resulting in inconsistent data periods for determining associations (i.e., patient experience ratings were linked with outcomes associated with a completely different patient population).

  2. Only assessed patient outcomes for mortality and readmissions across 3 conditions (acute myocardial infarction, pneumonia, and heart failure) rather than evaluating a more extensive set of clinical outcomes across all conditions commonly treated by hospitals.

In July, CMS released another star rating system designed to measure the overall quality of hospital care using additional measures beyond patient experiences. The industry has already expressed concerns over the new methodology indicating it puts certain hospitals, like academic medical centers and safety-net hospitals, at a disadvantage since the data is not risk-adjusted and does not account for socioeconomic factors3. More research will be needed to validate if these ratings provide a reasonable solution for measuring the quality of hospital care in an equitable manner.


  1. Wang D.E., Tsuawa Y., Figueroa J.F., Jha A.K. “Association between Centers for Medicare and Medicaid Services Hospital Star Rating and Patient Outcomes.” JAMA Internal Medicine. Volume 176, No 6, June 2016, pp. 848-850.

  2. Forthman, M.T., Gold, R.S., Dove, H.G. & Henderson, R.D. “Risk-Adjusted Indices for Measuring the Quality of Inpatient Care.” Quality Management in Health Care, Volume 19, No 3, Summer 2010, pp. 265-277.

  3. Punke, H. “CMS Releases Overall Hospital Star Ratings: 12 Things to Know.” Becker’s Infection Control and Clinical Quality. Published online July 27, 2016.

About Quantros
Quantros is a leading provider of software and services to advance healthcare quality and safety performance. Our Software-as a-Service (SaaS)-based applications help thousands of hospitals, retail pharmacies and some of the nation’s largest health systems to capture actionable intelligence they can use to improve outcomes, reduce risks and to reinforce their commitments to delivering safer, higher-quality value-based care. For more information visit

2020 CareChex Awards are Here!

Quantros prides itself in providing the most comprehensive and objective calculation of hospital and system quality performance in the industry. Take a minute and check out if your hospital or system won any awards.

Request Award Summary