Penn study suggests Yelp reviews can enhance government reports on hospital quality
PHILADELPHIA – Yelp reviews of hospitals cover topics not found in the federal government's survey of patients' hospital experiences, according to the results of a new study from researchers at the Perelman School of Medicine at the University of Pennsylvania. The additional information, which the authors say tends to be strongly linked to positive or negative reviews from Yelp contributors, could influence patient decision making on where to receive hospital care, and provide valuable information to hospital administrators, caregivers, and policymakers. The study is published today in the April issue of Health Affairs.
Yelp publishes online crowd-sourced reviews about local businesses and services. It is the 33rd most visited website in the United States, with 142 million unique monthly visitors. On the site, patients are able to give hospitals a ranking from one to five stars, and provide narrative components that often reflect the features of a hospital experience most important to them.
In the new study, researchers compared approximately 17,000 Yelp reviews of 1,352 hospitals to HCAHPS reviews of the same institutions. The HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) Survey (pronounced "H-Caps"), is a standardized survey and data-collection tool that has been in use since 2006 to measure patient perspectives of hospital care in 11 different categories.
"Nearly 75 percent of U.S. Internet users reported looking online for health information in 2012," said the study's senior author Raina M. Merchant, MD, MSHP, an assistant professor of Emergency Medicine at Penn's Perelman School of Medicine, and director of the Penn Social Media and Health Innovation Lab. "Forty-two percent reported looking at social media for health-related consumer reviews. Meanwhile, only six percent of Americans had heard of the government website where the HCAHPS survey is reported, as of 2008. This divergence presents an opportunity for online consumer reviews to augment and even improve formal rating systems such as HCAHPS and increase their use in consumer decision making."
The researchers used natural language processing techniques to analyze the text of Yelp narrative reviews of hospitals posted as of July 15, 2014 to produce underlying categories. For example, a post that contained the terms "pain, doctor, nurse, told, medication, meds, gave," was labeled under the category "pain medications." Where possible, the categories were assigned to corresponding HCAHPS domains. For example, "pain medications" was allotted to the HCAHPS domain Pain Control.
While the Yelp reviews included information about seven of the 11 HCAHPS domains, the research team uncovered 12 additional categories covered in the Yelp reviews that are not covered by the HCAHPS survey: cost of hospital visit, insurance and billing, ancillary testing, facilities, amenities, scheduling, compassion of staff, family member care, quality of nursing, quality of staff, quality of technical aspects of care, and specific type of medical care.
For example, "compassion of staff" is not an item on the HCAHPS survey but instead is broadly reported under responses to the question "How often did doctors [or nurses] communicate well with patients?" Furthermore, four of the top five Yelp topics most strongly associated with positive Yelp review ratings were not covered by HCAHPS domains. These included: caring doctors, nurses, and staff; comforting; surgery/ procedure and peri-op; and labor and delivery.
"These topics that are covered within the Yelp reviews are important because they relate to the interpersonal relationships of patients with physicians, nurses, and staff," said the study's lead author Benjamin L. Ranard, a junior fellow at the Penn Social Media and Health Innovation Lab, and a combined MD/MS in Health Policy Research student at the Perelman School of Medicine. "Prospective patients are likely to want to know how caring and comforting caregivers are in various departments of a hospital."
Two of the top five Yelp categories most strongly associated with negative Yelp review ratings – insurance and billing, and cost of hospital visit – are also not covered by HCAHPS domains.
Citing previous research, the authors note that formal surveys such as HCAHPS suffer from low response rates and typically entail significant delays between hospitalization and public reporting of results. Finally, even if the evaluations can supply an overall indication of patient satisfaction, they rarely identify the source of perceived problems. The researchers add that while reviews on social media sites are not currently randomized, are largely uncurated, unvalidated, and subject to gaming, they are free, continuously updated, and often reveal in precise detail what the problem or positive occurrence was that affected the patient's or family member's experience.
"Yelp reviews are in real time and often written by patients for patients," said Merchant. "In addition, patients' perceptions of what matters most to them can change over time. HCAHPS may not be able to respond as agilely to these kinds of changes as social media."
To make reviews more helpful to readers, the Penn team suggests Yelp could highlight the most helpful positive and negative reviews, a practice that is becoming increasingly popular on consumer web sites like Amazon.com.
HCAHPS survey results are publicly reported on the Centers for Medicare and Medicaid Services Hospital Compare website, which rates all US hospitals that receive Medicare payments on a variety of quality measures. HCAHPS survey scores serve as a partial basis for what CMS pays hospitals for Medicare services. Hospitals with poor HCAHPS and other scores may see cuts of up to two percent in their Medicare payments. HCAHPS scores are based on a random sample of eligible discharges.
Other Penn co-authors are Rachel M. Werner, Tadas Antanavicius, Robert J. Smith, Zachary F. Meisel, David A. Asch, and Lyle H. Ungar.
Funding from the National Institutes of Health (K23-10714038, K24-AG047908) was provided to Drs. Werner and Merchant.
Penn Medicine is one of the world's leading academic medical centers, dedicated to the related missions of medical education, biomedical research, and excellence in patient care. Penn Medicine consists of the Raymond and Ruth Perelman School of Medicine at the University of Pennsylvania (founded in 1765 as the nation's first medical school) and the University of Pennsylvania Health System, which together form a $5.3 billion enterprise.
The Perelman School of Medicine has been ranked among the top five medical schools in the United States for the past 18 years, according to U.S. News & World Report's survey of research-oriented medical schools. The School is consistently among the nation's top recipients of funding from the National Institutes of Health, with $373 million awarded in the 2015 fiscal year.
The University of Pennsylvania Health System's patient care facilities include: The Hospital of the University of Pennsylvania and Penn Presbyterian Medical Center — which are recognized as one of the nation's top "Honor Roll" hospitals by U.S. News & World Report — Chester County Hospital; Lancaster General Health; Penn Wissahickon Hospice; and Pennsylvania Hospital — the nation's first hospital, founded in 1751. Additional affiliated inpatient care facilities and services throughout the Philadelphia region include Chestnut Hill Hospital and Good Shepherd Penn Partners, a partnership between Good Shepherd Rehabilitation Network and Penn Medicine.
Penn Medicine is committed to improving lives and health through a variety of community-based programs and activities. In fiscal year 2015, Penn Medicine provided $253.3 million to benefit our community.