Client Satisfaction Surveys to Improve Outcome Management in Mental and Behavioral Health

‐

 

By Judith Ann Shields, RN, MSN, CHCQM; Hugh Sage, PhD; Alan M. Stillman, MSS;
Steve Bryant, BA; Norris Sonntag, MA; Ken Carabello, MSW; and Susan A. Nayda, MA

The Liberty QualityCare Client Satisfaction Survey is designed to provide an annual measurement of outcomes for a national, private, physician-owned corporation established in 1986. Liberty Healthcare Corporation provides healthcare management and staffing for acute care behavioral health, mental health, dual diagnosis with intellectual/developmental disabilities, correctional mental health, and primary care settings.

A small work group of advanced clinicians and executive administrators created the Liberty QualityCare Client Satisfaction Survey tool to be an early indicator of program performance problems and alpha tested it from 2005 through 2006. The tool was beta tested from 2007 through 2008 to improve its quality and to get more input from the clinicians at the two program sites that were involved in the alpha/beta testing.  

Following the testing, the leadership team has distributed the survey tool annually to three programs across the country in which leadership has full responsibility for clinical outcomes. These selected programs comprise approximately 23% of Liberty Healthcare Corporation’s business.

The survey provides invaluable feedback concerning treatment goals, access to program services, and insight into the client’s ability to maintain general health through activity and movement. Clients are able to add comments and questions on each annual survey, which brings the client outlook into the satisfaction process in a very real and bold way. The leadership team combines the data-driven nature of clinical outcomes with input from those who receive the treatment to make improvements in the programs.

As it reviews the annual satisfaction data, the leadership team is reminded of the adage, “Be careful what you wish for.” When an organization begins the journey of asking satisfaction questions, it must be willing to hear and read the responses. Satisfaction responses are anchored to the client’s perception of the quality of services provided and the client’s feelings of safety.

Methodology

The leadership team developed 11 survey questions (Figure 1) to measure four key areas of the treatment process:

  • How well the client’s perceived clinical needs match the program goals
  • Respectful treatment and availability of staff and services
  • Client safety and privacy
  • The opportunity for clients to remain active

For the leadership team, there are several advantages and programmatic benefits to addressing client satisfaction using this particular assessment tool. Regularly assessing clients in these four key areas of the treatment process allows the program leadership to manage outcomes with direct input from the end user of the services delivered. The client’s understanding of program goals is central to achieving successful clinical outcomes and maintaining a respectful environment that fosters privacy, health, and safety. Providing that environment facilitates access to services. The survey metric is vital to our quality performance initiatives.

The Liberty QualityCare Client Satisfaction Survey’s 11 questions are answered on a 5-point Likert scale, with responses of “strongly agree,” “agree,” “neither agree nor disagree,” “disagree,” and “strongly disagree.” The tool also includes four open-ended questions:

  •  What do you find beneficial about the services?
  •  What changes would you make to the services?
  •  What questions should be asked in the next Liberty QualityCare Client Satisfaction Survey?
  • What other comments do you have for the leadership team?

Each survey includes the following introduction:

Your opinion is important to us. There are eleven (11) statements below. For each statement, check ONLY ONE (1) of  the boxes that best matches how you feel: “Strongly Disagree”,  “Disagree”,  “Neither Agree nor Disagree”, “Agree”, OR “Strongly Agree”. Your responses are confidential and will be used to inform your program leadership team on how you responded to these questions so that your input is considered in improvements to your program.

The subjects are clients over the age of 18 who voluntarily agree to complete the Liberty QualityCare Client Satisfaction survey.

Procedure
Once each year, printed copies of the Liberty QualityCare Client Satisfaction Survey are distributed by the organization’s vice president of quality performance/quality improvement to clinical leaders from programs in various parts of the country for distribution to clients, with written assurances of confidentiality. The clinical leaders are given a deadline for returning the completed survey questionnaires to the vice president. Return address envelopes are provided to allow clients to mail completed surveys back individually, if they so desire. For any client who wants to complete the questionnaire but is functionally illiterate, a continuous quality improvement coordinator uses interviewing techniques and unique satisfaction recognition tools that allow the client either to verbally respond to the interview questions or select by face recognition the satisfaction level they have for each question.

The vice president of quality performance/quality improvement is responsible for aggregating the survey data from client respondents, tallying the results and outcomes into a summary report, which is distributed to the participating executive directors and Liberty Healthcare’s vice president of operations and senior vice president/chief operating officer. This process was established to avoid any concern that the client’s honest feedback would interfere with his or her treatment process, progress, or services.

Analysis
Liberty uses the Shewart/Deming Improvement circle of Plan, Do, Check/Study, Act (Deming, 1986; Shewart, 1939) to process the results of each annual survey. Each year, directors and clinical teams meet to review the outcome results and the summary report and to discuss intervention strategies and improvements to program goals. Areas of concern identified in the annual summary are reviewed by these clinical programs with specific input from all clinical providers.

These detailed and honest reviews point to the heart of the clinical programs, and the four key questions asked of our clients are the heart of the questionnaire. The humanity of the responses goes beyond the data-driven nature of outcomes management to the human spark that is at the center of all client-centered care and population health management. In comments such as “I just want to see my mom” and “I see ghosts in that group home you want me to go to,” staff members can identify with their clients’ need to be human as they struggle with persistent mental illness or behavioral disturbances in addition to medical diagnoses.

One major component of satisfaction with which clinicians and clients struggle is the ability of clients to stay active. The mental and physical health of clients work in tandem; they both significantly affect health status and subsequent resources. The mental and physical health status of clients work together to improve population health, and the clinical leadership continues to work hard to provide this delicate balance of safety and activity.

On several occasions, the results of the Liberty QualityCare survey were discussed at an annual meeting of the entire organization’s leadership group, which added objective input and clinical strategies that might not have been discussed or implemented by those who were directly involved. In addition, collaborative groups of clients and clinical providers discussed the clients’ perception of program goals, access to services, safety, respect from staff, and their ability to stay active and healthy.

Results
It is difficult to find comparison data for benchmarking this kind of survey. However, the leadership team reviewed data from the Association of Support Professionals (2007) that gives a benchmark for customer satisfaction response rates of between 17% and 23%. PeoplePulse (n.d.) reports a benchmark for client satisfaction response rates between 15% and 30%. The response rates for the Liberty QualityCare tool fluctuate because our clients are not required to complete the survey and the population can vary in volume from year to year (Figure 2, p. 28; Figure 3, p. 29). However, the leadership team considers Liberty QualityCare’s response rates representative of the clients’ satisfaction and strong enough to take tactical action steps.

From 2009 through 2014, the leadership team decided that the best use of clinical time was to aggregate the data according to the percentage of “agree” and “strongly agree” responses to each question. The team focused their performance improvement efforts on questions to which fewer than 70% of clients responded with “agree” or “strongly agree” (Figure 4).

In 2014 (year six), the leadership team concurred that it would be helpful also to include the mean Likert score and add the percentage of negative, neutral, and no responses in order to give a bigger picture that includes satisfaction, dissatisfaction, and neutrality concerning treatment processes, safety, and the ability to stay active (Figure 5).

Discussion
Having reviewed six years of Liberty QualityCare Client Satisfaction data, it is clear to the leadership team that the decision to use the annual client satisfaction tool as a key component of outcomes management has been highly beneficial in the following ways:

The tool is economical and a comprehensive measure of client satisfaction. Creating the Liberty QualityCare tool was cost-effective, and it is easy to apply to any program in which the leadership team has total responsibility for clinical outcome.

Leadership can compare performance over each year of completion. Following the completion of each annual summary report, the leaders and clinicians meet to discuss clinical strategies and tactics to improve satisfaction while still achieving the clinical outcomes expected for health improvement.

The Liberty QualityCare Client Satisfaction Survey provides one way to implement continuous quality ­improvement ­throughout the organization. The leadership team has become as data-driven with our clients’ satisfaction as with our entire outcome measurements programs. Satisfaction results are included in the annual summary report of quality performance and quality improvement, and strategies are applied equally to client satisfaction and other quality indicators. The Liberty QualityCare results have become a unifying process for divergent clinical programs and a tangible expression of the organization’s commitment to its clients. The annual results help to educate clinicians on the importance of satisfaction and activate “outside the box” thinking when it comes to action steps and tactics that could improve satisfaction.

Survey results identify key areas in need of improvement across the organization. As clinicians, the leadership team knows the importance of understanding the program goals. These results led them to increase clinical time with clients for explaining and obtaining immediate feedback about goals. As stated in Liberty Healthcare Corporation’s mission, being trustworthy is vital to the organization, and a key to trustworthiness is the client feeling respected by the clinical staff. Another equally important characteristic of trustworthiness is ensuring that clients feel safe and know who to go to when they lack this feeling of safety. Finally, the ability of clients to maintain an active lifestyle is very important to the organization since the leadership team takes pride in treating the whole individual and the entire population. Even when satisfaction scores wavered, the leadership team knew that performance excellence depended on remaining true to the mission and committed to the clients’ overall health.

The survey tool facilitates honest feedback with business objectivity. By allowing clients to express anonymous, frank, uncensored perceptions of their treatment, service quality, and safety, the leadership team brings clients’ perceptions into outcomes measurement and management. The leadership team has to be humble, and this strengthens the clinical programs and commitment to the organization’s mission. The anonymity of the surveys facilitates the business-like review of the responses, and the client comments documented in the final section of the tool humanize the clinical outcome process. The emotions that the results may trigger are focused in a razor-sharp manner to improve the business results by reviewing the perceptions of the vital end users of the clinical products. The leadership team often discusses client-centered work, and the questionnaire and subsequent results serve as annual reminders that the client’s perception is at the core of every outcome. Clients know what it takes to reach a positive outcome, and without that input, our outcomes will never be as good as they could be, and our clients may not be as safe as they should be.

The survey is good preparation for Joint Commission Health Care Staffing Services Certification. The Liberty QualityCare Client Satisfaction Survey and its results have been recognized in each Joint Commission biannual review since 2009. Having six years of data on our clients’ perception of their satisfaction with treatment, quality services, and safety has been recognized for excellence by The Joint Commission. In addition to having this client satisfaction outcome process identified in each final Joint Commission exit conference in 2006, 2008, and 2010, in 2011, Liberty Healthcare Corporation’s vice president of quality performance/quality improvement was asked to give a national presentation sponsored by The Joint Commission to outline this client satisfaction process as part of an overall performance improvement program (Shields, 2011).

Lessons Learned and Future Directions
Refresh the performance improvement process at intervals. Reviewing the data, the leadership team found that at the six-year mark, there should be an organized review of the performance improvement process and the entire client satisfaction program. The team sees a need for this review based on a slight decline in improvement for both the understanding of program goals and the clients’ ability to stay active. The data indicate that at year six, the successes started to decline slightly for these two measurements, which is a signal to review the client satisfaction program and evaluate whether any aspects of it need to be refreshed.

Shortcomings of the client satisfaction study. As new programs were integrated into our organization, the leadership team should have done a better job of highlighting the benefits of obtaining annual client satisfaction data, which would have increased the sample size and allowed these newer programs to benefit from the quality performance initiative. Resistance by program leaders to take on this client satisfaction initiative may have been unrecognized. The leadership team, therefore, may not have spent the time needed to challenge new leaders’ fears of failure, time commitment, and resources entailed in this commitment to client perceptions of care delivery.

Future directions. The leadership team will recognize an opportunity for improvement by increasing efforts to engage additional program leaders to participate in gathering client satisfaction data and to strengthen efforts to improve goal setting and clients’ activity levels.

In conclusion, asking for client satisfaction is difficult because the results are based on perception, but the process is workable. Because they are based on direct feedback from clients, the changes made to program strategy and the action steps taken to improve the program are most likely to be successful.

Acknowledgments
Special thanks to Drs. Adam Deming and Todd Casbon of the Indiana Sex Offender Monitoring and Management Program for their detailed reviews and advice.

Judith Shields is vice president of quality improvement/quality performance and corporate/privacy officer for Liberty Healthcare Corporation. Shields is board certified by the American Board of Quality and Utilization Review Physicians and certified as a nurse executive administrator by the American Nurses Association. She may be contacted at JudithS@LibertyHealth.com.

Hugh Sage is executive director of the Robert M. Greer Center in Enid, Oklahoma. Sage’s program is accredited by the National Association of Dual Diagnosis (NADD) and the Commission on the Accreditation of Rehabilitative Facilities, CCAC (CARF).

Alan Stillman is executive director of the California Conditional Release Program.

Steve Bryant is executive director of the Illinois Conditional Release Program.

Norris Sonntag is continuous quality improvement coordinator at the Robert M. Greer Center in Enid, Oklahoma.

Ken Carabello is vice president of operations for Liberty Healthcare Corporation. He was twice elected president of the Forensic Mental Health Association of California and is the recipient of the association’s Christine M. West Award for significant contributions to the field of forensic mental health.

Susan Nayda is senior vice president and chief operating officer for Liberty Healthcare Corporation. She is responsible for leading operations across diverse product lines, championing innovative solutions, and driving operational excellence to yield successful results and high value to customers. Nayda has over 25 years of experience managing healthcare and human services.

References

Brooks, I. (2007). Customer satisfaction benchmarking. Watertown, MA: The Association of Support Professionals.
Deming, W. (1986). Out of crisis. Cambridge, MA: MIT Center for Advanced Engineering Study.
PeoplePulse. (n.d.). Survey response rates. Retrieved from http://www.peoplepulse.com.au/Survey-Response-Rates.htm
Shewart, W. (1939). Statistical method from the viewpoint of quality control. New York: Dover.
Shields, J. (2011, May 10). Framework development for quality performance measurement systems. Panel discussion at the National Joint Commission Health Care Staffing Services (HCSS) Certification, Oakbrook, IL.
Shields, J. A., & Jennings, J. L. (2013). Using the Malcolm Baldrige “Are we making progress” survey for organizational self-assessment and performance improvement. Journal for Healthcare Quality, 35(4), 5.