Diversifying Electronic Cohort Research at UCSF

A community-engaged contest to select and support a diverse "eCohort" at UCSF

Computerized Assessment of Mental Status (CAMS): Using remote assessment to make mental health resources more accessible to underserved populations

Primary Author: Josh Woolley
Proposal Status: 

Computerized Assessment of Mental Status (CAMS): Using remote assessment to make mental health resources more accessible to underserved populations

The Problem

Every year, one in five adults in the U.S. experiences mental illness and over $190 billion in earnings are lost due to the disabling effects of serious mental illness. Members of racial and ethnic minorities face an even greater burden: they are less likely to have access to mental health services, more likely to use emergency rooms for their mental health care, and more likely to receive lower quality care. The overwhelming personal, social, and economic costs associated with mental illness, particularly in underserved communities, call for innovative approaches that address this critical gap in access. Remote assessment of mental health has the potential to transform mental health care delivery, but lack of robust assessment tools presents a major challenge to implementing scalable, cost-effective solutions.

The Solution

When a person seeks mental health care, the first thing clinicians do is a mental health status interview. Throughout this interview, the clinician looks for signs of mental health or illness by observing their behavior: the language they use, the quality of their voice, and how they move their face and body. Unfortunately, these mental health assessments are 1) expensive in terms of training and administration time, and 2) subjective, meaning that conclusions can vary by clinician and are vulnerable to implicit bias. To address these challenges, we have developed Computerized Assessment of Mental Status (CAMS), an easy-to-use, interactive, cloud-based application that extracts information about people’s behavior from video data in order to objectively assess their mental health status. CAMS has the potential to improve accessibility, precision, and cost-effectiveness of mental health care delivery, whether used for connecting first-time patients to the mental health resources they need, or monitoring people already in treatment.   

How it works

In CAMS, participants interact with a virtual assistant, responding to a series of questions and stimuli similar to what is used during in-person mental health assessments (Fig. 1). From these videos of rich social behavior, we extract three types of data: 1) natural language (what the participant says), 2) vocal signals (how they say it), and 3) visual signals (how they move their face and body). Together, these data provide information about a participant’s emotional state and thought process, providing a snapshot of their mental health. When compared to normed data matched on demographic variables, this snapshot can indicate mental health problems and the need for treatment. The more data we have to compare participants’ results to, the more accurate CAMS will be. To ensure that we are building a platform that will serve diverse communities it is critical that data from those populations are adequately represented in the CAMS data.  

 Fig. 1

Fig. 1. Data collection using CAMS. A virtual assistant guides the participant through a series of tasks designed to capture clinically relevant behavior. For example, participants are asked about the last time they felt really happy and what they are most worried about. Participants also watch a series of brief, emotionally evocative videos and answer questions about the videos as shown above. Throughout CAMS, video data of the participant is recorded using a phone’s built-in camera. 

The current proposal

Previous efforts to generalize mental health research to diverse communities, especially research using technology, have been largely ineffective due to lack of diverse populations used in the original research. By combining CAMS with the capabilities of the Eureka platform, our goal is to create a tool that can improve access and cost-effectiveness of mental health care for underserved communities. Enrolling a large number of people who are diverse with regards to race, ethnicity, SES, and age, is a priority for three reasons. First, the proposed project will provide a critical first test of the usability of CAMS in diverse populations, enabling us to identify ways CAMS can best serve a wide variety of communities. Second, the project will yield a large dataset that allows us to examine how behavior covaries with symptoms of mental illness and will be the basis for scientific publications that will have immediate impact on the field of mental health. Third, in the long term, the data we will collect in this project will be used in machine learning approaches to ensure that CAMS will be able to accurately assess the mental health status of people from diverse communities.  

To meet these goals, we propose to partner with community-based organizations and leverage the Eureka platform to gather video data via smartphones from a large diverse sample (N=2000) of adults using CAMS. Given the large target sample, we intend to recruit outside UCSF patient populations. In this entirely remote study, we would use the Eureka platform for online consenting, remote administration of mental health symptom self-report scales, and remote video data collection. There will be two assessment points at least six months apart during which self-reported symptoms (e.g. depression, psychosis) will be assessed and responses to CAMS will be video recorded. This repeated measures design will allow us to parse between-person and within-person variability in symptoms of mental illness and behavioral responses.  On the backend, we will use our suite of behavioral analysis tools (see Fig. 2) to identify behaviors that significantly relate to self-reported mental health symptoms and establish how these relationships may differ between diverse populations.  In the short term, these data will improve our understanding of how emotional and cognitive behavior relate to current and future symptoms of mental illness, and in the long term, will allow us to improve CAMS’ ability to bring much-needed mental health resources to diverse communities.  At this time, we will focus on English-speaking participants, but we are currently planning a Spanish language version of CAMS. 

 Fig. 2

Fig. 2.  Data extracted by CAMS. We analyze three types of data from participant videos to create a snapshot of mental health status.

Our team is led by Josh Woolley, a psychiatrist and Assistant Professor with extensive experience assessing social and emotional behavior in people with mental illness and designing and conducting clinical trials. He also has experience remotely recruiting participants with mental illness for online studies. Dr. Woolley has salary support through a Career Development Award and project funding through 2020. Dr. Bradley is a psychiatrist and VA Research Fellow with experience running clinical trials and recruiting participants with psychosis both within UCSF and across the Bay Area community. Dr. Anderson is a postdoctoral fellow in the UCSF department of psychiatry, and is an expert on the assessment of emotions. He has previously partnered with Bay Area organizations that make outdoors experiences more accessible to underserved communities to study how nature can improve the mental health of at-risk youth. Together, our team has salary support and grant funding to cover participant compensation, project coordination, and cloud-based data storage through December of 2020. We also re-submitted a National Science Foundation (NSF) grant, which received positive reviews, that would potentially fund CAMS for the next 5 years.  

Comments

Thanks for taking the time to add your proposal. Our selection committee has UCSF and non-UCSF community-based members and non-medical staff. While your proposal is well composed, a short, accessible summary will make it easier for everyone to engage. Could you share a 150-word plain-language summary of your proposal?

Thanks, Brian! We have added a brief plain-language summary (also pasting here):

As mental health providers, we learn a lot about how our patients are doing from what they say, how their voices sound, and what kind of facial expressions they make. These behaviors can help us understand whether a patient is improving on a new medication, or whether they might be in crisis and need to be hospitalized. Unfortunately, in-person mental health evaluations are expensive and unstandardized, so access to them is limited and they can be biased. This is one reason why many communities are underserved by the mental health care system. Our team is using technology to objectively evaluate patients’ mental health using videos that they record themselves. To make sure that the technology serves many communities, we hope to partner with community-based organizations and Eureka to analyze video data from a large group of people that is diverse in terms of race, ethnicity, and socioeconomic status.

Neat proposal!

Interesting proposal and agree that understanding the usability of CAMS in diverse populations could be very valuable. A couple of questions about how the rest of the proposal might work:

"In this entirely remote study, we would use the Eureka platform for online consenting, remote administration of mental health symptom self-report scales, and remote video data collection."  

Are you certain that Eureka currently has the capacity for remote video data collection? Have you considered other possibilities for collecting video? Could your existing solution be leveraged in conjunction with Eureka?

Also, could you expand on considerations the machine learning approaches described as possible in goal 3 might impact the diverse communities you plan to involve?


I assume there is a separate and independently-functioning CAMS app.  Native smartphone app?  iOS or Android or both or web portal only?  Eureka could be configured to send people to such an app.  I would think CAMS would have the video data collection functionality built in, and you don't need Eureka to do this?  Please clarify.

To clarify, there is indeed an independently functioning CAMS web app, but not native smartphone app. CAMS does have video data collection functionality built in, but if this could be incorporated into the Eureka platform that would be ideal. If that is not possible, Eureka can be used to direct participants to the CAMS web app.

Hi Patricia,

Thank you for these questions. I'll try to answer each below; please just let us know if you'd like any further clarification/elaboration.

The existing CAMS application already has the capacity for remote video data collection via a web portal. Ideally, we would like to incorporate video capture into the Eureka platform, and from our phone conversations with people on the Eureka team, it is possible that they might be interested in building out this functionality. This is exciting to us and would streamline data collection, but the project does not depend on it. We can always direct people from Eureka to the web app.  

Our goal is for CAMS to become a widely accessible tool that clinicians can use to help identify and track symptoms of mental health problems in people from diverse backgrounds. In the past, technology such as eye tracking and facial expression detection algorithms have been developed using data from non-diverse samples and were therefore limited in their ability to serve a broad range of populations. By collecting large amounts of data from diverse populations in these early stages of CAMS to be used as input in machine learning approaches we will be well-positioned to ultimately build a tool that will accurately identify mental health problems across diverse populations.  

Very interesting approach to evaluation of mental illness.  Will you be deliberately recruiting any UCSF patients (e.g., through clinic or EHR data querying), or just patients in the community?  Do the patients need to have symptoms of mental illness, or would you take anyone (in the interest of developing your "control" responses in CAMS)? 

Thank you, Mark. We are excited about the project and think it has the potential to be high-impact.

To answer your question, in order to develop the predictive power of CAMS we are interested in sampling people with a wide range of mental health statuses.  We would indeed recruit people in the community with symptoms of mental illness as well as those who do not endorse symptoms (to develop control responses, as you mentioned above). Since members of our team do have access to mental health clinics, our aim is to recruit from these settings as well. Ideally, this approach will yield a sample with no, mild, moderate, and more severe symptoms of mental illness as reported by standardized rating scales. We believe this will greatly faciliate the development of CAMS into a robust tool.

Questions:

1) would the CAMS be a stand alone assessment or be part of a larger assessment process that includes in-person / face to face interaction? 

2) As community members, if this is a stand alone assessment, we find it difficult to believe that community would want to be assessed in this manner. From a Latin@ standpoint, in person face to face interactions are so part of our cultural lifestyle. Even in mental health training, we know that relationship building is critical to receiving accurate information about a person. Rapport is so key to effective mental health interventions and ultimate treatment outcomes. There are so many nuances, includindg cultural cues, within any racial/ethnic group. 

Thank you for bringing up both of these questions, Angela. 

We do not believe that any remote assessment of a person's mental health status can ever be "stand alone." Ordering a panel of blood tests does not replace a visit to the primary care doctor's office--it provides critical, objective information that can lead to important adjustments in a patient's care. As psychiatrists, we are oriented first and foremost towards face-to-face interactions and building trust with patients over time. However, being solely dependent on in-person interactions for all of our data has major implications for some of our patients who, for example, have to skip a day of work to take a 2 hour bus trip each way in order to get to each appointment. While we do not wish to replace in-person interactions, we do want to complement them, as we believe this can improve access to close monitoring. Furthermore, we do not believe that subjective assessments alone are sufficient to serve our patients. When we read a fellow clinician's documentation describing a patient's affect at a visit one month ago as "somewhat labile", it is challenging to understand exactly what that means. Is that clinician's definition of "labile" the same as ours? Was there something about that interaction that could have led to a biased assessment of the patient's affect? How can we use this information to understand whether the patient is doing better or worse than they were one month ago? We believe there is value to bringing more objectivity into mental health assessment in order to guide treatment & bring us closer to the approaches used in other areas of medicine. Just as a primary care doctor reviews recent hemoglobin A1c results with a patient who has diabetes during a visit, we hope that in the future mental health care providers can sit with their patients and review data from assessments like CAMS.

Please let us know if this helps to address your concerns. 

Best,

Ellen, Josh, and Craig 

This is a very innovative idea!  I love the idea of using vocal and visual signals to capture non-spoken "language"!

As indicated in the proposal that the research team hopes to build "a platform that will serve diverse communities it is critical that data from those populations are adequately represented in the CAMS data". What about Asian Americans?  I hope there are plans for this study to include Asian Americans.   This method will be very useful in gathering data among Asian Americans, especially that mental health is such a stigmatized topic in the community. 

 

Thank you, Angela. We are excited to know that you think this could approach could serve the Asian American community.

It is really critical that Asian Americans are represented in the datasets that are used to build CAMS--making sure that we partner with people from different racial, ethnic, and cultural backgrounds is the only way that we think CAMS can be a useful tool. While technology that can capture these different behavioral signals is exciting, we know that leveraging it for effective patient care means that we have to avoid the well-known problem of using datasets from overwhelmingly white and male populations. We are looking forward to working with community-based organizations that can guide us in this process. 

Hi Ellen,

Glad to hear that the research team plans to include Asian Americans in the datasets.   

What are the team’s specific plans to collaborate with Asian community-based organizations?  

Has the team identified potential partners from the Asian American community?

Great idea!  It would be really helpful if we could have more objective ways of capturing mental status.  

Really interesting and potentially impactful. I have some comments and some questions.

1) As a primary care doctor, the phrase "mental status" does not mean "mental health" so the title threw me off--I thought you were doing a cognitive study.

2) For broad screening and assessment, in primary care, we use validated questionnaires such as PHQ-9 and GAD-7, and for CAMS to be used outside of psychiatric settings, whatever findings need to have some relations to these widely used tools. There is value in delineating more clearly the non-verbal and other types of communications, particularly in the context of culture and to train providers, but it would be hard to know what to do with all that information in terms of clinical care outside the psychiatric setting.

3) What are your plans to address people who have significant impairment in this study? Will participants receive any intervention at all? This is an important issue for participants from diverse communities, who often have trouble accessing care.

4) I think you do need to have in-person recruitment to have effective recruitment and retention with minorities. While the tool itself will hopefully save time and labor, the study will need the personal touch to recruit people who have distrust of the system and technology and view mental health issues as stigmatized. The only way to get around that is by having people they can relate to explain to them why this is important.

Tung, thank you for thoughtful questions. You raise points that we ourselves have grappled with when we think about where CAMS might be in 3-5 years: generalizing outside of psychiatric contexts; having a treatment pipeline set up to get resources to people who are in most need; interfacing with primary care physicians. These are all critical issues that will need to be done correctly if the long-term goal of CAMS is to be realized. However, as CAMS is still in early stages, we note that these are beyond the scope of the project for the next year. Our immediate goal is to run large number of people from diverse communities through the CAMS protocol to examine how expressive behaviors covary with symptoms of mental illness and how this relationship varies by cultural context. If CAMS is to be a useful tool for delivering high-quality mental health care to diverse communities in the future, we think this is the critical first step. It is worth stating explicitly that CAMS is not intended as a replacement for primary care physicians or mental health professionals. The phase of the project that we are proposing here is not designed to provide treatment (though we have crisis/emergency protocols in place to connect participants in all of our studies to urgent mental health care if necessary). Our intent is to build a tool that is cost-effective, scalable, and that fundamentally shifts the way we gather information about people’s mental status towards a more objective approach. While we look forward to the phase of the project that involves integrating CAMS into mental health and primary care settings, the current proposal is geared towards engaging the community in a data-gathering effort in order to make sure that CAMS can serve our patients’ needs in the future.

--Craig, Ellen, & Josh

Interesting concept!  Many of our current assessments have inherent limitations based on accessibility of potential normative samples to the researchers who are obtaining the normative data.  Using technology to break those barriers of access is a wonderful first step.

This virtual assistant would change triage, as we know it.  Thinking about the growing number of seniors, people using telehealth/home care, people living in rural areas, this could be so helpful to the social service industry.  I’m excited to see this proposal blossom.  

Thank you, Stephanie! We are also excited about the potential to make objective assessment more accessible for these populations.

Thank you for this very interesting proposal!  As your team mentions, implicit bias is a challenge in the assessment of patient health status, as well as in treatment of illness.  How might your team work to address implicit bais in study design and interpretation, including interpretation of community input?  Would your team be interested in, or capable of leveraging community input for designing how visual and audio data is interpreted by CAMS, or is the assumption that it is designed to perceive mental health cues without the implicit bias of researchers built into the app? How might the team beneft from the input of our community-based partners who could bring culturally-informed perspectives to the project, especially from communities disproportionately burdened by mental health disparities, byond assistance with recruitment?  

Thank you, Roberto. Implicit bias is such a critical issue in medicine that we are hoping to address with this study. Our approach is fundamentally data-driven; we are indeed aiming to minimize the impact of our bias that way and let the body of data, audio/video signals and people’s reports of their own symptoms, speak for itself. At this phase of development, we are not offering a treatment or intervention for the community, but rather asking the community to partner with us in a large data-gathering effort so that we can build an inclusive technology that will ultimately serve multiple communities. We believe this “data first” approach will yield the most impactful tool. Part of our motivation to structure the project this way comes out of experience using technology in our laboratory. We have eye-movement tracking equipment, for example, that can provide rich data to help us understand the pathophysiology of mental illness. But we’ve noticed that it only works reliably when used to track white people’s eyes--presumably because the technology was not developed with input and data from racially and ethnically diverse samples. Ideally, collecting large amounts of data from diverse communities in the proposed project, we help us identify markers of mental health problems that might be unique within each community and guide development of the technology itself. Given that each community has a unique perspective on interactions with the medical system and mental health care, we would definitely be interested in discussing our approach and plans with community representatives to get feedback and understand how we can strengthen the study.

--Craig, Ellen, & Josh

Thank you for this interesting and thought provoking proposal. I'm with my colleagues in that I believe there would need to be a relationship built in order for this to be effective, and I am intrigued by it. I also know that for Black/AA folks recording anything and sending it to "an institution" may be a very big ask due to much deserved criticism of medical instututions and structural and individual racism, and historic and not so historific horrific treatments. How might you see addressing this?

Thank you for this feedback, Monique. The comments and questions from you and your colleagues are really helping us to think critically about our project. We recognize that this is a big ask on our end, and that there are significant challenges to engaging community members in a large audio/video data-gathering effort. Particularly given the remote data collection approach, building trust with our participants is critical. We want to be transparent about the fact that we are not sure about the best way to do this, and would greatly appreciate input from different community representatives about strategies that might be helpful. We have thought about the possibility of holding Q&A sessions and demonstrations at community-based organizations and clinics to provide an informal forum for people to meet our team and discuss their concerns. We imagine, though, that there are other steps that we could take, and that the approach may vary depending on which community we are interfacing with. Hopefully we will have the opportunity to brainstorm more ideas with community representatives.

--Ellen, Craig, & Josh

Commenting is closed.