Context-Adaptive Multimodal Informatics During Mental Health Hospitalization

Feasibility and clinical utility of multisense for nonverbal feature extraction in psychosis

Pasted Graphic 3

Determining when a patient is ready for discharge is an open problem, as currently 20% of those hospitalized for psychiatric illness will be hospitalized again within 30 days. To address this issue, we are working to identify behavioral biomarkers for severe mental illness in inpatients hospitalized for psychosis at McLean. The Multisense project is a collaboration with Dr. Louis-Philippe Morency’s team at Carnegie Mellon University, where information is extracted from both audio and video of both clinician and patient during a series of semi-structured inpatient interviews. Multimodal analysis techniques are used both to predict clinical scale and discharge-readiness scores, as well as to visualize a summary of each interview and extract relevant interpretable features.