CHS: Small: Extracting affect and interaction information from primary care visits to support patient-provider interactions

Primary care is an integral part of an effective health care system that emphasizes continuous and preventive care. However, the United States faces a shortage of primary care physicians that is projected to get worse, a shortage most pronounced in clinics that serve rural and minority patients. This shortage can lead to stress and physician burnout, affecting both the shortage itself and the quality of patient-provider interactions. This project will develop methods for semi-automated analysis of primary care visits, with the long-term goal of detecting both momentary and longer-term stress, then providing feedback and reflection systems to help reduce stress and improve the quality of patient care and retention of physicians. This will require both advances in video processing to recognize aspects of how humans interact with both each other and with technology, novel models of how these interactions affect the group's affective state and relationship, and initial design inquiry into patients' and providers' needs during primary care visits. This, in turn, will require providing interdisciplinary educational opportunities for health service and engineering students. The project has two main aims. The first focuses on developing methods to extract data from clinical encounters about stress, burnout, and interaction, leveraging a large existing dataset of recorded primary care visits collected in several contexts. The team will develop techniques to recognize both non-verbal and verbal cues in interactions, including eye contact, facial expressions, posture and body language, turn-taking behaviors, and indicators of socio-emotional exchange such as facial mimicry. The team will also extract cues about how the people involved interact with technology through gaze analysis, evidence of typing or touchscreen use, and screen sharing/co-referencing behaviors. These methods will be validated by comparing their performance against human annotations, with the extracted data informing the second main aim around determining opportunities and requirements for developing tools to provide feedback and support reflection. This second aim will be pursued through user-centered design methods, including reviewing extracted data with clinical collaborators to evaluate its potential value, workflow analysis of patient-provider interactions informed by ethnographic observational methods, and thematic analysis of focus groups including both patients and care providers. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.