Analyzing Social Interaction through Behavioral Imaging
February 10, 2015, Electronic Imaging Conference, San Francisco—James Rehg from Georgia Tech talked about developing new tools to use the many available video analysis capabilities to evaluate social interactions in younger children. The growing capabilities of vision tools and other sensors provides an ability to measure and analyze behavior.
In the recent past, some of the analysis depended on CT or MRI scans and required a laboratory environment. The goal of their research is to move behavioral analysis from the lab to a larger scale and capture responses in the wild. The current focus is on autism as all treatment and analysis is hands-on observations and is very labor intensive.
The application of technology to the problem has three goals; early detection and intervention, treatment and measurements of treatment responses, and new research. They want to develop tools for large scale collection and analysis using mostly video data. The videos are enhanced with data from a variety of sensors on both the patient and the provider. The underlying protocols are designed to assess and probe the responses.
The first issue for an autistic child is a lack of social connections as in limited eye contact. The target is to sense subtle changes at a very young age, since early interventions show the most effectiveness. Ideally, you want to capture key social and behavioral responses between 18 and 36 months of age.
One technique is to measure response to the subject’s name being called out. Motion capture is through a Kinect video system and other software measures any turns and the time for the response. Other efforts are in place to move more workout of the lab. Wearable cameras mounted in eyeglass frames are good for eye contact evaluations and do not threaten the patients as other body-worn cameras do. In addition, there is an inherent reference from the practitioner’s point of view since the camera is located in the middle of the eyeglass frame.
Similar tools can also be used as front viewing camera for the patients to enable prediction of gaze during hand-eye coordination tasks. A head-mounted camera to gain a relative perspective of the head and implied eye directions. The evaluation software uses forward pixels to generate a locus of focus without requiring an eye tracking camera for the patient. The software creates a center prior that is as realistic as a gaze tracing on a monitor and is even better with wearable cameras as the child’s head can move freely.
People use their eyes to coordinate their hand motions but children haven’t developed a full sense of proprioception, the kinesthetic sense of where your body parts are without looking at them.
Other use cases for the main technologies include hand-eye coordination testing and coarse and fine motor skills. A set of tests single hand or both hands to be involved in a task as well as hands together or apart. The software can identify a manipulation point for the task and show changes in that focal point based on predictive models from the pixel information.
Gaze prediction is used a lot in autism work. the autistic children have challenges in controlling gaze and attention, so this technique is used to measure response. Unfortunately, there is no standard or instrument that can reliably and repeatably detect and measure gaze shift for frequency and duration. Use of wearable cameras is providing a first-person view of the patient and face detection, facial details, and eye tracking are all software tools that are readily available. this setup allows for easier detection of gaze changes and doesn’t require any sensors on the child for gaze tracking. Other sensors on a small vest can monitor body movements and physiological parameters on the child.
These imaging solutions are a means to measure social behaviors in an unobtrusive manner. The advances in wearable technologies also means that the adult doesn’t have to fuss with the technology, while the resulting videos can be used by other professionals and parents as well as baseline studies to generate trends. Future tools will be able to follow attention shifts, as eye contact is only the first step.
The tools are slowly converging to map well to the analyses of expert panels, so the increase in detail is helping the whole area. Even though the details are getting better, the tools still are not as good as humans. The move from lab to the wild is happening. The first-person perspective available with wearable cameras is enabling analysis of attention and helping to identify social roles. The ability to infer a 3-D projection of all people in an environment helps to identify focal points and develop maps and track attention over time.
The new tools will help to define social interaction features and a taxonomy of categories of interactions. Other associated activities will include mapping behavior to health, and using big data techniques to advance our knowledge of behaviors. There are possible applications to behavioral medicine and support for real-time interventions. The field has potential for lots of new work and may result in a new science of behavioral imaging.
