Music, Sensors, and Emotion
May 17, 2013, Music Meets Science Forum, SwissNex, San Francisco—Brennon Bortz from Virginia Technical University presented their research results in music, sensors, and emotion. The work involves many sensors and other technologies as well as a range of experiments.
Controlling music from biofeedback is now relatively easy. The difficulty is in tracking gestures and emotion during a performance with a live audience. In any performance, a good show causes a contagion of emotion. The listener to listener interactions are at least as important as the performer to listener interactions.
A difficulty in measuring the audience is that they are mobile and unconstrained, unlike the performers who are mostly limited to the stage. The shadow media is a shared service that enables co-creative expression over a distance. This is a video chat plus other sensors and special lighting. The projected shadows run from the person to a wall and evoke a sense of presence because it moves in tandem with the person.
The first experiment is to have two people in the same room who interact. The resulting responses including touching are recorded and analyzed. The same experiment is repeated with slit screens, and shadows only, allowing for totally remote interactions. The resulting video show the physical responses to a real or virtual touch both show increased arousal. Direct interactions, interactions across a split screen, or shadows only all give the same response.
The physiological data were captured with EEG, EKG, GSR (galvanic skin response), and EMG sensors for all subjects. With better algorithms, the number of sensors can be reduced for each participant, greatly reducing the amount of data generated. A new sensor set includes a pulse oximeter, EDA (Electrical Dermal Activity), temperature, a 3-axis accelerometer, and an audio output.
The experiments show that contagion and training can happen over a distance. Even extended distances from VTU to Waseda University in Japan are workable, so the lag and latency is not a factor. The software tools and frameworks are on iOS now and will be ported to Android soon. The framework platform and functions allow the experimenter to see the response of others
These results enable performance and audience tracking through correlated trends. Two performers are highly correlated and create emotion in motion in response to the music. So far, they have tested 10,000 people with 20,000 songs. The participants were instrumented and the data are now available for further analysis. One difficulty is that even with the large data set, there are large variances from person to person as well as for an individual.
The context and applications need more research. Some fairly universal functions like changes in volume or pitch lead to changes in response. The data are just the physical responses, so their next task is to map those responses to emotions. This is a hard task and requires a lot of data cleanup to make the data more useful. For example, the EDA values need to be taken from the arm or other part of the body, because the fingers give off sticky sweat.
