Mind Controlled Interactive Music

October 27, 2012, AES Convention, San Francisco—A panel looked at the technologies and capabilities of mind-controlled interactive music. These active input technologies enhance the contextual responsiveness and reality of a game. Adam Gazzaley from UCSF moderated the panel. Panel members included Nicolas Tomasino from IGN Entertainment, Al Gurlalei from UCSF, Kyle Machulis from Nonpolynomial Labs, and Jim Hedges from Zynga.

Gazzaley stated that game platforms are evolving to enable emotive, cognitive control into game engines like Unity. The pipeline for audio control matches the game and video streams. An emotive headset with 16 channels for EEG and signal processing technologies can infer a general sense of intent.

Machulis agreed that the street price of a consumer-grade EEG sensor is under $300.
Gurlalei stated that an EEG in the lab would use 64-128 channels, but the data from fewer sensors is sufficient for many game functions. Vendors need to provide an SDK so developers can access the raw data. Tomasino commented that the control functions are usually in space. The newer technologies are tapping into the mental states and facial musculature.

One challenge is that the digital pipeline is very long. You have to send data to the game from the headset, to the signal processor, and then to the game engine. The hardware costs are dropping and brain-computer interfaces are getting better. Many companies are working on the equipment and software, and dry electrodes are appearing. The game industry is using the EEG data to modulate the game content with information on mood and intent.
Gurlalei noted that functional MRIs and other research tools are showing that the brain changes over time, with disease, etc. The research is focused on therapeutics and creates studies to develop ways to change the brain. One path is through games targeted at changing memories and other cognitive functions. The combination of visual and audio stimulation plus adaptive algorithms in real time allow the mapping of behavior and neural activity to change the games. The player uses the visual and auditory cues to change behaviors. The technology to push real-time source localization allows people to step into a virtual brain image. The brain rhythms stimulate the hearing and can change behavior and control functions.
Hedges added the adaptive music generation starts with chips. The digital outputs align with game events and change with the play by tying game play to the brain state. The challenge is for the composer to develop a score and an adaptive mechanism that still flows with the game. The in-game engines need to have full bi-directional data flows to allow the game and player to respond to changes.
Machulis considered the nature of the feedback. For game control, the more senses you use, the better. Now the audio and video are linked. For example, the new Wii controller adds more capabilities to the game. A haptic controller needs to have mass and kick-back to address the touch senses. The issue of active versus passive feedback in games goes a long way back. Atari had an EEG accessory in ’82.
Direct control means greater integration into the game, but the costs have been too high for consumer electronics until recently. The addition of biometrics to run around and control the sound around the game needs better game-interface integration. One key, as seen in Second Life, is the need to match the experiential biometrics with the action. There are lots of possibilities in a game for an avatar to take the effects in context.

Audio bio-feedback better than visual?
Gurlalei responded that we know a lot about the visual system. Biofeedback that addresses the brain rhythms has potential for better entertainment. The high connectivity between audio and brain activity is well known. The brain uses separate functions for hearing and seeing. Mixing audio and video makes both better.

Integration is a challenge? Currently, there is no separate path for audio modifications in a game.
Hedges complained that we need new middleware so we can isolate the brain signals and use them as inputs to various functions. later, we can integrate the functions better. The middleware needs to be capable of influencing the game at the nano-scale. The other side is the actual synthesis of the sounds. The game platforms will need much more processing power to construct smooth transitions in the audio. There are no good tools for this type of work.

Neurosky uses a one point EEG. The software for audio-video feedback helps with the controls. Is it realistic to expect highly controllable interfaces to replace the existing operating modes? Fine-grain control is not easy, is it even possible?
Gurlalei agreed that fine-grain control is hard to do. If developers can make it easier, then games would be more fun. The signal processing requirements call for much better hardware to perform the many additional tasks like artifact reduction. Current work is looking at therapeutic efforts to modify an environment to create a new experience.

Will composers be out of jobs? Will they move to be more of a guide?
Hedges responded that linear media dictate emotion, but the linear mode is fixed. A non-linear mode can anticipate behaviors, but is susceptible to over adapting. The ideal solution is to monitor feelings during play. The games will still need composers, but their job will be more of a pointer and director than background fill.

Other biofeedback modes like EMG, galvanic?
Gurlalei considered that the other modes are relatively easy to collect, but suffer from a large temporal lag for the physiological parameters. An EEG is real-time data with spatial resolution. The issues are due to poor signal-noise ratios and no good software for the amount of signal processing needed.

Gazzelly objected that the EEG interface requires a lot of personal training to match the area of the brain with a control function. This training is a high barrier to more widespread adoption.