Sensory Gaming Platform Panel

May 6, 2015, NeuroGaming Conference, San Francisco—A panel moderated by Jim Gatta from the legal firm Pillsbury considered the evolving interfaces and emergence of sensory gaming platforms. Panelists were Tan Le from Emotive, Richard Marks from Sony Playstation Magin Labs, David Holz from Leap Motion, and Nick Thomas from Immersion.

The nature of changing interfaces offers the opportunity to make games more immersive. The technologies allow new ways to interact with games, and have the games become more personalized through brain feedback. Biggest inroads?

Le noted the technologies allow for closed-loop feedback especially when add to AR and VR as input devices. There are multiple and diverse approaches all enabled by low-cost.

Marks commented that VR advances include higher quality leading to greater immersion.

Holz added that VR and AR are almost really cool, and now adding inputs to more senses from just vision and sound to include emotive and contextual content through closed-loop feedback.

Thomas stated that more technology is moving closer to increase synergy will improve consumer adoption. The technologies are moving from R & D to real products.

Changes at companies?
Thomas mentioned that some of their products are the result of over 20 years of research. Now haptic capabilities are getting to market and getting greater adoption. Mobility is the new focus from earlier versions like Rumble. The latest haptic functions are nuanced 3-D actuators. A haptic language is coming into place. A recent study had some subjects express emotions through a haptics interface.

The emotions were communicated and recognized by untrained test subjects at a 60 percent correct rate, meaning the haptic inputs were able to transfer emotional content. The dynamic integration of emotions and a haptic layer will enable more apps leading to more adoption. The new sensory systems are then paralleled with audio to further increase immersion.

Gestures useful?
Holz commented that the latest motion capture interfaces are a change from a 3-D screen to something wearable as an entryway to total 3-D. The latest devices integrate the physical and VR spaces, as a wearable sensor can now track hand locations and movements and show the hands in the VR space. This new visualization is more like the real vision of your hands in physical space. The VR aspects are moving into AR allowing for a transparent perspective that integrates physical and virtual representations.

Marks added that their R&D group is also looking into eye tracking and other interfaces. The issue is not what is possible, but what to do with the technologies. Currently, the technologies are too broad and abstract in their application. They are looking at mixing the literal and abstract views. The apps for the PS4 have to integrate with the existing controller and platform. There are many changes to add more VR to the system. They do have a full headset and some VR content to be released soon.

Le offered their latest version is a second-generation product. A lot is involved with software updates even though the hardware is still important. The new software adds diagnostics and file services to enable some database functions. First, the brain adaptivity tasks, like a fitbit for the brain, also measures attention and stress management. The back-end analytics allow for trending. Second, a new environment to collect and share results to create a large narrative sample. The apps include software for research that can take advantage of the higher fidelity in the sensor array.

The biggest issue is that the classical approaches for brain function analysis are broken, due to the high attached labor and expensive hardware only allowing small samples. Getting more people involved will lead to better models that can reflect a general population, rather than the specialized sampling now taking place. They hope to distribute and accelerate research platforms.

Challenges?
Thomas opined that the business proof points are needed. There is a measurable ROI for new technologies and the addition of haptics. The industry needs to increase outreach, provide more a-b studies on haptics on gaming experiences. Improving metrics for parameters like immersion, especially for casual games is important.

Holz noted a transition from controllers to hands and controllers. There is a spectrum of actions possible as you change from a direct interaction on a device to 1:1 images of hands within a game architecture.

Marks added that they have to balance product costs versus encumbrance. The user experience determines which sensor applications are needed for a function.

Le contended that content and pricing matter. As more people can afford headsets, the problem is that an EEG headset is very visible. As a result, it is compelling to use in short bursts and provides insight, but to get more use of these devices, there is a great need for a killer app. All the sensor arrays are just platforms now, so the question is what is the right space for these technologies. A more openly available set of platforms may be the desired response.

Data collecting sensors raise an integration question about platform fallibility?
Le noted that reliability is a function of cost and is a multi-dimensional issues. The value add and getting a closed-loop user experience requires that the VR devices have much lower latencies and better motion tracking.

Marks data are a synergistic function due to the integration. The different user experience augments the traditional experience.

Holz considered that the hand and eye sensors are good technical considerations for better integration. The eye tracking allows for determining focus and the hands indicate intent. For example, you don’t have to hover to get focus on an object. Technology maturity provides functional reliability.

Thomas suggested that there is a cost-benefit tradeoff, and also a chicken-egg problem. The industry needs to change to better content and technologies. Reducing silos leads to connectivity to full immersion. Unity is integrating the technologies into their engine.

Reducing the cost of hardware?
Le noted that the EEG space started with machines starting at over $60k – $100k with the software. Now an equivalent sensor array costs less than $300. The issue is the availability of content to justify the sensors. Eventually, as higher volumes we should see $99 multi-channel, actively filtered headsets.

Thomas claimed that they have been developing software for over 20 years and the software complexity partially drives hardware costs. The industry needs to spend more on education and information.

Holz considered that the hardware can only see small changes. Consumerization of cameras and LEDs lowered the costs for all, but new applications and functions still need R&D.

Marks added that commoditizing of products drives the costs down for all.

Emotive headsets are conspicuous. Crowd source research leading to a more inconspicuous headset?
Le answered yes, but the key is getting to the third product cycle. They are on the second generation, so still need R&D for apps like telemedicine which are viable now but lack content. There are also form factor issues and questions on the quality of the data. There is a great need to fill out the brain database.

Commercial uses and different interfaces?
Thomas noted the technologies have universal value. There are lots of areas to look into and many potential areas for expansion.

Holz advocated inputs for all senses, and outputs for all senses. The haptic nature of human interfaces calls for new technologies that address energy, physics, and software.

Marks noted that the experiences are not found by the developer or designers. For example galvanic skin response could be great, but no user experience depends on it.

Game developers integrate EEG signals, etc. or aggregated functions?
Le suggested do both. The state data or the raw eeg signals are both sent to computer algorithms, so it depends on the level of interpreting wanted. Most standard functions are in the SDK. Some therapeutic functions need the raw data and diagnostics, so the researcher needs to build new algorithms.