Making the Kinect – IGIC
November 2, 2011, International Games Innovation Conference, Orange, CA–Zsolt Mathe from the Microsoft KInect group talked about the magic behind Kinect. He described the development process and the changes in approach needed to make a new user interface.
They started with hardware from PrimeSource and held a hack-a-thon to cobble together some software. The magic parts were the fusion of the technologies with new concepts of interfaces. The technology provided depth maps, but they had to create the skeletal tracking, biometric identification, and speech recognition with noise cancellation functions from scratch.
The development required many breakthroughs. It was hard to define the specifications, and even harder to define what constituted a bug. Examples of new bug types included identifying children, very tall people, and shadows. The technology uses an IR emitter and depth sensor to create depth maps. The images are from a pair of color sensors. The depth field is from 800 to 4000 mm
Sound capture is from an array of 4 microphones to enable noise cancellation. The mikes are arranged to focus beams for better noise immunity. The beams identify a rectangle in space for tracking and are integrated as part of the user interface. The noise cancellation is necessary for speech recognition, since the mikes are far from the users. All of the hardware and drivers needed extensive tuning to get reasonable responses. The software for the depth processing uses a look-up table to reduce the computational loading.
Putting the new interface into a game required the development of joint tracking and filtering code, and they had to define what a gesture represented. Skeletal tracking was reduced to 20 joints for a standing person, and 10 for a sitting person. Head tracking is a separate function. The resulting signals are represented by x,y,z points and the system can track two people and detect 6. Making the gestures and the speech work together to enable speech commands while moving was another challenge. Part of the processing stack is in the sensors, the balance is in the Xbox.
They had the machine view over 1 million images to create exemplars for people. Machine learning needs even more images to improve the tracking. The system uses probability of matching to make a track for gestures.
The challenges for the speech recognition is that they area working with untrained, distant speakers. Most voice recognition systems need known speakers in a quiet environment to get good results. The Xbox needs to work with anyone. Therefore, they designed an acoustic architecture to use the 4 input channels to define the background for noise and echo cancellation. They can beam form and direct the combined responses from the mike array to get fairly fine-grained spatial acoustic images the system is monophonic and is digitized ad 16 kHz in 16 bits. Speech generates an angle map and identifies the person. If the talker and the gesturing people are not in the same location, then the system tries to determine if more than one person is active. All of the audio processing is in the Xbox.
The Kinect is just the hardware and a data stream. Everything else is in the new software. They will be releasing an API and driver set so people can develop applications on PCs. The Kinect development kit will eventually be open to all platforms. Currently, the system has a game-focused and limited vocabulary.
