Augmented Video Communications
September 10, 2014, Intel Develop Forum, San Francisco—Stan Baran described the requirements and recommendations for HD video communications. The capabilities require numerous hardware and software functions that are highly interactive and open to a wide range of optimizations.
The basic application uses the capabilities of the base processors, optimized drivers, and a number of tools for media. The Intel Media SDK, SDK for OpenCL, and Performance and profiling tools are all used to optimize the final app. The result is a high-performance, minimal power application. Adding the RealSense SDK capabilities helps to differentiate the app.
Video communications can take advantage of the many graphics functions in Intel processors. Decode and Encode is performed in the MFX engine, while video pre-and post-processing and some encode are handled in the execution units. Media at the pixel level are handled in the backend cache as is image quality. Geometry tasks are processed in the execution units, and sampler and instructions are in the bus interface.
Video conferencing requires low latency encode and decode that is coupled to dynamic bit-rate and resolution control to adapt to the changing transmission channel. In addition, the function needs forced key frame generation, reference picture marking, and temporal scalabilty, which is developed from a per-frame quantization parameter with rolling I-frames.
In addition to conferencing, video is important for entertainment and gaming, reality books and interaction , and control and navigation. The tools provide use libraries and development environments to support Microsoft visual studio and eclipse IDEs to create apps for Windows, Linux, and Android. Much of the code is in OpenCL to allow greater usability and cross-platform developments.
Some of the optimizations for quality improvements with encode, decode, pre-and post-processing include constant quality and bit rate control modes that target use for best sped, balanced operations, and quality modes. The resulting system can speed up performance up to 5x compared to a x264 software encoder at similar quality.
Quality of service can be optimized with temporal scalability which can take multiple frame rates from a bit stream and define skippable frames.
Use mfxExtAvcTemporalLayersextended buffer
Encoder inserts prefix Network Abstraction Layer
Unique sequences use mfxExtCodingOptionSPSPPS)
A reference list selection can change the encoding based on feedback from the decoder.
Use mfxExtAVCRefListCtrlstructure
Use PreferredRefListand/or RejectRefList
Contrast with Long-Term Reference List
A partial list of pre/post processing functions includes:
- Color Space Conversion
- De-interlace
- Inverse Telecine
- Resizing -Scaling, Cropping
- Skin tone detection and enhancement
- Frame rate conversion
- Motion-adaptive noise reduction
- Detail -to enhance details and edges
- Image stabilization -upscale or boxing
- Automatic contrast enhancement
- Total color control
- Gamut correction
- Brightness, contrast, hue, saturation
Recommendations for power and performance call for moving as much of the video processing to the GPU and leave the data management and quality assessment issues to the CPU. In this way, most of the processing tasks are in the background and don’t impact the other conferencing functions. In addition, moving the video processing to the GPU reduces latency by over an order of magnitude compared to using the CPU while reducing total power.
Some functions like background segmentation use a mix of CPU and GPU for the segmenting process, then use the GPU for blend and render operations to reduce CPU load and overall latency. Then the CPU only has to manage the communications interfaces.
The computer vision functions from the communications can also be used for other purposes. Image recognition, motion analysis, and reconstruction and restoration all benefit from the GPU capabilities which have commonly used filters and statistics functions, and a dedicated motion estimation engine for video analysis and codecs. The optimizations from Itseez on the OpenCL and OpenCV allow a single code base for CPU and GPU while improving processing speeds.
Facial analysis is one key building block for many functions, and uses other building blocks for its own functionality. Detection and tracking requires defining region of interest for landmark detection. The landmark detection extracts key features of the image and maps those points to recognize a person. The region of interest also allows tracking.
The tracking function is used to perform image stabilization. An input sequence generates a global motion estimation which is used to compute trajectories and smoothing corrections. When the two functions are combined, the result is a stabilized sequence.
