Clouds, Pixels, and Video Streams
June 16, 2015, SMPTE Entertainment Technology in the Internet Age, Stanford, CA—Neil Hunt from Netflix talked about a bifurcation in trends that will affect all aspects of video content. Video is moving towards higher quality; bigger, brighter, and better, while also going to mobile devices, which are smaller, faster, and cheaper.
Netflix is adding about100 M hours of content a day. Much is going to Internet TV, but the problem is that few can find what they want to watch. The capabilities of more technology in the delivery chain enables greater personalization compared to linear TV or most cable offerings. The problem is how to curate the 10,000 channels available to the 10-12 that you really want to watch.
TV still dominates the landscape, but the Internet is much more agile. The Web delivery is first to use new standards, since it is not dependent on an SoC in the set-top box. Hardwired infrastructure has a very slow upgrade cycle, while the Internet technologies are field upgradeable and facilitate a narrow-cast 1:1 stream.
The move to 4KUHD is starting. It is possible to deliver 4K content now, albeit at the cost of requiring downloads, but by next year, expect to see streaming delivery. All new Netflix original programming is now done in 4K. The HEVC codecs are learning to optimize. Currently they only get about 10-20 percent greater compression than AVC versus the theoretical claims of over 50 percent. This level of progress in new codecs is disappointing.
One challenge is to decode at 60P, which is not possible now. One question coming up is do we need 30 and 60P formats? In general, the industry will be supporting all formats to increase their footprint. The advent of high dynamic range and wide color gamut will exacerbate the problems. Some HDR content exists, “Marco Polo” for example, but the challenge is to remaster the content for HD and a smaller color space.
The formats and color spaces are so different that conversion has to be manual on a frame-by frame basis. The conversion problems are not getting easier. The industry needs alternative standards or the proliferation of formats will continue, causing labor and file issues.
The new formats need full HDR pipelines and equipment. The biggest issue is to retain the director’s intent across the many formats and displays. Most TV sets are too bright and hot in rooms with colored lights. We need a standard for perceptual user experience that refers to calibrated equipment and standards. The TVs need to have a reference mode and an ambient light sensor that detects light intensity and color.
The broadband infrastructure needs to embrace adaptive streaming technologies. The bandwidth needed for acceptable images is as follows:
Image Min. B/W Min. B/W HDR
720 P 4 Mbps 6 Mbps
1080 P 6Mbps 8 Mbps
2K 8 11 Mbps 14 Mbps
4K 16 Mbps 20 Mbps
Few people complain about the quality of HD, but many complain about stalls and buffering. As a result, part time 4K is probably OK for many viewers. Also, some of the effects are mastered in 2K, so a 4K video may have some 2K content embedded in it.
The move to high frame rates is a different technology that is more realistic and compelling. The biggest issue is addresses is motion blur. The difficulty is getting the directors to take proper advantage of the technology, much like the differences in production caused by moving to 3-D. the move to 4K also changes the field of view. An SD 16 x 9 image at 3 H viewing distance encompasses a 33 degree field of view. A surround view of 48 x 9 at 3 H viewing distance results in a 100 degree field of view, completely filling a normal human visual range. It is possible to get a similar viewing experience by cropping an image to 16 x 3 or 16 x 9 at 3 H. These differences in user experience also change the acquisition processes.
While TVs are moving towards bigger, brighter, and better pixels, we cannot ignore the growth of the mobile market. Last year, the industry produced 1 B new tablets and 2.4 B new smart phones. There are about 3.5 B wireless broadband devices compared to about 800 M wire line devices.
The mobile view experience is very different than the fixed TV. There are lots of stops and go of the streams as bandwidth changes over time. The viewer is usually solo and on the move, and uses WiFi and cellular networks for viewing. This user needs a 480P image that uses less than 250 kbps. The average mobile user watches about 30 hours of content a month, about 4 GB total. In this market, HEVC is not effective. HEVC is interesting at the high-end TV space, but is not useful at the low end as it creates battery issues. The Google codec shows better battery life at the low end.
The next generation, high-efficiency encoders will stream, but not in real time. The mobile devices all suffer from the same characteristics: bright screens to overcome bright ambient, and poor audio. As a result, adaptive streaming for mobile doesn’t have to recover from lost data if the system has a 30-60 second buffer.
The capture for a small screen must address the image and viewing conditions. A 15 cm screen at a 50 cm viewing distance provides a 15 degree field of view. As a result, the mobile content should have simple images and few people for the best user experience. In general, faces are more important than splash screens.
Encoding the content in the cloud provides greater elasticity and provides more time for the various encoding and other functions for the mobile devices. A cloud workflow is all virtual, and the encoding should also be virtual. Asset tracking includes issues like language, sub-titles, captions, etc. and most of the functions are automated. Encoding is done in parallel streams for multiple bit rates and formats and all the content is checked with a predictive QC program. Multiple standards are OK for the high end like TV, but the mobile devices need to have much more lightweight processing.
The end display devices vary widely, and the cell phones create special issues. Other view screens for mobile devices like Occulus or other glasses are mostly niche, since they force the user to be a passive viewer who is totally isolated. They cause problems because they differ from environmental and social norms. Big screens are easier for social viewing, they allow interaction and change storytelling. The big screen is a parallel branch from the cinematic user experience.
The field of view for different devices takes advantage of the existing standards for metadata. The different perspectives from 15, 30, and 100 degree fields of view change the viewing experience. An 8K screen with a 100 degree field of view will take 40-50 Mbps data stream, but the concept of central acuity may allow lower resolution at the edges. One possibility may be to encode different resolutions by screen area.
While quality in flat panel displays has been getting better, and the new standards for TV sets look promising, they still don’t compare with CRTs and their 15 degree field of view. CRTs provided good images over the past 50 years, and the conversion to a panel scan is the same as that needed for converting cinema to TV.
Expansion to an international footprint is not as easy as it may seem. Although the TV sets are the same worldwide, global content has many language problems. For example, a French movie with Hungarian speakers is already subtitled into French. So how would you translate this content into English? How would you layer the subtitles? The new TV broadcast standards will allow interesting new formats. A narrative-only file for the blind is one possibility. The issue is that they already are trying to manage 150 assets per movie.
Although they have extensive data on their users, they are not sophisticated enough to use those data to manage the creative process. They give the directors total artistic control, which is better than changing the creative to map to an audience. Their content is not the same as linear episodic material, but they do use the data to help determine which shows are likely to be successful. To help the directors, they sign 2-year content contracts.
