Streaming Format Evolution — Does CMAF Give Us the Single Format?

The HTML5 video standards have been evolving for the last years to a point where they can be leveraged for premium video experiences. It evolved from the early struggle of finding the right codec to work across all browsers, to the gradual adoption and the maturity of Media Source Extensions (MSE) and Encrypted Media Extensions (EME) for DRM protected content.

Digital Rights Management at scale is essential for premium video content, which means we need a single streaming format

The reason for this statement is the volume that is normally associated with a premium video library. As an example, a video catalog of 200.000 titles highly benefits from a single format purely because of storage costs. The same is true for live streaming, where a single format leads to higher caching efficiency inside and outside of the CDN, which then results in an overall higher video quality with content cached closer to the end user. In addition, the live encoding / packaging requirements are higher with multiple formats. This is why the industry as a whole is looking for the single format that will play across all screens.

The Single Streaming Format

For non-DRM protected content the single format has been a reality, and it is Apple’s HTTP Live Streaming (HLS). The reason why the format prevailed is because it is the only streaming format that plays on iOS / TVOS devices and is approved for cellular networks, combined with Apple’s dominance in the devices market (nobody can afford to exclude iOS from their device strategy), and the public release of the HLS specifications to the IETF early on. The specs allowed every device manufacturer to support HLS streaming and play the same content as iOS.

Since the specs can be implemented differently, devices have quite a bit of quality variance for HLS playback. This is where solutions like Adobe Primetime come in, which can normalize the implementation quality and make it premium video friendly.

In parallel, MPEG-DASH evolved as an independent standard with the promise to unify the industry around one format, which succeeded everywhere except Apple devices.

The Obstacle of DRM

There are two main challenges with the single format and Digital Rights Management.

The use of multiple DRM systems is a studio requirement to reach all HTML5 browsers and devices

DRM fulfills a single purpose. Protect content from illegal redistribution and violating the terms of use of content owners, which are often movie studios. Unfortunately, there is not a single vendor DRM solution available across all HTML5 browsers and devices. Even though the HTML5 EME interface is standardized, it does not include the ability to use a DRM of choice. The DRM technology is included as part of the browser itself, and each browser supports a different DRM.

• Chrome – Widevine
• Internet Explorer – PlayReady
• Firefox – Adobe Access / Widevine
• Safari – Fairplay

To avoid having to package a different stream for each DRM system, the concept was introduced to use multiple DRM systems with a single streaming format. Common Encryption (CENC) for MPEG-DASH, and the unofficial HLS equivalent, HLS with Sample Based Encryption.

DRM systems in the market are not unified on the segment encryption mode, and were not on the segment container until recently

This simplified visualization demonstrates the layers involved.

1. Adaptive Protocol: This is can be often parsed by the player itself and is not tight to a specific platform, with the main exception of iOS (HLS)
2. Container format: This is can be often parsed by the player itself and is not tight to a specific platform, with the main exception of iOS (Only TS until recently, now fMP4 as well)
3. Encryption mode: The two common encryption modes are CBC (Cipher Block Chaining) and CTR (Counter Mode). In case of a hardware supported DRM, the mode is tied to a specific DRM on a platform.
4. Video codec: In most cases H.264 today, with HEVC and VP9 emerging for 4k content. For best performance, the underlying hardware should support the codec in hardware.

While the container + streaming formats can be often rewritten by the client, and platforms have unified around H.264, the DRM systems have chosen different encryption modes, which leads to the following challenge.

Modes supported in hardware

• PlayReady – MPEG-DASH/fMP4 -> CTR
• Widevine – MPEG-DASH/fMP4 -> CTR
• Fairplay – HLS/TS (announced with fMP4) -> CBC

Since the encryption mode is hardwired into the underlying DRM system, it is not possible to transmux the format on the client securely in the application layer/player, which means at least two formats have to be used to reach all browsers and devices.

The Introduction of Common Media Application Format (CMAF)

The challenge of at least requiring two different container formats, and therefore streaming protocols, has been identified as a problem by MPEG, and efforts have started to standardize on a single fMP4 based format, CMAF.

The core mission is defined as:

“Several MPEG technologies have been adopted for the majority of video delivered over the internet and other IP networks (cellular, cable, broadcast, etc.). Various organizations have taken MPEG’s core coding, file format and system standards, and combined them into their own specifications for their specific applications. While these specifications share major common parts, their differences result in both unnecessary duplication of engineering effort, and duplication of identical content in slightly different formats. The industry would benefit if application consortia could reference a single MPEG specification (a “common format”) that would allow a single media encoding to work across many applications and devices.”

(Quoted from the CMAF requirements document, which contains more detail.)

The streaming ecosystem has been trying to unify around a common format with MPEG-DASH, but the exclusion of Apple’s support was a challenge. This changed with the WWDC 2016 announcement of Apple starting to support fMP4 as additional container format for HLS, and their involvement in CMAF, which is trying to develop a standard fMP4 container. fMP4 can support both CBC and CTR, and therefore is compatible with most hardware DRM systems.
Since solutions with low level video engines, such as Adobe Primetime, can technically provide support for MPEG-DASH / HLS in combination with CMAF across most devices, we are getting pretty close to truly single DRM protected format – with one exception: the encryption mode, which is tied to the DRM system of the hardware.

Since Apple did not announce support for the CTR mode in Fairplay, and the other DRM systems have not indicated support for CBC, a truly single format for DRM is not the reality yet.

Conclusion

A single format format without DRM has been reality with HLS + TS in the past, and now could also be possible with HLS + fMP4 with Apple’s support and potentially added support by other devices and player video engines. Unfortunately, it is not yet possible when DRM content protection is required due to incompatible encryption modes. Nevertheless, it’s a promising new initiative in the industry and heading into the right direction, which deserves paying close attention.