I wrote this article for January 2017 issue of Widescreen Review Magazine. The magazine has been out for a few weeks so I thought I go ahead and put it online. Hope you find it interesting.
---
Politics Of Format Making
Amir Majidimehr
Traditionally I have written technical articles for Widescreen Review magazine. This is a departure in that I would like to share what goes into creating new video formats such as Blu-ray. I was fortunate to have been part of that process when I was at Microsoft. I must say, it is eye opening how the parts come together and the driving factors involved. The picture is not pretty. It goes along the lines of “how sausage is made.” That is, if you knew what all gets ground up in sausage, you may not like to eat it as much!
Let’s get specific with an example: The Blu-ray format has three mandatory video codecs: MPEG-2, and two next generation codecs: MPEG4 AVC/H.264, and SMPTE/Microsoft VC-1. On the audio side, there are two lossless audio codecs, Dolby TrueHD and DTS-HD Master Audio. Do you see something strange here? Three video codecs to do the same thing? Two lossless audio codecs again with the same purpose? Why the multiples? That is the story of this article.
Stepping way back, Blu-ray started off as a Japan-only format for recording high-definition broadcast MPEG-2 on optical media. Unlike in the United States, DVD recording of video was quite popular in Japan so it was a natural progression to enhance it to support high-definition broadcast television. The optical media recipe was changed and the recorder simply captured the already compressed stream and stored it on the new writeable disc format.
Strategically, Japanese consumer electronics companies liked this business model since it cut Hollywood out of the picture. With DVD, the players quickly became commoditized and sold at or below cost. Meanwhile, Hollywood studios sold movies with great profit margin. Heck, you could buy an entire DVD player for what it almost cost to buy a DVD movie! DVD fueled the success of Hollywood studios, with home video sales crossing over the revenues of theatrical release. The higher profits meant the amount of money the talent made went up and so did the cost of movie making. But let’s not go there as that is for another article.
Meanwhile another development occurred: the DVD Forum decided to develop HD DVD. Unlike Blu-ray the focus was on pre-recorded theatrical movie releases. The initial target was to fit HD movies in the same red laser DVD, taking advantage of next-generation codecs to stuff more bits in the same space as compared to MPEG-2. A call for new codecs was put out to match the fidelity of MPEG-2 at around 20 megabit per second (the exact number escapes me) at just 7 megabit per second or so. We (Microsoft) entered our video codec called WMV (Windows Media Video) and others proposed variations of MPEG-2 and then new, H.264/MPEG-4 AVC.
After two rounds of this randomized, blind tests WMV produced the best picture. To many people’s surprise, MPEG-4 AVC did not finish first and actually unperformed MPEG-2 in some tests due to its low-bitrate optimization for low bit rate/resolution Internet streaming. MPEG-4 AVC was later revised to improve its performance for high-definition, high-bitrate applications.
Meanwhile the results of the test were presented to the DVD Forum steering committee for selection of a video codec for HD DVD. We thought our job was done and WMV would be selected as the sole mandatory codec. But that was not remotely the case. First objection was fair, which called for video codecs to have an open specification. So we submitted WMV to the Society of Motion Pictures and Television Engineers (SMPTE) for standardization. They renamed the codec to VC-1 and published its specifications.
Again, we thought we were home free but that was premature or better put, naïve. To understand why, I need to tell you about the great bit of education I received in talking to an executive of a major consumer electronics company. He said their strategy was simple: first make money from selling players. Once the bottom fell out of that, sell silicon/chips that implement the technology. When that gets commoditized, make money from patents of the technology in the format. Wow, that is one well thought out strategy!
Let’s dig into how the rest of the sausage is made. While we had designed WMV/VC-1 in house, both MPEG-2 and MPEG-4 AVC/H.264 were created in standards organizations: ITU and MPEG. MPEG lost its leadership after creating the anemic version of MPEG-4 which was not all that better than MPEG-2. ITU, on the other hand, set a very high target of efficiency and got there with the creation of H.264. Not wanting to get a black eye, the MPEG committee quickly adopted H.264 as its own and renamed it MPEG-4 AVC. Never mind that the standard had little to do with MPEG-4. This is why you see me being specific here with always using the full phrase, MPEG-4 AVC. Just saying MPEG-4 is confusing and may very well mean the old inefficient version.
The process for creation of new codecs in these organizations involves joining the group and then sending technical people to help drive the specification. Major Consumer Electronics powerhouses of the 1980s were present when MPEG-2 was created. We are talking Panasonic, Sony, Thomson (RCA), Philips, etc. By patenting their new ideas and getting them implemented in the specification they assured themselves a future royalty stream.
Technologies developed by MPEG and ITU have incredible market power with almost force of law behind their adoption in many countries over proprietary standards created by other companies. Great example is MPEG-2 which quickly became a worldwide standard powering everything from Digital TV broadcast standards to cable, satellite and DVD among countless other applications. We are talking hundreds of millions of devices generating billions of dollars in patent revenues.
Note that the patents read on the specifications themselves. As such, even if you went and wrote your own implementation from scratch, you would still be owing royalties to patent holders. In other words, just because the specification is “open,” it does not mean it is “free.” Open simply says you can buy/download the specification for a format. It says nothing about what it costs to use and sell it commercially. It is the responsibility of the adopter of the specification to make sure fees are paid for any and all patents embodied in the specification.
Here is the most shocking and broken part of this process: there is no list of patent or patent holders in the specification! Yes, you read that right. MPEG (and ITU) create their standards from technical points of view. What all an implementer needs to license is left as an assignment to them!
What if you miss a few patent holders? If they show up and knock on your door with their hand out for royalties on products you have already shipped, can you claim lack of knowledge and be OK? Sadly you cannot. As they say, ignorance of the law is no excuse. In this case, not knowing someone has a patent that reads on your product is no excuse at all. You are still liable for back royalties and other damages to these other entities if their patents read on the specification.
Let’s say you do know the list of companies or at least the major patent holders. How would you as a start-up or a small company go and negotiate licenses with potentially dozens of companies? The legal fees and time and energy needed to do that would be exorbitant. Even if you could negotiate all of these contracts, you may not want to do so. Say each license holder asks for a five (5) percent royalty on your product. By itself that may be reasonable but by the time you get to 10 of those companies, half of your product cost is now consumed by patent royalties and you are not done yet!
This problem was recognized back when MPEG-2 was being rolled out. Most of the major patent holders agreed to offer their patents together in one license agreement and at a set price. This is called a “patent pool.” An organization called MPEG Licensing Association or MPEG-LA was created that administrated the process of bringing patent holders together and providing license agreements to adopting companies.
The solution was and remains highly imperfect however. Due to anti-trust concerns, there can be no mandatory force for any patent holder to join the patent pool. Participation in patent pools is therefore voluntary and therein lies the problem. Such was the case with Thompson (RCA), which decided to opt out of the MPEG-2 patent pool and pursue its own royalty program. And there can be one or hundred other companies or individuals out there that have patents that read on the technology still that are not in the patent pool.
Not being in the charity business, MPEG-LA set the royalty cost of MPEG-2 at $2.50 with no cap or quantity discount. This broke a bunch of business models such as distributing the MPEG-2 decoder in software-only players that were given away for free. Not having high assurance of not getting sued by any and all other patent holders out there also gave pause to inclusion of such technology in some products.
MPEG-2 proved to be a highly profitable business. The rest of the world was asleep at the helm when that happened. Not so when the development of H.264 started. A gold rush type of event occurred and large number of companies participated in the formation of that standard and got their bits inserted in it. So now we have the MPEG-2 problem times ten. The list of patent holders is likely much, much larger than MPEG-2. So while MPEG-LA also provides a license for MPEG-4/AVC, that license is likely not sufficient to give one comfort that there won’t be claimants for the rest of the intellectual property (IP) that is in there.
The good news is that the licensing cost of MPEG-4 AVC is much more reasonable than MPEG-2. We are talking cents rather than dollars with maximum caps on total yearly royalties. MPEG-LA also created a patent pool for VC-1 with similar cost to MPEG-4 AVC.
Now let’s go back to the HD-DVD story. Imagine you are an MPEG-2 patent holder and member of the steering (board) of the DVD Forum. Would you vote to jettison that old technology in favor of VC-1 or MPEG-4 AVC when the royalties are a fraction of MPEG-2? Of course not. How about selecting Microsoft technology at the expense of MPEG-4 AVC? That was not going to go down easily either. So what to do? Aha, let’s make all three codecs mandatory!
What is cool about that is that if you have the same patent that reads on all three codecs you get paid three times for it! Patent licenses are field specific so if the same IP is used in MPEG-2 and MPEG-4 AVC, the former license does not grant you any rights in the latter.
To make a long story short, HD DVD adopted all three codecs. Blu-ray then followed suit (with some other twists and turns I won’t go into) ending up with adopting the same three codecs.
Today almost all Blu-ray discs are produced with MPEG-4 AVC yet every time you buy hardware that plays the format, you deposit money for the other two video codecs in there. You can thank the above process and strategy for it.
On the audio side the situation was different, in that both proposals from Dolby and DTS were owned by the respective companies (with Dolby’s solution being based on Meridian’s MLP codec). And no major consumer electronics company had a stake in them. Importantly, while one can argue about which video codec is better, there is no such argument around lossless audio codecs since they produce the same fidelity as the uncompressed audio samples. So you would think that only one would be adopted but you would be wrong. Both were adopted because picking one could be seen as playing favorites. And if you are major conglomerates in the world, you worry about such things.
Fast forward to today when video is part and parcel of the Internet experience. The above process has no chance of working. MPEG-2, for example, has fees for commercial content. How would you charge and collect that for You Tube videos?
And what would be reasonable royalties for these technologies? Video technology is only a small part of the functionality of a phone or computer for example. If I am building a dedicated cable box then playing $2.50 may be justifiable but if I am giving away Android to phone customers as Google does, what does it mean to pay the same fee? What if you get sued by a new guy? The usual practices of just asking for “5 percent” of product royalty would be a killer in these software dominated, cost-nothing business models of the Internet. Should a computer being used by a bank teller be liable for $2.50 for MPEG-2 royalties? What if they don’t even use video in their line of business application?
Fortunately, there is a ray of hope with the latest ruling from the U.S. Supreme Court in the case of Apple versus Samsung. The court ruled that patent infringement awards must be relative to the functionality in the product as opposed to its entire value. This helps some but still does not solve the broken standards setting process where companies get to define a format, create almost compulsory pressure for the world to adopt, then collect nearly open-ended royalties against it. Nothing wrong with capitalism but this process goes way beyond what is reasonable in my view.
This is one big mess with no solution in sight. Designing a new codec that is completely free of many fundamental patents in video is hard if not impossible to do. We need patent reform to solve this problem at its core.
Amir Majidimehr is the founder of Madrona Digital (www.madronadigital.com), which specializes in custom home electronics. He started Madrona after he left Microsoft where he was the Vice President in charge of the division developing audio/video technologies. With more than 30 years in the technology industry, he brings a fresh perspective to the world of home electronics.
---
Politics Of Format Making
Amir Majidimehr
Traditionally I have written technical articles for Widescreen Review magazine. This is a departure in that I would like to share what goes into creating new video formats such as Blu-ray. I was fortunate to have been part of that process when I was at Microsoft. I must say, it is eye opening how the parts come together and the driving factors involved. The picture is not pretty. It goes along the lines of “how sausage is made.” That is, if you knew what all gets ground up in sausage, you may not like to eat it as much!
Let’s get specific with an example: The Blu-ray format has three mandatory video codecs: MPEG-2, and two next generation codecs: MPEG4 AVC/H.264, and SMPTE/Microsoft VC-1. On the audio side, there are two lossless audio codecs, Dolby TrueHD and DTS-HD Master Audio. Do you see something strange here? Three video codecs to do the same thing? Two lossless audio codecs again with the same purpose? Why the multiples? That is the story of this article.
Stepping way back, Blu-ray started off as a Japan-only format for recording high-definition broadcast MPEG-2 on optical media. Unlike in the United States, DVD recording of video was quite popular in Japan so it was a natural progression to enhance it to support high-definition broadcast television. The optical media recipe was changed and the recorder simply captured the already compressed stream and stored it on the new writeable disc format.
Strategically, Japanese consumer electronics companies liked this business model since it cut Hollywood out of the picture. With DVD, the players quickly became commoditized and sold at or below cost. Meanwhile, Hollywood studios sold movies with great profit margin. Heck, you could buy an entire DVD player for what it almost cost to buy a DVD movie! DVD fueled the success of Hollywood studios, with home video sales crossing over the revenues of theatrical release. The higher profits meant the amount of money the talent made went up and so did the cost of movie making. But let’s not go there as that is for another article.
Meanwhile another development occurred: the DVD Forum decided to develop HD DVD. Unlike Blu-ray the focus was on pre-recorded theatrical movie releases. The initial target was to fit HD movies in the same red laser DVD, taking advantage of next-generation codecs to stuff more bits in the same space as compared to MPEG-2. A call for new codecs was put out to match the fidelity of MPEG-2 at around 20 megabit per second (the exact number escapes me) at just 7 megabit per second or so. We (Microsoft) entered our video codec called WMV (Windows Media Video) and others proposed variations of MPEG-2 and then new, H.264/MPEG-4 AVC.
After two rounds of this randomized, blind tests WMV produced the best picture. To many people’s surprise, MPEG-4 AVC did not finish first and actually unperformed MPEG-2 in some tests due to its low-bitrate optimization for low bit rate/resolution Internet streaming. MPEG-4 AVC was later revised to improve its performance for high-definition, high-bitrate applications.
Meanwhile the results of the test were presented to the DVD Forum steering committee for selection of a video codec for HD DVD. We thought our job was done and WMV would be selected as the sole mandatory codec. But that was not remotely the case. First objection was fair, which called for video codecs to have an open specification. So we submitted WMV to the Society of Motion Pictures and Television Engineers (SMPTE) for standardization. They renamed the codec to VC-1 and published its specifications.
Again, we thought we were home free but that was premature or better put, naïve. To understand why, I need to tell you about the great bit of education I received in talking to an executive of a major consumer electronics company. He said their strategy was simple: first make money from selling players. Once the bottom fell out of that, sell silicon/chips that implement the technology. When that gets commoditized, make money from patents of the technology in the format. Wow, that is one well thought out strategy!
Let’s dig into how the rest of the sausage is made. While we had designed WMV/VC-1 in house, both MPEG-2 and MPEG-4 AVC/H.264 were created in standards organizations: ITU and MPEG. MPEG lost its leadership after creating the anemic version of MPEG-4 which was not all that better than MPEG-2. ITU, on the other hand, set a very high target of efficiency and got there with the creation of H.264. Not wanting to get a black eye, the MPEG committee quickly adopted H.264 as its own and renamed it MPEG-4 AVC. Never mind that the standard had little to do with MPEG-4. This is why you see me being specific here with always using the full phrase, MPEG-4 AVC. Just saying MPEG-4 is confusing and may very well mean the old inefficient version.
The process for creation of new codecs in these organizations involves joining the group and then sending technical people to help drive the specification. Major Consumer Electronics powerhouses of the 1980s were present when MPEG-2 was created. We are talking Panasonic, Sony, Thomson (RCA), Philips, etc. By patenting their new ideas and getting them implemented in the specification they assured themselves a future royalty stream.
Technologies developed by MPEG and ITU have incredible market power with almost force of law behind their adoption in many countries over proprietary standards created by other companies. Great example is MPEG-2 which quickly became a worldwide standard powering everything from Digital TV broadcast standards to cable, satellite and DVD among countless other applications. We are talking hundreds of millions of devices generating billions of dollars in patent revenues.
Note that the patents read on the specifications themselves. As such, even if you went and wrote your own implementation from scratch, you would still be owing royalties to patent holders. In other words, just because the specification is “open,” it does not mean it is “free.” Open simply says you can buy/download the specification for a format. It says nothing about what it costs to use and sell it commercially. It is the responsibility of the adopter of the specification to make sure fees are paid for any and all patents embodied in the specification.
Here is the most shocking and broken part of this process: there is no list of patent or patent holders in the specification! Yes, you read that right. MPEG (and ITU) create their standards from technical points of view. What all an implementer needs to license is left as an assignment to them!
What if you miss a few patent holders? If they show up and knock on your door with their hand out for royalties on products you have already shipped, can you claim lack of knowledge and be OK? Sadly you cannot. As they say, ignorance of the law is no excuse. In this case, not knowing someone has a patent that reads on your product is no excuse at all. You are still liable for back royalties and other damages to these other entities if their patents read on the specification.
Let’s say you do know the list of companies or at least the major patent holders. How would you as a start-up or a small company go and negotiate licenses with potentially dozens of companies? The legal fees and time and energy needed to do that would be exorbitant. Even if you could negotiate all of these contracts, you may not want to do so. Say each license holder asks for a five (5) percent royalty on your product. By itself that may be reasonable but by the time you get to 10 of those companies, half of your product cost is now consumed by patent royalties and you are not done yet!
This problem was recognized back when MPEG-2 was being rolled out. Most of the major patent holders agreed to offer their patents together in one license agreement and at a set price. This is called a “patent pool.” An organization called MPEG Licensing Association or MPEG-LA was created that administrated the process of bringing patent holders together and providing license agreements to adopting companies.
The solution was and remains highly imperfect however. Due to anti-trust concerns, there can be no mandatory force for any patent holder to join the patent pool. Participation in patent pools is therefore voluntary and therein lies the problem. Such was the case with Thompson (RCA), which decided to opt out of the MPEG-2 patent pool and pursue its own royalty program. And there can be one or hundred other companies or individuals out there that have patents that read on the technology still that are not in the patent pool.
Not being in the charity business, MPEG-LA set the royalty cost of MPEG-2 at $2.50 with no cap or quantity discount. This broke a bunch of business models such as distributing the MPEG-2 decoder in software-only players that were given away for free. Not having high assurance of not getting sued by any and all other patent holders out there also gave pause to inclusion of such technology in some products.
MPEG-2 proved to be a highly profitable business. The rest of the world was asleep at the helm when that happened. Not so when the development of H.264 started. A gold rush type of event occurred and large number of companies participated in the formation of that standard and got their bits inserted in it. So now we have the MPEG-2 problem times ten. The list of patent holders is likely much, much larger than MPEG-2. So while MPEG-LA also provides a license for MPEG-4/AVC, that license is likely not sufficient to give one comfort that there won’t be claimants for the rest of the intellectual property (IP) that is in there.
The good news is that the licensing cost of MPEG-4 AVC is much more reasonable than MPEG-2. We are talking cents rather than dollars with maximum caps on total yearly royalties. MPEG-LA also created a patent pool for VC-1 with similar cost to MPEG-4 AVC.
Now let’s go back to the HD-DVD story. Imagine you are an MPEG-2 patent holder and member of the steering (board) of the DVD Forum. Would you vote to jettison that old technology in favor of VC-1 or MPEG-4 AVC when the royalties are a fraction of MPEG-2? Of course not. How about selecting Microsoft technology at the expense of MPEG-4 AVC? That was not going to go down easily either. So what to do? Aha, let’s make all three codecs mandatory!
What is cool about that is that if you have the same patent that reads on all three codecs you get paid three times for it! Patent licenses are field specific so if the same IP is used in MPEG-2 and MPEG-4 AVC, the former license does not grant you any rights in the latter.
To make a long story short, HD DVD adopted all three codecs. Blu-ray then followed suit (with some other twists and turns I won’t go into) ending up with adopting the same three codecs.
Today almost all Blu-ray discs are produced with MPEG-4 AVC yet every time you buy hardware that plays the format, you deposit money for the other two video codecs in there. You can thank the above process and strategy for it.
On the audio side the situation was different, in that both proposals from Dolby and DTS were owned by the respective companies (with Dolby’s solution being based on Meridian’s MLP codec). And no major consumer electronics company had a stake in them. Importantly, while one can argue about which video codec is better, there is no such argument around lossless audio codecs since they produce the same fidelity as the uncompressed audio samples. So you would think that only one would be adopted but you would be wrong. Both were adopted because picking one could be seen as playing favorites. And if you are major conglomerates in the world, you worry about such things.
Fast forward to today when video is part and parcel of the Internet experience. The above process has no chance of working. MPEG-2, for example, has fees for commercial content. How would you charge and collect that for You Tube videos?
And what would be reasonable royalties for these technologies? Video technology is only a small part of the functionality of a phone or computer for example. If I am building a dedicated cable box then playing $2.50 may be justifiable but if I am giving away Android to phone customers as Google does, what does it mean to pay the same fee? What if you get sued by a new guy? The usual practices of just asking for “5 percent” of product royalty would be a killer in these software dominated, cost-nothing business models of the Internet. Should a computer being used by a bank teller be liable for $2.50 for MPEG-2 royalties? What if they don’t even use video in their line of business application?
Fortunately, there is a ray of hope with the latest ruling from the U.S. Supreme Court in the case of Apple versus Samsung. The court ruled that patent infringement awards must be relative to the functionality in the product as opposed to its entire value. This helps some but still does not solve the broken standards setting process where companies get to define a format, create almost compulsory pressure for the world to adopt, then collect nearly open-ended royalties against it. Nothing wrong with capitalism but this process goes way beyond what is reasonable in my view.
This is one big mess with no solution in sight. Designing a new codec that is completely free of many fundamental patents in video is hard if not impossible to do. We need patent reform to solve this problem at its core.
Amir Majidimehr is the founder of Madrona Digital (www.madronadigital.com), which specializes in custom home electronics. He started Madrona after he left Microsoft where he was the Vice President in charge of the division developing audio/video technologies. With more than 30 years in the technology industry, he brings a fresh perspective to the world of home electronics.
