It is All Software
By Amir Majidimehr
[Note: this is an article that was published in the Widescreen Review Magazine]
OK, it is not all software but most of it is. What am I talking about? I am talking about video! Yes, video. Please don’t flip the page. Let me explain with a bit of personal history.
My journey into video started when I worked for companies that were producing equipment for the television broadcast and editing industries. This was in the 1990s and the product we produced consisted of rack-mounted gear that costs tens of thousands of dollars, capable of performing real-time transformation on video which at the time was the job of “heavy” hardware. With the Internet coming on strong, I decided to change paths and manage engineering at a start-up that was trying to deliver video on the Internet. I say “trying” because this was in the era of 28 Kbps dial-up modems. To put this challenge in perspective, we are talking about a link speed that was 250 times slower than a typical broadband connection today! Video sent over a dial up modem, despite all the tricks we pulled to make it work, was postage stamped (160x120 pixels) at maybe 6 to 8 frames/second. Compare this to standard definition television at a resolution of 640x480 (square pixels), and 30 frames/second.
While the video experience over those modem connections was hugely underwhelming, how we accomplished it in some ways was way ahead of all the expensive equipment that we designed for the broadcast industry. This related to how the streaming system managed to harness the power of software to make video do more. Doing more meant allowing people to watch programming on demand without owning anything other than a personal computer. This extended to the “broadcast” end just the same with almost all of the functionality being powered in software with the small exception of the video capture card. Everything from compression to transmission over the Internet, reception and decoding in the client machine was performed using computer code.
Fast forward 15 years later and that kind of approach has become common. Whether it is YouTube video or Netflix/Hulu streaming of movies and TV programming, you are experiencing the power of software in the way it has transformed video creation, distribution and consumption. Even the once daunting task of encoding high-definition video in real time can be achieved in software using multiple-core CPUs. $20,000 worth of hardware has been shrunk into a $500 PC running the right software.
To appreciate the power of software in our modern lives, it is important to understand the challenges it presents. Video transmission by definition is a “real-time” operation. For a movie shot at 24 frames a second, we expect a video frame to be produced every 42 milliseconds (1/24) or else, the cadence will be wrong. Likewise audio will have to be continuous or we will hear pops, clicks and gaps. This is very ill suited to the design of the Internet, a communication medium which is designed with no guarantee of on-time delivery. Sophisticated software is then used to turn such a hostile channel into one that appears to deliver video frames reliably, approximating your cable and satellite which use dedicated communication channels.
Some of the clever techniques used in above systems include switching between video streams encoded at different quality levels. As the connection speed to the Internet naturally fluctuates the player will switch to different quality levels and hence, adapt to the available bandwidth and lower the chances of having to pause the video to catch up. We call this “adaptive streaming.” Compare this to your off-the-air digital signal which simply stops or falls apart if signal reception becomes poor. That is the difference between rigid hardware and flexible software.
You might think that all of this is only true of Internet technology but that is not so. Software has transformed the entire process of creating and mastering of content. Take the Blu-ray Disc format. The video and audio encoding at mastering time is entirely done in software. Gone are the old days of MPEG encoding where black box hardware products were used to produce the video stream. Today’s encoding tools are completely software based allowing the compressionist to not only encode the movie but also hand tune it frame by frame for the best fidelity.
Once the audio/video streams are compressed, they are “authored” (packaged) in a format compatible with the Blu-ray disc format before being sent to manufacturing of physical discs. The authoring tool naturally is in software.
Our journey continues on the playback side. There, the operation starts with software which deciphers the Blu-ray format to find the audio and video streams (reverse of authoring process above). Once there, the video stream is handed to dedicated hardware for decoding due to cost reasons. OK, you got me. There is some hardware here.
Actually, even that aspect is sometimes done using software assistance. Video decoding has two components, one of which is very numerically intensive and the other not. The former lends itself very well to hardware implementations due to the predictable and parallel nature of it. While software on a computer can do the same, and indeed does that routinely when you watch video on a PC, it is not very cost effective. The other part can also be implemented in hardware but it is more challenging due to complexity and the lack of parallelism. As a result, a “coprocessor” (computer CPU) is often used to drive the specialized hardware, bringing best of both worlds.
Audio decoding for Blu-ray for example can come in a myriad of formats ranging from lossy and lossless versions of Dolby and DTS, putting aside common Internet formats such as MP3, AAC and WMA. So for a long time now, audio processing has been performed using software running on a special type of microprocessor called a DSP. This is a type of CPU that is optimized for numerical computations (see http://www.audiosciencereview.com/f.../dsps-what-they-are-and-why-they-are-used.16/ for an overview). The DSP runs different programs depending on the format in use, generating digital audio samples ready to be sent out over HDMI or converted to analog using the Digital to Analog Converter (DAC) inside the player. So this is another score for software. Had it not been this way, we would have needed dedicated hardware for every format, increasing the cost and complexity of players and processors/AVRs.
We have gotten ahead of ourselves a bit because we have an important step to get past before we get to our audio/video streams, namely, content protection. Streams on Blu-ray Disc are encrypted. Unless a key is used to unlock them, the audio/video data cannot be extracted. The specification that governs this is called AACS. Due to its static nature, dedicated hardware could have been used to handle it but is not. The reason is that the same silicon used in Blu-ray players is also sold to companies making cable and satellite set-top boxes (with some small variations). Since content protection systems vary between these formats of video distribution, it is simpler and more cost effective to put the functionality in software.
One benefits of software approach to system design is that one can be modified without redesigning and remanufacturing the hardware. Well, that flexibility can be a determent in the case of copy protection technology as it can also allow someone to modify the code and hence get access to content that is not theirs. To safeguard against this, Blu-ray decoder silicon usually has a special subsystem called a “Secure Processor.” This is a microprocessor whose only job is to decrypt the content in such a manner as to make it very hard for someone to extract and expose the secrets (keys). In the case of Blu-ray, the secure processor loads the AACS code and executes it in a partitioned manner (i.e. no outside access) after which, it generates the audio/video streams ready to be sent out to decoders for those bits (the path is private so cannot be easily intercepted).
Blu-ray Disc has another optional content protection layer called BD+. This technology allows the movie to come with code that tells the player how to decipher it. The player loads this software into the secure processor where it manipulates the decrypted AACS stream after which, the audio/video data is available ready to be decoded. Since this code is written by whoever authored the movie, it is by definition software that needs to run in the player for the movie to be watched.
So far we have dealt with the “elementary streams” in the format (i.e. audio/video). But Blu-ray supports more features. There is a subsystem called BD-J that is used to implement “interactivity” and often, the menu system for the movie. The letter J stands for Java which is a programming environment that allows applications developed in it to (ostensibly) run on different hardware unchanged. The main motivation for it was to create a dynamic experience for movies, perhaps crudely approximating what could be done on a game console.
From code and complexity point of view, the Java subsystem is huge by consumer electronics standards, consisting of tens of thousands of lines of code. Alas, due to lack of certification, this subsystem can be buggy, much like programs in your computer, causing some titles to not run properly. You no doubt have seen discs that show unfriendly warnings instructing users to update the “firmware” (software) for their player. So no, not everything about software is good. Too much software can become unwieldy causing poor user experience.
Just in case you thought we were done with the software layers in Blu-ray, we are not. Yet another (optional) feature is BD-Live which supports Internet access together with such things as storing content locally in the player. This entails a ton of new functionality in the player from understanding the Internet protocols (TCP/IP), to “file system” code that knows how to read and write files to flash memory, and ability to run concurrent operation among others. Instead of trying to reinvent the wheel, player manufacturers resort to running a full blown operating system (OS) such as Linux which provides all of this functionality. This means that your player now has hundreds of thousands of lines of code just to play your movie! And oh, this is one of the reasons it takes a while for your player to power on because it is “booting” (loading) a ton of software just to say “hi, I am here” just like your computer.
While not part of Blu-ray Disc format, it has become the norm for the player to support a myriad of other features such as Internet streaming, playback of audio and video from networked PCs and servers in the home, etc. Naturally all of this is implemented in software. And yes, with thousands of lines of code added yet to the pile of code already there.
All of this takes a lot of computing player with the typical Blu-ray silicon consisting of dual GHz+ CPU cores, graphics acceleration, DSP for audio, and lots of memory and firmware space to contain all the software. Luckily price pressure is quite high so the cost of the silicon is incredibly low by computer standards with a full “system on a chip” Blu-ray silicon going as low as $5 in volume as of this writing.
So there you have it. What looks like a hardware player in reality relies on 80% software to get its job done. The role of software in our “hardware” devices will keep growing with the advent of many online services and the desire for consumer electronics companies to chase the coattails of tablet and phone manufacturers by supporting “apps.” I wish them the best of luck as they suffer though the immense effort it takes to write quality software at large scale!
Amir Majidimehr is the founder of audio/video/integration/automation company, Madrona Digital (madronadigital.com). Prior to that, he spent over 30 years in the computer and broadcast/consumer video industries at leading companies from Sony to Microsoft, always pushing to advance the state-of-the-art in delivery and consumption of digital media. Technologies developed in his teams are shipped in billions of devices from leading game consoles and phones to every PC in the world and are mandatory in such standards as Blu-ray. He retired as Corporate VP at Microsoft in 2007 to pursue other interests, including advancing the way we interconnect devices in our homes for better enjoyment of audio/video content.
By Amir Majidimehr
[Note: this is an article that was published in the Widescreen Review Magazine]
OK, it is not all software but most of it is. What am I talking about? I am talking about video! Yes, video. Please don’t flip the page. Let me explain with a bit of personal history.
My journey into video started when I worked for companies that were producing equipment for the television broadcast and editing industries. This was in the 1990s and the product we produced consisted of rack-mounted gear that costs tens of thousands of dollars, capable of performing real-time transformation on video which at the time was the job of “heavy” hardware. With the Internet coming on strong, I decided to change paths and manage engineering at a start-up that was trying to deliver video on the Internet. I say “trying” because this was in the era of 28 Kbps dial-up modems. To put this challenge in perspective, we are talking about a link speed that was 250 times slower than a typical broadband connection today! Video sent over a dial up modem, despite all the tricks we pulled to make it work, was postage stamped (160x120 pixels) at maybe 6 to 8 frames/second. Compare this to standard definition television at a resolution of 640x480 (square pixels), and 30 frames/second.
While the video experience over those modem connections was hugely underwhelming, how we accomplished it in some ways was way ahead of all the expensive equipment that we designed for the broadcast industry. This related to how the streaming system managed to harness the power of software to make video do more. Doing more meant allowing people to watch programming on demand without owning anything other than a personal computer. This extended to the “broadcast” end just the same with almost all of the functionality being powered in software with the small exception of the video capture card. Everything from compression to transmission over the Internet, reception and decoding in the client machine was performed using computer code.
Fast forward 15 years later and that kind of approach has become common. Whether it is YouTube video or Netflix/Hulu streaming of movies and TV programming, you are experiencing the power of software in the way it has transformed video creation, distribution and consumption. Even the once daunting task of encoding high-definition video in real time can be achieved in software using multiple-core CPUs. $20,000 worth of hardware has been shrunk into a $500 PC running the right software.
To appreciate the power of software in our modern lives, it is important to understand the challenges it presents. Video transmission by definition is a “real-time” operation. For a movie shot at 24 frames a second, we expect a video frame to be produced every 42 milliseconds (1/24) or else, the cadence will be wrong. Likewise audio will have to be continuous or we will hear pops, clicks and gaps. This is very ill suited to the design of the Internet, a communication medium which is designed with no guarantee of on-time delivery. Sophisticated software is then used to turn such a hostile channel into one that appears to deliver video frames reliably, approximating your cable and satellite which use dedicated communication channels.
Some of the clever techniques used in above systems include switching between video streams encoded at different quality levels. As the connection speed to the Internet naturally fluctuates the player will switch to different quality levels and hence, adapt to the available bandwidth and lower the chances of having to pause the video to catch up. We call this “adaptive streaming.” Compare this to your off-the-air digital signal which simply stops or falls apart if signal reception becomes poor. That is the difference between rigid hardware and flexible software.
You might think that all of this is only true of Internet technology but that is not so. Software has transformed the entire process of creating and mastering of content. Take the Blu-ray Disc format. The video and audio encoding at mastering time is entirely done in software. Gone are the old days of MPEG encoding where black box hardware products were used to produce the video stream. Today’s encoding tools are completely software based allowing the compressionist to not only encode the movie but also hand tune it frame by frame for the best fidelity.
Once the audio/video streams are compressed, they are “authored” (packaged) in a format compatible with the Blu-ray disc format before being sent to manufacturing of physical discs. The authoring tool naturally is in software.
Our journey continues on the playback side. There, the operation starts with software which deciphers the Blu-ray format to find the audio and video streams (reverse of authoring process above). Once there, the video stream is handed to dedicated hardware for decoding due to cost reasons. OK, you got me. There is some hardware here.
Actually, even that aspect is sometimes done using software assistance. Video decoding has two components, one of which is very numerically intensive and the other not. The former lends itself very well to hardware implementations due to the predictable and parallel nature of it. While software on a computer can do the same, and indeed does that routinely when you watch video on a PC, it is not very cost effective. The other part can also be implemented in hardware but it is more challenging due to complexity and the lack of parallelism. As a result, a “coprocessor” (computer CPU) is often used to drive the specialized hardware, bringing best of both worlds.Audio decoding for Blu-ray for example can come in a myriad of formats ranging from lossy and lossless versions of Dolby and DTS, putting aside common Internet formats such as MP3, AAC and WMA. So for a long time now, audio processing has been performed using software running on a special type of microprocessor called a DSP. This is a type of CPU that is optimized for numerical computations (see http://www.audiosciencereview.com/f.../dsps-what-they-are-and-why-they-are-used.16/ for an overview). The DSP runs different programs depending on the format in use, generating digital audio samples ready to be sent out over HDMI or converted to analog using the Digital to Analog Converter (DAC) inside the player. So this is another score for software. Had it not been this way, we would have needed dedicated hardware for every format, increasing the cost and complexity of players and processors/AVRs.
We have gotten ahead of ourselves a bit because we have an important step to get past before we get to our audio/video streams, namely, content protection. Streams on Blu-ray Disc are encrypted. Unless a key is used to unlock them, the audio/video data cannot be extracted. The specification that governs this is called AACS. Due to its static nature, dedicated hardware could have been used to handle it but is not. The reason is that the same silicon used in Blu-ray players is also sold to companies making cable and satellite set-top boxes (with some small variations). Since content protection systems vary between these formats of video distribution, it is simpler and more cost effective to put the functionality in software.
One benefits of software approach to system design is that one can be modified without redesigning and remanufacturing the hardware. Well, that flexibility can be a determent in the case of copy protection technology as it can also allow someone to modify the code and hence get access to content that is not theirs. To safeguard against this, Blu-ray decoder silicon usually has a special subsystem called a “Secure Processor.” This is a microprocessor whose only job is to decrypt the content in such a manner as to make it very hard for someone to extract and expose the secrets (keys). In the case of Blu-ray, the secure processor loads the AACS code and executes it in a partitioned manner (i.e. no outside access) after which, it generates the audio/video streams ready to be sent out to decoders for those bits (the path is private so cannot be easily intercepted).
Blu-ray Disc has another optional content protection layer called BD+. This technology allows the movie to come with code that tells the player how to decipher it. The player loads this software into the secure processor where it manipulates the decrypted AACS stream after which, the audio/video data is available ready to be decoded. Since this code is written by whoever authored the movie, it is by definition software that needs to run in the player for the movie to be watched.
So far we have dealt with the “elementary streams” in the format (i.e. audio/video). But Blu-ray supports more features. There is a subsystem called BD-J that is used to implement “interactivity” and often, the menu system for the movie. The letter J stands for Java which is a programming environment that allows applications developed in it to (ostensibly) run on different hardware unchanged. The main motivation for it was to create a dynamic experience for movies, perhaps crudely approximating what could be done on a game console.
From code and complexity point of view, the Java subsystem is huge by consumer electronics standards, consisting of tens of thousands of lines of code. Alas, due to lack of certification, this subsystem can be buggy, much like programs in your computer, causing some titles to not run properly. You no doubt have seen discs that show unfriendly warnings instructing users to update the “firmware” (software) for their player. So no, not everything about software is good. Too much software can become unwieldy causing poor user experience.
Just in case you thought we were done with the software layers in Blu-ray, we are not. Yet another (optional) feature is BD-Live which supports Internet access together with such things as storing content locally in the player. This entails a ton of new functionality in the player from understanding the Internet protocols (TCP/IP), to “file system” code that knows how to read and write files to flash memory, and ability to run concurrent operation among others. Instead of trying to reinvent the wheel, player manufacturers resort to running a full blown operating system (OS) such as Linux which provides all of this functionality. This means that your player now has hundreds of thousands of lines of code just to play your movie! And oh, this is one of the reasons it takes a while for your player to power on because it is “booting” (loading) a ton of software just to say “hi, I am here” just like your computer.
While not part of Blu-ray Disc format, it has become the norm for the player to support a myriad of other features such as Internet streaming, playback of audio and video from networked PCs and servers in the home, etc. Naturally all of this is implemented in software. And yes, with thousands of lines of code added yet to the pile of code already there.
All of this takes a lot of computing player with the typical Blu-ray silicon consisting of dual GHz+ CPU cores, graphics acceleration, DSP for audio, and lots of memory and firmware space to contain all the software. Luckily price pressure is quite high so the cost of the silicon is incredibly low by computer standards with a full “system on a chip” Blu-ray silicon going as low as $5 in volume as of this writing.
So there you have it. What looks like a hardware player in reality relies on 80% software to get its job done. The role of software in our “hardware” devices will keep growing with the advent of many online services and the desire for consumer electronics companies to chase the coattails of tablet and phone manufacturers by supporting “apps.” I wish them the best of luck as they suffer though the immense effort it takes to write quality software at large scale!
Amir Majidimehr is the founder of audio/video/integration/automation company, Madrona Digital (madronadigital.com). Prior to that, he spent over 30 years in the computer and broadcast/consumer video industries at leading companies from Sony to Microsoft, always pushing to advance the state-of-the-art in delivery and consumption of digital media. Technologies developed in his teams are shipped in billions of devices from leading game consoles and phones to every PC in the world and are mandatory in such standards as Blu-ray. He retired as Corporate VP at Microsoft in 2007 to pursue other interests, including advancing the way we interconnect devices in our homes for better enjoyment of audio/video content.
Last edited:
