You have heard the names: Windows, MacOS, iOS, Android, Linux. These are all examples of very popular operating systems. Question is, what is the function of an operating system? Why have they become the essential component of our computing world and more so, our home electronics?
The operating system is specialized software that sits between your computer hardware and the applications that run on top of it. The motivation for creating it was to make it easier for software developers to write their programs. By isolating the programs from the hardware, the job of the software developer was made easier. And importantly, programs could more readily be moved from one hardware platform to another. Think of how you can install the same software on multitudes of Mac or PC hardware. Without the operating system, the application would have had to know how to “talk” to each specific piece of hardware and if you changed that, it would stop working.
Even though the operating system is a fundamental part of your computer, it is not the first thing that runs. That job falls on another piece of code built into your computer hardware called the “boot loader.” The boot loader’s essential function is to locate the operating on the default storage device (hard disk or flash). Once there, it loads it into memory and then passes the control to it. At this point you usually see the logo of the operating system as it goes through its housekeeping chores before giving you control to use the system.
Modern operating systems allow multiple concurrent applications to load and be active at the same time. We call this “multi-tasking.” One of the key requirements here is to make sure each application thinks it has the entire computer dedicated to it. That way, if one program crashes, it can’t bring down the rest with it. And importantly, not allow a program to peer into another as to steal any sensitive information (think your password as you type it into another program). This partitioning is an important feature provided by the operating system.
Understanding the operating system requires knowledge of the hardware and more specifically, that of your computer CPU. The CPU has what is called a privileged mode which is active by default as your computer boots. This means that the operating system runs its code as privileged. The best way to understand what this mode does is to look its opposite. If the privileged mode is off, the software that is running cannot communicate with the computer hardware. If it for example attempts to directly read the bits from the hard disk rather than nicely asking the operating system, the CPU stops it and complains to the operating system. This in turn will cause the application to be shut down. We call this a “program fault.”
In addition to above restriction, non-privileged application code can only see the memory that it occupies. If it attempts to peer into the memory that it does not own, such as that of the operating system or other applications, it will again be forcefully shut down. Sometimes this occurs not because the application is trying to be malicious but due to some bug (software defect) that causes the application to access memory it doesn’t own resulting in its demise.
As implied, the operating system has none of these bounds. It can freely access all the hardware as it pleases. It is also all-seeing in that it can peer into any application that is running although it does not usually attempt to do that.
You may know that DVD copy protection was breached years ago. It is reported that the motivation for doing so was to enable playback of DVD on the Linux operating system. No, I did not change topics all of a sudden. Believe it or not, the reason DVD was hacked this way was due to the explanation I just gave on how the operating system can see application data. The reason requires digging a bit deeper into this story.
Content on DVD is “scrambled.” A key is used to reverse this scrambling allowing the player to get at the audio and video streams to decode and play them. It is the responsibility of the DVD playback application to hide this key sufficiently from the people trying to discover it. The second part to the story is the Linux operating system. Linux is “open source” meaning anyone can download the pieces to it and create their own version of it. Now let’s put these two pieces together. If the operating system can see inside any application it runs – such as the DVD player – and I am allowed to build my own version of the OS as is the case with Linux, there is no effective security. I can easily make those modifications to Linux, steal the key and distribute it to the world to bypass DVDs.
Because of this known limitation of Linux, no implementation existed for a DVD player on top of it due to liability arising from its secrets getting out. The story goes that the Norwegian programmer Jon Lech Johansen was frustrated by this fact and went searching to find the keys as to enable anyone then to build a DVD player for Linux. Turns out the job was not that hard as a software DVD player called Xing had done little to hide the key allowing him to discover it. He published the keys he discovered and with it, enabled an army of underground software developers building all manner of DVD “rippers” and players. Just when you thought the topic of operating system had nothing to do with audio/video!
Sometimes an operating system is not what it seems. Case in point is Android. You probably think Android is an operating system but it actually is not. If you have an Android device, the operating system is actually Linux underneath! Linux performs all the core functionality that I described earlier. What you know as Android is “middleware” which is yet another software layer between your applications and the Linux operating system. Linux by itself does not have a graphical interface, has no support for touch or voice interface, marketplace for downloading applications, etc. All of this is and then some is provided by Android. Think of Linux as the engine the in the car, quietly powering the system, and Android as everything else that makes a car. The terminology we use in the industry is to refer to the role Linux plays as the “kernel.” This is to distinguish it against the system applications and middleware running above it.
In the old days the kernel was “pure” in that it provided the core functionality that was needed but no more. This provided rock solid isolation between the hardware and applications resulting in high level of reliability as applications could never cause the system to crash. That line has been blurred in the case of features such as graphics and video decoding/display support in Windows. There, in the interest of providing maximum performance for games and multimedia applications such as video, direct access is allowed to the hardware for application programs. The down side is that applications can perform functions that could cause the entire computer to crash.
The kernel software is written in a programming language just like the applications. Its functionality though is quite specialized and requires extensive knowledge of its architecture and that of hardware. Modifications of it is challenging in that slightest mistake can cause the system to crash altogether. Contrast that with bugs in the application programs where just that program dies and your computer keeps going. Not so when the kernel faults.
Complicating the life of the kernel developer is that when something goes wrong there, it can manifest itself sometimes days or even months later! Imagine if you print one document and some piece of memory is corrupted after the printing is done. Nothing goes wrong then until you try to print another document a week later and bam, the kernel crashes. You think that is what caused the crash but in really it was an operation much earlier. I say that from personal experience, having had to trace kernel failures that took me weeks to trace.
How about a bit more on the history of operating systems? As I mentioned earlier, Android is based on Linux. Turns out because of that, it is actually a cousin of MacOS! The kernel that MacOS is based on is called Mach. Mach was developed at Carnegie Mellon University as a derivative of another operating system called UNIX. When the late Steve Jobs left Apple years ago, he started a company called NexT which used Mach for the workstations they were designing. Later when his company was bought and he returned to Apple, Mach became the kernel for the Mac OS, modernizing it and giving it such essential features as multi-tasking. For its part, Linux was also a variation of UNIX written from scratch to avoid needing a license to the original source code for UNIX which at the time, belonged to the then dominant phone company in US called AT&T. So putting it altogether, both operating systems trace their roots to UNIX.
I started my career in 1982, working on the UNIX kernel just as the operating system was taking off to power computers from mid-tier to high-end. I remember printing out its almost entirely undocumented source code and studying every line of trying to understand it. The listing was 250,000 lines of code and created one massive book, probably weighing some 20 to 30 pounds. I would carry that under my arm to work and home every day. I would spend most of my free time hand documenting its secret on that printed book. Thankfully I was married by then so I did not have to worry about looking like a textbook geek! As challenging as that work was, it was extremely rewarding both from the thrill of fixing and optimizing it but also giving me a solid foundation to understand the operation of our computers.
By the way, 250,000 lines may sound like a large program but in today’s world, it is nothing. The “little” media player application that ships as part of your operating system likely clocks at a million lines of code! Modern operating systems are beasts and have incredible breath of functionality and code to support them. With literally thousands of people working on them every year, they keep growing in an attempt to keep up with the latest hardware technologies and expectations for what an operating system should do. Despite that, the core concept of what they do is not hard to comprehend as I just explained.
An interesting development in operating systems is extensive use of them in consumer electronics devices. It used to be that TVs and DVD players were pretty simply devices and did not have an operating system. Today, they have so much functionality akin to a smaller computer that they almost always run an operating system with that choice almost always being Linux. This is why it takes a while for your equipment to turn on and shut off. It is the kernel booting and shutting down!
The operating system is specialized software that sits between your computer hardware and the applications that run on top of it. The motivation for creating it was to make it easier for software developers to write their programs. By isolating the programs from the hardware, the job of the software developer was made easier. And importantly, programs could more readily be moved from one hardware platform to another. Think of how you can install the same software on multitudes of Mac or PC hardware. Without the operating system, the application would have had to know how to “talk” to each specific piece of hardware and if you changed that, it would stop working.
Even though the operating system is a fundamental part of your computer, it is not the first thing that runs. That job falls on another piece of code built into your computer hardware called the “boot loader.” The boot loader’s essential function is to locate the operating on the default storage device (hard disk or flash). Once there, it loads it into memory and then passes the control to it. At this point you usually see the logo of the operating system as it goes through its housekeeping chores before giving you control to use the system.
Modern operating systems allow multiple concurrent applications to load and be active at the same time. We call this “multi-tasking.” One of the key requirements here is to make sure each application thinks it has the entire computer dedicated to it. That way, if one program crashes, it can’t bring down the rest with it. And importantly, not allow a program to peer into another as to steal any sensitive information (think your password as you type it into another program). This partitioning is an important feature provided by the operating system.
Understanding the operating system requires knowledge of the hardware and more specifically, that of your computer CPU. The CPU has what is called a privileged mode which is active by default as your computer boots. This means that the operating system runs its code as privileged. The best way to understand what this mode does is to look its opposite. If the privileged mode is off, the software that is running cannot communicate with the computer hardware. If it for example attempts to directly read the bits from the hard disk rather than nicely asking the operating system, the CPU stops it and complains to the operating system. This in turn will cause the application to be shut down. We call this a “program fault.”
In addition to above restriction, non-privileged application code can only see the memory that it occupies. If it attempts to peer into the memory that it does not own, such as that of the operating system or other applications, it will again be forcefully shut down. Sometimes this occurs not because the application is trying to be malicious but due to some bug (software defect) that causes the application to access memory it doesn’t own resulting in its demise.
As implied, the operating system has none of these bounds. It can freely access all the hardware as it pleases. It is also all-seeing in that it can peer into any application that is running although it does not usually attempt to do that.
You may know that DVD copy protection was breached years ago. It is reported that the motivation for doing so was to enable playback of DVD on the Linux operating system. No, I did not change topics all of a sudden. Believe it or not, the reason DVD was hacked this way was due to the explanation I just gave on how the operating system can see application data. The reason requires digging a bit deeper into this story.
Content on DVD is “scrambled.” A key is used to reverse this scrambling allowing the player to get at the audio and video streams to decode and play them. It is the responsibility of the DVD playback application to hide this key sufficiently from the people trying to discover it. The second part to the story is the Linux operating system. Linux is “open source” meaning anyone can download the pieces to it and create their own version of it. Now let’s put these two pieces together. If the operating system can see inside any application it runs – such as the DVD player – and I am allowed to build my own version of the OS as is the case with Linux, there is no effective security. I can easily make those modifications to Linux, steal the key and distribute it to the world to bypass DVDs.
Because of this known limitation of Linux, no implementation existed for a DVD player on top of it due to liability arising from its secrets getting out. The story goes that the Norwegian programmer Jon Lech Johansen was frustrated by this fact and went searching to find the keys as to enable anyone then to build a DVD player for Linux. Turns out the job was not that hard as a software DVD player called Xing had done little to hide the key allowing him to discover it. He published the keys he discovered and with it, enabled an army of underground software developers building all manner of DVD “rippers” and players. Just when you thought the topic of operating system had nothing to do with audio/video!
Sometimes an operating system is not what it seems. Case in point is Android. You probably think Android is an operating system but it actually is not. If you have an Android device, the operating system is actually Linux underneath! Linux performs all the core functionality that I described earlier. What you know as Android is “middleware” which is yet another software layer between your applications and the Linux operating system. Linux by itself does not have a graphical interface, has no support for touch or voice interface, marketplace for downloading applications, etc. All of this is and then some is provided by Android. Think of Linux as the engine the in the car, quietly powering the system, and Android as everything else that makes a car. The terminology we use in the industry is to refer to the role Linux plays as the “kernel.” This is to distinguish it against the system applications and middleware running above it.
In the old days the kernel was “pure” in that it provided the core functionality that was needed but no more. This provided rock solid isolation between the hardware and applications resulting in high level of reliability as applications could never cause the system to crash. That line has been blurred in the case of features such as graphics and video decoding/display support in Windows. There, in the interest of providing maximum performance for games and multimedia applications such as video, direct access is allowed to the hardware for application programs. The down side is that applications can perform functions that could cause the entire computer to crash.
The kernel software is written in a programming language just like the applications. Its functionality though is quite specialized and requires extensive knowledge of its architecture and that of hardware. Modifications of it is challenging in that slightest mistake can cause the system to crash altogether. Contrast that with bugs in the application programs where just that program dies and your computer keeps going. Not so when the kernel faults.
Complicating the life of the kernel developer is that when something goes wrong there, it can manifest itself sometimes days or even months later! Imagine if you print one document and some piece of memory is corrupted after the printing is done. Nothing goes wrong then until you try to print another document a week later and bam, the kernel crashes. You think that is what caused the crash but in really it was an operation much earlier. I say that from personal experience, having had to trace kernel failures that took me weeks to trace.
How about a bit more on the history of operating systems? As I mentioned earlier, Android is based on Linux. Turns out because of that, it is actually a cousin of MacOS! The kernel that MacOS is based on is called Mach. Mach was developed at Carnegie Mellon University as a derivative of another operating system called UNIX. When the late Steve Jobs left Apple years ago, he started a company called NexT which used Mach for the workstations they were designing. Later when his company was bought and he returned to Apple, Mach became the kernel for the Mac OS, modernizing it and giving it such essential features as multi-tasking. For its part, Linux was also a variation of UNIX written from scratch to avoid needing a license to the original source code for UNIX which at the time, belonged to the then dominant phone company in US called AT&T. So putting it altogether, both operating systems trace their roots to UNIX.
I started my career in 1982, working on the UNIX kernel just as the operating system was taking off to power computers from mid-tier to high-end. I remember printing out its almost entirely undocumented source code and studying every line of trying to understand it. The listing was 250,000 lines of code and created one massive book, probably weighing some 20 to 30 pounds. I would carry that under my arm to work and home every day. I would spend most of my free time hand documenting its secret on that printed book. Thankfully I was married by then so I did not have to worry about looking like a textbook geek! As challenging as that work was, it was extremely rewarding both from the thrill of fixing and optimizing it but also giving me a solid foundation to understand the operation of our computers.
By the way, 250,000 lines may sound like a large program but in today’s world, it is nothing. The “little” media player application that ships as part of your operating system likely clocks at a million lines of code! Modern operating systems are beasts and have incredible breath of functionality and code to support them. With literally thousands of people working on them every year, they keep growing in an attempt to keep up with the latest hardware technologies and expectations for what an operating system should do. Despite that, the core concept of what they do is not hard to comprehend as I just explained.
An interesting development in operating systems is extensive use of them in consumer electronics devices. It used to be that TVs and DVD players were pretty simply devices and did not have an operating system. Today, they have so much functionality akin to a smaller computer that they almost always run an operating system with that choice almost always being Linux. This is why it takes a while for your equipment to turn on and shut off. It is the kernel booting and shutting down!
