Linux dominates supercomputing

In addition to all its other successes, Linux dominates supercomputing: All 500 of the worlds fastest supercomputers run Linux, and that has been the case since November 2017, according to the TOP500 organization, which has been ranking the 500 most powerful computer systems since 1993. (A graph of Linux’ ascension is available on here.)

How did this happen?

A little history

Linux began life in 1991 as the personal project of 21-year-old Finnish student Linus Torvalds. I first became aware of it several years later while working at Johns Hopkins University’s physics and astronomy department where I managed the department’s network and a number of servers with the help of a couple grad students.

At the time, I was intrigued by Linux, but couldn’t imagine how dominant the OS would become simply because its source code was available to anyone who wanted to work with it. I could not forsee that a significant group of large companies would grasp its value, work together and innovate to make Linux what it is today. Intense collaboration was key to this success, including contributions from countless individuals and organizations that includes IBM, Intel, NVIDIA, Red Hat, Samsung, SUSE and many others. (Browsing the corporate members of the Linux Foundation is likely to make your jaw drop.)

The success of Linux can also be attributed to the fact that it is open source, non-proprietary, and extendable.

The TOP500

Twice a year, in June and November, TOP500 releases it a list of the 500 most powerful computer systems ranked by their performance on something called the LINPACK Benchmark, which calls for the computer being tested to solve a dense system of linear equations.

I have heard it said, though not been able to verify, that Linux runs on more than 90% of public clouds, more than 60% of embedded systems and IoT devices, as much as 99% of supercomputers and more than 80% of smartphones. If these claims are even close to the truth, it attests to the success and versatility of Linux

In the most recent TOP500 ranking, a Japanese supercomputer name Fugaku (derived from an alternate name for Mount Fuji) has taken the top spot and pushed the former leaders down a rank. Fugaku was co-developed by Riken and Fujitsu and uses Fujitsu’s 48-core A64FX ARM chip. This is the first time a computer based on ARM processors has topped the list.

The computer was fully assembled only in May but has already helped fight COVID-19 by sorting through more than 2,000 drugs that might effectively block the virus and found a dozen that show promise.

Containerization, AI and ML

Recently I had a chance to discuss Linux with Stefanie Chiras, vice president and general manager of the Red Hat Enterprise Linux business unit. She sees Linux as tightly linked to supercomputing because it provides the scale and flexibility to support high-performance computing and exascale computing systems – those that are capable of calculating at least 1,018 floating point operations per second (1 exaFLOPS). She also sees Linux as adding to the ongoing development of artificial intelligence and machine learning.

Chiras expects that containerization will enable more and more researchers and analysts to benefit from supercomputing power. And, as someone who has provided support to both scientists and analysts over the past few decades, I can appreciate what a difference this will make in their work.

In discussing supercomputers, Chiras pointed out that, beyond having a Linux operating system, the top three fastest – Fugaku, Summit and Sierra – are all built from commercial hardware. Summit and Sierra are Power Systems-based, while Fugaku is the first Arm-powered supercomputer to top the list. The days of purpose-built, custom hardware and software in supercomputing could be over even though new and demanding workloads like AI and complex modeling require greater and greater power.

(Chiras also wasn’t shy about mentioning that Red Hat Enterprise Linux is running on four of the top 10 supercomputers: the top three plus number nine, Marconi-100.)

The character of open source and the willingness of many companies to recognize its value and work together to develop it have made Linux the top OS for both supercomputers and micro-devices. We can expect continued improvements in how the OS is deployed – including getting supercomputing into the hands of a lot more scientists and engineers – as we move forward. As a person who has spent close to 40 years working with Unix and Linux systems, I couldn’t be more pleased.