Ten years ago, Amazon began due diligence on a startup called Elemental Technologies, as they were considering them as an acquisition to boost their streaming video service (now Amazon Prime Video). Elemental made software for compressing large video files and they had several national security contracts that made Amazon comfortable with cybersecurity considerations and the integration with AWS, as Amazon has many large contracts with agencies like the CIA.

A third-party contractor discovered some troubling issues on their first diligence pass which prompted Amazon to take a closer look at the hardware servers that Elemental customers installed in their networks. It turns out that these servers were assembled by Super Micro Computer Inc., a San Jose-based company (aka, Supermicro) that’s also the world’s biggest independent supplier of server motherboards.

A Closer Look

At the evaluation site, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon realized that these servers were probably installed at Federal agencies as well, and after notifying the Federal government, discovered that Elemental’s servers were networked throughout the Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships.

And Elemental was of course just one of hundreds of Supermicro customers.

During the ensuing probe, investigators discovered that the chips were designed to allow attackers to create a stealth doorway into any network connected to the tricked-out servers. Not surprisingly, investigators have traced the motherboards back to factories run by manufacturing subcontractors in (gasp!) China.

This attack scenario is far deadlier than the software-based back-doors that we all have grown accustomed to seeing. In fact, a well-done, nation-state-level hardware implant would be like wandering badge-less into the Pentagon unchallenged. Yet here we are with maybe 80% of the world’s servers sporting a motherboard with the additional chip. Completely beneath the radar.

Interdiction or Seeding

There are two ways for spies to alter the guts of computer equipment. One, known as interdiction, involves manipulating devices as they’re in transit from manufacturer to customer. This approach is favored by our own U.S. spy agencies. The second method involves seeding modifications into the core processors at birth.

China makes 75 percent of the world’s mobile phones and 90 percent of its PCs. To pull off a seeding attack would mean that the perpetrators would have to develop a deep understanding of a product’s design, then be able to manipulate components at the factory, and finally be able to ensure that the doctored devices would make it through the global logistics chain to the desired destination. All of which is as difficult as it sounds.

Yet, that is exactly what was found here. The chips had been inserted during the manufacturing process, by operatives from a unit of the People’s Liberation Army and their goal was clearly to gain long-term access to high-value corporate secrets and sensitive government networks.

Which they have done. The chips were designed to be as inconspicuous as possible, appearing more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment.

Depending on the board model, the chips varied slightly in size, suggesting that the attackers had supplied different factories with different batches. Many different factories with many different batches.

Uber-Control

The primary role of the chips was to manipulate the core operating instructions that tell the server what to do as data moves across a motherboard, and at a crucial moment, allow it to effectively edit the information queue, inject its own code or alter the order of the instructions the CPU was meant to follow.

The chips were then able to tell the device to communicate with one of several anonymous computers elsewhere on the Internet that were loaded with advanced code; and to prepare the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

This system could then let the attackers alter how the device functioned, line by line, however they wanted, leaving no one the wiser. As one example, consider the Linux operating system where code resides that validates a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code, so the server won’t check for a password any more, and voila, the secure machine is open to all users, authorized or otherwise.

That same chip can also steal encryption keys for secure communications, block security updates that would neutralize the attack, and open new pathways to the Internet.

Customer Surprises

In the early days, the investigation concluded that over 30 companies had been affected, including a major bank, large government contractors, and one of the world’s most valuable companies, Apple. Apple had been one of the top Supermicro customers and had planned to order an additional 30,000 of its servers for their new global network of data centers. But in the summer of 2015, Apple conducted their own investigation and discovered the rogue chips which led to an immediate cancellation of that order and severance of the business relationship.

The extent to which Apple’s existing servers are affected and how those networks have been compromised is unclear. Apple’s official statement was that no networks or servers had been compromised and nothing was at risk.

In with In-Q-Tel

In 2009, Elemental announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, along with a deal that enabled Elemental servers to be used in national security missions across the U.S. government, including the Department of Defense data centers, to process drone and surveillance-camera footage, on Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing.

NASA, both houses of Congress, and the Department of Homeland Security have also been and are still, customers.

Charles, the Big Guy.

Supermicro, founded in 1993 by Charles Liang, a Taiwanese engineer was an early beneficiary of the contract manufacturing outsourcing trend in the Valley and began by provided engineering design services along with access to supply chain factories in Taiwan and China.

Today, Supermicro sells more server motherboards than almost anyone else on the planet.

It also dominates the $multi-billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order servers for banks, hedge funds, cloud computing providers, and web-hosting services, throughout the world.

Its core product, the motherboards, are all manufactured by contractors in China.

Hundreds of full-time Taiwanese or Chinese engineers with overseas ties work at the San Jose facility where Mandarin is the chosen spoken language supplemented by the written Hanzi which “makes it easier for visiting Chinese partners to gain an understanding of their operations and designs.” The current investigation is still trying to determine whether active spies were planted inside Supermicro and/or any of its suppliers.

The easiest way to imagine Supermicro is to think of them as the Microsoft of the hardware world, with more than 900 customers in 100 countries by 2015, Supermicro potentially offered inroads to a massive collection of sensitive targets. Attacking their motherboards is like attacking Windows.

The reality is in fact that the security of the entire global technology supply chain has been compromised, though consumers and most companies don’t know it yet.

Way Back Then.

Way back in 2014, U.S. intelligence officials had specific and concrete evidence that China’s military was preparing to insert these rogue chips into Supermicro motherboards bound for U.S. companies. But, issuing a broad warning to Supermicro’s customers could have crippled the company, (still a major American hardware maker), and it wasn’t clear from the intelligence exactly whom the operation was targeting or what its ultimate aims were. Plus, without confirmation that anyone had been attacked, the FBI was limited in how it could respond.

So, we did nothing.

At last count, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan, one in Shanghai and subcontract manufacturers scattered throughout China.

Four of those subcontractors are now known to have been building out Supermicro motherboards for at least the last two years. Those subs operate the way all subs operate in China.

Plant managers at the subs are approached by people who claimed to represent Supermicro or who held positions connected to the Chinese government. These middlemen would request changes to the motherboards’ original designs, offering bribes where they encountered resistance or suspicion and even threatening factory managers with inspections that would surely shut down their plants. Once these “arrangements” were finalized, they would organize the rest of the supply chain back to Supermicro.

And we know all of this, yet we do business with them anyway.

Chinese Comedy

China comically wants the world to believe that their 2011 proposed international guarantees on hardware security was somehow meaningful and that their plea to “parties” making “gratuitous accusations and suspicions” must end, so that we can all Kumbaya and “conduct more constructive talk and collaboration together in building a peaceful, safe, open, cooperative and orderly cyberspace.” Right.

As far as Amazon is concerned, their security team, conducting its own investigation into AWS’s Beijing facilities found altered motherboards throughout those servers, including more sophisticated designs than they’d previously seen. In one case, the rogue chips were thin enough so that they’d been able to be embedded between the layers of fiberglass onto which the other components were attached.

While China has long been known to monitor banks, manufacturers, and ordinary citizens on its own soil, and the main customers of AWS’s China cloud were domestic companies or foreign entities with operations in China, the fact that the country was conducting those operations inside Amazon’s cloud presented a Gordian knot.

The dilemma was that it would be difficult to remove the equipment without alerting interested parties and that, even if they could devise a way, doing so would certainly alert the attackers that the chips had been discovered. So, they instead developed a method of monitoring the chips and in so doing, could not identify any traffic among the entry points and the attackers. Which only told them that the bad guys were saving the chips to mount a delayed attack or that they’d infiltrated other parts of the network before they started monitoring, neither of which was particularly reassuring.

The point is that this is all going on right now as the world ponders what to do about this invasion.

Apple and Amazon Leave Supermicro

When two of its major customers, Apple and Amazon bailed out on Supermicro in 2016, its CEO, Liang vaguely blamed it on pricing with the statement, “When customers asked for lower price, our people did not respond quickly enough.” Whether that sort of dissembling raises the question of complicity, I will leave to you, but it seems to me that Supermicro is making it harder to take seriously the company’s case for innocence.

Because concurrent with the malicious chips’ discovery in 2015 and the investigation continuing to unfold around it, Supermicro has suspiciously missed two deadlines to file quarterly and annual reports required by regulators and was delisted from the Nasdaq that summer. This seems like extraordinary behavior for a company growing at a breath-taking rate over the last five years ($3.2 Billion 2018 to $7.1 Billion in 2023).

Concessions? What Concessions?

In 2015, then President Obama and Chinese President Xi Jinping appeared together at the White House for a joint press conference touting a landmark deal on cybersecurity. China had agreed that it would no longer support the theft by hackers of U.S. intellectual property to benefit Chinese companies. But anyone who had been paying attention, including senior White House officials knew that China was happy to offer this concession because it had already developed advanced forms of cyber-theft supported by its monopoly of the Asian technology supply chain.

U.S. government officials and U.S. technology executives subsequently met in a Pentagon sponsored event to determine ways in which the U.S. could develop commercial products that could detect hardware implants. Everyone present knew that the hardware on topic was being produced by Supermicro. Why, in the name of national security did we not simply shut the company down?

I am acutely aware that it is unbelievably difficult to make decisions and execute any sort of rational sounding plan within the labyrinth we call the Federal government. There are hundreds of thousands of people in roles where they have been since the last several administrations, continuing to do the work that they are chartered to do regardless of who is in charge now or what their political agenda happens to be. There are tons of moving parts and hundreds of shadow agencies and agreements dependent on any directional movement, one way or the other. In short, it is not easy to get “easy” things done.

The Cheap Ship has finally Sailed.

Decades ago, we had made the decision to send advanced production work to Southeast Asia and low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Apple, for instance, made many of its most sophisticated electronics domestically up until 1992 when they closed their state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent that work to China.

The assumption that China would surely not do anything like spying in its factories to jeopardize its position as the workshop to the world became the guiding principle for continuing the folly. We seem to be intoxicated with this classic devil’s bargain as our somatic capitalism continues to opt for the supply chain we desire, even with all of the known global risks over accepting a lower capacity supply chain with greater security. Well, according to the current administration, that dog no longer hunts.

As the White House has focused on computer and networking hardware, including motherboards in its latest round of trade sanctions against China, we will now have to figure out how to shift our technology supply chains to other countries and/or bring them back to America.

The cheap chip ship has finally sailed, though it leaves behind more than 80% of the world’s servers infected with an incurable cyber-disease.

What we do next matters more than anything we have done in the past.