The U.S. spent $2.2 million on a cybersecurity system that wasn’t implemented — and might have stopped a major hack
This article was first posted on ProPublica.
As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.
The massive breach, which U.S. intelligence agencies say was “likely Russian in origin,” penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.
The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors”) that granted them access to an untold bounty of government and company secrets.
The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.
This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole”), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life’s work. In 2013, Popular Science named him as one of its “Brilliant Ten” scientists under 40.
Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn’t happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.
“In security, you almost never go from making something possible to impossible,” Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack.” Although the SolarWinds breach was a “really sneaky” approach, Cappos said, “in-toto definitely can protect against this. It’s very possible to catch it.”
In-toto’s system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto’s federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent.” Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure” and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it’s impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately,” said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies.”
Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams, head of Microsoft’s initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world.” (After Williams’ initial response, Microsoft declined to comment further.)
One senator blasted the government’s failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains.”
The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what’s done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn’t tampered with in transit.
