In 2017, a power failure caused critical computer systems to shut down at Delta Airlines, leading to almost 700 flight cancellations and thousands of delays, reducing Delta’s flight throughput for the day by approximately 60%. Unlike many other system characteristics, reliability and security are emergent properties of a system’s design, i.e., they are often determined by the interaction of many components. Even a harmless-looking update to one component might compromise the security of the entire system.
Security of software supply chain systems came under scanner during the Solarwinds attack last month. According to Microsoft’s investigation, the hackers managed to view source code in several source code repositories. The Solarwinds hack was so well-planned that the hackers even compromised the secured servers of government institutions too.
According to estimates, the hackers infiltrated dozens of government and private networks via management software provider SolarWinds. The attackers breached multiple layers of the supply chain, gaining access to as many as 250 networks.
The Data Security Council of India (DSCI) said the cyberattacks could be carried out in several ways: Computer-network , Supply-chain, Social-networking, Radio networks for GPS etc. Of which, software supply chains are presumed to inflict more damage to organisations. It represents the links across organisations—an individual company can only do so much on their own.
Fog Of War
Sizing up potential adversaries can be a complex and nuanced exercise. One way of preparing for attacks is to map out all the possible steps a hacker might take to launch an attack. Some security researchers use formalised frameworks like the Cyber Kill Chain as mentioned above. Here are a few thumb rules to remember:
- Size doesn’t matter. Even smaller organisations are prone to attacks and will be used as pawns to launch a bigger attack. Be vigilant always.
- Get the basics right. Attackers sometimes use basic attacks like phishing instead of firmware backdoors.
- Attackers can go to any lengths to achieve their goals.
- Consider the insider risk. Employees in higher positions are a big vulnerability.
In the next section, we shall take a look at a few of best practices recommended by the security experts in their work.
Software supply chain attacks are well-documented; each new episode reveals new challenges. The SolarWinds event is profoundly concerning. On the other hand, it also highlights the opportunities for government, industry, and other stakeholders to collaborate on best practices and build effective technology to improve the software ecosystem. So, to address the knowledge gaps in the industry and to assist organisations in thwarting future attacks, reliability experts at Google have published a book discussing the best practices for building secure and reliable systems.
Google uses both Go and C++ for their interoperability with well-tested cryptographic libraries, exhibit excellent performance. Go was picked to address the long history of memory-related vulnerabilities in code. Whereas C++ is not memory-safe, its interoperability for critical subcomponents of the system makes it the best choice.
The authors reiterated the code would inevitably have bugs, and one can avoid these common security vulnerabilities and reliability issues by using hardened frameworks and libraries. “Manual code reviews won’t find every issue, and no reviewer will catch every security problem that an attacker could potentially exploit,” the authors warned.
According to the experts, a better approach is to handle security and reliability in common frameworks, languages, and libraries. “Ideally, libraries only expose an interface that makes writing code with common classes of security vulnerabilities impossible. Multiple applications can use each library or framework. When domain experts fix an issue, they remove it from all the applications the framework supports, allowing this engineering approach to scale better.” A centralised hardened framework also diminishes the chances of future vulnerabilities. However, it is still possible for attackers to discover an unforeseen class of attacks or find mistakes in the implementation of the framework. And, that’s where common frameworks and libraries come in. One has to discover the vulnerability instead of devouring the codebase.
Defense At Scale
Large organisations, such as Google, should be able to afford global scale technical infrastructure designed to provide security through the entire information processing life cycle. This infrastructure ensures a safe time on the internet. The security of the infrastructure needs to be designed in progressive layers starting from the physical security of data centres, extending to the security of hardware and software that underpin the infrastructure. Finally, technical constraints and processes should be in place to support operational security. Companies like Google invest heavily in securing its infrastructure with hundreds of engineers dedicated to security and privacy distributed across all of Google. Integrating independent encryption layers for sensitive data is another common mechanism for top-notch defense.
Protect The Boot Process
At Google’s data centres, machines boot a known firmware/software stack, cryptographically verify this stack and then gain (or fail to gain) access to resources on the network based on that verification status. Titan integrates with this process and offers additional layers of protection. For example, during Titan chips’ manufacturing process, unique keying material for each chip is generated. The provenance information is stored in a registry database. The database contents are cryptographically protected using keys maintained in an offline quorum-based Titan Certification Authority (CA).
Handling Technical Debt
The authors said the pressure to meet market demands could lead system developers and maintainers to cut corners and accumulate technical debt. No matter how simple and elegant the initial design, systems rarely remain unchanged over time.
For example, back in 2018, YouTube went down for an hour due to a small change in a generic logging library. An update to the logging system that looked harmless by the code reviewer ended up crashing YouTube. The developers didn’t fully realise the impact. Under production load, the update caused YouTube servers to run out of memory and crash. As the failures shifted user traffic toward other healthy servers, cascading failures brought the entire service to a halt. This is why the experts advocate for a simple and elegant initial design as systems rarely remain unchanged over time. Since new feature requirements, changes in scale, and evolution of the underlying infrastructure all tend to introduce complexity, there is a need to keep up with evolving attacks as new adversaries can also increase system complexity.
The experts suggest measuring the probability of negative events differently for reliability and security. One can reason about the reliability of a composition of systems and plan engineering work by assuming independence of failures across the individual components, making it challenging to make an assessment. According to the authors, adversarial testing typically performed from the perspective of a defined adversary—can also be used to evaluate a system’s resistance to particular kinds of attacks, the effectiveness of attack detection mechanisms, and the potential consequences.
Nuggets Of Wisdom
“If the system design is as simple as possible, it improves the ability to assess both the reliability and the security of a system.”
“Load is a function of the volume and the average cost of requests to the system. To achieve resilience, processing should be less and cheap.”
“A properly reviewed code can help to mitigate the risk of an insider pushing a malicious binary to production.”
“The capability to push changes quickly is a double-edged sword: while this capability can help close vulnerabilities quickly, it can also introduce bugs.”
Find out more about building secure systems here.
Subscribe to our Newsletter
Get the latest updates and relevant offers by sharing your email.