What does it mean when I hear about the Spectre and Meltdown vulnerabilities?
In simple terms, these vulnerabilities allow unauthorized access to sensitive memory content on computing systems that should never be allowed. Successful exploitation of these vulnerabilities can compromise the core system integrity and all multi-tenant users or processes on a system; think public/private clouds, virtualization, and even desktop programs. These vulnerabilities take advantage of design flaws embedded in many types of processors (not only Intel). Most of the systems in your IT environment and at home are at risk until remediation activities have been completely implemented. The fixes to these issues will be complex, requiring a significant investment in time, resources, and productivity loss when compared to the canonical “Patch Tuesday” many organizations are accustomed to.
To protect IT environments, there are many things that companies will need to do. It not only includes managing the changes to the technical platforms but the impact on your support team’s operations and communications to your customers about how you are protecting their data. The key is to get organized, establish a plan, don’t underestimate effort, and dedicate resources to remediation activities immediately. Depending on your current IT infrastructure (cloud vs. hybrid vs. on-premise internal), you will have to coordinate internal and vendor resources to achieve success.
As your company works to address these vulnerabilities, you should take the following steps:
- Take time to understand the different types of computing systems and computing layers (e.g., virtualization, containers, web browsers) that are affected.
- Develop a remediation plan, including prioritizing the more critical assets and identifying resources that will need to be allocated to assist in the remediation activities.
- Develop a communications plan for your organization, employees, and customers, as applicable.
Three questions to ask yourself and what to do:
1. I use public cloud vendors for hosting services, what should I expect them to do to protect me?
Fortunately, Microsoft, Amazon, Google, and others were somewhat ahead or quick to respond to the public disclosure of these security vulnerabilities. These vendors have well established and practiced plans in place to ensure a timely and structured response to security events. Today, most vendors have already implemented the existing patches to their infrastructure or plan to do so in the immediate future.
It’s important to recognize that while hosting vendors are quickly patching the underlying infrastructure (i.e. bare metal), this does not address the security vulnerabilities at all computing layers (e.g., virtual machines, containers, and user or application processes). Understanding your vendors’ shared responsibility model and their guidance for the services your organization leverages will provide you with a series of actions to ensure that all IT systems are patched for these vulnerabilities.
Quick references to guidance provided by Microsoft, Amazon, Google, VMware, and Red Hat are below.
2. What do I need to do for any infrastructure that we maintain internally for either customer facing products or our corporate services?
For on-premises or collocated IT systems, the remediation activities can be even more challenging as all computing layers will need to be patched, including bare-metal platforms. In a common scenario leveraging virtualization technology in the data center, the hardware platform (e.g., Dell, HP, Cisco bare-metal), virtualization platform (e.g., VMware, Hyper-V), and virtual machines (i.e. Microsoft, Red Hat) will need to be patched. This may require multiple maintenance windows and downtime for each critical business system. Furthermore, over the coming weeks and months vendors may release new patches to more completely address the vulnerabilities or side-effects of previously released patches (i.e. performance issues) thus requiring additional resource time and maintenance windows.
While some patches for Microsoft Windows systems (Windows Server, SQL Server, and workstations) have already been released, Microsoft and other credible sources have reported that the patches are bricking systems (e.g., blue screen of death), causing noticeable performance slowdowns on older systems, and may be incompatible with common anti-virus software packages. This can sound terrible for IT system administrators and IT executives, but it’s important to understand that the patches released over the coming days will be an iterative and drawn-out series of updates as vendors and IT organizations across the globe race to secure IT systems, understand potential impacts, and optimize previously released patches.
Below are quick references to guidance provided by major IT vendors:
3. Will there be performance impacts when we apply the fixes?
While early reports were speculative, vendors (e.g., Microsoft, Red Hat) have confirmed measurable impact to system performance. Transactional databases, commonly used for web-based applications, appear to be among the more impacted systems reported to date. Other applications or processes that require constant transactions with the system’s memory and CPU will likely be impacted. In the coming days, we expect to see a significant amount of press and objective reports of impact other organizations are experiencing as they work to implement security patches.
As your organization and vendors implement the remediation actions to address these vulnerabilities, it’s crucial to have the appropriate system performance metrics being captured throughout the entire process. Of all the system metrics that can be recorded, the most important will be granular CPU metrics. This will include not only CPU load but also per-socket/processor metrics and per-CPU usage type (e.g., system, idle, user, kernel). Collection of these detailed metrics will allow your organization, specifically IT system administrators, to more accurately review, analyze, and report the impact security patches may be affecting IT systems.
Our security team assists clients in remediation planning and execution. If you’ve already been compromised, we have threat hunting capabilities. Contact a member of our Security team to have a discussion, we are here to help. As information on these vulnerabilities continues to become available, our team will be updating this post. Be sure to check back for updates.
Contact Tommy Borchers, Senior Manager
firstname.lastname@example.org or (312) 980-9441