As discussed in last week’s posting, central to the device maintenance process and keeping devices secure after they’ve been deployed is the ongoing monitoring and managing of CVEs that affect your product components. Therefore, it’s essential to have a clear view of relevant CVEs because there are many moving parts that need to be managed.
Adam Boone: Along those lines, you mentioned monitoring patches and software upgrades as one of the moving parts to be managed in a security maintenance program. What’s the challenge there?
Akshay Bhat: Patch management alone is always challenging, especially if you have a large number of open source components. You need to evaluate when to apply a patch, how the patch affects other components, what testing needs to be conducted, whether a patched component can be backported to earlier versions, and so on.
But add to that the pressure of dealing with a high severity CVE, one that might be putting your customers at risk right now, and patch management gets even more challenging.
We guide our customers to follow a consistent process in a CVE-driven patching scenario. Once a given CVE has been identified, your team will investigate the fixes and so may review logs, bug tracker reporters, and other documentation distributed via the mailing lists for upstream packages.
DIY CVE Patching
The right fix will depend on many factors. There might be a patch for a set of issues including the specific CVE. Or it may make more sense to upgrade versions because a newer version is not exposed to the CVE.
We always urge our developer customers to upgrade to the latest version of software components whenever possible. The latest version will address CVEs and possibly other security vulnerabilities that might not be part of the CVE dictionary. This is particularly true for the Linux kernel where upgrading to the latest LTS release is the best bet to keep it secure.
But you also need to consider earlier versions of your code and whether any fixes can be easily backported without breaking product functionality.
The decisions become even more complex if patches or upgrades become available for multiple components in your product at the same time. Analyzing potential conflicts and regression testing become essential steps to manage that type of scenario.
Adam: So is it fair to say that time is the enemy when it comes to monitoring CVEs and addressing them?
Akshay: Yes, time is the enemy in more ways than one.
For example, there often is a gap of days or weeks between the time a vulnerability has been first disclosed, it has been fully analyzed and a patch has been issued.
That’s your “vulnerability window,” which is the period during which the black-hat hacking community has become aware of the vulnerability and is working hard on designing exploits to take advantage of it. If the exploit emerges and spreads before the full analysis is available or the patches are issued, then you may be fully vulnerable to a damaging attack.
Adam: Given all this complexity, what’s your advice to engineering managers looking to boil all this down into a simple, manageable process?
Akshay: The first step is recognizing that you need that process at all.
Security maintenance and CVE management workflows should be a central part of all product maintenance reviews and processes. Surprising numbers of companies don’t take that active approach but only focus on CVEs when there is a major problem, like a customer with a security breach.
Once you have established it, the process itself should feature a few key elements.
First, you need tools for monitoring and filtering CVEs based on your actual, accurate product components. As I mentioned earlier, you need something less prone to false positives and CVE misses than the Yocto CVE monitoring feature.
Then you need workspace and communication tools to support CVE investigation, triage, prioritization and collaboration.
That should include the ability for teams to share findings, status and actions, and especially to learn from one another as the process matures. Your process also should include reports and dashboards that enable you to see the big picture and understand the CVE counts by product, the progress in fixing them, and the changes over time.
Then, you need to automate the patch monitoring and management process as much as possible. That involves automatically identifying patches and upgrades that pertain to the specific product components and enabling your team to assess them as a fix for a particular CVE.
At Timesys we have built these elements into an automated security maintenance platform, Vigiles, which provides CVE monitoring, investigation and mitigation tools.
Your team can try Vigiles Basic for free here: https://timesys.com/security/vigiles/
Ultimately, embedded system security really needs to be addressed in a layered approach. That is, you need to focus on what we call “Secure by Design” along with the “Stay Secure” principles like what we have been discussing here.
So you may devote a portion of your product design effort to security features like Secure Boot and establishing a Chain of Trust. You may focus on device hardening to reduce your product’s attack surface to minimize ways attackers can compromise it.
These design steps then need to be complemented by Stay Secure processes that include the security maintenance practices we have outlined here. Taken together, these strategies will enable embedded system product developers to bring more secure products to market and keep them secure over the product lifecycle.
Learn more about Timesys embedded open source security solutions here.
About Timesys
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.
