Timesys’ Director of Engineering, Akshay Bhat, presented a session on Open Source Security at the Embedded Linux Conference North America 2019 in August. For this two-part Q&A interview, our VP of Marketing Adam Boone asked Akshay to share his views on the challenges and best practices for maintaining security in Open Source Embedded System products.
Adam Boone: Why should product developers and engineering managers be familiar with CVEs and make an effort to monitor them?
Akshay Bhat: I think everyone recognizes it is important to bring products to market that are secure and that stay secure throughout their deployment lifecycles.
Who wants to deliver a product that is going to put a customer or user at risk of a data breach, system hijacking, or other security issue now or two years from now?
This is especially important today as embedded systems products are being deployed in greater numbers than ever before, and in locations and configurations we have never seen before. Plus, these systems are supporting critical processes and essential services we all depend on. Think about the Internet of Things, Industrial Internet of Things, industrial control systems, utility control systems and so on.
And perhaps most critically, more and more of these embedded systems contain Linux and other open source components. While open source is a fantastic way to get products to market quickly, you need to make sure these components are secure so that customers are not put at risk.
So anyone working on the software for an embedded system product today needs to know about Common Vulnerabilities and Exposures, or CVEs, that are announced and maintained in the National Vulnerability Database. The NVD is maintained by the US National Institute of Standards and Technology, or NIST.
The challenge, of course, is how much effort you can expend in monitoring, managing and fixing vulnerabilities if they happen to affect your products.
While many of us know about the NVD, there actually are many other potential sources of important vulnerabilities, such as manufacturer security bulletins, security mailing lists, and bug lists. And there are what you might call silent bug fixes in which a manufacturer fixes a bug without issuing a public notification.
In a perfect world, you would be tracking all these sources of security info and then perform static analysis of your software to assess the security risk. But the amount of time and resources invested in that would be massive, and there is what they call the point of diminishing returns. At a certain level, a vulnerability poses such a low level of risk that it makes no sense to expend any time to investigate and fix it.
The challenge of how to deploy your resources and how to prioritize security issues is even greater when you consider that there are something like 300 new vulnerabilities disclosed in the NVD every week.
So we advise our customers to make security maintenance a part of the product development and product maintenance processes, and central to that effort is monitoring and managing CVEs.
Adam: That’s sound advice. Overall, what approaches do you see developers and engineering managers taking to monitoring and addressing CVEs?
Akshay: Well, the easiest approach is to do nothing at all and hope for the best. But that’s clearly a non-starter and very frankly, irresponsible.
A common approach to dealing with CVEs is to monitor, analyze and address them manually, in a Do-It-Yourself (DIY) process.
While that might seem straight-forward, it’s really the least effective and most costly approach, when you consider the time and effort it takes to analyze 300 CVEs every week. On top of that, the same team needs to be monitoring patches and software versions of all components across all your products.
And oh, by the way, these same people probably are involved in actually delivering product and enhancements to market. So they need to find some time to work on that too, right?
It’s really overwhelming to do the manual approach and expect to have a clear view of vulnerabilities affecting your products because there are so many moving parts.
Adam: What does that mean? What are the moving parts?
Akshay: On the one hand, there are the contents of your product itself. You need to have current information on all packages and versions across all the products in development and in maintenance or support. That of course can change at any time, so you need a way to constantly monitor your own product components.
On the other hand, there is the flood of CVEs that you need to review, sift through and match to your product contents.
Then, there are the updates, patches, upgrades and enhancements released by third parties who support the components in your product. All three of these sources of data need to be monitored and analyzed (Figure 1).
Figure 1: Do-It-Yourself (DIY) CVE monitoring process
When you arrive at a list of the CVEs affecting your products, you then need to triage the list, analyzing them to determine which are the most pressing to address because they pose the greatest risk for your products based on their configurations and how they are deployed.
All these things need to be accomplished and repeated in a consistent way. And that’s before you have actually addressed even a single vulnerability.
Adam: CVE monitoring is built in to Yocto. Doesn’t that make the process simpler if you use Yocto?
Akshay: That can help. But the process of pinpointing the specific components that are affected in your builds is still very cumbersome in Yocto’s CVE readouts.
And even worse, our analysis of Yocto CVE monitoring shows an excessive number of false positives and CVE misses. That’s because much of the NVD data has naming inconsistencies, simple typos in data records, outdated information or missing data.
Any of these can cause Yocto CVE monitoring to report a CVE match where none exists or miss CVEs that should have been a match.
Adam: So let’s say we have assembled a list of CVEs we believe affect our product line. What’s next? How do you triage and prioritize them?
Akshay: The step of triaging the CVEs is just like in the hospital or medical care setting. Emergency room doctors will evaluate patients coming in and move those with the most dire, pressing conditions to the top of the list for care.
With CVEs, you analyze all the CVEs in that batch and figure out which ones pose the biggest risk and must be fixed right away. Others might be less pressing, with less likelihood of causing a security problem for a customer. And others might be not important at all and can be safely ignored.
When you do this triage analysis, you will look at things like the Common Vulnerability Scoring System, or CVSS. That’s the commonly used model for scoring a given CVE in a standard way to arrive at a severity rating.
CVSS ratings may rank Low, Medium, High or Critical. The rating then provides some good initial guidance for your comparison and prioritization of one CVE versus another.
But then you also need to consider a lot of other factors.
For example, does the CVE’s attack vector matter in your product’s deployment? In other words, for an attacker to exploit the CVE, will they be able to launch an attack against your product using that vector?
A common example is a CVE that uses the network attack vector, such as gaining unauthorized remote access to a device. If your product is not going to be deployed as a networked device, or will be on its own entirely isolated, air-gapped network, then this CVE may not be as important to address, even if it is has a Critical CVSS rating.
Similarly the configuration of your system and components may make a given CVE irrelevant because a process required for an exploit of the CVE is not activated in your product.
Another consideration for triaging your list of CVEs is whether a known exploit of a CVE is in the wild.
Sometimes a CVE comes to light because a hacker has already built an exploit to take advantage of a vulnerability that they found on their own. Unfortunately, it might take a damaging security breach in a customer for the CVE to be identified and reported.
But many CVEs are found and reported before that happens because security researchers, bug bounty programs, and responsible disclosure programs are notifying vendors, notification list managers, and security database authorities. That gives product developers a head-start to address CVEs before an exploit appears to take advantage of them.
Similarly, a reason to prioritize CVEs higher is whether an easily available patch is already available and known and whether a single CVE fix can address multiple of your products that have components in common. That’s low-hanging fruit to fix with minimal effort and so may make sense to take care of quickly.
In part 2 of this post, we’ll continue the discussion about maintaining security in open source embedded system products as Akshay addresses the challenges with patch management and talks about best practices for security maintenance and CVE management workflows.
About Timesys
Timesys has extensive experience with embedded system development and lifecycle management. Timesys has been instrumental in working with global leader semiconductor manufacturers with smart, quick and quality solutions for highly complex systems with accelerated product innovation and multiple product variants.
