Microsoft has revealed how it’s applying machine learning to the challenge of correctly identifying which bug reports are actually security-related.
Its goal is to correctly identify security bugs at scale using a machine-learning model to analyze just the label of bug reports.
According to Microsoft, its 47,000 developers generate about 30,000 bugs a month, but only some of the flaws have security implications that need to be addressed during the development cycle.
Microsoft says its machine-learning model correctly distinguishes between security and non-security bugs 99% of the time. It can also accurately identify critical security bugs 97% of the time.
The model allows Microsoft to label and prioritize bugs without necessarily throwing more human resources at the challenge. Fortunately for Microsoft, it has a trove of 13 million work items and bugs it’s collected since 2001 to train its machine-learning model on.
Microsoft used a supervised learning approach to teach a machine-learning model how to classify data from pre-labeled data and then used that model to label data that wasn’t already classified.
Importantly, the classifier is able to classify bug reports just from the title of the bug report, allowing it to get around the problem of handling sensitive information within bug reports such as passwords or personal information.
“We train classifiers for the identification of security bug reports (SBRs) based solely on the title of the reports,” explain Mayana Pereira, a Microsoft data scientist, and Scott Christiansen from Microsoft’s Customer Security and Trust division in a new paper titled Identifying Security Bug Reports Based Solely on Report Titles and Noisy Data.
“To the best of our knowledge this is the first work to do so. Previous works either used the complete bug report or enhanced the bug report with additional complementary features,” they write.
“Classifying bugs based solely on the tile is particularly relevant when the complete bug reports cannot be made available due to privacy concerns. For example, it is notorious the case of bug reports that contain passwords and other sensitive data.”
Microsoft still relies on security experts who are involved in training, retraining, and evaluating the model, as well as approving training data that its data scientists fed into the machine-learning model.
“By applying machine learning to our data, we accurately classify which work items are security bugs 99% of the time. The model is also 97% accurate at labeling critical and non-critical security bugs. This level of accuracy gives us confidence that we are catching more security vulnerabilities before they are exploited,” Pereira and Christiansen said in a blogpost.
Microsoft plans to share its methodology on GitHub in the coming months.
