The benefits of AI in cybersecurity are hard to dispute. When implemented properly, AI enhances detection and response, improves access controls, aids with threat hunting and other capabilities, and uses machine learning to continually improve response.
At the same time, AI is the tool of choice for malware creators who are developing AI-proof hacks against the security systems that deploy AI. Not to mention, the developers of AI-based security systems can accidentally introduce bias into AI-enabled security.
AI Bias
RiskSense CEO Srinivas Mukkamala, Ph.D., earned his doctorate in computational intelligence for large-scale intrusions. He explains that AI bias occurs in at least three places in the AI development and AI data consumption level: humans who are developing it, data that is being used to develop the models, and algorithms behind the AI decision-making program.
"You can actually introduce algorithmic bias that creates sloppy false positive and false negative rates because the program may try to balance the dataset by adding more data when there is not enough data available to the AI system," Mukkamala says.
Take, for example, AI that is based on the national vulnerability database. Of the more than 134,000 vulnerabilities listed as of December 2019, just over 24,000 are critical or high. A much smaller percent of those are capable of a remote code execution or privilege escalation, and only a few hundred of these vulnerabilities are commonly exploited, Mukkamala adds.
In fact, when RiskSense analysts examined a number of AI algorithms against the national vulnerability database, they reported between 25 and 30% false positive and false negative rates.
"Don't just trust your vendors that their AI is good. You have to ask, what is your AI model trained on, who built it, what were that person's biases?" continues Mukkamala, who recently moderated a panel on this topic at SINET. "The AI expertise needs to understand software risk, data modeling and mathematical modeling. And that modeling should be free from bias, which is the only way to remove false positives and false negatives from the equation."
AI Turned Against Us
In his blog about weaponizing AI security against organizations using this technology, Isaac Ben-Israel, director of the ICRC and chairman of Cyberweek, outlines ways AI-savvy malware could inject false data into the AI system and disrupt patterns used for machine-learning decision trees.
Ben-Israel also describes a process called "bobbing and weaving," wherein attackers input innocuous routines that make the AI system deem them normal machine behaviors; then later, attackers come back to exploit those routines without notice. Even changing log files can confuse the AI system, he adds.
Diana Kelley, a Microsoft CTO, said during an CNBC broadcast that AI must be resilient in order for it to remain accurate and stand up against AI-resistant attack methods.
AI Protecting AI
Microsoft's philosophy is that the AI/ML system itself should be able to detect bias in its systems. For instance, the system should discern maliciously introduced data from benign events, as well as have its own built-in self-forensics capabilities for transparency and accountability of the AI.
To do that, Microsoft recommends developing AI in a way that it can detect and overcome accidental or intentionally introduced bias in the datasets it trains on. The AI system itself must ensure the data it trains on is accurate and clean, while also detecting red-flagged words and phrases that would indicate bias.
Organizations purchasing and utilizing AI in their security modules need to monitor the quality of their data and changes to AI data input. And they should remember that AI is no replacement for the human analyst, but rather an augmentation.
"There's a lot of anxiety around AI right now," says Mukkamala. "What we're struggling with is not well-defined, and we're really just in the development phase of the process when it comes to AI in vulnerability and threat management."
