Harnessing artificial intelligence to build an army of virtual analysts

Enterprises of all types and sizes are continually probed and targeted by cyber attackers. It doesn’t matter whether they are after the company’s or their customers’ information, or are trying to find ways in so that they can commit fraud, what matters is that many are succeeding.

So far, the security industry’s attempts to stop them have not been enough, but maybe this situation will finally change.

An innovative combination

PatternEx, a startup that gathered a team of AI researcher from MIT CSAIL as well as security and distributed systems experts, is poised to shake up things in the user and entity behavior analytics market.

On Wednesday, the company launched its Threat Prediction Platform, which combines the ability of machines to extract patterns from massive volumes of data with the capability of human analysts to understand the implications of these patterns.

Their goal was to make a system capable of mimicking the knowledge and intuition of human security analysts so that attacks can be detected in real time.

The platform can go through millions of events per day and can make an increasingly better evaluation of whether they are anomalous, malicious or benign. The company’s human analysts aren’t overwhelmed with an avalanche of unnecessary alerts and don’t end up burned out.

A platform that never stops learning and adapting

Let’s face it, most companies don’t have the budget to employ an army of analysts – but this is just what PatternEx is offering.

“The whole purpose of this product is to make the analyst(s) you have super efficient,” Uday Veeramachaneni, one of the co-founders and the current CEO of the company, told me.

The platform can effectively work with just one analyst at the helm. It “learns” how to mimic the analyst with the help of the analyst himself. The whole process, from start to end, looks like this:

The inputed raw data comes from the company’s networking devices – firewalls, proxies, etc. The system’s algorithms create behavior predictions, detect rare events (and unusual behaviors), and point them out to the analyst.

The analyst looks at the provided information and identifies malicious events. He labels them and this feedback is absorbed by the system. The algorithms then start creating models that will allow the platform to predict the very next day whether an anomalous new event is one (already labeled) attack or another, or whether it is benign.

On the second day, the analyst comes in and the platform shows that it has detected five attacks of one type. The analyst looks at the evidence and says: “These three are attacks of this type, the fourth one is benign, and the fifth one is an altogether new type of attack.” He then labels the latter, that feedback is again inserted in the system and the models update themselves. As time goes by, they learn to discriminate between a great many types of attacks and benign events.

“The analyst is always training the system because there are always newer attacks,” says Veeramachaneni. “At some point the system trains itself so well and becomes so very accurate that the analyst can get a bit more comfortable.”

Additional help comes from the fact that once this solution is deployed by many companies, the models that are learned by the system at each of these can be aggregated and shared, creating a network effect.

“The more customers you have the more training you get, the more training you get, the more accurate you become, and the system starts detecting newer and newer attacks more speedily and more accurately,” he pointed out, and made sure to note that no actual data about the customer or belonging to the customer is shared.

The training of the system doesn’t have to begin on the first day of deployment. Most companies keep the needed logs for weeks if not months, and they can be fed into the models, as well as compared with the knowledge of past attacks in that period of time. This allows the system to start working initially and start identifying specific attacks from the very first day, and the training can continue from there.

“This is one approach we use with a lot of customers,” says Veeramachaneni. “The other one is to install the software, extract the data and feed it into the system on the first day in real-time, and on the second day the system knows what’s the ‘normal’ situation and what’s abnormal behavior, and you can start screening through these events.”

Real-time alerts of ongoing attacks allow the analyst to implement incident response if needed. Sometimes that means just picking up the phone and contacting an employee to see whether he or she is doing the thing that triggered the alert, and shutting the machine down if they aren’t.

In large scale real-time environments, e.g. e-commerce, the reaction has to be even faster, and automated workflows have to be put in place so that they can be started immediately after the attack is detected in order to thwart it.

The platform is currently geared towards breach and fraud detection.

A tried and tested solution

It’s interesting to note that, unlike most other companies, PatternEx had to enter the market even before the product was finished, as they needed the data provided by customers to perfect it.

They have been working on the platform for the last two years, and have deployed it at several Fortune 500 companies.

It proved to be extremely effective – it has 10 times better detection rates and 5 times fewer false positives than other user behavior analytics solutions.

“The most frustrating thing in infosec is that the data to detect malicious behavior often already exists in enterprise infrastructures today,” notes Veeramachaneni. “The human analysts can detect it, but analysts are difficult to hire and are not scalable.”

He believes their technology is the right solution for the problem.