Latest SELabs Testing Report Omits Next-Gen Endpoint Products

UK product testing laboratory SELabs recently published three test reports on endpoint security products. SecurityWeek looked particularly at the report, Enterprise Endpoint Protection April – June 2016, which included a high proportion of ransomware within its test samples. Kaspersky Lab handled these samples best, scoring 100% in the ‘total accuracy’ result. Microsoft System Center Endpoint Protection fared worst, scoring just 77%.

SELabs is a Croydon-based UK product testing organization founded and run by Simon Edwards. Edwards was previously the technical director of Dennis Technology Labs and is a former chairman of the board of the Anti-Malware Testing Standards Organization (AMTSO). He is steeped in the theory and practice of testing anti-malware products — and is well-regarded in the industry.

Two points are immediately apparent from the report: only six products are included; and there are no next-generation products at all. Given some of the recent antipathy between next-gen and traditional AV vendors, this could be seen a surprising. SELabs director Simon Edwards explained that modern testing is very labor-intensive. For this reason there is a limit to the number of products that can be accommodated. “We aim to include the most popular products, as we determine them to be,” he told SecurityWeek. “This is based on market share and our own perception of which products businesses seem to care most about.”

This places a slight question mark over the objective value of the results. A CISO looking for the best solution for an endpoint problem will get an objective comparison of a subjective list of contenders. While the reason for a small number of participants is valid, the question remains whether this small number invalidates the results.

Opinions on this vary. SecurityWeek spoke to several vendors who were not included. Luis Corrons, technical director at PandaLabs was uncertain, suggesting that the value of limited participants limits the value of the results to the customers using or considering those products. “Obviously, the more the better,” he told SecurityWeek.

Edwards did, however, get support from ESET senior research fellow David Harley, and F-Secure security advisor Sean Sullivan. Harley said, “One of the strengths of an SELabs test is that it doesn’t try to throw a million samples at every conceivable security product: SE is very choosy about the selection of samples… and seems to go out of its way to avoid comparing apples and oranges in the products it tests against.” His point is that SELabs’ selection of market leading enterprise vendors for enterprise tests is reasonable and effective.

Sullivan commented, “The participants that were tested have plenty of customers, and it’s always useful to have more information produced by serious professional testers. Furthermore, the results can be cross-referenced with other testing companies to validate their results.”

Responses from the excluded next-generation companies was blunt. When asked if it was realistic to call these test results ‘Enterprise Endpoint Protection’ Chad Skipper, VP of Product Testing and Certification at Cylance, simply replied, “No, it’s not realistic.”

SentinelOne’s CMO Scott Gainey’s position is more complex — but basically, no tests of whatever number of participants is realistic unless the full mix of threats, including current unknowns, file-less threats, scripts and javascript threats are all included.

Although the reason for a limited number of participants in this test process is understandable, it doesn’t specifically explain the lack of any next-gen vendors. Edwards told us, “Historically we’ve found so-called ‘next-gen’ vendors to be unwilling to submit to third-party testing. There can be a variety of reasons, some of which are more reasonable than others. We do work privately with some of the best-known of these newer brands and there will be some public results coming soon. 

“Since the beginning of the year I’ve noticed a much greater interest in testing coming from these companies. That said, one major player steadfastly refuses to engage with any tester that I am aware of, so it’s not all progress.” 

This is probably fair. Cylance said, “Yes, we are warming to the idea.” SentinelOne’s Gainey commented that he had been contacted by a different testing company a couple of months ago. He provided a SentinelOne account and offered time with an engineer — but that nothing came of it. Edwards told SecurityWeek that if SELabs gets a similar opportunity, “I guarantee we’ll test it.” The will is growing on both sides; but we’re not quite there yet.

Gainey added that SentinelOne had been the first next-gen vendor to submit itself to AV Test for both Windows and Mac OSX last year. “We’re currently working with NSS on an extensive ‘advanced endpoint protection’ test,” he said. “NSS are trying to do real world testing that utilize not just the usual samples — they’re trying to pull down new and unknown malware and trying to be real-world by using file-less attacks, scripts and javascript.”

In reality, SELabs isn’t quite there yet, but is working towards it.”Twenty-five percent of the test included what we call ‘targeted attacks’, which include creating infected Word documents, PDFs and other similar ‘hacking’ attacks,” said Edwards. “These are similar tactics to those used by ‘APTs’.

“We plan to include harder threats, that use AV evasion techniques, at some stage but for now some products are so bad at handling these, particularly in the consumer space, that we don’t really need to raise the bar much to see a differentiation in products. We will move this part of the test upwards over time, though.”

Gainey responded, “We’re going to lean more towards testers that use a combination of both known and unknown threats as being more real world — otherwise the results create a false sense of security. All of the tested products will be at or around 100%, which we know to be wrong. That will just create a false sense of security for users.” He added, “I’d like to be tested against traditional AV vendors, because at the end of the day we’re here to replace them. My goal is to show that our approach to detection is going to be superior to what they can provide. If the tests just pull down known samples it’s not going to show much difference between any of the products.”

Interestingly, SentinelOne is willing to back its product financially. In late July, the company announced a guarantee against the failure of its product against ransomware. This will pay customers up to $1,000 per endpoint to a maximum of $1 million per company in the event of SentinelOne failing to stop the attack. “I’d like to see other vendors, not just endpoint vendors but network security, and web security vendors take a similar approach,” Gainey told SecurityWeek. “We’ve all lived through these 100% detection claims that we know are bogus. If you really are truly that good then you should have no qualms whatsoever offering a guarantee against your product.”

In the meantime we are limited to the results of third-party testing. SELabs has a high technical reputation, and there is no criticism of its process. Nevertheless, the lack of any next-gen vendors among the participants suggests that calling the report ‘Enterprise Endpoint Protection’ is probably unrealistic. Furthermore, the small number of participants means that all that can really be said is that ‘this product performed better than that product in detecting these threats’. It would be unrealistic to suggest that any one product is better than any other product that was tested. 

*Correction: Error in material provided to SecurityWeek by SentinelOne: it was not SELabs that ignored the offer of a SentinelOne account.

view counter

Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines.

Previous Columns by Kevin Townsend:

Tags: