A while back I had a chat with Ed Skoudis, who with Tom Liston created Spycar. The log of the chat is available here. In this conversation Ed agreed that Spycar is not a comprehensive test for anti-spyware products, and that it was not designed to be such a test.
Randy says:
OK, so to be clear. Spycar is not about revealing the quality of a solution. Spycar is a suite of tools that a person can use to reveal the behavioral characteristics of the product they are testing. Once they have this knowledge then they can select single or overlapping security products based upon the capabilities of the products and their needs.
Ed says:
Well, said, Randy. I agree.
This is because, as Ed tells "Spycar was created to help people test whether they have rudimentary (and in the newer version more complex) behavior based defenses."
This is not the same as testing to see how good your anti-spyware program is. I asked Ed about people who assume that a product must detect Spycar to be good.
Randy says:
So, if one of our customers says "you don't detect Spycar therefore you are not a good product" what would you tell them?
Ed says:
I would tell them that the tool they have doesn't do behavior-based detection. If they want that kind of detection (with its strengths and weaknesses ), they will have to augment what they have. Or, they could opt to stay with a sig and heuristic approach
So, how good is Spycar at evaluating anti-spyware software?
Ed further elaborates that ""Spycar is just testing for one aspect of anti-malware defenses. There are a multitude of other things to consider too."
Ed didn't say it, but I'll say that if you want to evaluate a behavior blocking program Spycar is a really cool tool to help you, but you have to know what you are doing. Behavior blocking can be very useful for a skilled user, but for an average user it can be quite confusing. Highly skilled users are who Spycar is designed for.
Randy says:
So the people you aim spycar at are expected to know that they are testing a specific type of condition for a specific defensive approach and would hopefully know better than to try to apply the results of a behavior detector to a signature based solution.
Ed says:
Yes, yes, yes! And that's why the spycar website is so darn wordy. We try to be very clear about what spycar is doing and why.
As Mary Landesman points out at http://antivirus.about.com/b/a/257711.htm "behavior blocking is best kept in the hands of an experienced user who can understand and respond appropriately to the types of alerts it delivers. And behavior blocking can't clean up an infection that's already present, which is where signature-based scanners have a distinct upper hand."
At ESET we believe that NOD32 should detect and block bad programs, not detect everything and then make you figure out if the toolbar you want to install is good or bad –good toolbars and spyware ridden ones will trigger behavior blockers. If you are an advanced user who wants to add another level of security to your system you might consider Ed's approach:
Ed says:
I'm a belt and suspenders guy myself (referring to infosec philosophies, not my wardrobe!). I do rather like to see a combo approach, including signature, heuristics, and behavior detection
I did find that Spycar performs another useful test. If a reviewer tries to tell you how good an anti-spyware solution is by testing it with Spycar alone, Spycar has revealed a person who does not know how to use Spycar and is not qualified to review anti-spyware products!
Randy Abrams
Director of Technical Education
ESET LLC
