So the people at untangle.com decide to “test” anti-virus product in an effort to prove their dedication to open source zealousness. I’m not against open source, but if you want to promote it then be honest about it.
First untangle grabs a few samples of “viruses” that they know CLAM AV will detect. Unfortunately 1 out of 3 samples they use in the 18 sample test is obviously not a virus at all.
The first indicator of incompetence is the size of the test set. It is fine to use a set of 18 samples for a specific reason, but when you draw the conclusion from that set that Clam “is excellent”, you obviously don’t understand how to test or interpret results. They even say their test is not a zero-day test, not a functionality test and not a coverage test and without all of this data they conclude that their pet virus scanner is excellent.
This second indicator was the inclusion of EICAR test files. In their report (http://blog.untangle.com/?p=96), untangle says “The first set was a basic test set (from eicar.org) that is a universal virus test.” This is completely incorrect. To call the EICAR test file a universal virus test is to boast of gross ignorance. The EICAR test set is an optional detection used to test if the scanner is functioning. The best scanner in the world could choose not to detect EICAR test files while the worst one could only detect them. Which is the better scanner? EICAR doesn’t tell you anything you can use to conclude that something is excellent in detection.
Finally, these people who cannot competently test software, and who run blatantly biased and incompetent tests are putting viruses up on their web site for anyone to download. I certainly hope this violates the terms of service of their provider.
In summary, Untangled.com appears to be a simple virus distribution web site, with irresponsible folks who know nothing of testing and even less of how to draw accurate conclusions from the results of a test, but are shills for open source.
It is curious that they won’t divulge the name of the “certification” body that wouldn’t test Clam for them. My guess is that they were asking a company that charges $15,000 per test to do it for free, but these open source zealots are closed lipped it seems :)
Randy Abrams
Director of Technical Education
