Before joining ESET, I worked for Microsoft for over 12 years. Much of that time it was my job to make sure that Microsoft did not release any infected software. Properly selecting anti-virus software was essential. Proper testing of anti-virus software is time consuming, very tedious, requires significant resources, and takes some skill and knowledge. I did not have the time to do the testing I needed so it was important for me to know what tests were good and what tests to ignore.
There are some critical elements that make a reliable test. First, are the files being tested relevant. It really doesn't matter if a product detects "simulated" viruses - that isn't what you need defense for. Intelligent testers don't use simulated viruses and never have. Are the samples being tested against in the wild or did some fool write them to test scanners? Anyone with a little programming knowledge can write a virus that isn't detected by most anti-virus products and any scanner can be defeated if it is targeted. Writing viruses to test scanners is generally the hallmark of an incompetent tester. Next, you need to know if the tester is really testing viruses or just says they do. If the tester does not replicate and test all samples to ensure that they are what they should be, the tester doesn't really have a clue what is being tested and the results are generally unreliable. If the tester says they select samples because an anti-virus scanner told them it was a virus then the tester is willing to accept false positives and rate products poorly if they actually perform well. There is no replacement for a tester knowing what they are testing.
There are several other factors in performing a test properly, however problems with samples and are consistently found in the really bad tests. Usually the problems are samples that do not work, too small a sample set for statistical significance, and use of simulated threats. In some cases the testers continue on to reach conclusions that their own tests do not support.
For useful test information I recommend www.virusbtn.com, www.av-test.or, www.av-comparatives.org (in-the-wild tests), www.icsalabs.org, and http://www.westcoastlabs.org.
Randy Abrams
Director of Technical Education
ESET LLC
