It has happened before, it just happened again and it will happen in the future. It is inevitable! Some company that needs to get some press coverage or public visibility will release yet another statement on how worthless Anti-Virus is, based on its own dysfunctional test.
For this “test”, they used the VirusTotal service. VirusTotal is a great service, but not suitable for testing Anti-Virus performance in this way. For a start, VirusTotal only uses the command-line scanner versions of the products that support VirusTotal. Even if a new threat is not detected by the command-line scanner, it does not mean that the threat would not be stopped by another Security Suite module. Nowadays there is no mainstream vendor anymore that only offers its customers an on demand command-line scanner: modern solutions are Security Suites with many additional components such as Anti-Phishing, Anti-Spam, IDS, (H)IPS, etc., all kinds of security technologies NOT reflected in the VirusTotal result.
Furthermore, some of the solutions included in VirusTotal are configured according to the parameters requested by the vendor, with a more aggressive level of heuristic detection than the official end-user default configuration would offer. This may result in a detection whereas the official end-user configuration would never detect the threat, so VirusTotal can be really, really misleading if used to compare performance between products..
Having the different products run with different parameters, resulting in different levels of heuristic paranoia is common practice during anti-malware tests, but then these differing parameters are the result of using so-called out-of-the-box settings: in other words, as the vendors have prepared them for what they consider to be the best end-user experience. It’s not unreasonable to test this way as long as its clear to the audience that what is being tested is not absolute detection, but configuration philosophy. A better way of testing overall detection is to modify individual products so that the level of heuristic aggression is equivalent for each product. However, tweaking some products and not for others will make the result equivalent to comparing apples and oranges. VirusTotal is unsuitable for comparing products because essentially, this is what it does. But that, of course, is fine because comparing products is not what VirusTotal is for. The importance of comparing like with like in the testing context has been described in 2008 already in the “Guidelines to the Fundamental Principles of Testing” document of the Anti?Malware Testing Standards Organization (AMTSO).
Similarly it is not correct to compare consumer versions against corporate versions. Imperva actually suggests for companies to use the free Avast! solution. Whereas there is nothing wrong with the Avast! product by itself, the free version is meant for consumers and not for businesses. These two branches of the security tree should not be directly compared to each other. In a corporate environment anti-virus solutions work with a central management system for enterprise wide reporting and configuration, an integration which is absent in consumer products. In corporate environments, thresholds for heuristics may be set higher than in the free consumer-targeted versions. In other words, heuristic detection in a corporate environment is likely to be less aggressive, to lessen the risk of false positives. A false positive in a corporate environment can make the entire network inoperable and cause significantly more damage: for a business, even transient inconvenience can be expensive. A lot of the time, the thresholds for heuristics are set much lower for free antivirus products. A false positive in a free consumer product is generally seen as not so severe in impact, and feedback from an affected system may provide valuable data for the corporate version.
David Harley has published a paper that analyses in detail the problems with one test that displayed a certain inconsistency between the test objective described and the methodology used.
Julio Canto of VirusTotal/Hispasec Sistema together with ESET’s David Harley wrote a paper on this topic called “Man, Myth, Malware and Multi-Scanning”. In the paper they documented: “VirusTotal was not designed as a tool to perform AV comparative analyses, but to check suspicious samples with multiple engines, and to help AV labs by forwarding them the malware they failed to detect."
"How not to use VT: VirusTotal uses a group of very heterogeneous engines. AV products may implement roughly equivalent functionality in enormously different ways, and VT doesn’t exercise all the layers of functionality that may be present in a modern security product. VirusTotal uses command-line versions: that also affects execution context, which may mean that a product fails to detect something it would detect in a more realistic context. It uses the parameters that AV vendors indicate: if you think of this as a (pseudo)test, then consider that you’re testing vendor philosophy in terms of default configurations, not objective performance. Some products are targeted for the gateway: gateway products are likely to be configured according to very different presumptions to those that govern desktop product configuration."
"Conclusion
VirusTotal is self-described as a TOOL, not a SOLUTION: it’s a highly collaborative enterprise, allowing the industry and users to help each other. As with any other tool (especially other public multi-scanner sites), it’s better suited to some contexts than others. It can be used for useful research or can be misused for purposes for which it was never intended, and the reader must have a minimum of knowledge and understanding to interpret the results correctly. With tools that are less impartial in origin, and/or less comprehensively documented, the risk of misunderstanding and misuse is even greater."
These are just some examples illustrating why using VirusTotal for antivirus testing is a bad idea. Our colleagues on the Prevx team also made an entry in their blog discussing the matter.
Why is anti-virus not a waste of money? The service, the support, the timely updates, the research into (future) threats, etc. There is NO such thing as a free anti-virus. Because of the work that is put into those ‘free’ products, the developer needs to get some return on his investment (ROI). Most often this is done by installing “complementary” toolbars, utilities containing adware-like functionality, and so forth, where the client is monitored and served with information “you need.” These add-on programs, subsidize the cost of the “free” anti-virus potentially at the expense of your privacy. Additionally, once an anti-virus company includes a toolbar with their “free” offering, they may be pressured by the toolbar vendor to exclude detection of other products bundling the toolbar vendor’s software, which may be more intrusive in nature and cross the line from grey into black.
The following two white papers provide additional information about such questionable software:
There is an unfortunate tendency to believe, especially amongst consumers, that anti-virus software serves as a magic forcefield which protects their computer from goblins and other things which go bump in the night. While this sometimes is the case, anti-virus software is more often like automobile insurance: You may not like purchasing it, but when an “accident” occurs, you’ll be glad that you bought the plan with the best support.
