From time to time we are asked to provide samples or malicious URLs to individuals and groups who are not in the full-time testing business. We do, of course, share such material with other actors in the security industry who are within our web of trust, but are not usually able to honor requests from other groups for the following reasons.
Bias and Statistical significance
Firstly, there's nothing to stop us, or any of our competitors, sending a reviewer essentially unimportant links or samples, knowing that it's unlikely that the competition will have exactly the same malware derived from exactly the same sources. This really proves nothing about the detection capabilities of our product or the competition: it simply enables the tester to gather samples that may or may not be valid.
Secondly, the small sample sets with which occasional tests are normally carried out are statistically uninformative. We see some tens of thousands of such active links daily (usually because we have detected some malware being downloaded from them) and process over 100,000 unique samples daily: clearly, turnover on this scale is impractical for an occasional tester. It’s also quite likely that out of any set of links that we forward in good faith, a proportion will be ‘dead’ by the time a reviewer comes to test them.
Ethical considerations
It is, of course, hard for us as a company not to offer help with the acquisition of samples, as non-participation is likely to be interpreted negatively by both the reviewing organization and by the reader. Equally clearly, it is nice to have any (positive) publicity that can be generated by the review. However we generally feel that we are unable to help, not because we believe we would do particularly badly in a specific test, and not only because we often believe the tests in question to be seriously flawed, but also because of the ethical objections to providing malicious samples and links. Such provision is at best a ‘gray’ area, and goes against the normal practices of the industry (at least, from the more traditional anti-virus vendors).
ESET does share samples with bona-fide vendors, researchers and law enforcement agencies, but this is only done in specific circumstances under tightly-controlled conditions with vetted individuals representing respected testers and other organizations. In principle, we don't see a real difference between supplying malicious URLs and supplying actual samples of malware.
Contrary to the popular myth, this practice, which is common among anti-malware vendors, is not adhered to because vendors are somehow conspiring to prevent anyone finding out how ineffective all the products are. In reality, we have always tried to act in such a way as to prevent the possibility of our actions ever causing others to become infected or affected by malware. Were we to pass samples to an individual or group whose trustworthiness and/or competence we are unable to evaluate effectively, we would be ignoring the risk of such undesirable side-effects. Certainly, we do not object when testers commission a recognized independent testing company such as AV-Test or AV-Comparatives to perform testing on their behalf, as they have established methods of handling and dealing with the samples that would be unlikely to cause any collateral damage.
Furthermore, while we obviously have an interest in doing well in independent tests, we don't feel that contributing to tests that may be biased or misleading is good for the consumer or for the industry. Often we have no visibility into the methodology used, or consider the methodology as described to be inadequate to meet the needs of the test and test audience. For example, we don’t consider tests based on validation (or even worse, actual testing) by submitting samples to VirusTotal and similar facilities to constitute a fair test of a modern scanner’s capabilities. Similarly, the use virtual machines, though convenient for the tester, is complicated by the fact that a large percentage of modern malware is able to detect that it is running in a virtual machine (of whatever stripe), and either exits, or acts abnormally in such circumstances, therefore adding a further layer of difficulty to establishing the sample as truly active malware.
Where Next?
We are all too aware that some aspiring testers will find this position unhelpful, and may feel obliged to find other sources of malware to test with: this is unfortunate, because ready sources of malware are rarely reliable, but we can’t accept “if I don’t do it, someone else will” as an ethically sound position. We are, however, more than happy to discuss these issues further, and help anyone with a bona fide interest in testing in any way we can, subject to the ethical and moral framework in which we have to operate, as a responsible security vendor.
We also recommend that anyone with a serious interest in anti-malware testing consider joining or aligning with the aims of the Anti-Malware Testing Standards Organization (AMTSO), and at least to check out the AMTSO web site at http://www.amtso.org, where an expanding range of informational resources is becoming available.
Resources
AMTSO Dynamic Testing Practices RFC, AMTSO Fundamental Principles of Testing RFC: http://www.amtso.org/
Testing-related papers and other resources: http://www.amtso.org/
EICAR test file: http://www.eicar.org/anti_virus_test_file.htm
David Harley, Robert Slade: “Chapter 9: Product Evaluation & Testing” in “Viruses Revealed” (Osborne, 2001) by Harley, Slade & Gattiker
Andrew Lee, David Harley: “Chapter 10: Antimalware Evaluation and Testing” in “AVIEN Malware Defense Guide for the Enterprise (Syngress, 2007) ed. Harley
David Harley
Director of Malware Intelligence
