[Since the owner of the blog described below interpreted this blog as a personal attack and marketing BS, I've removed information that identifies his blog. Which is a pity, because his blogs on the topic actually include useful information. I'm not withdrawing the whole blog, because it isn't marketing and it isn't about our product: it's about an important principle, which is that VirusTotal is not (and is not meant to be) a comparative detection performance tool.]

I just came across a note on a specialist list pointing to low vendor detection rate on a Waledac file. Happily, the VirusTotal report quoted did indicate that we're one of the vendors who do recognize it. However, there was also a pointer to another blog that while it's well worth reading, includes some comments that need a response.

More specifically, it includes a dump of and link to a VirusTotal report that indicates that only four out forty products detected a specific version of Waledac (apparently we weren't one of them). The author points to this as proof that "4 of 40 anti-virus products know to block this program!" Sorry, but that isn't so. 

First of all, as I've pointed out many times before, VirusTotal is not designed to measure the ability of specific products to detect specific malware. In other words "not detected by this product according to a VT report" is not the same as "this product does not know how to block this program!"

  • A scanner may be perfectly able to detect a sample in "real life" using a form of dynamic analysis that isn't triggered by VT's command-line scanning.
  • A scanner may not be able to detect a sample using the configuration in the VT setup, yet be able to detect it using a different (for example, non-default) configuration.
  • A VT report is a snapshot of a moment in time. Even if the first two considerations don't apply (and you can't know whether they do without considerable inside knowledge), it tells you nothing about the result you might get five minutes later. The report on the page in question describes detection at 2009.07.03 14:38:46 (UTC): searching for the same hash value got me a report generated at 2009.07.03 20:16:56 (UTC) indicating that ten products detected it (including ours). Clearly, those products might have been updated at any time during the intermediate few hours. 

(A hash is a sort of shorthand way of representing the physical qualities of the sample as a checksum. In theory, every file has a unique hash value, though in real life this isn't quite true.)

As I've also pointed out before, it isn't the fault of Hispasec, who supply the free VirusTotal service, and have themselves pointed out that it simply isn't designed to evaluate the detection performance of individual products. (I'm almost as tired of this misconception as they are, and I will be looking at this topic in detail very soon.)

What this probably does indicate, though (at any rate up to a point) is that, as I remarked in my previous blog, the bad guys are watching us as carefully as we're watching them, and detection will fluctuate as they try to tweak their code to evade detection by the products that are best able to detect it.

My main issue with the blog post, however, is that he claims that this is a demonstration of how AV works. He apparently believes that we only start to detect malware when enough people complain about it to force us to add detection. Actually, we use heuristic analysis and automated processes rather than wait for people to send us malware to analyse and insist that we write a signature for it. Even companies that are more reliant on signature detection also use a variety of techniques to speed up sample processing, and that's one of the main reasons why some are so anxious to get that process into “the cloud".

He goes on to imply that "the virus" wasn't known by most people to be viral because it hadn't been spammed out yet. Actually, Pierre-Marc had already written about this the day before his blog, so that plainly isn't so. Naturally, there's been a lot of information shared by ourselves and other researchers in order to mitigate the effects of the attack. Believe it or not, there's a lot of cooperation in this industry, and we don't aim to allow competitive advantage (or vested interests)  to take precedence over the welfare of the community. Well, most of us don't. :)

Finally he suggests that there would be few people in AV labs to write definitions on a Saturday evening after the new wave of attacks hit. Actually, this is not a 5/7, 9-5 industry. Of course a lot of people work office hours, but the people who maintain the software can't do that, especially when they know that an attack is on the way. Though not many of them work 24 hours every day.

David Harley
Director of Malware Intelligence (who doesn't write signatures, he's happy to say, but is still working on a Sunday. Again...)