Early in 2017, Kevin Townsend invited me (among others) to comment on vendor hype (in general, not with reference to any particular vendor). You can find the article he eventually wrote for Security Week – including some comments from me – here: Fighting Cyber Security FUD and Hype. He led with a couple of quotes from Ian Levy, Technical Director of the UK’s National Cyber Security Center (NCSC), accusing the security industry of (in Kevin's words) 'overhyping the cybersecurity threat to sell under-achieving products'. I certainly wouldn't discourage you from reading the article, in which a wide range of people at the sharp end of vendor marketing express their opinions.
However, since he didn't use all my commentary – it's a bad habit of mine, providing journalists with more wordage than they actually need! – and since it's a topic to which I've devoted quite a bit of thought (and wordage) over the years, I thought I'd expand here on my original commentary. Since this is a vendor blog, I'll forgive you if you are skeptical of my ability to be impartial. However, one reason I've worked with ESET for so long is that the company is scrupulous about allowing researchers to stay hands-off from marketing, so I don't normally talk about specific products (ESET's or anyone else's). That doesn't mean that I don't understand the need for marketing, though.
A better mousetrap
I'm going to quote myself (actually, myself and former colleague Randy Abrams), from a paper written for an AVAR conference a few years ago on – among other things – ethical and professional conduct in security writing):
What are the cornerstones of success in the anti-malware industry? Sound technology, of course. Sound marketing, too: a good product is of little use if no-one knows enough to go out and buy it. There are other factors too: sound after-sales and support service, for instance, influences the deployment, maintenance and efficacy of a product or service.
They say that if you invent a better mousetrap, Amazon users will beat a path to your door. Or something like that. The trouble is, that the security world is packed with products that have their own merits – well, there are some that have no merit at all, but I'm not about to name names – but the average consumer isn't in a position to evaluate those comparative merits and demerits. Nor, to be honest, are some reviewers and testers, but some have made a highly commendable effort to raise the standard of testing (dramatically, in some cases). I do believe that AMTSO deserve credit for facilitating that rise in standards. (I'll come back to testing shortly.)
Still, there's acceptable, ethical marketing, and then there's hype.
Kevin Townsend suggested (and I don't disagree) that there are two main varieties of vendor hype:
- We are the best thing since sliced bread and will solve all of your problems
- Our competitors are totally useless
The 100% mousetrap
If you're familiar with my writing, you won't be surprised to hear that I regard marketing based on the promise of 100% protection in perpetuity to be not only ethically suspect (and usually technically indefensible) but actively dangerous. It's an attractive idea, the hope that once you buy a given product, all your security problems will be over. But it's not the world we live in.
Some marketing – well, a lot of marketing – in the security sector takes the position that all you have to do is buy X, and then you never have to take any of the responsibility for your own security. Such claims are not necessarily based on deliberate deception. Often, they are based on lack of technical knowledge on the part of people with a marketing agenda. And often they are not: in combination with the 'slag off the competition' approach so beloved of startups in the security sector, half-truths and misconceptions are often initiated and spread by people who are clearly knowledgeable enough to know better.
Emotive language
To address another point Kevin raised, there's no doubt that emotive language relating to warfare and/or epidemiology has long been a staple of security-related marketing (and journalism!). I can't say I like it – much of my career in security has been devoted to damping down the fires of hyperbole and advocating less drama and more precision – but I'm most concerned by the misuse of language in ways that are actually deceptive rather than just everyday sloppiness.
It annoys me (quite disproportionately) when a system is described as 'infected' when 'compromised [by some form of trojan]' would be more accurate, or when people use 'virus' as a synonym for 'malware'. That may seem just pedantry – well, it is pedantry – but it also irritates me in that use of the words 'virus' and 'infect' can be used to set up the reader to expect the cyber-equivalent of Ebola, where much malware is closer to the common cold. I also regard it as frankly deceptive when people evade the distinction between 'successful attacks' and 'attempted attacks' because saying 'ten million systems were attacked' is more dramatic and makes a marketing point more effectively than adding that '0.001% of attacks were actually successful' ...
This doesn't mean that I underestimate the possible impact that some attacks, such as ransomware or phishing fraud, can have on individuals or organizations. After the WannaCryptor (WannaCry) attacks, no-one should be in any doubt as to the potential ransomware has for harm. And yes, I do think a good, properly configured security product is a sound investment when it comes to protecting your data from ransomware, but it's equally important to maintain frequent offline backups so that it's less likely to be disastrous if malware manages to get past security software.
Testing and evaluation
But let's talk about testing, since it's a subject much on my mind at this moment, because of a conference paper I've been working on for several weeks. (More about that another time …)
While test results are among the tools that marketing departments use for marketing leverage, they're not the whole toolbox. However, test results do provide leverage, and it's probable that more people get their test result information filtered through vendor press releases (directly or indirectly) than they do from an independent tester.
Probably any security researcher has seen instances where selective quoting and paraphrasing of test report content has made it sound far more favorable to a given product than is supported by the test statistics. Where a range of products of broadly similar abilities are tested, testers already face the issue of basing a report on the magnification of minor differences. On the other hand, we're seeing an ongoing move away from large sample sets to far smaller sets, which is – provided the samples are properly selected, validated and weighted – not a bad thing. (However, we're also seeing a resurgence of the bad old idea that testing is so easy that anyone can do it, with guidance and off-the-peg samples from vendors or their associates…)
Try before you buy
An idea that comes up several times in Kevin's article explores the necessity of 'trying before you buy'. Inevitably, since much of the market consists of products whose detection performance may not differ dramatically from that of their competitors, on average, so while detection test results are an important component of marketing strategies, other performance issues may be as important, or even more important.
The trouble with 'try before you buy' is that you only get to experience the product in a limited range of situations, and the less you know about the threat and counterthreat technology, the less you'll get out of a trial period. While it's perfectly reasonable to try to benefit from the experience from others in forums and through personal discussion, your indirect experience is filtered through a third party's experience and understanding, and that party's needs may be quite different to yours.
There is a point to evaluation/trial versions, of course. For instance, an independent tester may, perfectly reasonably, award stars or a best-of-breed classification to a product that performs outstandingly in its tests, but you might then find that it's totally incompatible with one of your in-house systems or a third-party product that is vital to your business processes. I don't think this is a common occurrence, but it could happen, because no tester can guarantee that a product they test will work flawlessly in every environment.
When it comes to detection testing, however, the accuracy of your evaluation is dependent on (among other things) the validity of your test set. The real value of the test is determined (among other things) by such complex and interrelated factors as the selection and validation of samples, weighting/scoring, appropriate configuration and so on. Keeping those particular juggling clubs in the air requires skill, knowledge, and resources. If you buy into the model of 'testing is so easy, you can do it yourself', pushed so hard by certain security companies over the years, you're less in control of the methodology than they are.
If you simply rely uncritically on a sample set from unknown sources (especially if those sources may be a vendor whose product is under test), you're letting the supplier of the samples control the test. And if you know that the source of samples for testing is a vendor whose product you're evaluating? Well, I just don't understand why you would do that.
- The vendor can't give you samples it doesn't have.
- The vendor is unlikely to give you samples its products don't detect.
- The vendor will be tempted to give you samples it knows other vendors won't detect.
In fact, some professional testers have been known to solicit samples from vendors and encourage those vendors to offer samples that other vendors probably don't have. And I can see some point to that, though it's not a point that works to the advantage of the tester's audience. It can happen with a reasonably well-weighted and selected set of samples that a selection of reasonably competent products will score around the same number of detections. (Can and does happen.) So a tester (or reviewer – not always the same person) may feel compelled to exaggerate small differences in detection rates in order to get a more dramatic detection ranking for the 'editor's choice'.
But why don't other companies have the same samples? Well, here are some possible (hypothetical) reasons:
- Unlike most mainstream vendors, the vendor doesn't share samples with other companies. Established AV companies have tended to prioritize the good of the community rather than competitive advantage, by sharing samples. Maybe not always all samples, and maybe not always immediately.
- The sample wasn't around long enough to pose a threat in the real world.
- The sample was generated specifically to offer to testers and potential customers in the hope and expectation that only the donor's product would detect it. There was a recent case where a high percentage of samples were found not to contain malicious code. While this is claimed to have been a side-effect of a process intended to generate 'new' samples of pre-existing malware rather than deliberate deception, it demonstrates how misleading the results can be when samples used for testing and evaluation are not properly validated before use.
