RE: how broken are antivirus products?

The anti-virus industry -does- keep its samples to itself. Industry-standard testing is very stringent and the methodology -is- somewhat under-documented. And yes, it -is- very difficult for an independent researcher to get into the charmed circle. I think there probably is an element of defensive obstructionism and preserving the mystique sometimes. But there are also good reasons for doubting the ability of Joe Sixpack to test competently, even if Joe has access to a sufficiency of samples. To talk of "proven live samples", in itself, suggests that that you may be underestimating either the stringency of competent testing or the importance of adhering even to obvious first principles like using replicated samples rather than samples from another source (however trustworthy the source).

AV testing is rather different to the full/partial/nondisclosure  debate. Anyone can disclose a bug (though many bug reports do turn out to be erroneous, and there are far too many glory hunters in the bug-hunting business for comfort), and how and when you disclose it is, in the end, a matter of personal
standards and ethical viewpoint. Anyone can test an anti-virus product, too, but they cannot do detection testing competently unless they have an unusual degree of expertise and an adequate sample set. The AV industry does not usually give out samples to people who haven't earned trust (yes, I know there are glaring exceptions!). That means trust in their competence as well as their moral standards. Some may suspect that sometimes this works to the industry's advantage in making it harder to DIY, but it's better than applying no standards. Some might feel that people in the security industry should be excepted from such rigid criteria. I'm afraid I don't: much harm has been done by the tendency of some security people to assume that expertise in one area automatically makes them experts in anti-virus.

-- 
David Harley
http://www.sherpasoft.org.uk/-----Original Message-----
From: Bruce Ediger [mailto:eballen1@qwest.net]
Sent: 19 November 2002 20:57
To: focus-virus@securityfocus.com
Subject: RE: how broken are antivirus products?




On Mon, 18 Nov 2002, Schmehl, Paul L wrote:



> This is hashed over repeatedly amongst AV professionals, and the only



Might I gently suggest that what's written above indicates a situation
exists rather like what spawned the open source and "full disclosure"
movements?

To an outsider like myself (and maybe even Jose Nazario?) it could appear
that an anointed few control the "proven live samples" and laboratories,
and the methods of testing.  One could argue that Joe Sixpacks like
myself can't double-check the official testing without getting "roundly
criticized".

In the past, system vendors could ignore people who reported bugs or
wanted usability or other changes.  Years of this basically spawned the
open source movement.  In the past, system vendors ignored or hushed up
reports of security problems.  Years of this spawned the full disclosure
movement.  At least, that's how I understand it.

My humble, and perhaps redundant, advice to people finding themselves in
some kind of position of authority is to act carefully when defending
the privileges of the authority.

I would warn that roundly criticizing even the most mild attempts
to double-check official results could lead to loss of faith in the
authority, or even open revolt against it.
This e-mail is confidential and privileged.  If you are not the intended
recipient please accept our apologies;  please do not disclose, copy or
distribute information in this e-mail or take any action in reliance on its
contents:  to do so is strictly prohibited and may be unlawful.  Please
inform us that this message has gone astray before deleting it.  Thank you
for your co-operation.