RE: ICSA [WAS: Re: Intrusion Prevention]

From: <smarkle(at)icsalabs.com>
Date: Mon Dec 30 2002 - 16:29:39 EST

>On 12/29/02 Greg Shipley wrote:
>Over the past six years Neohapsis Labs has been testing products in the

All - I have remained silent on this list for years. I am interested in helping mature an Industry. That is what ICSA Labs does and IDS has been one of my responsibilities since early 1999. After cutting through the stinging criticism and saber rattling, I have chosen to respond only to the paragraph above. Any vendor that knows the ICSA Labs testing methodology knows that for over ten years we have perfected pass/fail certification testing with evolving test methodology and criteria. We did this when everyone else argued that it was the wrong approach. This is the standard, and it is in fact the ICSA Labs approach that has been mirrored by other test labs.

>On 1/18/01 Greg Shipley wrote:
>[edit] Don't get me wrong, I think there is a huge need for 3rd-party
involvement, and dare I say it, "certification."
>IMHO, there are some fronts to this that are REALLY important on. For
example, I've heard that the ICSA team is working on >IPSEC *compliance* and interoperability testing. Ok, that's huge, as anyone who has worked with multi-vendor VPN
>deployments knows that the VPN space is a mess on that front.

>The problem is, I question whether or not people are being mislead, and how
much good some of these certifications (like the >firewall one) really do. Ultimately, does this type of "branding" help provide for a false sense of security? [end]

The problem, clearly stated by Greg, is whether people are being misled. The answer is emphatically NO. The ICSA Labs NIDS test is geared toward three different network types. ICSA Labs has never mirrored the 1999 Neohapsis test, nor will we - it was flawed. We have built a real network to test NIDS. We have always used working exploits that are targeting a victim machine that is vulnerable to each specific attack. We have also included the first false positive test...ever. You may be a bit beyond, however, your F-1 vs. Garbage Truck analogy reminds me of the tortoise and the hare. You may have gone farther in terms of performance but you yourself have admitted errors caused by the pace. This is where people have been misled. They read a magazine article that states vendor x has the best NIDS. End-users do not need to know who has the best product in a snap-shot-in-time lab test, they need to know the best product for their real live environment. That is where ICSA Labs NIDS testing and certification has excelled and IMNSHO will never be caught.

Greg - I sincerely ask you to contact me off-line and discuss a possible visit to the ICSA labs. It is evident by your post that you do not have a complete knowledge of what we do. This thread has also included reference to the ICSA Labs Firewall program. I have asked one of our most vocal critics in the past to give you his opinion on the current state of the ICSA Labs Firewall program. Look for a post in the near future on that subject.

Scott Markle
IDS Program Manager
ICSA Labs

---

This message is intended only for the use of the intended recipient and may contain information that is PRIVILEGED and/or CONFIDENTIAL. If you are not the intended recipient, you are hereby notified that any use, dissemination, disclosure or copying of this communication is strictly prohibited. If you have received this communication in error, please destroy all copies of this message and its attachments and notify us immediately.

---

Received on Mon Dec 30 18:01:25 2002