Webscarab development continues

Hi folks,

After some discussion with Ingo Struck about structuring packages, and so on, I would like to invite you to look at my proposed structure (and initial implementation) for the new WebScarab.

You will see obvious links back to the current Exodus code, as I have reused what seems worth reusing from my previous efforts.

You can find the latest archive at
http://home.intekom.co.za/rdawes/webscarab-20030728-0820.jar. There is also a link from my exodus page (http://home.intekom.co.za/rdawes/exodus.html) which will be updated as I progress. (Look for it after the BOLD section where I explain that future development will go into WebScarab :-)

This .jar should be runnable, and provides:

• the webscarab framework (user interface independent),
• the WebScarabPlugin framework (user interface independent),
• a Proxy WebScarabPlugin implementation (no SSL yet)
• the ProxyPlugin framework (user interface independent),
• two sample ProxyPlugins (ManualEdit and RevealHidden) (not quite user interface independent - see the comments in the ManualEdit proxyplugin)
• a sample Swing webscarab UI, with some panels that interact with the underlying plugins

Part of the model is also implemented:
* "Conversation" holds what we know about a particular conversation,
including the Request and Response. It will eventually hold a parsed version of the Response content (as flexibly as possible, to cope with various content-types - I would appreciate help here!)
* "URLInfo" holds what we know about a particular URL. It is a summary of
all the Conversations that have been seen (analogous to the Site view panel in Exodus, I guess). E.g. it will record the various methods seen, that generated anything other than "method not supported", list the total (content) bytes received as responses to requests for that URL, checksums of the content, etc.

Each WebScarabPlugin gets a chance to analyse a Conversation as it is seen, and can summarise whatever information it wants to into the URLInfo. The presentation layer will then need to show a column with that information in it, or save it out, or whatever. This is currently implemented as a Property class, so you can use a fairly arbitrary string to index your information.

Major things that need to be implemented still:

• HTML parser - I'm thinking of a Tokeniser approach, that could return an array of Tags, which each plugin can iterate through. The Tags will be used to extract Links (for use by the Spider), find XSS, ODBC error messages, etc
• Readers and Writers (so we can save and resume a session)
• a decent conversation cache, so we can dump the raw requests and responses to save memory, but read them back if requested.
• Various views into the model - showing conversation history (a table of Conversations, effectively), URL properties, etc
• Various plugins - such as those from the current Exodus, as well as others. In particular the Spider will be a good one to get started on.
• a "shared browser state", that can be used by the Spider and Proxy plugins to synchronise Cookies (if the Proxy sees a Set-Cookie, the Spider can use it for future requests, if the Spider sees a Set-Cookie, the Proxy will inject it into future requests, as well as back to the browser)
• interfaces to the above

Do you need help?

In addition to our free technical library, Pantek provides professional IT services to companies who utilize Open Source software: Learn more about Pantek's Technical Support offerings Place an order for 24/7/365 Technical Support here Learn more about Pantek, who we are, and what we do Chat with a Live Operator Related securityfocus.com Links bugtraq certification focus-bsd focus-ids focus-ih focus-linux focus-ms focus-sun focus-unix-other focus-virus forensics incidents libnet linux-secnews ms-secnews pen-test secpapers secprog sectools secureshell security-basics securityjobs sf-news vuln-dev webappsec Request more information about Pantek

Any volunteers?

All comments are welcome!

Rogan

P.S. I originally sent this mail on Monday morning, but it didn't go through. Since then Ingo and I have been busy checking the code into the Sourceforge CVS repository, under webscarab. However, the build scripts have not yet been updated to reflect the new code. Consequently, it is unlikely that everything will build successfully at this point. The code in CVS is essentially the same as that in the .jar mentioned above, so please get that if you want to see how it works so far. Thanks!

-- 
"Using encryption on the Internet is the equivalent of arranging an 
armored car to deliver credit card information from someone living 
in a cardboard box to someone living on a park bench."
  - Gene Spafford
-- 
Deloitte & Touche Security Services Group
Tel: +27(11)806-6216     Fax: +27(11)806-5202     Cell: +27(82)784-9498
-- 

Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by clicking here or by copying and pasting the following address into your Internet browser's address bar: 
http://www.Deloitte.co.za/Disc.htm. The Disclaimer is deemed to form part of the content of this email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre(at)Deloitte.co.za.