Wanted: Testers for indexed searching in Autopsy and Sleuthkit

From: Paul Bakker <bakker(at)fox-it.com>
Date: Tue May 13 2003 - 09:16:12 EDT

 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I work at a company doing Forensic IT investigations in the Netherlands called Fox-IT (http://www.fox-it.com). We are working on an all-Linux environment for Forensic research.

As the main Forensic tool we would like to use Autopsy/Sleuthkit. As it is missing some features in comparison to (commercial) Windows products, we've decided to contribute and add some new features to Autopsy and Sleuthkit. We're doing this in cooperation with Brian Carrier of @stake.

One of the major missing features is indexed searching. Indexed searching greatly speeds up searches for words during investigations.

We created a first implementation for indexed searching in Autopsy and Sleuthkit. This e-mail is to inform the users of this new addition and to request testers as the code is still in beta. After the addition has been successfully tested it will be submitted for integration in Autopsy and Sleuthkit.

Indexed searching requires the creation of two additional files (And thus will require additional diskspace). The total size of these files is comparable to the size of the strings file generated from an image. For the creation however, twice the space of the strings file is required.

It has been tested on a Debian Linux system and on a number of forensic images. The speedup for searching is very great (Searches on a 5 Gb image file for a single word in less than 1 second (Resulting in 11866 hits), compared to 168 seconds using the regular grepping on the strings file).

For the indexed search two files are required: a "mangled strings" file and an "index" file. The creation of the "mangled strings" file requires the strings file for the image. The process is split in two parts (but are combined within Autopsy) and takes about 68 minutes to complete for a 3.5 Gb strings file, resulting in a 4.0 Gb "mangled strings" file. During the proces about 8.5 Gb of temporary space is required! The creation of the "index" file requires the "mangled strings" file and takes about 5 minutes to complete for the aforementioned 4.0 Gb file. The resulting "index" file is only 5 Mb in size.

Features:

• Tools for Indexed searching in sleuthkit.
• Creation of necessary files integrated into Autopsy interface.
• Indexed Search field (At the bottom of the "Keyword search" page).
• Case insensitive searching.

There are still some limitations:

• Only the ASCII character set is recognized for indexing. This is because the meaning of Unicode characters depends on the context. This makes it very hard to index these. If somebody knows how this could be integrated in the utilities, I will gladly add the functionality.
• Only able to search for single words. Option for combining multiple searches will be added in a later version (In addition to the option to recall search results).
• Only start of words can be searched. e.g. the original word is "baseball". A search for "base" will match, a search for "ball" will not. In the future I will expand the indexing functionality. This will require a lot of additional diskspace (So this option comes at a price).
• No regex searches possible. (It is almost impossible to combine indexed searching with regex.)

The available patches are for Autopsy 1.71 and Sleuthkit 1.61. They add a first (beta) version of indexed searching to Autopsy.

It is still in beta and therefore I would greatly appreciate it if people would test the indexed searching on other machines and images and send their problems, feedback and feature requests to me.

All feedback is appreciated! My goal is to add useful features (like indexed searching) to Autopsy and Sleuthkit. This requires feedback! ;-)

Please send an e-mail to me if you'd like to test the patches.

• -- Paul Bakker

Fox-IT Experts in IT Security!
Haagweg 137
2281 AG RIJSWIJK
T 070 336 9999
F 070 336 9990
I www.fox-it.com
E bakker@fox-it.com
57A6 C5EA 55E4 CC1C A967 B13C F8C0 C0FB 8135 E225

Disclaimer: This email may contain confidential information. If this message is not addressed to you, you may not retain or use the information in it for any purpose. If you have received it in error, please notify the sender and delete this message. We try to screen out viruses but take no responsibility if this email contains a virus.

-----BEGIN PGP SIGNATURE-----
Version: PGP 7.1.1

iQA/AwUBPsDvgPjAwPuBNeIlEQJmYACg3csr2FHGtHNqXbRiVHrIHZ3vHHEAnR40 EZz6BZYC6cQGB8xUA+V3mXmm
=Rmsg
-----END PGP SIGNATURE-----

---

This list is provided by the SecurityFocus ARIS analyzer service. For more information on this free incident handling, management and tracking system please see: http://aris.securityfocus.comReceived on Tue May 13 09:19:45 2003