Background
I usually do a lot of data carving. With 500 gig drives becoming the norm in machines, the recovered files I see from data carving is huge. Nothing like having to review 10000+ jpegs and having to review each one. I had a lot of issues trying to find something to review that many images. After trying many programs and some hacks to break up the images into smaller subsets. I decided to write my own set of tools for processing the files recovered from data carving.
Data Carver Processors
The Data Carver Processors are a combination of Perl scripts and other programs that are designed to break up the recovered files into manageable chunks. As the script runs over the files, it will create a series of web pages with thumbnails and a second web page for each file that contains plug-in output like metadata, hashes, and etc. The scripts, for the most part, will not process damaged files. If a file is damaged, there will be no image for it on the web page. Just some text on the file.
Currently I have something for images, video, pdfs, and documents. The first one to be released is the Image processor. I will be releasing the others as I have them finished and docs created for them. Should be over the course of the Aug. and Sept.
Image-processor
Point this script at a directory full of images and it does:
- Creates a series of web pages that contains a thumbnail of all readable images
- Gathers details about the files such as Exif data
- Sort images based upon nudity; Creates a CSV file with results in addition to the output in the web pages; Currently only detects "white" skin
- Review images using StegDetect (only does JPEGs); Creates a CSV file with results in addition to the output in the web pages
- Create your own plug-ins
Requirements
Perl modules: Getopt::Long, Pod::Usage, Image::ExifTool, Image::Magick, Imager::SkinDetector, File::Basename, Config::IniFiles
Libraries and packages installed: Imagemagick and Stegdetect
Stegdetect Notes: In order to get Stegdetect to compile using gcc 4.x, download the patch at Stegdetect 0.6.
Example commandline: 1) cd into the directory where you want the output put into 2) /appl/data_carver_processors/image-processor.pl -inputdir /foremost/output/jpeg -output index -plugindir /appl/data_carver_processors/image-plugins -ini /appl/data_carver_processors/data_processor.ini -nudity -stego
INI File
The INI file (data_processor.ini) contains the user configurable options for each one of the data processor scripts.
Each line has a comment before the parameter. See the INI file for more details.
Plug-ins
The script will only except plug-ins written in Perl. Just place your plug-in into the appropriate directory (i.e. image-plug-ins). Each plug-in has to be able to accept one command line argument which is -i {file name}. The Image Processor script will execute the plug-in and pass the -i option and file name to the plug-in. The output will be captured and placed into the web page for that file.
An example Perl plug-in
#!/usr/bin/perl
#use warnings;
use Getopt::Std;
use File::Basename;
use Digest::SHA qw(sha512);
sub usage() {
print <<EOM;
usage: $0 [options] [directory]
options:
-i file Input file
EOM
exit;
}
usage() unless getopts("ht:o:i:p:");
usage() if $opt_h;
$file = $opt_i;
$sha = Digest::SHA->new;
$sha->addfile($file);
$digest = $sha->hexdigest;
$digest =~ tr/[a-z]/[A-Z]/;
print "SHA512: $digest\n";
An example Perl plug-in with shell commands
#!/usr/bin/perl
#use warnings;
use Getopt::Std;
use File::Basename;
sub usage() {
print <<EOM;
usage: $0 [options] [directory]
options:
-i file Input file
EOM
exit;
}
usage() unless getopts("hi:");
usage() if $opt_h;
$file = $opt_i;
$digest = `md5sum $file`;
$digest =~ tr/[a-z]/[A-Z]/;
print "MD5: $digest\n";
For those of you who are not into writing Perl scripts, take a look a the first line that has a $digest in it. The shell command is between the ` `. Feel free to replace the md5sum with anything else you want to run. You should remove the 2nd $digest line, unless you want the output in upper case. Lastly, alter the print statement by changing the MD5 to whatever you want to call your plug-in. Save the file and place it in the plug-in directory.
Testing The Plugin
If you end up writing your own plug-ins, you can always test your plug-in by doing:
# ./{plug-in file name}.pl -i {test file name}
If the output is what you expect, then your ready to run it with the processor.
Please send me any plug-ins you write. I will be happy to include them with the download.
CSV Files
The CSV files that get generated with the -nudity and -stego options can be found under the directory tn.
nudity_review.csv
If you gave the Image processor the -nudity option it will create a file called nudity_review.csv. The contents contain the file name and score as per the Imager::SkinDetector. An example file looks like this:
/appl/scripts/data_carve_processors/test/data/Battlestar.jpg,9.43998543211 /appl/scripts/data_carve_processors/test/data/Nmap_Matrix_Screen_Huge.jpg,0.235281046114735 /appl/scripts/data_carve_processors/test/data/annie-360x184.png,23.9994074147233 /appl/scripts/data_carve_processors/test/data/annie-720x368.png,23.6893463389346 /appl/scripts/data_carve_processors/test/data/axm-v3-10-p25.jpg,14.7481563652296 /appl/scripts/data_carve_processors/test/data/nmap-matrix2log-cropped.gif,6.15041880852475 /appl/scripts/data_carve_processors/test/data/nmap-matrixhax0r3c.gif,0 /appl/scripts/data_carve_processors/test/data/nmap-matrixhax0r3c.png,0 /appl/scripts/data_carve_processors/test/data/nmap_matrix5.png,0 /appl/scripts/data_carve_processors/test/data/nmap_matrix6.png,0
stego.csv
The stego.csv file will get created when the -stego is given. Again, this contains the file name and output from StegDetect. Here is an example:
/appl/scripts/data_carve_processors/test/data/Battlestar.jpg : skipped (false positive likely) /appl/scripts/data_carve_processors/test/data/Nmap_Matrix_Screen_Huge.jpg : negative /appl/scripts/data_carve_processors/test/data/axm-v3-10-p25.jpg : negative
INI File
The INI file (data_processor.ini) contains the user configurable options for each one of the data processor scripts.
Each line has a comment before the parameter. See the INI file for more details.
Other Notes
Feedback: Please send me an email with any features/plug-ins you would like to see. If you find any errors with the scripts, let me know. I am also interested any plug-ins you want to share.
Errors: As the script runs over the files you may see some errors outputted. The errors are from the programs running on the recovered files. Not all of the files that the data carvers recover are good files. Hence, the errors.
License: GPL 2.0
Download at: data_carver_processors.tar.gz
Contact: cs[at]citadelsystems.net
Keven Murphy, GCFA Gold #24, is the Senior Forensics/Incident Handler to General Dynamics Land Systems.
