Let me start by noting how much fun I had while investigating and analyzing everything for this forensics challenge, I was able to apply many different techniques, from analyzing logs to file carving and network forensics.
The first thing I did was grab the images and pcap files from the DFRWS site and mounted those images on my workstation for analysis:
asm forensics # mount -o loop,noexec,ro /forensics/nssal-linux-side-fs.dd /mnt/ps3asm forensics # mount -o loop,noexec,ro /forensics/jhuisi-linux-side-fs.dd /mnt/ps3-jhs
At this point I tried generating a timeline but I ended up with two huge files, 240 MB and 75 MB, it would not be easy to go through that by hand so I tried to narrow down the date range by analyzing the images I had and trying to figure out when the suspicious activity started.
I started with "nssal-linux-side-fs.dd" and looked at the /etc/shadow file to see what users were on the system and I found these 3 user names:
root:$6$0GL.Dibt$.2dVOk9BcGVR6NJKdl.ZacR63CHB0hhJ3DBDRg/dT/XeULxdblHdOJWrjfP0QyInFcWlxXNreuIHre9VyMxxb1:14258:0:99999:7:::nssal:$1$Hf2Vk/7k$HikXkhyPtzxzJSXvSqTYf0:14217:0:99999:7:::jhuisi:$6$HqshGlKp$PwsgCEwcQ2u57aMkHvwLB25gWGrQSC.jvnPE5aOTAFbNdoLCs0GWVvXStAXWmWnMUfePK5BKY0u0q5RPtQVqU.:14314:0:99999:7:::
It would be possible to decrypt the password for the user nssal since it uses MD5 but it wouldn't be very useful to us. What I did was go through that user's home directory and I found some images that looked like Mardi Gras pictures (~4MB total). Nothing too suspicious there, so I went through the other user's home directory. There aren't many files there except .bash_history:
asm jhuisi # pwd /mnt/ps3/home/jhuisi asm jhuisi # cat .bash_history ls cd Examples ls cd cd .. ls cd nssal/ ls cd Images/ ls cd ls ./backdoor ls -l ~/backdoor ls -l sudo nssal ./backd00r su netstat -an netstat -an | more telnet localhost 45541 ps auxww | grep klog ps auxww | grep back netstat -an | more exit
Ok, this looks interesting. The user seems to have run a backdoor with root privileges, possibly listening on port TCP 45541 and with a process name of klog. These would be some good keywords for a string search. Using the stat command you can see that the history file has a modification date of 2009-03-11. This would be a good time to look at the system logs and find login times and ssh session start times, the files i looked at where /var/log/wtmp, /var/log/auth.log, /var/log/messages and /var/log/ConsoleKit/history.
asm log # last -f wtmp<p>nssal pts/3 128.220.249.83 Fri Aug 25 14:17 - 07:02 (195+16:44) jhuisi pts/2 128.220.249.83 Wed Oct 8 01:01 - 19:50 (34373+18:48 nssal pts/1 137.30.251.152 Wed Jul 16 11:01 - 03:01 (-18057+-7:0</p>asm log # grep 128.220.249.83 auth.log<p>Mar 11 11:49:43 nssal-ps3 sshd[3208]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT! Mar 11 11:49:45 nssal-ps3 sshd[3208]: Accepted password for jhuisi from 128.220.249.83 port 51874 ssh2 Mar 11 11:52:00 nssal-ps3 sshd[3266]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT! Mar 11 11:52:02 nssal-ps3 sshd[3266]: Accepted password for jhuisi from 128.220.249.83 port 51942 ssh2 Mar 11 11:56:09 nssal-ps3 sshd[3326]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT! Mar 11 11:56:11 nssal-ps3 sshd[3326]: Accepted password for jhuisi from 128.220.249.83 port 51075 ssh2 Mar 11 12:02:26 nssal-ps3 sshd[3369]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT! Mar 11 12:02:30 nssal-ps3 sshd[3369]: Accepted password for nssal from 128.220.249.83 port 51109 ssh2</p>asm ConsoleKit # tail history<p>1236791463.488 type=SEAT_SESSION_REMOVED : seat-id='Seat5′ session-id='Session5′ session-type=" session-x11-display=" session-x11-display-device=" session-display-device='/dev/ssh' session-remote-host-name='128.220.249.83′ session-is-local=FALSE session-unix-user=1000 session-creation-time='2009-03-11T17:02:30.355257Z'</p>
I found a few lastlog entries for both nssal and jhuisi and two IP addresses, I also found successful logins from both users on 2009-03-11.I decided it was time to have a look at the other PS3's image (jhuisi-linux-side-fs.dd). Again, I began by looking at the user's home directories, in this case they were jhuisi and goatboy. Once again I found suspicious files and activity.
asm goatboy # cat /mnt/ps3-jhs/home/goatboy/.bash_history<p>ls mkdir Recipes ls cd Recipes/ ls tar xvf recipes.tar ls customers/ mv Tír\ na\ nÓg customers/ ls rm recipes.tar exit ls ls -l Recipes/ exit ls cd Examples ls cd .. clear ls cd Recipes ls cat bateman\'s talk ytalk ntalk write jhuisi ls cat custmers write jhuisi cat customers cd customers ls clear ls -l cat Tír\ na\ nÓg ls write jhuisi I feel younger already! write jhuisi write write jhuisi Nice doing business with you! clear ls cd clear ls</p>asm jhuisi # pwd<p>/mnt/ps3-jhs/home/jhuisi</p>asm jhuisi # ls<p>Desktop Documents Examples Music Pictures Public Templates Videos a.out backd00r.c</p>asm jhuisi # strings a.out...<p>root klogd -x jhuisi Backdoor by darkXside Enter the second password. Password accepted! :Welcome!psyBNC@lam3rz.de NOTICE * :psyBNC2.3.2-4 [backdoor]# /dev/.tty01 chdir exit See ya later...</p>asm log # grep goatboy auth.log...<p>Mar 11 12:49:25 ps3 sshd[5382]: Accepted password for goatboy from 137.30.123.40 port 34612 ssh2 Mar 11 12:49:25 ps3 sshd[5382]: pam_unix(sshd:session): session opened for user goatboy by (uid=0) ...</p>
Ok, now it's getting more interesting, we found what apparently was the source code for the backdoor that was used and its binary version, and also some interesting bash_history and ssh sessions for user goatboy. I have to admit, at first I thought it could not be that easy to find the source and binary for the backdoor, maybe it had been left there as a decoy, so I thought I had to go back and create my timeline with a start date of 2009-03-11, take a good look at it and then do some string searching on the file systems.I used The SleuthKit's fls to create what's called a body file from the images and then created a timeline with mactime.
asm forensics # fls -r -m / nssal-linux-side-fs.dd nssal-linux-side-fs.dd > nssal-linux-side-fs.bodyfileasm forensics # fls -r -m / jhuisi-linux-side-fs.dd > jhuisi-linux-side-fs.bodyfileasm forensics # mactime -d -b nssal-linux-side-fs.bodyfile -z CST6CDT 2009-03-11 > nssal-linux-side-fs.timelineasm forensics # mactime -d -b jhuisi-linux-side-fs.bodyfile -z EST5EDT 2009-03-11 > jhuisi-linux-side-fs.timeline
Why did I use timezones with mactime? Because the timeline needs to be on the same timezone than the system it was created for. How did I find out the timezone for each system?
asm etc # strings /mnt/ps3/etc/localtime<p>... CST6CDT,M3.2.0,M11.1.0</p>asm etc # strings /mnt/ps3-jhs/etc/localtime ... EST5EDT,M3.2.0,M11.1.0
Mactime generated two files (the -d option tells it to index files in comma separated format) which I opened with OpenOffice's Calc.Looking at the timelines from both systems I found evidence of a few interesting facts. On March 11, 2009 at approximately 12:49 EST (11:49 CST) sftp-server was executed on the John's Hopkins PS3 and the sftp client was executed on NSSAL's PS3 which would indicate a file transfer took place. There's evidence on the PCAP files and system log files to back this up.
asm etc # tail -100 /mnt/ps3/var/log/auth.log<p>Mar 11 11:49:43 nssal-ps3 sshd[3208]: reverse mapping checking getaddrinfo for ps3.isi.jhu.edu.249.220.128.in-addr.arpa [128.220.249.83] failed - POSSIBLE BREAK-IN ATTEMPT! Mar 11 11:49:45 nssal-ps3 sshd[3208]: Accepted password for jhuisi from 128.220.249.83 port 51874 ssh2 Mar 11 11:49:45 nssal-ps3 sshd[3208]: pam_unix(sshd:session): session opened for user jhuisi by (uid=0)</p>asm etc # tail -100 /mnt/ps3-jhs/var/log/auth.log<p>Mar 11 12:49:25 ps3 sshd[5382]: Accepted password for goatboy from 137.30.123.40 port 34612 ssh2 Mar 11 12:49:25 ps3 sshd[5382]: pam_unix(sshd:session): session opened for user goatboy by (uid=0) Mar 11 12:49:25 ps3 sshd[5411]: subsystem request for sftp</p>Extracted from jhuisi-capture-1.pcap165720 2009-03-11 13:49:28.800357 137.30.123.40 128.220.249.83 TCP 34612 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1380 TSV=213441 TSER=0 WS=2165721 2009-03-11 13:49:28.800455 128.220.249.83 137.30.123.40 TCP 22 > 34612 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=219936 TSER=213441 WS=2 165722 2009-03-11 13:49:28.860085 137.30.123.40 128.220.249.83 TCP 34612 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=213456 TSER=219936 165723 2009-03-11 13:49:28.883103 128.220.249.83 137.30.123.40 SSHv2 Server Protocol: SSH-2.0-OpenSSH_5.1p1 Debian-3ubuntu1\r ... 166834 2009-03-11 13:52:24.069081 128.220.249.83 137.30.123.40 TCP 51942 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=263753 TSER=0 WS=2 166837 2009-03-11 13:52:24.118586 137.30.123.40 128.220.249.83 TCP 22 > 51942 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=257270 TSER=263753 WS=2 166838 2009-03-11 13:52:24.118676 128.220.249.83 137.30.123.40 TCP 51942 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=263765 TSER=257270 Session size = ~4MB (possible SFTP transfer)<p></p>Extracted from nssal-capture-2.pcap<p>34671 2009-03-11 13:49:02.101469 137.30.123.40 128.220.249.83 TCP 34612 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=213441 TSER=0 WS=2 34672 2009-03-11 13:49:02.161136 128.220.249.83 137.30.123.40 TCP 22 > 34612 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=219936 TSER=213441 WS=2 34673 2009-03-11 13:49:02.161235 137.30.123.40 128.220.249.83 TCP 34612 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=213456 TSER=219936 34674 2009-03-11 13:49:02.236815 128.220.249.83 137.30.123.40 SSHv2 Server Protocol: SSH-2.0-OpenSSH_5.1p1 Debian-3ubuntu1\r ... 35804 2009-03-11 13:51:57.409715 128.220.249.83 137.30.123.40 TCP 51942 > 22 [SYN] Seq=0 Win=5840 Len=0 MSS=1380 TSV=263753 TSER=0 WS=2 35805 2009-03-11 13:51:57.409862 137.30.123.40 128.220.249.83 TCP 22 > 51942 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=257270 TSER=263753 WS=2 35807 2009-03-11 13:51:57.459273 128.220.249.83 137.30.123.40 TCP 51942 > 22 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=263765 TSER=257270</p> Session size = ~4MB (possible SFTP transfer)
I could tell from this network traffic that there was SSH/SFTP activity at that time on both systems, however it wasn't possible to see what was being transferred because everything was encrypted. I suspect that it was an sftp file transfer of the Mardi Gras pictures considering the size in bytes of the session which is almost the same as the total size of the Images directory on nssal's home directory. Wireshark's 'Follow TCP Stream' feature was very useful to determine when that particular session started and ended, I didn't include the FIN packets from when the session was closed here but you can easily find them using that particular Wireshark feature or doing some filtering.
asm etc # ls -alh /mnt/ps3/home/nssal/Images/<p>total 3.9M</p>
It isn't over yet.. there was more to see. I mentioned earlier that I had possibly found the source code for a backdoor that looked like it was used. I did some string searching and file carving and was able to recover different versions of the same backdoor, but all the evidence points to that same file that was sitting there in plain sight. My dirty words list was:
asm forensics # cat keywords.txt<p>Mardi grass nssal jhuisi goatboy backd00r #define password 45541 137.30.251.152 128.220.249.83</p>
In order to recover that deleted backdoor source code i had to extract the unallocated space from the images i had, TSK's blks is what I used.
asm forensics # blkls nssal-linux-side-fs.dd > nssal-linux-side-fs.unalloc asm forensics # blkls jhuisi-linux-side-fs.dd > jhuisi-linux-side-fs.unalloc
Then I extracted strings from unallocated space using srch_strings with the "-t d" option to get a byte offset for each string
asm forensics # srch_strings -a -t d nssal-linux-side-fs.unalloc > nssal-linux-side-fs.unalloc.asc asm forensics # srch_strings -a -t d jhuisi-linux-side-fs.unalloc > jhuisi-linux-side-fs.unalloc.asc
Now that I had strings from unallocated space I was able to go through it and look for suspicious strings. There's several ways to do this, you could manually go through the file using the less command or you could be smart and use your dirty words list and grep
asm forensics # grep -i -f keywords.txt jhuisi-linux-side-fs.unalloc.asc ... 3762245958 #define password "q1w2e3r4" ...
I found many interesting strings but after a lot of trial and error I was able to extract a source code for the backdoor. This line
3762245958 #define password "q1w2e3r4"
looked like part of a C code so i tried to recover the file that contained that string from unallocated space. I used fsstat to get the file system's block size from the image, its 4096, this is needed to calculate the block number (in unallocated space) where this string was found, then I used blkcalc to calculate the block number on the original image and blkcat to extract the block's content.
asm forensics # expr 3762245958 / 4096<p>918517 asm forensics # blkcalc jhuisi-linux-side-fs.dd -u 918517 1273872 asm forensics # blkcat jhuisi-linux-side-fs.dd 1273872 > backd00r.c asm forensics # file backd00r.c backd00r.c: ASCII C program text, with CRLF line terminators</p> There you have it. The source code for the backdoor that was used, if you look at the source you will find the lines
<p>#define PORT 45541</p> <p>#define HIDE "klogd -x"</p>
<p>...</p><p>servaddr.sin_port = htons(PORT);</p><p>...</p>memset(argv[0], ' ', strlen(argv[0])); strcpy(argv[0], HIDE);<p>...</p>
You'll see that the backdoor listens on port TCP 45541 and attempts to "hide" itself by running with the process name "klogd -x" which for the untrained eye would seem like a normal process on a Linux box. By realizing the port it uses and what I mentioned earlier in jhuisi's bash history file I thought it would be a good idea to look at the pcap file again and look for more evidence in terms of network traffic involving the two PS3's over TCP port 45541.
Extracted from jhuisi-capture-1.pcap:
173999 2009-03-11 14:10:26.035241 128.220.249.83 137.30.123.40 TCP 56515 > 45541 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=534240 TSER=0 WS=2 174000 2009-03-11 14:10:26.084796 137.30.123.40 128.220.249.83 TCP 45541 > 56515 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=527756 TSER=534240 WS=2 174002 2009-03-11 14:10:26.084964 128.220.249.83 137.30.123.40 TCP 56515 > 45541 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=534252 TSER=527756
We can clearly see here a TCP connection being established between the two systems over TCP port 45541, again I used wireshark's Follow TCP Stream here to be able to see what was going on, and this time everything was in clear text, nothing was encrypted.
jhuisi Backdoor by darkXside Enter the second password. mac Password accepted! [backdoor]# ls backd00r Examples [backdoor]# ls backd00r Examples [backdoor]# rm backd00r [backdoor]# ls Examples [backdoor]# cd .. [backdoor]# cd nssal [backdoor]# ls channels crosstool d Desktop Documents downloads dump Examples file find.py first first100 Images kmem memdump memdump.new.kernel memdump-powerpc.tar memdump.tct mem.find.pics mem.hithere mod nulls off.c ppu-binutils-2.18.50-21.ppc.rpm ppu-gcc43-4.3.2-2.ppc.rpm ps3git Public ram.dd Recipes screenshots strings.hithere Templates test Videos [backdoor]# who nssal tty7 2009-03-11 11:31 (:0) nssal pts/0 2009-03-11 11:32 (:0.0) nssal pts/1 2009-03-11 11:47 (:0.0) [backdoor]# ls channels crosstool d Desktop Documents downloads dump Examples file find.py first first100 Images kmem memdump memdump.new.kernel memdump-powerpc.tar memdump.tct mem.find.pics mem.hithere mod nulls off.c ppu-binutils-2.18.50-21.ppc.rpm ppu-gcc43-4.3.2-2.ppc.rpm ps3git Public ram.dd Recipes screenshots strings.hithere Templates test Videos [backdoor]# cd Recipes [backdoor]# ls andromachi bateman's stanley's stoughton's [backdoor]# cd .. [backdoor]# ls channels crosstool d Desktop Documents downloads dump Examples file find.py first first100 Images kmem memdump.new.kernel memdump-powerpc.tar memdump.tct mem.find.pics mem.hithere mod nulls off.c ppu-binutils-2.18.50-21.ppc.rpm ppu-gcc43-4.3.2-2.ppc.rpm ps3git Public ram.dd Recipes screenshots strings.hithere Templates test Videos [backdoor]# ls Videos [backdoor]# ls screenshots [backdoor]# ls Images 3316820191_4737c3edf4.jpg 3317048368_639213e24b_b.jpg 3317820492_cefb7ca452_b.jpg 3318492402_731ae5cdc3_b.jpg 3318589824_35fe706451_b.jpg 3320345810_6acc8185b2_b.jpg 3321856153_0a2c9577bd_b.jpg 3322040743_08a3b99acd_b.jpg 3322064459_d61e14dea5_b.jpg 3323556994_59a3982d61_b.jpg 3323673964_94e64ebddd_b.jpg [backdoor]# fdisk -l [backdoor]# ls -l /media total 4 lrwxrwxrwx 1 root root 6 2008-12-03 18:14 cdrom -> cdrom0 drwxr-xr-x 2 root root 4096 2008-12-03 18:14 cdrom0 [backdoor]# ls -l /mnt total 4 drwxr-xr-x 2 root root 4096 2009-03-06 15:01 usb [backdoor]# ls -l /mnt/usb total 0 [backdoor]#
We finally found more evidence of illicit access to that system. I didn't have time to look at the USB drive's image, it would be interesting to see what's in there. There's also probably more data to be extracted from the packet captures, such as images or HTTP traffic, it could easily be done using a tool like NetworkMiner.—Juan is a Network Security Engineer for an IT company in Uruguay. He holds a GCIA certification, Security+ and CCNA. He is working towards getting his GCFA.
