Tutorial: Payload Dumper

This tutorial discusses the plugin payloadDumper, a tcpflow replacement. A plugin which does not match with the T2 traffic mining philosophy. You select first the relevant flows and then look into more detail. But a good lad we know wanted it badly, so we wrote it. In future it will do much more than tcpflow, e.g. SCTP, layer 2, etc

Preparation

Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.

First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the default plugin folder ~/.tranalyzer/plugins. Just as a precaution if you have some old plugins or files there. If you like to keep them, please copy them away.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$

Then compile the following plugins

$ t2build tranalyzer2 basicFlow payloadDumper txtSink
...
BUILD SUCCESSFUL

$

If you did not create a separate data and results directory yet, please do it now in another cmd window, it facilitates your workflow:

$ mkdir ~/data ~/results
$

Download the sample pcap here: faf-exercise.pcap. Now you’re all set.

payloadDumper

Let’s look at the plugin configuration first:

$ payloaDumper
$ vi src/payloadDumper.h

You can choose whether TCP and/or UDP is your target, extract port pairs, set a maximum for the payload length to be extracted and choose the name of the files and their location. We leave everything as default for the time being.

Now run t2 on the supplied pcap.

$ t2 -r ~/data/ faf-exercise.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.11 (Anteater), Tarantula. PID: 15510
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.11
    02: payloadDumper, 0.8.11
    03: txtSink, 0.8.11
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406084 (406.08 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51072 (51.07 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.015180 sec
Finished unloading flow memory. Time: 0.015190 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
payloadDumper: pldStat=0x03
payloadDumper: Number of non zero content dumped flows: 71 [97.26%]
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 73
Number of processed A flows: 37 [50.68%]
Number of processed B flows: 36 [49.32%]
Number of request     flows: 36 [49.32%]
Number of reply       flows: 37 [50.68%]
Total   A/B    flow asymmetry: 0.01
Total req/rply flow asymmetry: -0.01
Number of processed   packets/flows: 80.85
Number of processed A packets/flows: 53.68
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregated flowStat=0x0400000000004000
[INF] IPv4 flows

So the aggregated status and the number of flows being dumped denotes no errors and that two flows were empty, so we don’t count them.

$ tawk -V pldStat=0x03
The pldStat column with value 0x03 is to be interpreted as follows:

   bit | pldStat | Description
   =============================================================================
     0 | 0x01    | Match for this flow
     1 | 0x02    | dump payload for this flow

In the flow file you see the plpStat for each flow being extracted.

$ cd ~/results
$ tcol faf-exercise_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP           srcIPCC  srcIPOrg                       srcPort  dstIP           dstIPCC  dstIPOrg                       dstPort  l4Proto  pldStat
A     1        0x0400000000004000  1258544215.037210  1258544215.372742  0.335532   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1258     77.67.44.206    gb       "Akamai Technologies"          80       6        0x03
B     1        0x0400000000004001  1258544215.202900  1258544215.537951  0.335051   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              77.67.44.206    gb       "Akamai Technologies"          80       192.168.1.104   07       "Private network"              1258     6        0x03
A     2        0x0400000000004000  1258544216.385370  1258544216.723144  0.337774   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1259     77.67.44.206    gb       "Akamai Technologies"          80       6        0x03
B     2        0x0400000000004001  1258544216.551313  1258544216.888595  0.337282   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              77.67.44.206    gb       "Akamai Technologies"          80       192.168.1.104   07       "Private network"              1259     6        0x03
A     3        0x0400000000004000  1258544216.908284  1258544217.008468  0.100184   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1260     198.189.255.75  us       "California State University"  80       6        0x03
B     3        0x0400000000004001  1258544216.915576  1258544217.008019  0.092443   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              198.189.255.75  us       "California State University"  80       192.168.1.104   07       "Private network"              1260     6        0x03
A     4        0x0400000000004000  1258544217.003718  1258544217.348506  0.344788   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104   07       "Private network"              1261     77.67.44.206    gb       "Akamai Technologies"          80       6        0x03
B     4        0x0400000000004001  1258544217.169421  1258544217.513942  0.344521   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              77.67.44.206    gb       "Akamai Technologies"          80       192.168.1.104   07       "Private network"              1261     6        0x03
...
A     36       0x0400000000004000  1258594163.408285  1258594191.015208  27.606923  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49330    143.166.11.10   us       "Dell"                         64334    6        0x01
B     36       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         64334    192.168.1.105   07       "Private network"              49330    6        0x03
A     35       0x0400000000004000  1258594162.928342  1258594185.618346  22.690004  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105   07       "Private network"              49329    143.166.11.10   us       "Dell"                         21       6        0x03
B     35       0x0400000000004001  1258594163.008594  1258594185.427515  22.418921  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         21       192.168.1.105   07       "Private network"              49329    6        0x03
A     37       0x0400000000004001  1258594491.683288  1258594491.683288  0.000000   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10   us       "Dell"                         21       192.168.1.105   07       "Private network"              49329    6        0x01

Note that flow 36 A and 37 B were not dumped because the content is 0. You may load basicStat to verify that fact. Now look under /tmp/ for the extracted payload:

$ cd /tmp/payloadDumper
$ ls
10_A  11_A  12_A  13_A  14_A  15_A  16_A  17_A  18_A  19_A  1_A  20_A  21_A  22_A  23_A  24_A  25_A  26_A  27_A  28_A  29_A  2_A  30_A  31_A  32_A  33_A  34_A  35_A  36_B  3_B  4_B  5_B  6_B  7_B  8_B  9_B
10_B  11_B  12_B  13_B  14_B  15_B  16_B  17_B  18_B  19_B  1_B  20_B  21_B  22_B  23_B  24_B  25_B  26_B  27_B  28_B  29_B  2_B  30_B  31_B  32_B  33_B  34_B  35_B  3_A   4_A  5_A  6_A  7_A  8_A  9_A

Let’s have a look at one of the files:

$ cat 10_A
GET /thunderbird/2.0.0.23/start/ HTTP/1.1
Host: www.mozillamessaging.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

GET /en-US/thunderbird/2.0.0.23/start/ HTTP/1.1
Host: www.mozillamessaging.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

GET /style/dalvay/bg-header-small.jpg HTTP/1.1
Host: www.mozillamessaging.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/

GET /style/dalvay/main-feature.jpg HTTP/1.1
Host: www.mozillamessaging.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/

Here you are. As homework change PLDUMP_NAME to 1, recompile and rerun T2, now the file names look like tcpflow. Play around with PLDUMP_TCP_PORTS.

Don’t forget to reset the plugin configuration for the next tutorial.

$ t2conf payloadDumper --reset && t2build payloadDumper
...
$

Have fun analyzing.