Tutorial: Payload Dumper

This tutorial discusses the plugin payloadDumper, a tcpflow replacement. A plugin which does not match with the T2 traffic mining philosophy. You select first the relevant flows and then look into more detail. But a good lad we know wanted it badly, so we wrote it. In future it will do much more than tcpflow, e.g., layer 2, etc

Preparation

Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.

First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the default plugin folder ~/.tranalyzer/plugins. Just as a precaution if you have some old plugins or files there. If you like to keep them, please copy them away.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$

Then compile the following plugins

$ t2build tranalyzer2 basicFlow payloadDumper txtSink
...
BUILD SUCCESSFUL

$

If you did not create a separate data and results directory yet, please do it now in another cmd window, it facilitates your workflow:

$ mkdir ~/data ~/results
$

Download the sample pcap here: faf-exercise.pcap. Now you’re all set.

payloadDumper

Let’s look at the plugin default configuration first:

$ payloaDumper
$ vi src/payloadDumper.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */

#define PLDUMP_L2          0 // Extract payload for layer 2 flows (require ETH_ACTIVATE > 0)
#define PLDUMP_TCP         1 // Extract payload for TCP flows
#define PLDUMP_UDP         1 // Extract payload for UDP flows
#define PLDUMP_SCTP        0 // Extract payload for SCTP stream flows (require SCTP_ACTIVATE = 1)

#define PLDUMP_ETHERTYPES {} // Only extract L2 payloads for those ethertypes, e.g., {0x2000,0x2003}
#define PLDUMP_TCP_PORTS  {} // Only extract TCP payloads on those ports, e.g., {80,8080}
#define PLDUMP_UDP_PORTS  {} // Only extract UDP payloads on those ports, e.g., {80,8080}
#define PLDUMP_SCTP_PORTS {} // Only extract SCTP payloads on those ports, e.g., {80,8080}

#define PLDUMP_MAX_BYTES   0 // Max number of bytes per flow to dump (use 0 for no limits)
#define PLDUMP_START_OFF   0 // Start dumping bytes at a specific offset (L2 and UDP only)

#define PLDUMP_RMDIR       1 // Empty PLDUMP_FOLDER before starting

#define PLDUMP_NAMES       0 // Format for filenames:
                             //    0: flowInd '_' [AB]
                             //    1: srcIP.srcPort-dstIP.dstPort-l4Proto[_sctpStream],
                             //       srcMac-dstMac-etherType
                             //    2: Same as 1, but prefixed with timestampT

#define PLDUMP_FOLDER    "/tmp/payloadDumper"  // output folder
#define PLDUMP_PREFIX    ""                    // prefix for output files
#define PLDUMP_SUFFIX    ""                    // suffix for output files

/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...

You can choose whether layer 2, TCP, SCTP and/or UDP is your target. You can also choose to only extract data for specific ports or ethertypes, set a maximum for the payload length to be extracted and choose the name of the files and their location. By default the plugin extracts all payload from TCP and UDP.

Note that the following core switches found in networkheaders.h must be activated for layer 2 and SCTP respectively:

  • For layer 2: ETH_ACTIVATE must be 1 or 2
  • For SCTP, SCTP_ACTIVATE must be 1.

The PLDUMP_START_OFF defines the offset to start the extraction, this is experimental currently.

Let’s say we are only interested in FTP and SMTP packets to be extracted. So the appropriate t2conf command is the following:

$ t2conf payloadDumper -D PLDUMP_TCP_PORTS="{25,21}"
$ t2build payloadDumper
...
$

Note that the port row has to be enclosed by ".

Now run t2 on the supplied pcap.

$ t2 -r ~/data/faf-exercise.pcap -w ~/results/ -s
================================================================================
Tranalyzer 0.8.13 (Anteater), Tarantula. PID: 21247
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.13
    02: payloadDumper, 0.8.13
    03: txtSink, 0.8.13
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.007433 sec
Finished unloading flow memory. Time: 0.007444 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
payloadDumper: Aggregated pldStat=0x03
payloadDumper: Number of non zero content dumped flows: 44 [60.27%]
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 73
Number of processed A flows: 37 [50.68%]
Number of processed B flows: 36 [49.32%]
Number of request     flows: 36 [49.32%]
Number of reply       flows: 37 [50.68%]
Total   A/B    flow asymmetry: 0.01
Total req/rply flow asymmetry: -0.01
Number of processed   packets/flows: 80.85
Number of processed A packets/flows: 53.68
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregated flowStat=0x0400000000004000
[INF] IPv4 flows
$ tawk -V pldStat=0x03
The pldStat column with value 0x03 is to be interpreted as follows:

   bit | pldStat | Description
   =============================================================================
     0 | 0x01    | Match for this flow
     1 | 0x02    | dump payload for this flow

Let’s show only the flows which extract content:

$ cd ~/results
$ tawk 'bitsanyset($pldStat,0x02)' faf-exercise_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  pldStat
A     12       0x0400000000004000  1258563573.941668  1258563576.594009  2.652341   1           3        eth:ipv4:tcp  00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800              192.168.1.103  07       "Private network"  1397     192.168.1.1    07       "Private network"  25       6        0x03
B     12       0x0400000000004001  1258563573.941709  1258563576.594045  2.652336   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800              192.168.1.1    07       "Private network"  25       192.168.1.103  07       "Private network"  1397     6        0x03
A     13       0x0400000000004000  1258565030.304653  1258565030.420837  0.116184   1           3        eth:ipv4:tcp  00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800              192.168.1.103  07       "Private network"  1749     192.168.1.1    07       "Private network"  25       6        0x03
B     13       0x0400000000004001  1258565030.304696  1258565030.420877  0.116181   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800              192.168.1.1    07       "Private network"  25       192.168.1.103  07       "Private network"  1749     6        0x03
A     14       0x0400000000004000  1258565174.919134  1258565175.037809  0.118675   1           3        eth:ipv4:tcp  00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800              192.168.1.103  07       "Private network"  1755     192.168.1.1    07       "Private network"  25       6        0x03
B     14       0x0400000000004001  1258565174.919179  1258565175.037828  0.118649   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800              192.168.1.1    07       "Private network"  25       192.168.1.103  07       "Private network"  1755     6        0x03
...
B     32       0x0400000000004001  1258583614.298161  1258583615.323218  1.025057   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:63:58:a6  0x0800              192.168.1.1    07       "Private network"  25       192.168.1.102  07       "Private network"  1911     6        0x03
A     35       0x0400000000004000  1258594162.928342  1258594185.618346  22.690004  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49329    143.166.11.10  us       "Dell"             21       6        0x03
B     35       0x0400000000004001  1258594163.008594  1258594185.427515  22.418921  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             21       192.168.1.105  07       "Private network"  49329    6        0x03

In the packet file we select now the packets which are dumped or only matched the rule.

$ tawk 'bitsanyset($pldStat,0x03)' faf-exercise_packets.txt | tcol
%pktNo  flowInd  flowStat            time               pktIAT     pktTrip    flowDuration  numHdrs  hdrDesc       ethVlanID  srcMac             dstMac             ethType  srcIP          srcIPCC  srcIPOrg         srcPort  dstIP          dstIPCC  dstIPOrg         dstPort  l4Proto  pldStat  l7Content
247     12       0x0400000000004000  1258563573.941668  0.000000   0.000000   0.000000      3        eth:ipv4:tcp             00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800   192.168.1.103  07       Private network  1397     192.168.1.1    07       Private network  25       6        0x01
248     12       0x0400000000004001  1258563573.941709  0.000000   0.000041   0.000000      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800   192.168.1.1    07       Private network  25       192.168.1.103  07       Private network  1397     6        0x01
249     12       0x0400000000004000  1258563573.942161  0.000493   0.000452   0.000493      3        eth:ipv4:tcp             00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800   192.168.1.103  07       Private network  1397     192.168.1.1    07       Private network  25       6        0x01
250     12       0x0400000000004001  1258563574.317353  0.375644   0.375192   0.375644      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800   192.168.1.1    07       Private network  25       192.168.1.103  07       Private network  1397     6        0x03     220 domex ESMTP Postfix\r\n
251     12       0x0400000000004000  1258563574.319182  0.377021   0.001829   0.377514      3        eth:ipv4:tcp             00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800   192.168.1.103  07       Private network  1397     192.168.1.1    07       Private network  25       6        0x03     EHLO m57pat\r\n
252     12       0x0400000000004001  1258563574.319211  0.001858   0.000029   0.377502      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800   192.168.1.1    07       Private network  25       192.168.1.103  07       Private network  1397     6        0x01
253     12       0x0400000000004001  1258563574.319380  0.000169   0.000198   0.377671      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800   192.168.1.1    07       Private network  25       192.168.1.103  07       Private network  1397     6        0x03     250-domex\r\n250-PIPELINING\r\n250-SIZE 10485760\r\n250-VRFY\r\n250-ETRN\r\n250-AUTH LOGIN PLAIN CRAM-MD5 GSSAPI\r\n250-STARTTLS\r\n250-ENHANCEDSTATUSCODES\r\n250-8BITMIME\r\n250 DSN\r\n
254     12       0x0400000000004000  1258563574.319928  0.000746   0.000548   0.378260      3        eth:ipv4:tcp             00:0b:db:63:5b:d4  00:19:e3:e7:5d:23  0x0800   192.168.1.103  07       Private network  1397     192.168.1.1    07       Private network  25       6        0x03     STARTTLS\r\n
255     12       0x0400000000004001  1258563574.319950  0.000570   0.000022   0.378241      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:63:5b:d4  0x0800   192.168.1.1    07       Private network  25       192.168.1.103  07       Private network  1397     6        0x01
...
1279    35       0x0400000000004001  1258594163.407582  0.080461   0.079737   0.398988      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             21       192.168.1.105  07       Private network  49329    6        0x03     227 Entering Passive Mode (143,166,11,10,251,78)\r\n
1283    35       0x0400000000004000  1258594163.487490  0.159645   0.079908   0.559148      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49329    143.166.11.10  us       Dell             21       6        0x03     SIZE /video/R79733.EXE\r\n
1284    35       0x0400000000004001  1258594163.565990  0.158408   0.078500   0.557396      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             21       192.168.1.105  07       Private network  49329    6        0x03     213 4255056\r\n
1285    35       0x0400000000004000  1258594163.566694  0.079204   0.000704   0.638352      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49329    143.166.11.10  us       Dell             21       6        0x03     RETR /video/R79733.EXE\r\n
1286    35       0x0400000000004001  1258594163.644188  0.078198   0.077494   0.635594      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             21       192.168.1.105  07       Private network  49329    6        0x03     125 Data connection already open; Transfer starting.\r\n
1303    35       0x0400000000004000  1258594163.838277  0.271583   0.194089   0.909935      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49329    143.166.11.10  us       Dell             21       6        0x01
5898    35       0x0400000000004001  1258594185.427515  21.783327  21.589237  22.418921     3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             21       192.168.1.105  07       Private network  49329    6        0x03     226 Transfer complete.\r\n
5900    35       0x0400000000004000  1258594185.618346  21.780069  0.190831   22.690004     3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49329    143.166.11.10  us       Dell             21       6        0x01
5902    37       0x0400000000004001  1258594491.683288  0.000000   0.000000   0.000000      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             21       192.168.1.105  07       Private network  49329    6        0x01

Note that packet 247-249,252,255… 1303,5900,5902 match our rule, but were not extracted due to no payload. You may load basicStat to verify that fact. Now look under /tmp/ for the extracted payload:

$ cd /tmp/payloadDumper
$ ls
12_A  13_A  14_A  15_A  16_A  17_A  18_A  19_A  20_A  21_A  22_A  23_A  24_A  25_A  26_A  27_A  28_A  29_A  30_A  31_A  32_A  35_A
12_B  13_B  14_B  15_B  16_B  17_B  18_B  19_B  20_B  21_B  22_B  23_B  24_B  25_B  26_B  27_B  28_B  29_B  30_B  31_B  32_B  35_B
$

Let’s have a look at one of the files:

$ cat 35_A
USER anonymous
PASS IEUser@
TYPE I
PASV
SIZE /video/R79733.EXE
RETR /video/R79733.EXE
$

Oups, what is that? Murky….

As homework try look into file 28_A and then change PLDUMP_NAME to 1, recompile and rerun T2, now the file names look like tcpflow. Play around with PLDUMP_TCP_PORTS extract some ports e.g. 80 traffic.

Another method to extract the content and display it like Wireshark’s Follow TCP/UDP stream is to use tawk and the follow_stream() function on the packet file. Refer to the packet mode tutorial for a practical example.

Don’t forget to reset the plugin configuration for the next tutorial.

$ t2conf payloadDumper --reset && t2build payloadDumper
...
$

Have fun analyzing.