Tutorial: PCAP extraction

This tutorial describes the reduction of pcaps to the very significant packets to answer a specific question. Did it happen to you that your pcap was in the Tbyte range and you had no clue what’s in it and loading it into Wireshark is already at 1GByte cumbersome.

So the task is to reduce the pcap to the very significant part, hence downsize it to a manageable size. This is what we do everyday, so I had to find a way to solve that problem. There a two options:

  • pcapd
  • findexer

pcapd is older and extract packets into a new pcap according to flow indices in different operational modes of the Anteater. E.g. if T2 is in alarm mode pcaps are only extracted if an alarm in an internal signalling block globalWarn is set. If was designed for maximum flexibility to enable the user configure T2 into an intelligent flow based IDS. So mostly running on an interface. Forensic guys might have several pcaps, and always different questions, then pcapd has to be invoked every time if different flows have to be extracted. This is avoided by findexer. This plugin indexes all packets in the pcap. Whenever your question changes you can select flows and store them into a pcap without running T2 again, hence the drill down process is much faster.

Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.

First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile the standard plugins. Just as a precaution if you did not do any tutorials beforehand.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build
...
BUILD SUCCESSFUL

Compiling the basicFlow took now a bit longer, because the subnetfile for geolocation had to rebuilt. Moreover the ‘-e’ option also removes all subnetfiles.

Another method is to just remove the .so files, and preserve the old subnetfiles:

$ rm ~/.tranalyzer/plugins/*.so
$ t2build protoStats

Then the compilation will be considerable faster, as the subnetfile already exists. If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

$ mkdir ~/data
$ mkdir ~/results
$ cd data

pcapd and findexer were made for reduction of really large pcaps, but we cannot supply here TBytes for you. So download the sample pcap if did not do it already: faf-exercise.pcap. And imagine it is very very big. Or just use your own if you need something larger.

Please extract it under your data folder. Sniffing directly from the interface is required, so have your sudo PW ready. Now you are all set for T2 monitoring experiments.

How to select relevant flows for pcap extraction

In order to find interesting flows it is always good practise to look first at the end report and protocol statistics. The rest of the standard plugins also provide pertinent information in the end report. If PLUGIN_REPORT 1, which it is by default.

$ t2 -r ~/data/faf-exercise.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 8402
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: protoStats, 0.8.2
    02: basicFlow, 0.8.3
    03: macRecorder, 0.8.2
    04: portClassifier, 0.8.2
    05: basicStats, 0.8.3
    06: tcpFlags, 0.8.2
    07: tcpStates, 0.8.2
    08: icmpDecode, 0.8.2
    09: connStat, 0.8.2
    10: txtSink, 0.8.2
[INF] basicFlow: IPv4 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 2816237 (2.82 M)
[INF] basicFlow: IPv6 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 36634 (36.63 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.151647 sec
Finished unloading flow memory. Time: 0.151722 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 143.166.11.10: 3101 (3.10 K) [52.54%] packets
basicStats: Biggest Talker: 143.166.11.10: 4268858 (4.27 M) [85.49%] bytes
tcpFlags: Aggregated IP anomaly flags : 0x0046
tcpFlags: Aggregated TCP anomaly flags: 0xbc07
tcpFlags: Number of TCP scans, succ scans, retries: 0, 0, 2
tcpFlags: Number WinSz below 1: 4 [0.07%]
tcpStates: Aggregated anomaly flags: 0x4a
connStat: Max unique number of IP source connections: 25
connStat: Max unique number of IP destination connections: 26
connStat: Max unique number of IP source/destination connections: 10
connStat: Max unique number of source IP / destination port connections: 18
connStat: prtcon/sdcon, prtcon/scon: 1.800000, 0.720000
connStat: Source IP with max connections: 192.168.1.104: 2 connections
connStat: Destination IP with max connections: 77.67.44.206: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 72
Number of processed A flows: 36 [50.00%]
Number of processed B flows: 36 [50.00%]
Number of request     flows: 36 [50.00%]
Number of reply       flows: 36 [50.00%]
Total   A/B    flow asymmetry: 0.00
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 81.97
Number of processed A packets/flows: 55.17
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B packets/s: 0.12
Number of processed A   packets/s: 0.04
Number of processed   B packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.43 GB [0.63%]
Aggregate flow status: 0x0000000000004000
[INF] IPv4

Only IPv4 TCP flows. All symmetric, all bandwidth measure the same, so we have all the packets. Good. Not much traffic, average packet load indicates downloads, and who is responsible? Probably the biggest talker. Some retries, and the window size hit 0, not a major problem. The number of connections is not worrying, so no indication for bots or P2P, which should not happen in our network, right?

Looking into the protoStats file reveal three well known ports FTP, SMTP and HTTP and one unassigned biggest talker.

$ cat faf-exercise_protocols.txt
# Total packets: 5902
# L2/3 Protocol Packets Percentage      Description
0x0800  5902    100.000 Internet Protocol version 4 (IPv4)


# Total IPv4 packets: 5902
# Total IPv6 packets: 0
# L4 Protocol   Packets Percentage      Description
6       5902    100.000 Transmission Control Protocol (TCP)


# Total TCP packets: 5902
# Port  Packets Percentage      Description
21      22      0.373   File Transfer [Control]
25      894     15.147  Simple Mail Transfer Protocol (SMTP)
80      371     6.286   World Wide Web HTTP
64334   4615    78.194


# Total UDP packets: 0
$

Lets assume ftp and http is unusual for our network, all is encrypted. So we like to learn more about ftpDecode and httpSniffer. The mail we ignore for now, the reader may look into that for himself, then he understands, why I left it out.

$ t2build ftpDecode httpSniffer
...
$

And execute T2 with that configuration:

$ t2 -r ~/data/faf-exercise.pcap -w ~/results/
...
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 143.166.11.10: 3101 (3.10 K) [52.54%] packets
basicStats: Biggest Talker: 143.166.11.10: 4268858 (4.27 M) [85.49%] bytes
tcpFlags: Aggregated IP anomaly flags : 0x0046
tcpFlags: Aggregated TCP anomaly flags: 0xbc07
tcpFlags: Number of TCP scans, succ scans, retries: 0, 0, 2
tcpFlags: Number WinSz below 1: 4 [0.07%]
tcpStates: Aggregated anomaly flags: 0x4a
ftpDecode: Anomaly flags: 0x03
ftpDecode: Number of FTP packets: 4634 (4.63 K) [78.52%]
httpSniffer: Number of HTTP packets: 3821 (3.82 K) [64.74%]
httpSniffer: Number of HTTP GET  requests: 16 [0.42%]
httpSniffer: Aggregated status flags : 0x002c
httpSniffer: Aggregated anomaly flags: 0x5000
httpSniffer: Aggregated content flags: 0x0010
httpSniffer: Aggregated mime type    : 0x0045
connStat: Max unique number of IP source connections: 25
connStat: Max unique number of IP destination connections: 26
connStat: Max unique number of IP source/destination connections: 10
connStat: Max unique number of source IP / destination port connections: 18
connStat: prtcon/sdcon, prtcon/scon: 1.800000, 0.720000
connStat: Source IP with max connections: 192.168.1.104: 2 connections
connStat: Destination IP with max connections: 77.67.44.206: 1 connections
--------------------------------------------------------------------------------
...
$

I extracted the interesting part for us now. If you want to decode the hex flags, either look into the .h file of the respective plugin or use tawk -V. If you do not know exactly the name of the variable, just start with e.g. tcp and hit tab two times to trigger the autocompletion:

$ tawk -V tcp
tcpAnomaly  tcpFlags    tcpFStat    tcpOptions  tcpStates

To question the code of the tcpAnomaly flags invoke the following tawk -V command:

$ tawk -V tcpAnomaly=0xbc07
The tcpAnomaly column with value 0xbc07 is to be interpreted as follows:

   bit | tcpAnomaly | Description
   =============================================================================
     0 | 0x0001     | FIN-ACK flag
     1 | 0x0002     | SYN-ACK flag
     2 | 0x0004     | RST-ACK flag
    10 | 0x0400     | Sequence Number retry
    11 | 0x0800     | Sequence Number out of order
    12 | 0x1000     | Sequence mess in flow order due to pcap packet loss
    13 | 0x2000     | Sequence number jump forward
    15 | 0x8000     | Duplicate ACK
$

The same for ftp status:

$ tawk -V ftpStat=0x03
The ftpStat column with value 0x03 is to be interpreted as follows:

   bit | ftpStat | Description
   =============================================================================
     0 | 0x01   | FTP port found
     1 | 0x02   | FTP passive parent flow
$

and the http Anomaly Flags:

$ $ tawk -V httpAFlags=0x5000
The httpAFlags column with value 0x5000 is to be interpreted as follows:

   bit | httpAFlags | Description
   =============================================================================
    12 | 0x1000     | Possible EXE download
    14 | 0x4000     | HTTP 1.0
$

And we are interested what kind of info can be downloaded in the HTTP traffic:

$ tawk -V httpHeadMimes=0x0045
The httpHeadMimes column with value 0x0045 is to be interpreted as follows:

   bit | httpHeadMimes | Description
   =============================================================================
     0 | 0x0001        | Application
     2 | 0x0004        | Image
     6 | 0x0040        | Text
$

So Applications, Images and text. Let’s look at some pictures. So there is HTTP 1.0, very odd for 2009, and we download an executable. Security guys should be interested now. And we have passive FTP around. In our network, no we only use SSH and SCP, why is that?

So this is our question in tawk to the flow file:

$ tawk '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) print; else {split($httpImg_Vid_Aud_Msg_Txt_App_Unk,A,"_"); if (A[1]>0) print;}}' faf-exercise_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration    numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP           srcIPCC  srcIPWho               srcPort  dstIP           dstIPCC  dstIPWho            dstPort  l4Proto  macPairs  srcMac_dstMac_numP                        srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT       aveIAT       stdIAT       pktps       bytps     pktAsm       bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipTOS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpTmS  tcpTmER  tcpEcI  tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStates  icmpStat  icmpTCcnt  icmpBFTypH_TypL_Code          icmpTmGtw   icmpEchoSuccRatio  icmpPFindex  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpUsrNum  ftpPwNum  ftpCNum  ftpUsr       ftpPw      ftpC                                                  httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts     httpURL                                   httpMimes                   httpCookies  httpImages                                                                              httpVideos  httpAudios  httpMsgs  httpAppl                                                                                           httpText                                                                               httpPunk  httpBdyURL  httpUsrAg                                                                                                             httpXFor  httpRefrr  httpVia        httpLoc                                                              httpServ                 httpPwr      connSip  connDip  connSipDip  connSipDprt  connF
B     3        0x0000000000004001  1258544216.915576  1258544217.008019  0.092443    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1260     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_73    Apple_Dell         80             http          73          18           95603        319           0         1380      1309.63     274.7284    0       0.021251     0.001266342  0.003059902  789.6758    1034183   0.6043956    0.9933488   0x01c0    1           42177       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  73          95602           0               2           319                    1               5840          6432         5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007292       0.006796          0.007834          0.007299494       0.0001743189         0.008467049   0.002675676      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7avi1777u1705ff.bin_1_3_1_0"                                                                                                                                                                                                                                                                                                                                                                                                "Apache"                              1        2        7           13           13
B     5        0x0000000000004001  1258544217.357036  1258544217.413505  0.056469    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1262     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_26    Apple_Dell         80             http          26          9            30820        320           0         1380      1185.385    425.1664    0       0.017243     0.002171885  0.00369694   460.4296    545786.2  0.4857143    0.9794477   0x01c0    1           20242       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  26          30819           0               2           320                    1               5840          6431.921     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007285       0.007036          0.00757           0.007321462       0.0001213493         0.008815795   0.002582306      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7iavi2511u2510ff.bin_1_5_1_0"                                                                                                                                                                                                                                                                                                                                                                                               "Apache"                              1        2        5           9            9
B     7        0x0000000000004001  1258544217.763049  1258544217.791016  0.027967    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1264     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_7     Apple_Dell         80             http          7           6            5268         317           0         1380      752.5714    567.7107    0       0.017745     0.003995285  0.005150571  250.295     188364.9  0.07692308   0.8864816   0x01c0    1           53026       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  7           5267            0               2           317                    1               5840          6362.352     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007303       0.007039          0.007577          0.007375143       0.0001950087         0.009821476   0.003135755      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_x8xplsb2_118c8.bin_1_7_1_0"                                                                                                                                                                                                                                                                                                                                                                                                  "Apache"                              1        2        3           5            5
B     9        0x0000000000004001  1258544218.137543  1258544218.165782  0.028239    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1266     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_5     Apple_Dell         80             http          5           6            1690         320           0         1380      338         441.2029    0       0.017993     0.005647799  0.005497161  177.0601    59846.31  -0.09090909  0.681592    0x01c0    1           30104       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  5           1689            0               2           320                    1               5840          6289.861     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007285       0.007037          0.007543          0.007288          0.0001869964         0.009793667   0.003212614      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_x8xplsc_149d148c8.bin_1_9_1_0"                                                                                                                                                                                                                                                                                                                                                                                               "Apache"                              1        1        1           1            1
B     11       0x0000000000004001  1258562478.266384  1258562509.653978  31.387594   1           3        eth:ipv4:tcp  0          63.245.221.11   ff       "mozilla corporation"  80       192.168.1.104   02       "private_reserved"  1384     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_8     Apple_Dell         80             http          8           7            4184         476           0         1380      523         579.3142    0       23.04        3.923449     7.138864     0.2548778   133.3011  0.06666667   0.7957082   0x01c0    1           59415       52        52        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  7           4183            1               2           476                    1               5840          6383.246     5840         6432         0               1              1                  0             0x1b      0x0403      1             4          0x00000016  1380    0      0       0        0       0.000000  0.061556       0.020611          23.03958          2.930275          7.038339             4.101549      7.522526         0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x0000      0x00         0x0004         0x0010      0_0           1          200         0_1_0_1_0_0_0_0_0_1                          1_0_0_0_0_0_0                                                                            "image/png"                              "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_1_11_1_0"                                                                                                                                                                                                                                                                                                                                                                                                               "1.1 varnish"                                                                       "Apache/2.2.3 (CentOS)"               1        1        3           3            3
B     10       0x0000000000004001  1258562467.754689  1258562509.653962  41.899273   1           3        eth:ipv4:tcp  0          63.245.221.11   ff       "mozilla corporation"  80       192.168.1.104   02       "private_reserved"  1379     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_18    Apple_Dell         80             http          18          14           15606        1801          0         1380      867         568.1608    0       22.97829     2.327737     5.515566     0.4296017   372.4647  0.125        0.7930717   0x01c0    1           55605       52        52        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  17          15605           1               5           1801                   1               5840          9616.309     5840         9648         0               4              1                  0             0x1b      0x0403      1             4          0x00000016  1380    0      0       0        0       0.000000  0.005547       7.1e-05           22.97788          1.313044          5.097706             2.65127       5.881811         0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x0000      0x00         0x0044         0x0010      0_0           2          302;200     0_1_1_1_1_0_0_0_0_2                          2_0_0_0_2_0_0                                                                            "text/html";"image/jpeg"                 "_style_dalvay_bg-header-small.jpg_1_10_4_0";"_style_dalvay_main-feature.jpg_1_10_5_1"                                                                                                                                       "_thunderbird_2.0.0.23_start__1_10_1_0";"_en-US_thunderbird_2.0.0.23_start__1_10_2_1"                                                                                                                                                                   "1.1 varnish"  "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"  "Apache/2.2.3 (CentOS)"  "PHP/5.1.6"  1        1        1           1            1
A     33       0x0000000000004000  1258587444.865917  1258587445.631435  0.765518    1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1908     198.189.255.75  us       "--"                80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_24    Dell_Apple         80             http          24          74           319          97078         0         319       13.29167    53.98931    0       0.574143     0.03189658   0.1023528    31.35132    416.7113  -0.5102041   -0.9934495  0x0140    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  21          318             2               19          95698                  0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x8001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000183          0.010698          0.0008450834      0.001741902          0.007304      -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"  "/softw/90/update/u7avi1778u1705z7.bin"                                                                                                                                                                                                                                                                                                                                                                                      "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                 1        1        4           4            4
B     33       0x0000000000004001  1258587444.873221  1258587445.638482  0.765261    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1908     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_74    Apple_Dell         80             http          74          24           97078        319           0         1380      1311.865    268.8814    0       0.573915     0.01034136   0.06162005   96.69904    126856.1  0.5102041    0.9934495   0x01c0    1           33497       57        57        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  73          99837           0               2           319                    1               5840          6432         5840         6432         0               1              1                  0             0x1b      0x3803      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007304       0.007047          0.573488          0.01674978        0.06118462           0.01759486    0.06120941       0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_2_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7avi1778u1705z7.bin_1_33_1_0";"_softw_90_update_u7avi1778u1705z7.bin_1_33_5_1"                                                                                                                                                                                                                                                                                                                                              "Apache"                              1        1        3           3            3
A     34       0x0000000000004000  1258587445.990733  1258587446.040428  0.049695    1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1910     198.189.255.75  us       "--"                80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_8     Dell_Apple         80             http          8           19           320          21634         0         320       40          88.41758    0       0.009989977  0.006211873  0.002938669  160.982     6439.28   -0.4074074   -0.9708481  0x0140    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  5           319             2               4           21634                  0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000402          0.01045102        0.001683127       0.002765731          0.007517      -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"  "/softw/90/update/u7iavi2512u2511z7.bin"                                                                                                                                                                                                                                                                                                                                                                                     "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                 1        1        2           2            2
B     34       0x0000000000004001  1258587445.998250  1258587446.047471  0.049221    1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"  1910     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_19    Apple_Dell         80             http          19          8            21634        320           0         1380      1138.632    455.4652    0       0.01752198   0.002590577  0.003994342  386.0141    439527.8  0.4074074    0.9708481   0x01c0    1           40938       57        57        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  19          21633           0               2           320                    1               5840          6431.036     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007517       0.007043          0.007573          0.007338999       0.0001176775         0.009022127   0.002768233      0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                            "application/octet-stream"                                                                                                                                         "_softw_90_update_u7iavi2512u2511z7.bin_1_34_1_0"                                                                                                                                                                                                                                                                                                                                                                                              "Apache"                              1        1        1           1            1
A     36       0x0000000000004000  1258594163.408285  1258594191.015208  27.606923   1           3        eth:ipv4:tcp  0          192.168.1.105   02       "private_reserved"     49330    143.166.11.10   us       "arin"              64334    6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_1514  Dell_Apple         64334          unknown       1514        3101         0            4268858       0         0         0           0           0       5.58724      0.01823444   0.1478493    54.84132    0         -0.3438787   -1          0x0160    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1512        0               1               931         4255057                478             32768         181608       0            259440       253             194            348                0.002642008   0x16      0x8004      511           1536       0x0000003e  1460    2      0       0        0       0.000000  0              2e-06             5.587702          0.004777336       0.1447472            0.078742      -1               0x42       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     35                                                                           0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    1        1        4           2            2
B     36       0x0000000000004001  1258594163.487027  1258594185.427506  21.940479   1           3        eth:ipv4:tcp  0          143.166.11.10   us       "arin"                 64334    192.168.1.105   02       "private_reserved"  49330    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_3101  Apple_Dell         64334          unknown       3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109      0.007075286  0.02745561   141.3369    194565.4  0.3438787    1           0x01c0    1           65535       111       111       0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3014        5234855         43              0           0                      1               8192          64860        8192         64860        0               1              1                  0             0x1b      0x3803      1             4          0x00000016  1380    0      0       0        0       0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.04343516    0.1503238        0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     35                                                                           0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    1        1        3           1            1
A     35       0x0000000000004000  1258594162.928342  1258594185.618346  22.690004   1           3        eth:ipv4:tcp  0          192.168.1.105   02       "private_reserved"     49329    143.166.11.10   us       "arin"              21       6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_11    Dell_Apple         21             ftp           11          11           92           1231          0         24        8.363636    8.41835     0       21.78007     2.062728     5.945361     0.484795    4.054649  0            -0.8609222  0x0140    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  9           91              1               8           1231                   0               8192          62176.56     8192         64860        8               1              2                  0             0x1a      0x0000      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.00045           0.194089          0.04297619        0.07021572           0.08025199    -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x01     36           USER;PASS;TYPE;PASV;SIZE;RETR                                   1          1         2        "anonymous"  "IEUser@"  "I";"/video/R79733.EXE"                               0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    1        1        2           2            2
B     35       0x0000000000004001  1258594163.008594  1258594491.683288  328.674694  1           3        eth:ipv4:tcp  0          143.166.11.10   us       "arin"                 21       192.168.1.105   02       "private_reserved"  49329    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_11    Apple_Dell         21             ftp           11          11           1231         92            0         950       111.9091    232.9224    0       306.2558     29.87952     83.53862     0.03346774  3.745345  0            0.8609222   0x01c0    1           26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  11          1230            0               6           92                     1               4140          4214.603     4140         4232         0               6              1                  0             0x1e      0x0006      1             2          0x00000014  1380    0      0       0        0       0.000000  0.08025199     0.077494          306.0649          29.85102          83.48595             29.89399      83.48597         0x42       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x41     36                                          220;331;230;200;227;213;125;226  0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    1        1        1           1            1

If this file is very long, you might be interested to first reduce the pcap size. This is what we discuss in the following section.

Extract flows with pcapd plugin

Extracting flow based information from the pcap to produce a downsized pcap can be achieved by the pcapd plugin. It functions in various modes, being configured in pcap.h.

cd ~/tranalyzer2/trunk/pcap/src
$ cat pcap.h
...

The constant PD_SPLIT controls the size of the resulting pcap files if the -W option is selected in the T2 command line. Not important here, so leave it at default setting. PD_MODE controls the basic mode of operation. The idea behind the first two modes (0 or 1) is to use awk to extract flows of interest and then the pcapd plugin to create a PCAP with all those flows. Those two modes require -e option in the T2 command line. The format of the file must be as follows: * 0 The first column must be the flow index, the rest is ignored * 1 Reads directly from flow file, so the second column must be the flow index Lines starting with %, #, a space or a tab are ignored, along with empty lines.

Flows whose index appears in the -e file will be written to a file named PREFIX_PD_SUFFIX, where PREFIX is the value given to T2 in the -e option. Note that if `PD_EQ 0, then flows whose index does not appear in the file will be dumped.

PD_MODE 2 activates a special alarm mode where any plugin can trigger pcapd to dump a flow. Will be discussed in the tutorial alarm mode.

PD_MODE 3/4 are interface modes, working in conjunction with the -W option of T2. Not important here, will be discussed in a later tutorial to come. PD_MAX_FD controls the number of active file handles in mode 4.

We leave everything at default, because we work now on a pcap and generated a flow index file: nudel. Hence, we need the -e option in the T2 command line later on.

In order to accelerate the extraction process it is recommended to build pcapd under another directory, so that only this plugin can be loaded. Hence, if another question has to be applied to the pcap only the plugin folder can be selected by the -p option. Because pcapd is only needed temporary, it can be buiilt under /tmp.

$ t2build -p /tmp pcapd
...
$

Move to your results window or change to the results directory and produce a file which contains all interesting flow indexes selected earlier by the awk. Just add the variable $flowInd in the print statement:

$ cd ~/results
$ tawk -H '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) print $flowInd; else {split($httpImg_Vid_Aud_Msg_Txt_App_Unk,A,"_"); if (A[1]>0) print $flowInd;}}' faf-exercise_flows.txt | tcol > nudel
$ cat nudel
3
5
7
9
11
10
33
33
34
34
36
36
35
35

And tell pcapd with the -e option that it should load the flow index file from the results directory:

$ t2 -r ~/data/faf-exercise.pcap -e ~/results/nudel -p /tmp
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 15053
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: pcapd, 0.8.2
[INF] pcapd: 14 flow indices
Processing file: /home/wurst/data/faf-exercise.pcap
...
--------------------------------------------------------------------------------
pcapd: number of packets extracted: 4958 (4.96 K) [84.01%]
--------------------------------------------------------------------------------
...
$

As you can see T2 produces a new pcap called nudel.pcap. Rerun T2 on the new pcap.

$ ls
faf-exercise_flows.txt  faf-exercise_headers.txt  faf-exercise_packets.txt  faf-exercise_protocols.txt
nudel  nudel.pcap
$
$ t2 -r ~/results/nudel.pcap -w ~/results/ -s
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 19084
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: protoStats, 0.8.2
    02: basicFlow, 0.8.3
    03: macRecorder, 0.8.2
    04: portClassifier, 0.8.2
    05: basicStats, 0.8.3
    06: tcpFlags, 0.8.2
    07: tcpStates, 0.8.2
    08: icmpDecode, 0.8.2
    09: ftpDecode, 0.8.2
    10: httpSniffer, 0.8.2
    11: connStat, 0.8.2
    12: txtSink, 0.8.2
[INF] basicFlow: IPv4 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 2816247 (2.82 M)
[INF] basicFlow: IPv6 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 36634 (36.63 K)
Processing file: /home/wurst/results/nudel.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544216.908284 sec (Wed 18 Nov 2009 11:36:56 GMT)
Dump stop : 1258594191.015208 sec (Thu 19 Nov 2009 01:29:51 GMT)
Total dump duration: 49974.106924 sec (13h 52m 54s)
Finished processing. Elapsed time: 0.144406 sec
Finished unloading flow memory. Time: 0.144577 sec
Percentage completed: 100.00%
Number of processed packets: 4958 (4.96 K)
Number of processed bytes: 4834396 (4.83 M)
Number of raw bytes: 4834396 (4.83 M)
Number of pcap bytes: 4913748 (4.91 M)
Number of IPv4 packets: 4958 (4.96 K) [100.00%]
Number of A packets: 1617 (1.62 K) [32.61%]
Number of B packets: 3341 (3.34 K) [67.39%]
Number of A bytes: 111930 (111.93 K) [2.32%]
Number of B bytes: 4722466 (4.72 M) [97.68%]
Average A packet load: 69.22
Average B packet load: 1413.49 (1.41 K)
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 143.166.11.10: 3101 (3.10 K) [62.55%] packets
basicStats: Biggest Talker: 143.166.11.10: 4268858 (4.27 M) [88.30%] bytes
tcpFlags: Aggregated IP anomaly flags : 0x0046
tcpFlags: Aggregated TCP anomaly flags: 0xbc07
tcpFlags: Number of TCP scans, succ scans, retries: 0, 0, 2
tcpFlags: Number WinSz below 1: 4 [0.08%]
tcpStates: Aggregated anomaly flags: 0x42
ftpDecode: Anomaly flags: 0x43
ftpDecode: Number of FTP packets: 4633 (4.63 K) [93.44%]
httpSniffer: Number of HTTP packets: 3329 (3.33 K) [67.14%]
httpSniffer: Number of HTTP GET  requests: 11 [0.33%]
httpSniffer: Aggregated status flags : 0x002c
httpSniffer: Aggregated anomaly flags: 0x5000
httpSniffer: Aggregated content flags: 0x0010
httpSniffer: Aggregated mime type    : 0x0045
connStat: Max unique number of IP source connections: 4
connStat: Max unique number of IP destination connections: 4
connStat: Max unique number of IP source/destination connections: 8
connStat: Max unique number of source IP / destination port connections: 8
connStat: prtcon/sdcon, prtcon/scon: 1.000000, 2.000000
connStat: Source IP with max connections: 192.168.1.104: 1 connections
connStat: Destination IP with max connections: 198.189.255.75: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 4958 (4.96 K) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 20
Number of processed A flows: 10 [50.00%]
Number of processed B flows: 10 [50.00%]
Number of request     flows: 10 [50.00%]
Number of reply       flows: 10 [50.00%]
Total   A/B    flow asymmetry: 0.00
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 247.90
Number of processed A packets/flows: 161.70
Number of processed B packets/flows: 334.10
Number of processed total packets/s: 0.10
Number of processed A+B packets/s: 0.10
Number of processed A   packets/s: 0.03
Number of processed   B packets/s: 0.07
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 774 b/s
Average full bandwidth : 772 b/s
Max number of flows in memory: 8 [0.00%]
Memory usage: 0.43 GB [0.64%]
Aggregate flow status: 0x0000000000004000
[INF] IPv4
$

Much less flows and packets. If you look at the resulting flow file, you actually see more flows than in our request on the original flow file above. Because not in every flow direction certain flags are set. So you also get the return or orig flow now, which preserves context of a communication and avoids unnecessary further tawk questions.

$ tcol nudel_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP           srcIPCC  srcIPWho               srcPort  dstIP           dstIPCC  dstIPWho               dstPort  l4Proto  macPairs  srcMac_dstMac_numP                        srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT       aveIAT       stdIAT       pktps      bytps     pktAsm       bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipTOS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpTmS  tcpTmER  tcpEcI  tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStates  icmpStat  icmpTCcnt  icmpBFTypH_TypL_Code          icmpTmGtw   icmpEchoSuccRatio  icmpPFindex  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpUsrNum  ftpPwNum  ftpCNum  ftpUsr       ftpPw      ftpC                                                  httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts                   httpURL                                                                                                                                   httpMimes                   httpCookies  httpImages                                                                            httpVideos  httpAudios  httpMsgs  httpAppl                                                                                         httpText                                                                             httpPunk  httpBdyURL  httpUsrAg                                                                                                             httpXFor  httpRefrr                                                            httpVia        httpLoc                                                              httpServ                 httpPwr      connSip  connDip  connSipDip  connSipDprt  connF
A     1        0x0000000000004000  1258544216.908284  1258544217.008468  0.100184   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1260     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_18    Dell_Apple         80             http          18          73           319          95603         0         319       17.72222    61.73868    0       0.013738     0.005565777  0.00363383   179.6694   3184.141  -0.6043956   -0.9933488  0x0140    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  15          318             2               14          95603                  0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              7.8e-05           0.014188          0.001167556       0.002669991          0.007292      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7avi1777u1705ff.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        8           8            8
B     1        0x0000000000004001  1258544216.915576  1258544217.008019  0.092443   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1260     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_73    Apple_Dell         80             http          73          18           95603        319           0         1380      1309.63     274.7284    0       0.021251     0.001266342  0.003059902  789.6758   1034183   0.6043956    0.9933488   0x01c0    1           42177       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  73          95602           0               2           319                    1               5840          6432         5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007292       0.006796          0.007834          0.007299494       0.0001743189         0.008467049   0.002675676      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7avi1777u1705ff.bin_1_1_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                      "Apache"                              1        1        7           7            7
A     2        0x0000000000004000  1258544217.349751  1258544217.413719  0.063968   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1262     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_9     Dell_Apple         80             http          9           26           320          30820         0         320       35.55556    84.1992     0       0.009743     0.007107555  0.002012674  140.6953   5002.501  -0.4857143   -0.9794477  0x0140    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  6           319             2               6           30820                  0               65535         65300.4      65075        65535        1               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000214          0.010205          0.001494333       0.002579453          0.007285      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7iavi2511u2510ff.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        6           6            6
B     2        0x0000000000004001  1258544217.357036  1258544217.413505  0.056469   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1262     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_26    Apple_Dell         80             http          26          9            30820        320           0         1380      1185.385    425.1664    0       0.017243     0.002171885  0.00369694   460.4296   545786.2  0.4857143    0.9794477   0x01c0    1           20242       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  26          30819           0               2           320                    1               5840          6431.921     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007285       0.007036          0.00757           0.007321462       0.0001213493         0.008815795   0.002582306      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7iavi2511u2510ff.bin_1_2_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"                              1        1        5           5            5
A     3        0x0000000000004000  1258544217.755746  1258544217.791475  0.035729   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1264     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_6     Dell_Apple         80             http          6           7            317          5268          0         317       52.83333    98.10986    0       0.010241     0.005954833  0.002865114  167.9308   8872.345  -0.07692308  -0.8864816  0x0140    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3           316             2               2           5268                   0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000423          0.010686          0.002446333       0.003129686          0.007303      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/x8xplsb2_118c8.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        4           4            4
B     3        0x0000000000004001  1258544217.763049  1258544217.791016  0.027967   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1264     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_7     Apple_Dell         80             http          7           6            5268         317           0         1380      752.5714    567.7107    0       0.017745     0.003995285  0.005150571  250.295    188364.9  0.07692308   0.8864816   0x01c0    1           53026       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  7           5267            0               2           317                    1               5840          6362.352     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007303       0.007039          0.007577          0.007375143       0.0001950087         0.009821476   0.003135755      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_x8xplsb2_118c8.bin_1_3_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                        "Apache"                              1        1        3           3            3
A     4        0x0000000000004000  1258544218.130258  1258544218.166240  0.035982   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1266     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_6     Dell_Apple         80             http          6           5            320          1690          0         320       53.33333    99.03834    0       0.01049      0.005997     0.00290767   166.75     8893.335  0.09090909   -0.681592   0x0140    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3           319             2               2           1690                   0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000456          0.010956          0.002505667       0.003207167          0.007285      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/x8xplsc_149d148c8.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        2           2            2
B     4        0x0000000000004001  1258544218.137543  1258544218.165782  0.028239   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1266     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_5     Apple_Dell         80             http          5           6            1690         320           0         1380      338         441.2029    0       0.017993     0.005647799  0.005497161  177.0601   59846.31  -0.09090909  0.681592    0x01c0    1           30104       57        57        0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  5           1689            0               2           320                    1               5840          6289.861     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007285       0.007037          0.007543          0.007288          0.0001869964         0.009793667   0.003212614      0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x1000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_x8xplsc_149d148c8.bin_1_4_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"                              1        1        1           1            1
A     6        0x0000000000004000  1258562478.204828  1258562509.633367  31.428539  1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1384     63.245.221.11   ff       "mozilla corporation"  80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_7     Dell_Apple         80             http          7           8            476          4184          0         476       68          138.7919    0       23.04004     4.489791     7.379        0.2227275  15.14547  -0.06666667  -0.7957082  0x0160    1           109         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  4           475             2               3           4184                   1               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000406          8.196325          1.171274          2.65522              0.061556      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_1_0_0                          0_0_0_0_0_0_0                    "www.mozillamessaging.com"  "/en-US/thunderbird/2.0.0.23/start/thunderbird_startpage.png"                                                                                                                                                                                                                                                                                                                                                                                                                                                          "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23"                              "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"                                                                                                                            1        1        4           4            4
B     6        0x0000000000004001  1258562478.266384  1258562509.653978  31.387594  1           3        eth:ipv4:tcp  0          63.245.221.11   ff       "mozilla corporation"  80       192.168.1.104   02       "private_reserved"     1384     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_8     Apple_Dell         80             http          8           7            4184         476           0         1380      523         579.3142    0       23.04        3.923449     7.138864     0.2548778  133.3011  0.06666667   0.7957082   0x01c0    1           59415       52        52        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  7           4183            1               2           476                    1               5840          6383.246     5840         6432         0               1              1                  0             0x1b      0x0403      1             4          0x00000016  1380    0      0       0        0       0.000000  0.061556       0.020611          23.03958          2.930275          7.038339             4.101549      7.522526         0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x0000      0x00         0x0004         0x0010      0_0           1          200         0_1_0_1_0_0_0_0_0_1                          1_0_0_0_0_0_0                                                                                                                                                                                          "image/png"                              "_en-US_thunderbird_2.0.0.23_start_thunderbird_startpage.png_1_6_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                                    "1.1 varnish"                                                                       "Apache/2.2.3 (CentOS)"               1        1        3           3            3
A     5        0x0000000000004000  1258562467.749142  1258562509.633370  41.884228  1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1379     63.245.221.11   ff       "mozilla corporation"  80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_14    Dell_Apple         80             http          14          18           1801         15606         0         455       128.6429    183.7502    0       22.97809     2.991731     6.053773     0.3342547  42.99948  -0.125       -0.7930717  0x01e0    1           91          128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  11          1800            2               8           15606                  1               65535         65497.7      64737        65535        2               2              3                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000186          9.952202          1.338226          2.934125             0.005547      -1               0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x0000      0x02         0x0000         0x0010      4_0           0                      4_0_0_0_0_1_0_1_0_0                          0_0_0_0_0_0_0                    "www.mozillamessaging.com"  "/thunderbird/2.0.0.23/start/";"/en-US/thunderbird/2.0.0.23/start/";"/style/dalvay/bg-header-small.jpg";"/style/dalvay/main-feature.jpg"                                                                                                                                                                                                                                                                                                                                                                               "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23"                              "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"                                                                                                                            1        1        2           2            2
B     5        0x0000000000004001  1258562467.754689  1258562509.653962  41.899273  1           3        eth:ipv4:tcp  0          63.245.221.11   ff       "mozilla corporation"  80       192.168.1.104   02       "private_reserved"     1379     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_18    Apple_Dell         80             http          18          14           15606        1801          0         1380      867         568.1608    0       22.97829     2.327737     5.515566     0.4296017  372.4647  0.125        0.7930717   0x01c0    1           55605       52        52        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  17          15605           1               5           1801                   1               5840          9616.309     5840         9648         0               4              1                  0             0x1b      0x0403      1             4          0x00000016  1380    0      0       0        0       0.000000  0.005547       7.1e-05           22.97788          1.313044          5.097706             2.65127       5.881811         0x00       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x0000      0x00         0x0044         0x0010      0_0           2          302;200     0_1_1_1_1_0_0_0_0_2                          2_0_0_0_2_0_0                                                                                                                                                                                          "text/html";"image/jpeg"                 "_style_dalvay_bg-header-small.jpg_1_5_4_0";"_style_dalvay_main-feature.jpg_1_5_5_1"                                                                                                                                     "_thunderbird_2.0.0.23_start__1_5_1_0";"_en-US_thunderbird_2.0.0.23_start__1_5_2_1"                                                                                                                                                                                                                             "1.1 varnish"  "http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/"  "Apache/2.2.3 (CentOS)"  "PHP/5.1.6"  1        1        1           1            1
A     7        0x0000000000004000  1258587444.865917  1258587445.631435  0.765518   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1908     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_24    Dell_Apple         80             http          24          74           319          97078         0         319       13.29167    53.98931    0       0.574143     0.03189658   0.1023528    31.35132   416.7113  -0.5102041   -0.9934495  0x0140    1           2           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  21          318             2               19          95698                  0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x8001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000183          0.010698          0.0008450834      0.001741902          0.007304      -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7avi1778u1705z7.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        4           4            4
B     7        0x0000000000004001  1258587444.873221  1258587445.638482  0.765261   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1908     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_74    Apple_Dell         80             http          74          24           97078        319           0         1380      1311.865    268.8814    0       0.573915     0.01034136   0.06162005   96.69904   126856.1  0.5102041    0.9934495   0x01c0    1           33497       57        57        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  73          99837           0               2           319                    1               5840          6432         5840         6432         0               1              1                  0             0x1b      0x3803      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007304       0.007047          0.573488          0.01674978        0.06118462           0.01759486    0.06120941       0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_2_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7avi1778u1705z7.bin_1_7_1_0";"_softw_90_update_u7avi1778u1705z7.bin_1_7_5_1"                                                                                                                                                                                                                                                                                                                                                                                                      "Apache"                              1        1        3           3            3
A     8        0x0000000000004000  1258587445.990733  1258587446.040428  0.049695   1           3        eth:ipv4:tcp  0          192.168.1.104   02       "private_reserved"     1910     198.189.255.75  us       "--"                   80       6        1         00:0b:db:4f:6b:10_00:19:e3:e7:5d:23_8     Dell_Apple         80             http          8           19           320          21634         0         320       40          88.41758    0       0.009989977  0.006211873  0.002938669  160.982    6439.28   -0.4074074   -0.9708481  0x0140    1           1           128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  5           319             2               4           21634                  0               65535         65535        65535        65535        0               0              0                  0             0x1b      0x0001      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.000402          0.01045102        0.001683127       0.002765731          0.007517      -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x006c    0x4000      0x02         0x0000         0x0010      1_0           0                      1_0_0_0_0_1_0_0_0_0                          0_0_0_0_0_0_0                    "aa.avg.com"                "/softw/90/update/u7iavi2512u2511z7.bin"                                                                                                                                                                                                                                                                                                                                                                                                                                                                               "AVGINET9-WXPPX86 90 AVI=270.14.72/2511 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA="                                                                                                                                                                                                           1        1        2           2            2
B     8        0x0000000000004001  1258587445.998250  1258587446.047471  0.049221   1           3        eth:ipv4:tcp  0          198.189.255.75  us       "--"                   80       192.168.1.104   02       "private_reserved"     1910     6        1         00:19:e3:e7:5d:23_00:0b:db:4f:6b:10_19    Apple_Dell         80             http          19          8            21634        320           0         1380      1138.632    455.4652    0       0.01752198   0.002590577  0.003994342  386.0141   439527.8  0.4074074    0.9708481   0x01c0    1           40938       57        57        0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  19          21633           0               2           320                    1               5840          6431.036     5840         6432         0               1              1                  0             0x1b      0x0003      1             4          0x00000016  1380    0      0       0        0       0.000000  0.007517       0.007043          0.007573          0.007338999       0.0001176775         0.009022127   0.002768233      0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x00                                                                                  0          0         0                                                                                      0x0068    0x5000      0x00         0x0001         0x0010      0_0           1          200         0_0_0_1_0_0_0_0_0_1                          0_0_0_0_0_1_0                                                                                                                                                                                          "application/octet-stream"                                                                                                                                       "_softw_90_update_u7iavi2512u2511z7.bin_1_8_1_0"                                                                                                                                                                                                                                                                                                                                                                                                                                                     "Apache"                              1        1        1           1            1
A     10       0x0000000000004000  1258594163.408285  1258594191.015208  27.606923  1           3        eth:ipv4:tcp  0          192.168.1.105   02       "private_reserved"     49330    143.166.11.10   us       "arin"                 64334    6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_1514  Dell_Apple         64334          unknown       1514        3101         0            4268858       0         0         0           0           0       5.58724      0.01823444   0.1478493    54.84132   0         -0.3438787   -1          0x0160    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1512        0               1               931         4255057                478             32768         181608       0            259440       253             194            348                0.002642008   0x16      0x8004      511           1536       0x0000003e  1460    2      0       0        0       0.000000  0              2e-06             5.587702          0.004777336       0.1447472            0.078742      -1               0x42       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     9                                                                            0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1        1        4           2            2
B     10       0x0000000000004001  1258594163.487027  1258594185.427506  21.940479  1           3        eth:ipv4:tcp  0          143.166.11.10   us       "arin"                 64334    192.168.1.105   02       "private_reserved"     49330    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_3101  Apple_Dell         64334          unknown       3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109      0.007075286  0.02745561   141.3369   194565.4  0.3438787    1           0x01c0    1           65535       111       111       0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3014        5234855         43              0           0                      1               8192          64860        8192         64860        0               1              1                  0             0x1b      0x3803      1             4          0x00000016  1380    0      0       0        0       0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.04343516    0.1503238        0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     9                                                                            0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1        1        3           1            1
A     9        0x0000000000004000  1258594162.928342  1258594185.618346  22.690004  1           3        eth:ipv4:tcp  0          192.168.1.105   02       "private_reserved"     49329    143.166.11.10   us       "arin"                 21       6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_11    Dell_Apple         21             ftp           11          10           92           1231          0         24        8.363636    8.41835     0       21.78007     2.062728     5.945361     0.484795   4.054649  0.04761905   -0.8609222  0x0140    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  9           91              1               8           1231                   0               8192          62176.56     8192         64860        8               1              2                  0             0x1a      0x0000      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.00045           0.194089          0.04297619        0.07021572           0.08025199    -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x01     10           USER;PASS;TYPE;PASV;SIZE;RETR                                   1          1         2        "anonymous"  "IEUser@"  "I";"/video/R79733.EXE"                               0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1        1        2           2            2
B     9        0x0000000000004001  1258594163.008594  1258594185.427515  22.418921  1           3        eth:ipv4:tcp  0          143.166.11.10   us       "arin"                 21       192.168.1.105   02       "private_reserved"     49329    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_10    Apple_Dell         21             ftp           10          11           1231         92            0         950       123.1       241.7142    0       21.78333     2.241892     6.179659     0.4460518  54.90898  -0.04761905  0.8609222   0x01c0    1           26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  10          1206            0               6           92                     1               4140          4207.147     4140         4232         0               6              1                  0             0x1a      0x0002      1             2          0x00000014  1380    0      0       0        0       0.000000  0.08025199     0.077494          21.58924          2.22962           6.122048             2.272596      6.122451         0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x41     10                                          220;331;230;200;227;213;125;226  0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      1        1        1           1            1

As we run t2 with the -s option, there is also a packet file.

$ tcol nudel_packets.txt
...

Have a look into it and see what you find. Have fun.

The superior way: findexer plugin

If you do not want to build an alarming tool for anomalies which extracts packets from on plugin request or if you have a stack of questions to a large pcap, and you do not want to rerun T2 all over again several times, the findexer plugin is the tool of choice. So remove pcapd from /tmp and compile findexer in your plugin repository and run T2.

$ t2build -p /tmp -u pcapd
...
$ t2build findexer
...
$ t2 -r ~/data/faf-exercise.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 28049
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: protoStats, 0.8.2
    02: basicFlow, 0.8.3
    03: macRecorder, 0.8.2
    04: portClassifier, 0.8.2
    05: basicStats, 0.8.3
    06: tcpFlags, 0.8.2
    07: tcpStates, 0.8.2
    08: icmpDecode, 0.8.2
    09: ftpDecode, 0.8.2
    10: httpSniffer, 0.8.2
    11: connStat, 0.8.2
    12: txtSink, 0.8.2
    13: findexer, 0.8.2
[INF] basicFlow: IPv4 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 2816244 (2.82 M)
[INF] basicFlow: IPv6 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 36634 (36.63 K)
Processing file: /home/wurst/data/faf-exercise.pcap
...
$

findexer produces a binary faf-exercise_flows.xer file, which contains all references between flow indexes and packets. Move to your results bash window or move to the results directory and do a ls then you spot it.

$ cd ~/results
$ ls
faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt  nudel  nudel.pcap
$

Now invoke the tawk which selects the flows from the pcapd section and add the -x option to produce a pcap.

$ tawk -H -x knoedel.pcap '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) print $flowInd; else {split($httpImg_Vid_Aud_Msg_Txt_App_Unk,A,"_"); if (A[1]>0) print $flowInd;}}' faf-exercise_flows.txt
[INFO] Processing findexer : faf-exercise_flows.xer
[INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap
[INFO] Extracted 4959 packets.
$

and ls will show you the knoedel.pcap file.

$ ls
faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt  knoedel.pcap  nudel  nudel.pcap
$

You may now run T2 on the knoedel.pcap and you will see only the selected flows as in the pcapd section. The main difference is that you can rerun the tawk above with a different question and you will get right away a new pcap without reinvoking T2. E.g. a troubleshooting question: Select all flows where the window size was reported to 0, so the receiver could not process more packets anymore. User the tcp Window size threshold ratio variable: $tcpWinSzThRt in tcpFlags, threshold is 0 by default. I leave this exercise to the reader.

Findexer and wireshark export

There are people who like to work with wireshark or tshark instead of t2 packet files. Fair enough, a bit more complicated if you have a lot of data.

So tawk comes to the rescue. Invoke the -k option with the same question as above:

$ tawk -H -k '{ if (bitsanyset($ftpStat, 0x0f) || bitsanyset($httpAFlags, 0x5000)) print $flowInd; else {split($httpImg_Vid_Aud_Msg_Txt_App_Unk,A,"_"); if (A[1]>0) print $flowInd;}}' faf-exercise_flows.txt
[INFO] Processing findexer : faf-exercise_flows.xer
[INFO] Processing pcap 1/1 : /home/wurst/data/faf-exercise.pcap
[INFO] Extracted 4959 packets.
$

If wireshark complains that you do not have rights as a user then use the following command

$ sudo gpasswd -a YOUR_USERNAME wireshark
$

and log off and on again. Then try the the tawk above again and you will have wireshark open with the knoedel.pcap.

tawk pcap export to wireshark
tawk pcap export to wireshark

How cool is that?

Management of humongous flow files: ffsplit

Very large pcaps produce large flow files, if you do not aggregate enought or if you have too many plugins. Right! There is a way to split these files. One way is to use the -W option in T2 which produces more smaller flow files instead of one large file.

If you are interested in segregating several protocols, that is possible with ffsplit:

tawk -d ffsplit
ffsplit(s, k, h):

  Split the input file into smaller more manageable files.

  The files to create can be specified as argument to the function (one comma
  separated string) (See `s`).

  If no argument is specified, creates one file per column whose name ends
  with Stat, e.g., dnsStat, and one for pwxType (pw) and covertChannels (cc)

  `s` Valid arguments for 's' are:
    - arp        - bgp        - dhcp        - dns        - ftp        - http
    - icmp       - igmp       - irc         - modbus     - nat        - ntp
    - ospf       - pop        - radius      - sctp       - smb        - smtp
    - ss         - stp        - syslog      - tftp       - voip       - vrrp

    - pw (cleartext passwords)
    - cc (covert channels)

  `h` Valid arguments for 'h' are:
    - if h is empty or omitted: keep the default columns
        default = "dir,flowInd,timeFirst,timeLast,duration,ethVlanID,
                srcIP,srcIP4,srcIP6,dstIP,dstIP4,dstIP6,srcPort,dstPort,l4Proto"
    - if h is a non-existent column, e.g., 0: keep only relevant fields
    - if h starts with '+', e.g., "+flowStat,hdrDesc", keep default fields
      and the ones specified
    - else keep fields specified in h, e.g., "srcIP,dstIP"

  Parameters:
    - s: a comma separated list of files to create (see `s`).
    - k: 0: keep all columns, 1: keep only relevant columns
    - h: comma separated list of fields to keep in each file (see `h`)

  Dependencies:
    - None

  Examples:
    - tawk 'ffsplit()' file.txt
    - tawk 'ffsplit("dns")' file.txt
    - tawk 'ffsplit("cc,dns,http")' file.txt
    - tawk 'ffsplit("", 1)' file.txt
    - tawk 'ffsplit("dns,http", 1, "+flowStat,hdrDesc")' file.txt
    - tawk 'ffsplit("", 1, "srcIP,dstIP")' file.txt
$

Let’s try the simplest way first, extraction by protocols:

$ tawk 'ffsplit()' faf-exercise_flows.txt
$ ls
faf-exercise_flows_ftp.txt   faf-exercise_flows_icmp.txt  faf-exercise_flows.xer    faf-exercise_icmpStats.txt  knoedel.pcap  nudel.pcap
faf-exercise_flows_http.txt  faf-exercise_flows.txt       faf-exercise_headers.txt  faf-exercise_protocols.txt  nudel
$

So you see that now for each existing protocol a different flow file is produced. Let’s look into ftp:

$ tcol faf-exercise_flows_ftp.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration    numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP          srcIPCC  srcIPWho            srcPort  dstIP          dstIPCC  dstIPWho            dstPort  l4Proto  macPairs  srcMac_dstMac_numP                        srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps       bytps     pktAsm      bytAsm      tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipTOS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D             tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpInitWinSz  tcpAveWinSz  tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzThRt  tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpTmS  tcpTmER  tcpEcI  tcpBtm    tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStates  icmpStat  icmpTCcnt  icmpBFTypH_TypL_Code          icmpTmGtw   icmpEchoSuccRatio  icmpPFindex  ftpStat  ftpCDFindex  ftpCC                          ftpRC                            ftpUsrNum  ftpPwNum  ftpCNum  ftpUsr       ftpPw      ftpC                                                  httpStat  httpAFlags  httpMethods  httpHeadMimes  httpCFlags  httpGet_Post  httpRSCnt  httpRSCode  httpURL_Via_Loc_Srv_Pwr_UAg_XFr_Ref_Cky_Mim  httpImg_Vid_Aud_Msg_Txt_App_Unk  httpHosts  httpURL  httpMimes  httpCookies  httpImages  httpVideos  httpAudios  httpMsgs  httpAppl  httpText  httpPunk  httpBdyURL  httpUsrAg  httpXFor  httpRefrr  httpVia  httpLoc  httpServ  httpPwr  connSip  connDip  connSipDip  connSipDprt  connF
A     36       0x0000000000004000  1258594163.408285  1258594191.015208  27.606923   1           3        eth:ipv4:tcp  0          192.168.1.105  02       "private_reserved"  49330    143.166.11.10  us       "arin"              64334    6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_1514  Dell_Apple         64334          unknown       1514        3101         0            4268858       0         0         0           0           0       5.58724   0.01823444   0.1478493   54.84132    0         -0.3438787  -1          0x0160    1           223         128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1512        0               1               931         4255057                478             32768         181608       0            259440       253             194            348                0.002642008   0x16      0x8004      511           1536       0x0000003e  1460    2      0       0        0       0.000000  0              2e-06             5.587702          0.004777336       0.1447472            0.078742      -1               0x42       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     35                                                                           0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                             1        1        4           2            2
B     36       0x0000000000004001  1258594163.487027  1258594185.427506  21.940479   1           3        eth:ipv4:tcp  0          143.166.11.10  us       "arin"              64334    192.168.1.105  02       "private_reserved"  49330    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_3101  Apple_Dell         64334          unknown       3101        1514         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561  141.3369    194565.4  0.3438787   1           0x01c0    1           65535       111       111       0         0x00   0x0042   0         0x00_0x00000000  0_0            0x00000000_0x00000000  3014        5234855         43              0           0                      1               8192          64860        8192         64860        0               1              1                  0             0x1b      0x3803      1             4          0x00000016  1380    0      0       0        0       0.000000  0.078742       1e-06             0.67088           0.03865783        0.04056465           0.04343516    0.1503238        0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x02     35                                                                           0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                             1        1        3           1            1
A     35       0x0000000000004000  1258594162.928342  1258594185.618346  22.690004   1           3        eth:ipv4:tcp  0          192.168.1.105  02       "private_reserved"  49329    143.166.11.10  us       "arin"              21       6        1         00:08:74:38:01:b4_00:19:e3:e7:5d:23_11    Dell_Apple         21             ftp           11          11           92           1231          0         24        8.363636    8.41835     0       21.78007  2.062728     5.945361    0.484795    4.054649  0           -0.8609222  0x0140    1           2328        128       128       0         0x00   0x0040   0         0x00_0x00000000  0_0            0x00000000_0x00000000  9           91              1               8           1231                   0               8192          62176.56     8192         64860        8               1              2                  0             0x1a      0x0000      1             4          0x00000016  1460    0      0       0        0       0.000000  0              0.00045           0.194089          0.04297619        0.07021572           0.08025199    -1               0x02       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x01     36           USER;PASS;TYPE;PASV;SIZE;RETR                                   1          1         2        "anonymous"  "IEUser@"  "I";"/video/R79733.EXE"                               0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                             1        1        2           2            2
B     35       0x0000000000004001  1258594163.008594  1258594491.683288  328.674694  1           3        eth:ipv4:tcp  0          143.166.11.10  us       "arin"              21       192.168.1.105  02       "private_reserved"  49329    6        1         00:19:e3:e7:5d:23_00:08:74:38:01:b4_11    Apple_Dell         21             ftp           11          11           1231         92            0         950       111.9091    232.9224    0       306.2558  29.87952     83.53862    0.03346774  3.745345  0           0.8609222   0x01c0    1           26560       239       239       0         0x00   0x0044   0         0x00_0x00000000  0_0            0x00000000_0x00000000  11          1230            0               6           92                     1               4140          4214.603     4140         4232         0               6              1                  0             0x1e      0x0006      1             2          0x00000014  1380    0      0       0        0       0.000000  0.08025199     0.077494          306.0649          29.85102          83.48595             29.89399      83.48597         0x42       0x00      0          0x00000000_0x00000000_0x0000  0x00000000  0                  0            0x41     36                                          220;331;230;200;227;213;125;226  0          0         1                                "125 Data connection already open; Transfer startin"  0x0000    0x0000      0x00         0x0000         0x0010      0_0           0                      0_0_0_0_0_0_0_0_0_0                          0_0_0_0_0_0_0                                                                                                                                                                                                                             1        1        1           1            1

yep all ftp. Does that help you? Give us feedback please. If somebody is interested why it is so easy to produce all kinds of functionality with tawk. There is a tool anael built, the fextractor which does all the work extracting packets according to any question, using the *.xer file.

Working on findexed pcaps: fextractor

Imagine you have only a distinct selection of pcaps to analyze, and they are big, so that you do not want to run tranalyzer every time you change your question, or you select certain flows. And imagine you want to script everything yourselve without using tawk -k because you want to do your own thing. Fair enough, then the fextractor is your tool of choice. Note that tawk -k uses fextractor as well.

Let’s have a look at the help first

fextractor -h
Usage: fextractor -r INPUT[:start][,end] (-w OUTPUT | -n) [OPTIONS]... \
            [[DIR@]FLOWINDEX[:start][,end]]...

Extract the flows FLOWINDEX using the _flows.xer INTPUT generated by Tranalyzer2 findexer plugin.
Alternatively use a list of findexer files generated by Tranalyzer2 -W option from index start
to end. The extracted flows are written to the OUTPUT pcap.

An optional packet range can be provided on each command line FLOWINDEX to only extract packets
in the range [start, end] of this flow. If start or end are ommitted, they are replaced by,
respectively, the first and the last available packets in the flow. The FLOWINDEX can also
optionally be prefixed with a direction A or B, by default both directions are extracted.

OPTIONS:
  -r INPUT[:start][,end]
            either read packet indexes from a single _flows.xer file named INPUT
            or read packet indexes from multiple _flows.xer files prefixed by INPUT
            and with suffix in range [start, end]. If start or end are ommitted,
            they are replaced by, respectively, first and last available XER files.
  -w OUTPUT write packets to pcap file OUTPUT
            OUTPUT "-" means that the PCAP is written to stdout.
  -f        overwrite OUTPUT if it already exists
  -n        print oldest PCAP still available, its first packet timestamp and exit
  -h        print this help message
  -i FILE   read flow indexes from FILE. FILE can either be in _flows.txt format
            (flow index in 2nd tab-separated column), or have one flow index per line.
            FILE "-" means that flows are read from stdin.
  -b        by default when FILE is in _flows.txt format, only directions present in
            it are extracted, this option force both directions to be extracted even if
            only the A or B direction is present in the flow file.
  -s N      skip the first N PCAPs
  -p DIR    search pcaps in DIR
            should only be set if pcaps were moved since Tranalyzer2 was run

The -r allows you to process any pcap which is indexed by findexer. If you have several files which are indexed, e.g. nudel1, nudel2, ... then use the colon notation. The -w option denotes the location and name where the extracted pcap should be written. The -i option denotes the flow index file. From our earlier experiment with pcapd above we still have our flow index file: nudel. So move to your results folder and invoke fextractor using the already generated findexer faf-exercise_flows.xer file.

$ cd ~/results
$ fextractor -i nudel -r ./faf-exercise_flows.xer -w ./nudelxer.pcap
$ ls
faf-exercise_flows.txt  faf-exercise_flows.xer  faf-exercise_headers.txt  faf-exercise_icmpStats.txt  faf-exercise_protocols.txt
nudel  nudel.pcap  nudelxer.pcap
$

Now you may change the nudel file and reinvoke the same fextractor command without running T2 ever again, if you are only interested in the same pcap. Have fun experimenting.