Tutorial: Basic Analysis

WTF is the Anteater?

Tranalyzer2(T2) was created at a swiss operator out of the need that standard cisco netflow did not supply the fields we needed for our troubleshooting and /security work. We needed all kinds of encapsulated protocols, content info, advanced statistics and an easy way to extract information for traffic mining. And most important a tool which can digest really large pcaps and runs stable on an interface. Therefore, only code and functionality which is needed by the user is added. THat should explain, why a lot of T2 is controlled by compiler switches, making it adapable and lightweigth. But no worries, we made compiling on different infrastructure easy for you.

Having also students with us, we saw they always reinventing the wheel when it came to traffic analysis, so in 2008 T2 became open source. Since then practical ideas from people working in the field and in research inspired the path of the Anteater.

This tutorial will teach you about the basic configuration, usage, basic plugins and post-processing philosophy. So, let’s first look at the basic protocol and output modes.

T2 operational modes

By default T2 operates in the following basic protocol modes:

  • IPv4
  • IPv6
  • Layer2

By default since the 0.8.0 version T2 operates concurrently in all protocol modes and feeds output into the same files. If you are only interested in IPv4 and decapsulation of protocols such as L2TP, GRE, IPvxinIPvy, etc is not relevant, T2 can easily be configured to do only this. Moreover L4 protocols support is supplied e.g. SCTP which transforms all streams into extra flows, if enabled in networkHeaders.h. We will discuss at the end of the tutorial and in the T2 Kungfu (coming soon).

T2 is capable to produce the following concurrent jobs.

  • Flow
  • Packet
  • Monitoring
  • Alarm

Lets have a quick look at these.

Flow

The most prominent one is flow, where traffic is aggregated into so called flows to process large amount of traffic. A flow is defined in T2 as A and opposite B Flow which are linked by a uniq flowIndex, a 64 bit number. The default aggregation of T2 flows is

(vlan, srcIP, srcPort, dstIP, dstPort, L3protocol)

which covers most cases in corporate networks, as VLANs are very common. It can be extended to

(srcEther, dstEther, ethertype, vlan, srcIP, srcPort, dstIP, dstPort, sctpChannel, L3protocol)

or reduced to aggregating all traffic into a few flows, defining only several networks without vlan, ports and protocols. The advanced flow aggregation modes will be discussed when intoducing T2 core operations in section T2 basic Kungfu

Each plugin added to T2 will produce additional columns in the flow file, producing an output easy to process for any script language or standard tools, such as Excel or SPSS. All this is discussed under

Packet

The packet mode’s output format is as scripting friendly as the flow output and thought as a drill down instrument, which links back to flows and L7 content via the flowIndex. We will discuss it in detail in the tutorial Packet Mode.

Monitoring

Network managers often need certain time sampled parameters, such as number of packets or bandwidth. T2 reports into standard tools. All aspects will be discussed in the tutorial Monitoring.

Alarm

Sometimes L3-4 or content driven rules or even a custom build AI classifier defines what is interesting for the user. Hence, the alarm mode enables each plugin to control flow processing and release to output. This mode is discussed in detail under section Alarm Mode

Hoto Anteater

To get started download Tranalyzer and unpack the tar ball, don’t forget to set the right x and y to the newest release: (BTW: lm means linux, mac tested.)

$ tar -xf tranalyzer2-0.8.2lm1.tar.gz
$ cd /tranalyzer2-0.8.2lm1
$ ls
documentation.pdf  README  setup.sh  trunk

You see the link to the documentation, the README (compilation, dependencies for different OS), the setup script and the trunk, which contains the Core, plugings and scripts. If you are a rookie to T2, use the setup.sh script, it will install all tools, links and environment variables for you and compile T2 with the standard basic plugins.

$ ./setup.sh
...

If compilation fails, it will tell you what is missing, then refer to the README or copy the appropriate dependencies from here. If nothing works, look under faq. if that does not solve your problem write to the Anteater. He will definitely help you.

If setup is successful then you may start T2 for a test:

$ t2 -h
Tranalyzer 0.8.2 (Anteater), Tarantula - High performance flow based network traffic analyzer

Usage:
    tranalyzer [OPTION...] <INPUT$

Input:
    -i IFACE     Listen on interface IFACE
    -r PCAP      Read packets from PCAP file or from stdin if PCAP is "-"
    -R FILE      Process every PCAP file listed in FILE
    -D EXPR[:SCHR][,STOP]
                 Process every PCAP file whose name matches EXPR, up to an
                 optional last index STOP. If STOP is omitted, then Tranalyzer
                 never stops. EXPR can be a filename, e.g., file.pcap0, or an
                 expression, such as "dump*.pcap00", where the star matches
                 anything (note the quotes to prevent the shell from
                 interpreting the expression). SCHR can be used to specify the
                 the last character before the index (default: 'p')

Output:
    -w PREFIX    Append PREFIX to any output file produced. If omitted, then
                 output is diverted to stdout
    -W PREFIX[:SIZE][,START]
                 Like -w, but fragment flow files according to SIZE, producing
                 files starting with index START. SIZE can be specified in bytes
                 (default), KB ('K'), MB ('M') or GB ('G'). Scientific notation,
                 i.e., 1e5 or 1E5 (=100000), can be used as well. If a 'f' is
                 appended, e.g., 10Kf, then SIZE denotes the number of flows.
    -l           Print end report in PREFIX_log.txt instead of stdout
    -s           Packet forensics mode

Optional arguments:
    -p PATH      Load plugins from path PATH instead of ~/.tranalyzer/plugins
    -b FILE      Use plugin list FILE instead of plugin_folder/plugins.txt
    -e FILE      Creates a PCAP file by extracting all packets belonging to
                 flow indexes listed in FILE
    -f FACTOR    Sets hash multiplication factor
    -x ID        Sensor ID
    -c CPU       Bind tranalyzer to one core. If CPU is 0 then OS selects the
                 core to bind
    -F FILE      Read BPF filter from FILE

    -v           Show the version of the program and exit

    -h           Show help options and exit

Remaining arguments:
    BPF          Berkeley Packet Filter command, as in tcpdump

If you cannot wait and like to try it now on an interface, go ahead and use the -i option. Here we will read from pacps, so the -r, -R or -D options are relevant. While the latter two are only being used if more than one pcap is to be analysed, for this tutorial -r is option of choice. The -w option defines where the flow files will be written to. If you omit -w T2 writes to the folder of the pcap. The rest is currently not important.

and create a separate data and results directory, which is good practise for analysis and mining jobs.

$ mkdir ~/data
$ mkdir ~/results
$ cd data

Download the pcap annoloc2.pcap and extract it under your data folder

$ tar -xf annoloc2.pcap
$ cd ~/results

Now apply the pcap to the Anteater:

$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 1061
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: protoStats, 0.8.2
    02: basicFlow, 0.8.3
    03: macRecorder, 0.8.2
    04: portClassifier, 0.8.2
    05: basicStats, 0.8.3
    06: tcpFlags, 0.8.2
    07: tcpStates, 0.8.2
    08: icmpDecode, 0.8.2
    09: connStat, 0.8.2
    10: txtSink, 0.8.2
[INF] basicFlow: IPv4 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 2816170 (2.82 M)
[INF] basicFlow: IPv6 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 36634 (36.63 K)
Processing file: /home/youraccount/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 0.891050 sec
Finished unloading flow memory. Time: 1.277270 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218588 (1.22 M) [99.96%]
Number of IPv6 packets: 180 [0.01%]
Number of A packets: 561580 (561.58 K) [46.07%]
Number of B packets: 657435 (657.43 K) [53.93%]
Number of A bytes: 29273340 (29.27 M) [45.68%]
Number of B bytes: 34809386 (34.81 M) [54.32%]
Average A packet load: 52.13
Average B packet load: 52.95
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 138.212.189.38: 23601 (23.60 K) [1.94%] packets
basicStats: Biggest Talker: 138.212.189.38: 33733962 (33.73 M) [52.64%] bytes
tcpFlags: Aggregated IP anomaly flags : 0x3966
tcpFlags: Aggregated TCP anomaly flags: 0xff47
tcpFlags: Number of TCP scans, succ scans, retries: 238, 671, 933
tcpFlags: Number WinSz below 1: 2415 (2.42 K) [0.25%]
tcpStates: Aggregated anomaly flags: 0xdf
icmpDecode: Number of ICMP echo request packets: 224 [7.32%]
icmpDecode: Number of ICMP echo reply packets: 191 [6.24%]
icmpDecode: ICMP echo reply / request ratio: 0.85
connStat: Max unique number of IP source connections: 4311 (4.31 K)
connStat: Max unique number of IP destination connections: 2922 (2.92 K)
connStat: Max unique number of IP source/destination connections: 182
connStat: Max unique number of source IP / destination port connections: 413
connStat: prtcon/sdcon, prtcon/scon: 2.269231, 0.095801
connStat: Source IP with max connections: 138.212.189.66: 369 connections
connStat: Destination IP with max connections: 138.212.184.235: 403 connections
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 20 [0.00%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17605 (17.61 K)
Number of processed A flows: 9997 (10.00 K) [56.79%]
Number of processed B flows: 7608 (7.61 K) [43.21%]
Number of request     flows: 9469 (9.47 K) [53.79%]
Number of reply       flows: 8136 (8.14 K) [46.21%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.08
Number of processed   packets/flows: 69.24
Number of processed A packets/flows: 56.17
Number of processed B packets/flows: 86.41
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A   packets/s: 22508.91 (22.51 K)
Number of processed   B packets/s: 26350.92 (26.35 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 705.63
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270445248 b/s (270.45 Mb/s)
Max number of flows in memory: 15222 (15.22 K) [5.81%]
Memory usage: 0.45 GB [0.67%]
Aggregate flow status: 0x000018fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] IPv4
[INF] IPv6
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP flows
[INF] Ethernet flows
[INF] ARP flows

T2 produces an end report, which serves as an initial assessment of the pcap content and anomalies. Aftger basic packet, byte statistics each plugin adds some stistical or hex coded info between the —- lines which will be discussed later. Moreover flow based statistics will is supplied to assess the traffic seen on the wire. At the end certain protocol based info and warnings about traffic content will be supplied to alert the user. Thus, an initial assessment is possible without even looking into flows or packets which is essential when dealing with large quantities of traffic.

All plugins reside under trunk and own a src (.h,.c), doc (.tex,pdf) and test (autotesting) directory. Important for now is the doc, where you will find a pdf describing the plugin or you look at the global trunk/doc/documentation.pdf. The rest will be discussed later.

To give you a basic introduction to the traffic mining art using tranalyzer is the primary goal of this tutorial, so lets start with the very basics; have fun!

Basic Flow based Plugins

For beginners let’s start with the very basic flow plugins and only use flow based text output, aka the extended netflow7 flow output:

  • tranalyzer2: Anteater’s core
  • basicFlow: Flow output definition + geo labeling + encapsulation info
  • basicStats: Basic descriptive statistics including Traffic Mining extensions
  • txtSink: Produces a text tab separated file: _flows.txt

so to unload unnecessary compiled plugins invoke:

t2build -u protoStats macRecorder portClassifier tcpFlags tcpStates icmpDecode connStat

Plugin 'protoStats'


Plugin 'macRecorder'


Plugin 'portClassifier'


Plugin 'tcpFlags'


Plugin 'tcpStates'


Plugin 'icmpDecode'


Plugin 'connStat'


BUILD SUCCESSFUL
$

Now restart the Anteater and have a look what changed in the end report:

$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.2 (Anteater), Tarantula. PID: 1389
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.3
    02: basicStats, 0.8.3
    03: txtSink, 0.8.2
[INF] basicFlow: IPv4 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 2816170 (2.82 M)
[INF] basicFlow: IPv6 Ver: 3, Rev: 20190114, Range Mode: 0, subnet ranges loaded: 36634 (36.63 K)
Processing file: /home/youraccount/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 0.521489 sec
Finished unloading flow memory. Time: 0.682393 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218588 (1.22 M) [99.96%]
Number of IPv6 packets: 180 [0.01%]
Number of A packets: 564228 (564.23 K) [46.29%]
Number of B packets: 654787 (654.79 K) [53.71%]
Number of A bytes: 29447896 (29.45 M) [45.95%]
Number of B bytes: 34634830 (34.63 M) [54.05%]
Average A packet load: 52.19
Average B packet load: 52.89
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 138.212.189.38: 23601 (23.60 K) [1.94%] packets
basicStats: Biggest Talker: 138.212.189.38: 33733962 (33.73 M) [52.64%] bytes
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 20 [0.00%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17102 (17.10 K)
Number of processed A flows: 9721 (9.72 K) [56.84%]
Number of processed B flows: 7381 (7.38 K) [43.16%]
Number of request     flows: 9678 (9.68 K) [56.59%]
Number of reply       flows: 7424 (7.42 K) [43.41%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 71.28
Number of processed A packets/flows: 58.04
Number of processed B packets/flows: 88.71
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A   packets/s: 22615.05 (22.61 K)
Number of processed   B packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 685.47
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270445248 b/s (270.45 Mb/s)
Max number of flows in memory: 17102 (17.10 K) [6.52%]
Memory usage: 0.37 GB [0.55%]
Aggregate flow status: 0x000018fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] IPv4
[INF] IPv6
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP flows
[INF] Ethernet flows
[INF] ARP flows

T2 produces an end report, which serves as an initial assessment of the pcap content and anomalies. Each plugin adds some info between the —- lines. BasicStats tell you about the biggest talker regarding traffic volume, which is one of the first features which is relevant for understanding large traffic pcaps. There are also biggest talker in regard to number of connections. We will come back later to that topic under chapter Simple example for Graph output.

So its an old pcap from 2002 afternoon, ipv4/6 + ethernet traffic and the payload is snapped. At the bottom, you see [WRN] and [INFO]. It is decoded from the Aggregate flow status, which denoted the ‘ored’ info from all flows. There are packets snapped down to the L3 header, framgments without header, or end. The difference between the snapped bandwidth and the full raw bandwidth denotes that either the snaplength was small, maybe the default, or somebody actually mangled with the packet content. The average packet load is symmetric for A and B flow, very odd. The protocols being used indicate that the traffic is either corporate or the wild. So if you want good traffic with content for your job, I wouldnt trust that pcap. You can extract way more info, which we will do in the tutorial T2 Kungfu.

T2 produced also the following files

$ ls
annoloc2_flows.txt	annoloc2_headers.txt

The header file contains information about the columns of the flow file, such as name, column position, T2 config, pcap file, etc This information makes it easier to reproduce results from different experiments and it is good doc.

# Date: 1541443480.086865 sec (Mon 05 Nov 2018 19:44:40 CET)
# Tranalyzer 0.8.2 (Anteater), Tarantula.
# sensorID: 666
# PID: 17031
# Command line: ./tranalyzer -r /home/yourname/data/film.pcap -w /home/yourname/result
# HW Info: eierfeile;Linux;4.18.16-arch1-1-ARCH;#1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018;x86_64
#
# Plugins loaded:
# 00: basicFlow, version 0.8.2
# 01: basicStats, version 0.8.1
# 03: txtSink, version 0.8.1
#
# Col No.       Type    Name    Description
1       C       dir     Flow direction
2       U64     flowInd Flow index
3       H64     flowStat        Flow status and warnings
4       U64.U32 timeFirst       Date time of first packet
5       U64.U32 timeLast        Date time of last packet
6       U64.U32 duration        Flow duration
7       U8      numHdrDesc      Number of different headers descriptions
8       U8:R    numHdrs Number of headers (depth) in hdrDesc
9       SC:R    hdrDesc Headers description
10      U16:R   ethVlanID       VLAN IDs
11      IPX     srcIP   Source IP address
12      SC      srcIPCC Source IP country code
13      S       srcIPWho        Source IP who
14      U16     srcPort Source port
15      IPX     dstIP   Destination IP address
...

Now compare it with the flow file, the first flow flowInd 6 is printed below. Until l4Proto the columns originate from basicFlow. After that until bytAsym is basicStats. I picked some interesting flows which demonstrate T2 ops when traffic is mangled with. To extract flows you can use any combination of columns in the flow file.

$ tawk 'bitsanyset($flowStat,0x0000080f00000000)' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc                                                                                              ethVlanID  srcIP                                    srcIPCC  srcIPWho                srcPort  dstIP                                    dstIPCC  dstIPWho                dstPort  l4Proto  numPktsSnt  numPktsRcvd	numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT        aveIAT        stdIAT        pktps       bytps         pktAsm         bytAsm 
A     59       0x0000000200004000  1022171701.692762  1022171701.692762  0.000000   1           3        eth:ipv4:icmp                                                                                        0          138.212.187.10                           jp       "asahi kasei corpora"   0        201.116.148.149                          mx       "--"                    0        1        1           0		28           0             28        28        28          0           0       0             0             0             0           0             1              1
...
A     888      0x0000080200028000  1022171701.810764  1022171701.810764  0.000000   1           4        eth:ipv4:ipv6:UNK(133)                                                                               0          e499:578c:5090:81d0:891b:0:101:80a       --       "--"                    0        514:2343:2e3c:512::c100                  --       "--"                    0        133      1           0            55304       0             55304     55304     55304       0           0       0             0             0             0           0             1              1
...
A     3898     0x0000080a00005000  1022171706.645144  1022171726.589552  19.944408  4           4;4;4;4  eth:ipv4:gre:UNK(0xaa18);eth:ipv4:gre:UNK(0xe805);eth:ipv4:gre:UNK(0x1d11);eth:ipv4:gre:UNK(0x0e0f)  0          201.9.4.49                               br       "--"                    0        138.212.191.213                          jp       "asahi kasei corpora"   0        47       4           5            137 	     256           0         86        34.25       26.7212     0       11.01577      4.986102      3.825814      0.2005575   6.869093      -0.1111111     -0.302799
B     3898     0x0001080a00005001  1022171706.645835  1022171726.447349  19.801514  4           4;4;4;4  eth:ipv4:gre:UNK(0x8b00);eth:ipv4:gre:UNK(0x1400);eth:ipv4:gre:UNK(0x900a);eth:ipv4:gre:UNK(0xe6ef)  0          138.212.191.213                          jp       "asahi kasei corpora"   0        201.9.4.49                               br       "--"                    0        47       5           4            256 	     137           0         234       51.2        81.84808    0       10.97614      3.960303      3.838957      0.252506    12.9283       0.1111111      0.302799
...
A     1041     0x0000000000004000  1022171701.876636  1022171726.639226  24.762590  1           3        eth:ipv4:tcp                                                                                         0          133.26.84.187                            jp       "meiji university"      4766     138.212.187.109                          jp       "asahi kasei corpora"   80       6        1692        2729        0            3970812       0         0         0           0           0       0.614191      0.01463511    0.04658939    68.32888    0             -0.2345623     -1
B     1041     0x0000000200004001  1022171701.877349  1022171726.639232  24.761883  1           3        eth:ipv4:tcp                                                                                         0          138.212.187.109                          jp       "asahi kasei corpora"   80       133.26.84.187                            jp       "meiji university"      4766     6        2729        1692        3970812      0             0         1460      1455.043    70.65761    0       0.480045      0.009073597   0.03961426    110.2097    160359.9      0.2345623      1

If you don’t like tabs as a separator, change “” to any character in utils/bin2txt.h and recompile txtSink.

Nevertheless, tabs are standard separators in most of the bash tools.

We use a lot of hex coded status variables because each info in the flow has to be multiplied by the number of flows T2 has to hold in memory and you will experience that selecting flows will be way easier with hex coding. Each bit has a meaning, please refer to the basicFlow.pdf under doc/ or type

$ tawk -V flowStat=0x0001080a00005001
The flowStat column with value 0x0001080a00005001 is to be interpreted as follows:

   bit | flowStat            | Description
   =============================================================================
     0 | 0x00000000 00000001 | Inverted flow, did not initiate connection
    12 | 0x00000000 00001000 | GRE v1/2
    14 | 0x00000000 00004000 | IPv4
    33 | 0x00000002 00000000 | Acquired packet length < packet length in L3 header
    35 | 0x00000008 00000000 | Acquired packet length < minimal L4 Header
    43 | 0x00000800 00000000 | Stop dissecting
    48 | 0x00010000 00000000 | Header description overrun

A single A Flow can be also the answering flow if the flowStat bit 0 is set. T2 sets this bit according to L4/7 info to the best of his knowledge. We will come back to that topic when discussing icmp flows.

Now try to select flows yourself, lets say all flows of source port 443 and having an acquire packet length issue and where T2 stopped disecting to prevent overrunning the pcap memory. A bitwise AND of flowStat and a mask is required and a selection of srcPort 443:

$ tawk 'bitsanyset($flowStat,0x0000080f00000000) && sport(443)' | tcol
B  4075  0x0000000200004001  1022171707.227811  1022171708.640243  1.412432  1  3  eth:ipv4:tcp  0  70.128.234.203  us  "--"  443  138.212.190.164  jp  "asahi kasei corpora"  1328  6  13  12  3907  917  0  536  300.5385  198.701  0  0.506266  0.1086486  0.118678  9.203983  2766.151  0.04  0.6198176

Play around a bit and you will discover how easily it is to select flows in question.

If you only deal with IP and are only interested in simple Netflow5 output, just cd trunk or use tran, a bash alias.

$ tran
$ cd basicFlow/src
$ vi basicFlow.h

change the following constants to 0

then move to basicStats

$ tran 
$ cd basicStats/src
$ vi basicStats.h

and change these constants to 0 and recompile

Run t2

$ t2build basicFlow basicStats
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...

and here you have your netflow5 output.

$ tcol annoloc2_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration   srcIP                                    srcPort  dstIP                                    dstPort  l4Proto  numPktsSnt  numBytesSnt
A     59       0x0000000200004000  1022171701.692762  1022171701.692762  0.000000   138.212.187.10                           0        201.116.148.149                          0        1        1           28
...
A     888      0x0000080200028000  1022171701.810764  1022171701.810764  0.000000   e499:578c:5090:81d0:891b:0:101:80a       0        514:2343:2e3c:512::c100                  0        133      1           55304
...
A     3898     0x0000080a00005000  1022171706.645144  1022171726.589552  19.944408  201.9.4.49                               0        138.212.191.213                          0        47       4           137
B     3898     0x0000080a00005001  1022171706.645835  1022171726.447349  19.801514  138.212.191.213                          0        201.9.4.49                               0        47       5           256
...
A     1041     0x0000000000004000  1022171701.876636  1022171726.639226  24.762590  133.26.84.187                            4766     138.212.187.109                          80       6        1692        0
B     1041     0x0000000200004001  1022171701.877349  1022171726.639232  24.761883  138.212.187.109                          80       133.26.84.187                            4766     6        2729        3970812

The flowStat you always need to identify flows. If you do not want it, or if you dont dare to edit .h files then just use cut

$ cut -f 1,3- annoloc2_flows.txt

and flowInd and flowStat are gone. There are more tricks with tawk, being discussed in the Post processing with TAWK tutorial.

Using graphviz you can now plot a simple netmap

Reset basicFlow and basicStats to default configuration (flîp the changed bits to 1 and recompile using t2build)

Now we are adding L4 information which does the following jobs:

  • tcpFlags: IP,UDP,TCP aggregated flags and anomaly status
  • tcpStates: TCP statemachine and RFC check, it also terminates tcp flows after a reset or fin

compile them and run T2

$ t2build tcpFlags tcpStates
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
--------------------------------------------------------------------------------
tcpFlags: Anomaly flags IP, TCP: 0x3966, 0xff47
tcpFlags: Number of TCP scans, succ scans, retries: 238, 671, 933
tcpFlags: Number WinSz below 1: 2415 (2.42 K) [0.25%]
tcpStates: Aggregated anomaly flags: 0xdf
--------------------------------------------------------------------------------
...

now you have netflow9/10++.

$ tcol annoloc2_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc                                                                                              ethVlanID  srcIP                                    srcIPCC  srcIPWho                srcPort  dstIP                                    dstIPCC  dstIPWho                dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT   maxIAT        aveIAT        stdIAT        pktps       bytps         pktAsm         bytAsm        tcpFStat  ipMindIPID  ipMaxdIPID  ipMinTTL  ipMaxTTL  ipTTLChg  ipTOS  ipFlags  ipOptCnt  ipOptCpCl_Num    ip6OptCntHH_D  ip6OptHH_D           tcpPSeqCnt  tcpSeqSntBytes  tcpSeqFaultCnt  tcpPAckCnt  tcpFlwLssAckRcvdBytes  tcpAckFaultCnt  tcpInitWinSz  tcpAveWinSz   tcpMinWinSz  tcpMaxWinSz  tcpWinSzDwnCnt  tcpWinSzUpCnt  tcpWinSzChgDirCnt  tcpWinSzMin   tcpFlags  tcpAnomaly  tcpOptPktCnt  tcpOptCnt  tcpOptions  tcpMSS  tcpWS  tcpTmS      tcpTmER     tcpEcI  tcpBtm         tcpSSASAATrip  tcpRTTAckTripMin  tcpRTTAckTripMax  tcpRTTAckTripAve  tcpRTTAckTripJitAve  tcpRTTSseqAA  tcpRTTAckJitAve  tcpStates
... 
A     3943     0x0000080a00005000  1022171706.645144  1022171726.589552  19.944408  4           4;4;4;4  eth:ipv4:gre:UNK(0xaa18);eth:ipv4:gre:UNK(0xe805);eth:ipv4:gre:UNK(0x1d11);eth:ipv4:gre:UNK(0x0e0f)  0          201.9.4.49                               br       "--"                    0        138.212.191.213                          jp       "asahi kasei corpora"   0        47       4           5            137           256           0         86        34.25       26.7212     0       11.01577      4.986102      3.825814      0.2005575   6.869093      -0.1111111     -0.302799      0x0100    4           114         118     118       0         0x00   0x1800   0         0x00_0x00000000  0_0            0x00000000_0x00000000  0           0               0               0           0                      0               0            0             0            0            0               0              0                  0             0x00      0x0000      0             0          0x00000000     0       0      0           0           0   0.000000           0              0.038939          8.783088          2.241058          3.104003             0             -1               0x00
B     3943     0x0001080a00005001  1022171706.645835  1022171726.447349  19.801514  4           4;4;4;4  eth:ipv4:gre:UNK(0x8b00);eth:ipv4:gre:UNK(0x1400);eth:ipv4:gre:UNK(0x900a);eth:ipv4:gre:UNK(0xe6ef)  0          138.212.191.213                          jp       "asahi kasei corpora"   0        201.9.4.49                               br       "--"                    0        47       5           4            256           137           0         234       51.2        81.84808    0       10.97614      3.960303      3.838957      0.252506    12.9283       0.1111111      0.302799       0x0100    1           1           64      64        0         0x00   0x1840   0         0x00_0x00000000  0_0            0x00000000_0x00000000  0           0               0               0           0                      0               0            0             0            0            0               0              0                  0             0x00      0x0000      0             0          0x00000000     0       0      0           0           0   0.000000           0              0.000691          10.97683          2.196429          3.342015             4.437487      4.56113          0x00
...
A     1041     0x0000000000004000  1022171701.876636  1022171726.639226  24.762590  1  	        3        eth:ipv4:tcp                                                                                         0          133.26.84.187                            jp       "meiji university"      4766     138.212.187.109                          jp       "asahi kasei corpora"   80       6        1692        2729         0            3970812        0         0         0           0           0       0.614191      0.01463511    0.04658939   68.32888     0            -0.2345623      -1             0x01c0    1           1173        120     120       0         0x00   0x2804   0         0x00_0x00000000  0_0            0x00000000_0x00000000  1690        0               2               1326        3814580                344             17520        17520         0            17520        126             38             67                 0.003546099   0x90      0x8000      0             0          0x00000000     0       0      0           0           0   0.000000           0              0                 0.335601          0.01491516        0.03908561           0             -1               0x03
B     1041     0x0000000200004001  1022171701.877349  1022171726.639232  24.761883  1           3        eth:ipv4:tcp                                                                                         0          138.212.187.109                          jp       "asahi kasei corpora"   80       133.26.84.187                            jp       "meiji university"      4766     6        2729        1692         3970812      0              0         1460      1455.043    70.65761    0       0.480045      0.009073597   0.03961426   110.2097     160359.9      0.2345623      1              0x0140    1           8           64      64        0         0x00   0x1840   0         0x00_0x00000000  0_0            0x00000000_0x00000000  2653        4535488         64              0           0                      2               6432         6432          6432         6432         0               0              0                  0             0x98      0xb800      0             0          0x00000000     0       0      0           0           0   0.000000           0              0                 0.415909          0.00327119        0.01852777           0.01818635    0.04325463       0x03

Note that between the lines of the endreport now additional aggregated fields appear:

  • tcpFlags: ipFlags, tcpFlags, tcpAnomaly, tcpWinSzMin and tcpStates, all kinds of info for troubleshooting and security purposes
  • tcpStates: aggregated anomaly flags, denoting deviations from RFC The hex numbers denote aggregated anomaly output, where each bit has a specific meaning. They are documented under each plugin folder or under trunk/doc/documents.pdf Another way is just using tawk again:
tawk -V ipFlags=0x3966
The ipFlags column with value 0x3966 is to be interpreted as follows:

   bit | ipFlags | Description
   =============================================================================
     1 | 0x0002  | IPv4 packets out of order
     2 | 0x0004  | IPv4 ID roll over
     5 | 0x0020  | More Fragment bit
     6 | 0x0040  | IPv4: Don't Fragment bit, IPv6: reserve bit
     8 | 0x0100  | Fragmentation position error
    11 | 0x0800  | L4 checksum error
    12 | 0x1000  | L3 header length snapped
    13 | 0x2000  | Packet interdistance = 0
$ tawk -V tcpAnomaly=0xff47
The tcpAnomaly column with value 0xff47 is to be interpreted as follows:

   bit | tcpAnomaly | Description
   =============================================================================
     0 | 0x0001     | FIN-ACK flag
     1 | 0x0002     | SYN-ACK flag
     2 | 0x0004     | RST-ACK flag
     6 | 0x0040     | Null flag, potential NULL scan packet, or malicious channel
     8 | 0x0100     | L4 option field corrupt or not acquired
     9 | 0x0200     | SYN retransmission
    10 | 0x0400     | Sequence Number retry
    11 | 0x0800     | Sequence Number out of order
    12 | 0x1000     | Sequence mess in flow order due to pcap packet loss
    13 | 0x2000     | Sequence number jump forward
    14 | 0x4000     | ACK number out of order
    15 | 0x8000     | Duplicate ACK
$ tawk -V tcpStates=0xdf
The tcpStates column with value 0xdf is to be interpreted as follows:

   bit | tcpStates | Description
   =============================================================================
     0 | 0x01      | Malformed connection establishment
     1 | 0x02      | Malformed teardown
     2 | 0x04      | Malformed flags during established connection
     3 | 0x08      | Packets detected after teardown
     4 | 0x10      | Packets detected after reset
     6 | 0x40      | Reset from sender
     7 | 0x80      | Potential evil behavior (scan)

tcpFlags is built for traffic forensics and troubleshooting. It countains information about L3 header and issues, such as fragmentation, L4 error/flow control and Bandwidth/Round Trip Times(RTT) and some nitty gritty tricks for security guys. Any section can be disabled in tcpFlow.h

For you currently the following extraction of tcpFlags.h is relevant for the beginning:

So you can switch off the RTT estimation, calculation of checksums, the tcp windowsize features or the tricks with tcp seq/ack numbers. Although fragmentaiton in IPv4 today is mostly fishy, if you are not interested switch it off. The code becomes then smaller and faster.

Lets go over the most important fields you need to understand for a start.

There are still OS which increment the IPID by 1. This is a formidable feature to detect the load of a machine. Hence, T2 provides ipMindIPID and ipMaxdIPID column which denotes the min/max difference of IPIDs between packets. If the differences are large and we are sure of the 1 increment, several connections from that IP distribute packets. So, every connection will have jumps / flow. The ipMin/MaxTTL gives you an indication how far your sniffing tap is from the senders IP address and whether several routing paths are involved.

ipFlags contains information about packet abnormalities and fragmentation mishaps. To see the meaning of the bits invoke: tcpFlags is the standard netflow aggregation of the flags in the tcp header. So you can assess the communication state of the flow during observation. The A flow with flowINdex 1041 has the value 0x2804

$ tawk -V ipFlags=0x01c0
The ipFlags column is to be interpreted as follows:

   bit | ipFlags | Description
   =============================================================================
     6 | 0x0040  | IPv4: Don't Fragment bit, IPv6: reserve bit
     7 | 0x0080  | Reserve bit
     8 | 0x0100  | Fragmentation position error

The L4 Checksum error originates from the

$ tawk -V tcpFlags
The tcpFlags column is to be interpreted as follows:

   bit | tcpFlags | Description
   =============================================================================
     0 | 0x01     | FIN: No more data, finish connection
     1 | 0x02     | SYN: Synchronize sequence numbers
     2 | 0x04     | RST: Reset connection
     3 | 0x08     | PSH: Push data
     4 | 0x10     | ACK: Acknowledgement field value valid
     5 | 0x20     | URG: Urgent pointer valid
     6 | 0x40     | ECE: ECN-Echo
     7 | 0x80     | CWR: Congestion Window Reduced flag is set

tcpAnomaly contains flags of combination of flags and abnormalities about sequence numbers.

$ tawk -V tcpAnomaly=0x2804
The tcpAnomaly column is to be interpreted as follows:

   bit | tcpAnomaly | Description
   =============================================================================
     2 | 0x0004     | RST-ACK flag
    11 | 0x0800     | Sequence Number out of order
    13 | 0x2000     | Sequence number jump forward
$ tawk -V tcpStates
The tcpStates column is to be interpreted as follows:

   bit | tcpStates | Description
   =============================================================================
     0 | 0x01      | Malformed connection establishment
     1 | 0x02      | Malformed teardown
     2 | 0x04      | Malformed flags during established connection
     3 | 0x08      | Packets detected after teardown
     4 | 0x10      | Packets detected after reset
     6 | 0x40      | Reset from sender
     7 | 0x80      | Potential evil behavior (scan)

if you want to see the MACs from the sending interface

Basic Traffic volume and connection analysis

To acquire an overview about networks and their communication a graphical output can be helpful. graphviz is a wonderful program to produce al kinds of graphs. T2 supplies a conversion example script grphvz which you may expand for your own purposes.

One basic approach is to look into the connection matrix or simpler the connections beetween nodes. In the script the graph edges are tagged with

  • flowStat direction bit, land of origin, tcpAnomaly, srcPort-dstPort, pktsSnt, bytesSnt.
  • Initiating flow: green, Response Flow: red
  • Width: number of bytesSnt

So apply the already generated flow file to grphvz, convert the resulting .dot file to jpg and display it with eog or better feh. You may also use the interactive program dotty.

$ head -n 43 annoloc2_flows.txt > a_flows.txt
$ grphvz a_flows.txt
$ dotty a_flows_graph.dot

or if you like a picture, use dot

$ dot -Tjpg a_flows_graph.dot -o a_flows_graph.jpg
$ feh a_flows_graph.jpg
graphviz example: extracted the first 43 flows from annoloce_flows.txt
graphviz example: extracted the first 43 flows from annoloce_flows.txt

If we had the full traffic plotted then you could identify large or biggest talkers, just by looking for the arrow with the largest width. But, note that with larger number of flows the performance of graphviz dot degrades rapidly. We produced a netgrapher which can handle very large connection matrices. Unfortunately this is not open source. If you are interested contact us here.

Another method to find biggest talkers is to reverse sort with tawk. Note that the number 4 in the tawk statement below denotes the number of lines to display. If you omit it, all lines will be displayed.

$ tawk 't2sort(numPktsSnt,4)' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP           srcIPCC  srcIPWho               srcPort  dstIP            dstIPCC  dstIPWho               dstPort  l4Proto    srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz ...
B     90       0x0000000200004001  1022171701.699480  1022171726.636773  24.937293  1           3        eth:ipv4:tcp  0          138.212.189.38  jp       "asahi kasei corpora"  139      138.212.86.201   jp       "asahi kasei network"  3429     6        139     23601       12342        33733962     42462         103       1460  ...
A     90       0x0000000200004000  1022171701.699996  1022171726.637210  24.937214  1           3        eth:ipv4:tcp  0          138.212.86.201  jp       "asahi kasei network"  3429     138.212.189.38   jp       "asahi kasei corpora"  139      6        1         00:d0:02:6d:78:00_00:00:21:d2:cc:72_12342  Ditech_SC&C        139            netbios-ssn   12342       23601        42462        33733962      0         63    ...
B     6267     0x0000000200004001  1022171714.045827  1022171722.457644  8.411817   1           3        eth:ipv4:tcp  0          139.45.174.202  us       "nl-legaco-139-45-0-"  56071    138.212.190.117  jp       "asahi kasei corpora"  3837     6        1         00:d0:02:6d:78:00_00:10:5a:c5:96:1a_10159  Ditech_3com        56071          unknown       10159       5692         14821880     0             0         1460  ...
B     3585     0x0000000200004001  1022171705.686717  1022171714.043794  8.357077   1           3        eth:ipv4:tcp  0          139.45.174.202  us       "nl-legaco-139-45-0-"  56070    138.212.190.117  jp       "asahi kasei corpora"  3820     6        1         00:d0:02:6d:78:00_00:10:5a:c5:96:1a_10048  Ditech_3com        56070          unknown       10048       5709         14656900     0             0         1460 ...

or

$ tawk 't2sort(numBytesSnt,4)' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP           srcIPCC  srcIPWho               srcPort  dstIP            dstIPCC  dstIPWho               dstPort  l4Proto  macPairs  srcMac_dstMac_numP                         srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz ...
B     90       0x0000000200004001  1022171701.699480  1022171726.636773  24.937293  1           3        eth:ipv4:tcp  0          138.212.189.38  jp       "asahi kasei corpora"  139      138.212.86.201   jp       "asahi kasei network"  3429     6        1         00:00:21:d2:cc:72_00:d0:02:6d:78:00_23601  SC&C_Ditech        139            netbios-ssn   23601       12342        33733962     42462         103       1460  ...
B     6267     0x0000000200004001  1022171714.045827  1022171722.457644  8.411817   1           3        eth:ipv4:tcp  0          139.45.174.202  us       "nl-legaco-139-45-0-"  56071    138.212.190.117  jp       "asahi kasei corpora"  3837     6        1         00:d0:02:6d:78:00_00:10:5a:c5:96:1a_10159  Ditech_3com        56071          unknown       10159       5692         14821880     0             0         1460  ... 
B     3585     0x0000000200004001  1022171705.686717  1022171714.043794  8.357077   1           3        eth:ipv4:tcp  0          139.45.174.202  us       "nl-legaco-139-45-0-"  56070    138.212.190.117  jp       "asahi kasei corpora"  3820     6        1         00:d0:02:6d:78:00_00:10:5a:c5:96:1a_10048  Ditech_3com        56070          unknown       10048       5709         14656900     0             0         1460  ...
A     326      0x0000000200004000  1022171701.712093  1022171726.638722  24.926629  1           3        eth:ipv4:tcp  0          19.59.134.250   us       "--"                   65230    138.212.187.240  jp       "asahi kasei corpora"  58290    6        1         00:d0:02:6d:78:00_00:50:bf:08:44:81_9459   Ditech_Metallig    58290          unknown       9459        5223         13696632     0             1448      1448 ...
$ t2build connStat
$ t2build -R
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.1 (Anteater), Tarantula. PID: 10282
================================================================================
...
--------------------------------------------------------------------------------
basicStats: Biggest Packet Talker IP: 138.212.189.38, Number of Packets: 23601 (23.60 K)
basicStats: Biggest Byte Talker IP: 138.212.189.38, Number of Bytes: 33733962 (33.73 M)
...
connStat: Max unique number of IP scon, dcon, sdcon, prtcon: 4311 (4.31 K), 2922 (2.92 K), 182, 413
connStat: prtcon/sdcon, prtcon/scon: 2.269231, 0.095801
connStat: Source IP with max connections: 138.212.189.66: 369 connections
connStat: Destination IP with max connections: 138.212.184.235: 403 connections
--------------------------------------------------------------------------------
...
...
$ tawk '{ print $timeFirst, $srcIP, $connSip}' annoloc2_flows.txt | t2plot -t "Simple connStat anomaly graph" -sy 0:250 -sx 0:40 -v 60,75 -r 1 -lz
connStat anomaly graph log scale zoomed: $timeFirst, $srcIP, $connSip
connStat anomaly graph log scale zoomed: $timeFirst, $srcIP, $connSip
$ tawk 'host("138.212.189.66")' annoloc2_flows.txt > annoloc2_ip.txt
$ t2timeline -r -ws 700,400 annoloc2_ip.txt
Timeline of IP 138.212.189.66, annoloce_flows.txt
Timeline of IP 138.212.189.66, annoloce_flows.txt

It adds four columns:

Global statistical plugins

After inspecting the T2 end we have a good overview about the pcap state, certain abnormalities and statistics. As each network has its specific protocol statistics, T2 provides several global plugins which produce specific protocols statistics.

protoStat and icmpDecode are standard to be scrutinized after inspecting the endreport. protoStat generates annoloc2_protocols.txt which is sorted according to Layer2-4 protocol numbers.

$ less annoloc2_protocols.txt
# Total Ether packets captured: 1219015                        
# L2/3 Protocol                          Packets  Percentage   Description
0x0800                                   1218588       99.965  Internet Protocol version 4 (IPv4)
0x0806                                   247            0.020  Address Resolution Protocol (ARP)
0x86dd                                   180            0.015  Internet Protocol version 6 (IPv6)

# Total IPv4 packets captured: 1218588                         
# Total IPv6 packets captured: 180                             
# L4 Protocol                            Packets  Percentage   Description
1                                        3059           0.251  Internet Control Message Protocol
2                                        12             0.001  Internet Group Management Protocol
6                                        948743        77.844  Transmission Control Protocol
17                                       266900        21.899  User Datagram Protocol
22                                       1              0.000  XEROX NS IDP
23                                       1              0.000  Trunk-1
28                                       1              0.000  Internet Reliable Transaction
47                                       20             0.002  General Routing Encapsulation
48                                       1              0.000  Mobile Host Routing Protocol
58                                       11             0.001  ICMP for IPv6
59                                       1              0.000  No Next Header for IPv6
64                                       1              0.000  SATNET and Backroom EXPAK
...

# Port                                   Packets  Percentage   Description
13                                       2              0.000  Daytime (RFC 867)
20                                       120418        12.692  File Transfer [Default Data]
21                                       2082           0.219  File Transfer [Control]
22                                       3793           0.400  The Secure Shell (SSH) Protocol
23                                       309            0.033  Telnet
25                                       134            0.014  Simple Mail Transfer
49                                       175            0.018  Login Host Protocol (TACACS)
53                                       8              0.001  Domain Name Server
65                                       13             0.001  TACACS-Database Service

Here as well the biggest protocol talker is interesting to begin an analysis. The script protStat sorts the protocols file according to number of packets. The -p option defines the lower limit of probability to display, we selected 1%.

$ protStat -p=1 annoloc2_protocols.txt
L2/3 Protocol	Packets	Probability[%]	Description
0x0800	1218588	     99.965	Internet Protocol version 4 (IPv4)

L4 Protocol	Packets	Probability[%]	Description
6	948743	     77.844	Transmission Control Protocol
17	266900	     21.899	User Datagram Protocol

TCP Port	Packets	Probability[%]	Description
139	203627	     21.463	NETBIOS Session Service
20	120418	     12.692	File Transfer [Default Data]
80	73283	      7.724	World Wide Web HTTP
445	27611	      2.910	Microsoft-DS
4662	26586	      2.802	OrbitNet Message Service
1214	20708	      2.183	KAZAA
56071	15851	      1.671	
56070	15757	      1.661	
58290	14682	      1.548	
6699	13711	      1.445	
81	10937	      1.153	Cobalt cube web access or trojan

UDP Port	Packets	Probability[%]	Description
27005	34284	     12.845	FLEX LM (1-10)
27960	24798	      9.291	
7777	15241	      5.710	cbt
28920	14301	      5.358	
10007	11847	      4.439	MVS Capacity
27115	11220	      4.204	
12203	10654	      3.992	
27963	8591	      3.219	
28015	8458	      3.169	
27016	7948	      2.978	
27116	7508	      2.813	
27025	7347	      2.753	
1111	7312	      2.740	LM Social Server
28910	6865	      2.572	
27035	6511	      2.439	
27961	4869	      1.824	
7000	3879	      1.453	file server itself
28901	3619	      1.356	
1028	3570	      1.338	
62626	3364	      1.260	
61996	3324	      1.245	
28001	2984	      1.118	
53	2928	      1.097	Domain Name Server

UDP-Lite Port	Packets	Probability[%]	Description

SCTP Port	Packets	Probability[%]	Description

We have 0.25% icmp traffic, which is not abnormal for that type of traffic. Often it is necessary to look at the icmp messages in detail because some may indicate problems or even malicious behaviour.

$ lsx -S -x 22  annoloc2_icmpStats.txt
Total number of ICMP messages: 3070 (3.07 K) [0.25%]

Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]

ICMP echo reply / request ratio: 0.853

# ICMP Type     	Code    		Packets Percentage
ICMP_ECHOREQUEST        -       		224     7.323
ICMP_ECHOREPLY  	-       		191     6.244
ICMP_SOURCE_QUENCH      -       		0       0.000
ICMP_TRACEROUTE 	-       		0       0.000
ICMP_DEST_UNREACH       ICMP_NET_UNREACH        0       0.000
ICMP_DEST_UNREACH       ICMP_HOST_UNREACH       25      0.817
ICMP_DEST_UNREACH       ICMP_PROT_UNREACH       0       0.000
ICMP_DEST_UNREACH       ICMP_PORT_UNREACH       2603    85.093
ICMP_DEST_UNREACH       ICMP_FRAG_NEEDED        0       0.000
ICMP_DEST_UNREACH       ICMP_SR_FAILED  	0       0.000
ICMP_DEST_UNREACH       ICMP_NET_UNKNOWN        0       0.000
ICMP_DEST_UNREACH       ICMP_HOST_UNKNOWN       0       0.000
ICMP_DEST_UNREACH       ICMP_HOST_ISOLATED      0       0.000
ICMP_DEST_UNREACH       ICMP_NET_ANO    	0       0.000
ICMP_DEST_UNREACH       ICMP_HOST_ANO   	0       0.000
ICMP_DEST_UNREACH       ICMP_NET_UNR_TOS        0       0.000
ICMP_DEST_UNREACH       ICMP_HOST_UNR_TOS       0       0.000
ICMP_DEST_UNREACH       ICMP_PKT_FILTERED       0       0.000
ICMP_DEST_UNREACH       ICMP_PREC_VIOLATION     0       0.000
ICMP_DEST_UNREACH       ICMP_PREC_CUTOFF        0       0.000
ICMP_REDIRECT   	ICMP_REDIR_NET  	0       0.000
ICMP_REDIRECT   	ICMP_REDIR_HOST 	0       0.000
ICMP_REDIRECT   	ICMP_REDIR_NETTOS       0       0.000
ICMP_REDIRECT   	ICMP_REDIR_HOSTTOS      0       0.000
ICMP_TIME_EXCEEDED      ICMP_EXC_TTL    	14      0.458
ICMP_TIME_EXCEEDED      ICMP_EXC_FRAGTIME       2       0.065

# ICMPv6 Type   Code    Packets Percentage
ICMP6_ECHOREQUEST       -       		0       0.000
ICMP6_ECHOREPLY 	-       		0       0.000
ICMP6_PKT_TOO_BIG       -       		0       0.000
ICMP6_DEST_UNREACH      ICMP6_NO_ROUTE  	0       0.000
ICMP6_DEST_UNREACH      ICMP6_COMM_PROHIBIT     0       0.000
ICMP6_DEST_UNREACH      ICMP6_BEYOND_SCOPE      0       0.000
ICMP6_DEST_UNREACH      ICMP6_ADDR_UNREACH      0       0.000
ICMP6_DEST_UNREACH      ICMP6_PORT_UNREACH      0       0.000
ICMP6_DEST_UNREACH      ICMP6_SR_FAILED 	0       0.000
ICMP6_DEST_UNREACH      ICMP6_REJECT    	0       0.000
ICMP6_DEST_UNREACH      ICMP6_ERROR_HDR 	0       0.000
ICMP6_TIME_EXCEEDED     ICMP6_EXC_HOPS  	0       0.000
ICMP6_TIME_EXCEEDED     ICMP6_EXC_FRAGTIME      0       0.000
ICMP6_PARAM_PROBLEM     ICMP6_ERR_HDR   	0       0.000
ICMP6_PARAM_PROBLEM     ICMP6_UNRECO_NEXT_HDR   0       0.000
ICMP6_PARAM_PROBLEM     ICMP6_UNRECO_IP6_OPT    0       0.000
ICMP6_RTER_ADVERT       -       		5       45.455
ICMP6_NBOR_SOLICIT      -       		3       27.273
ICMP6_NBOR_ADVERT       -       		3       27.273

Now lets find all host who send icmp messages:

$ tawk 'icmp()' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc        ethVlanID  srcIP            srcIPCC  srcIPWho               srcPort  dstIP            dstIPCC  dstIPWho               dstPort  l4Proto  tcpStates  icmpStat  icmpTCcnt  icmpBFTypH_TypL_Code          icmptmgtw   icmpEchoSuccRatio  icmpPFindex
A     59       0x0000000200004001  1022171701.692762  1022171701.692762  0.000000   1           3        eth:ipv4:icmp  0          138.212.187.10   jp       "asahi kasei corpora"  0        201.116.148.149  mx       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  0
A     896      0x0000000200004001  1022171701.812425  1022171701.812425  0.000000   1           3        eth:ipv4:icmp  0          138.212.189.88   jp       "asahi kasei corpora"  0        201.116.161.83   mx       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  893
A     1073     0x0000000200004001  1022171701.889357  1022171701.889357  0.000000   1           3        eth:ipv4:icmp  0          138.212.184.71   jp       "asahi kasei corpora"  0        146.208.9.41     us       "arin"                 0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1056
A     1181     0x0000000200004001  1022171701.956543  1022171701.956543  0.000000   1           3        eth:ipv4:icmp  0          201.118.86.105   mx       "--"                   0        138.212.189.66   jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0002  0x00000000  0                  1170
A     1208     0x0000000200004001  1022171701.980834  1022171701.980834  0.000000   1           3        eth:ipv4:icmp  0          138.213.40.91    ff       "apnic"                0        138.212.189.66   jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1184
A     1236     0x0000000200004001  1022171702.009674  1022171702.009674  0.000000   1           3        eth:ipv4:icmp  0          138.212.184.71   jp       "asahi kasei corpora"  0        36.237.77.156    tw       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1226
A     1561     0x0000000200004001  1022171702.247453  1022171702.247453  0.000000   1           3        eth:ipv4:icmp  0          138.212.186.88   jp       "asahi kasei corpora"  0        201.19.77.72     br       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1559
A     1576     0x0000000200004001  1022171702.265015  1022171702.265015  0.000000   1           3        eth:ipv4:icmp  0          138.212.191.25   jp       "asahi kasei corpora"  0        19.50.144.156    us       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1574
A     1722     0x0000000200004001  1022171702.396273  1022171702.396273  0.000000   1           3        eth:ipv4:icmp  0          138.212.190.25   jp       "asahi kasei corpora"  0        19.6.20.159      us       "searched the apnic "  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1715
A     1745     0x0000000200004001  1022171702.417049  1022171702.417049  0.000000   1           3        eth:ipv4:icmp  0          138.212.187.10   jp       "asahi kasei corpora"  0        65.171.40.80     ff       "sprint"               0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1744
A     1754     0x0000000200004001  1022171702.423157  1022171702.423157  0.000000   1           3        eth:ipv4:icmp  0          138.212.187.10   jp       "asahi kasei corpora"  0        193.108.29.243   lv       "ripencc"              0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1752
A     1824     0x0000000200004001  1022171702.510250  1022171702.510250  0.000000   1           3        eth:ipv4:icmp  0          138.212.187.10   jp       "asahi kasei corpora"  0        138.213.33.28    ff       "apnic"                0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1819
A     1881     0x0000000200004001  1022171722.772690  1022171722.785414  0.012724   1           3        eth:ipv4:icmp  0          193.133.161.22   gb       "--"                   0        138.212.191.75   jp       "asahi kasei corpora"  0        1        0x00       0x01      9          0x00000000_0x00000008_0x0008  0x00000000  0                  7889
B     1881     0x0000000200004001  1022171702.597916  1022171702.597916  0.000000   1           3        eth:ipv4:icmp  0          138.212.191.75   jp       "asahi kasei corpora"  0        193.133.161.22   gb       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1880
A     1909     0x0000000200004001  1022171702.623420  1022171702.623420  0.000000   1           3        eth:ipv4:icmp  0          201.74.106.234   br       "--"                   0        138.212.187.11   jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1876
A     1990     0x0000000200004001  1022171702.721365  1022171702.721365  0.000000   1           3        eth:ipv4:icmp  0          139.97.6.149     fi       "elisa oyj"            0        138.212.189.66   jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1978
A     1999     0x0000000200004001  1022171702.739522  1022171702.739522  0.000000   1           3        eth:ipv4:icmp  0          138.212.189.88   jp       "asahi kasei corpora"  0        216.218.79.22    us       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  1998
A     2040     0x0000000200004001  1022171702.768754  1022171702.768754  0.000000   1           3        eth:ipv4:icmp  0          138.212.187.10   jp       "asahi kasei corpora"  0        201.108.14.212   mx       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000008_0x0008  0x00000000  0                  2014
A     2066     0x0000000200004000  1022171702.799287  1022171702.799287  0.000000   1           3        eth:ipv4:icmp  0          70.101.52.210    us       "--"                   0        138.212.184.246  jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x99bb0002  1                  0

By scrolling to the right you see the icmpBFTypH_TypL_Code bit field. So we are interested in ICMP_HOST_UNREACH and ICMP_PORT_UNREACH. So the 3rd should be 3

$ tawk '{ split($icmpBFTypH_TypL_Code,A,"_"); if (bitsanyset(A[3],0x3)) print }' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc          ethVlanID  srcIP                                    srcIPCC  srcIPWho               srcPort  dstIP                                    dstIPCC  dstIPWho               dstPort  l4Proto  tcpStates  icmpStat  icmpTCcnt  icmpBFTypH_TypL_Code          icmptmgtw   icmpEchoSuccRatio  icmpPFindex
A     1181     0x0000000200004001  1022171701.956543  1022171701.956543  0.000000   1           3        eth:ipv4:icmp    0          201.118.86.105                           mx       "--"                   0        138.212.189.66                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0002  0x00000000  0                  1170
A     2066     0x0000000200004000  1022171702.799287  1022171702.799287  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.246                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x99bb0002  1                  0
B     2066     0x0000000200004001  1022171702.799877  1022171702.799877  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.246                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000001_0x0001  0x99bb0002  0                  0
A     2067     0x0000000200004000  1022171702.800596  1022171702.800596  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.247                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x99bc0002  1                  0
B     2067     0x0000000200004001  1022171702.800830  1022171702.800830  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.247                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000001_0x0001  0x99bc0002  0                  0
A     2069     0x0000000200004000  1022171702.801985  1022171702.801985  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.244                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x99bd0002  1                  0
B     2069     0x0000000200004001  1022171702.803416  1022171702.803416  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.244                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000001_0x0001  0x99bd0002  0                  0
A     2751     0x0000000200004000  1022171703.870541  1022171703.870541  0.000000   1           3        eth:ipv4:icmp    0          138.212.189.44                           jp       "asahi kasei corpora"  0        201.98.147.38                            mx       "--"                   0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x00006ac3  1                  0
B     2751     0x0000000200004001  1022171703.898733  1022171703.898733  0.000000   1           3        eth:ipv4:icmp    0          201.98.147.38                            mx       "--"                   0        138.212.189.44                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000001_0x0001  0x00006ac3  0                  0
A     2777     0x0000000200004000  1022171703.912653  1022171703.912653  0.000000   1           3        eth:ipv4:icmp    0          138.212.189.44                           jp       "asahi kasei corpora"  0        217.12.211.19                            ua       "itl company"          0        1        0x00       0x01      1          0x00000000_0x00000100_0x0001  0x00006ac4  1                  0
B     2777     0x0000000200004001  1022171703.918949  1022171703.918949  0.000000   1           3        eth:ipv4:icmp    0          217.12.211.19                            ua       "itl company"          0        138.212.189.44                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000001_0x0001  0x00006ac4  0                  0
A     1581     0x0000000200004000  1022171702.276213  1022171704.296096  2.019883   1           3        eth:ipv4:icmp    0          200.83.66.22                             cl       "--"                   0        138.212.188.197                          jp       "asahi kasei corpora"  0        1        0x00       0x01      3          0x00000000_0x00000100_0x0001  0x0004cf6c  1                  0
B     1581     0x0000000200004001  1022171702.276503  1022171704.296913  2.020410   1           3        eth:ipv4:icmp    0          138.212.188.197                          jp       "asahi kasei corpora"  0        200.83.66.22                             cl       "--"                   0        1        0x00       0x01      3          0x00000000_0x00000001_0x0001  0x0004cf6c  0                  0
A     3131     0x0000000200004001  1022171704.596259  1022171704.596259  0.000000   1           3        eth:ipv4:icmp    0          200.9.115.105                            jm       "arin"                 0        138.212.189.66                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000800_0x0001  0x00000000  0                  3120
A     3710     0x0000000200004001  1022171706.008768  1022171706.008768  0.000000   1           3        eth:ipv4:icmp    0          138.212.190.107                          jp       "asahi kasei corpora"  0        55.54.217.39                             us       "dod network informa"  0        1        0x00       0x01      1          0x00000000_0x00000008_0x0002  0x00000000  0                  2196
A     1751     0x0000100200004000  1022171702.420004  1022171706.422258  4.002254   1           3        eth:ipv4:icmp    0          138.212.189.177                          jp       "asahi kasei corpora"  0        138.212.109.236                          jp       "asahi kasei network"  0        1        0x00       0x01      5          0x00000000_0x00000100_0x0001  0x00043ce9  1                  0
B     1751     0x0000000200004001  1022171702.420110  1022171706.422380  4.002270   1           3        eth:ipv4:icmp    0          138.212.109.236                          jp       "asahi kasei network"  0        138.212.189.177                          jp       "asahi kasei corpora"  0        1        0x00       0x01      5          0x00000000_0x00000001_0x0001  0x00043ce9  0                  0
A     3888     0x0000000200008001  1022171706.464670  1022171706.464670  0.000000   1           3        eth:ipv6:icmpv6  0          2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf  --       "--"                   0        fe80::31e1:c7ff:d5fa:684c                01       "private_reserved"     0        58       0x00       0x01      1          0x00000100_0x00000000_0x0001  0x00000000  0                  0
B     3888     0x0000000200008001  1022171706.464331  1022171706.464331  0.000000   1           3        eth:ipv6:icmpv6  0          fe80::31e1:c7ff:d5fa:684c                01       "private_reserved"     0        2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf  --       "--"                   0        58       0x00       0x01      1          0x00000080_0x00000000_0x0001  0x00000000  0                  0
A     1971     0x0000100200004000  1022171702.703042  1022171706.951177  4.248135   1           3        eth:ipv4:icmp    0          138.212.189.172                          jp       "asahi kasei corpora"  0        219.41.251.166                           jp       "--"                   0        1        0x00       0x01      6          0x00000000_0x00000100_0x0001  0x000259cd  1                  0
B     1971     0x0000000200004001  1022171702.709337  1022171706.957427  4.248090   1           3        eth:ipv4:icmp    0          219.41.251.166                           jp       "--"                   0        138.212.189.172                          jp       "asahi kasei corpora"  0        1        0x00       0x01      6          0x00000000_0x00000001_0x0001  0x000259cd  0                  0
A     3160     0x0000000200004001  1022171704.641708  1022171707.619668  2.977960   1           3        eth:ipv4:icmp    0          58.170.137.34                            au       "--"                   0        138.212.190.120                          jp       "asahi kasei corpora"  0        1        0x00       0x01      2          0x00000000_0x00000800_0x0001  0x00000000  0                  3065
A     4385     0x0000000200008001  1022171707.757245  1022171707.757245  0.000000   1           3        eth:ipv6:icmpv6  0          2001:70e8:d3ce:e200:de29:8cff:c041:14c7  --       "--"                   0        fe80::31c2:57ff:e822:64da                01       "private_reserved"     0        58       0x00       0x01      1          0x00000100_0x00000000_0x0001  0x00000000  0                  0
B     4385     0x0000000200008001  1022171707.756560  1022171707.756560  0.000000   1           3        eth:ipv6:icmpv6  0          fe80::31c2:57ff:e822:64da                01       "private_reserved"     0        2001:70e8:d3ce:e200:de29:8cff:c041:14c7  --       "--"                   0        58       0x00       0x01      1          0x00000080_0x00000000_0x0001  0x00000000  0                  0
A     4564     0x0000000200004001  1022171708.319511  1022171708.319511  0.000000   1           3        eth:ipv4:icmp    0          138.212.249.49                           jp       "--"                   0        138.212.186.108                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          0x00000000_0x00000800_0x0001  0x00000000  0                  4450

The bitfields are useful for selecting flows, but if you like a bit more human readability, set **ICMP_TC_MD* to 0, recompile and rerun T2.

$ t2build icmpDecode
...
BUILD SUCCESSFUL

$ t2 -r ~/data/annoloc2.pcap -w ~/results
$ tawk '{ if ($icmpTCcnt > 0) print }' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc          ethVlanID  srcIP                                    srcIPCC  srcIPWho               srcPort  dstIP                                    dstIPCC  dstIPWho               dstPort  l4Proto  tcpStates  icmpStat  icmpTCcnt  icmpType_Code                            icmptmgtw   icmpEchoSuccRatio  icmpPFindex
A     59       0x0000000200004001  1022171701.692762  1022171701.692762  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        201.116.148.149                          mx       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  0
A     896      0x0000000200004001  1022171701.812425  1022171701.812425  0.000000   1           3        eth:ipv4:icmp    0          138.212.189.88                           jp       "asahi kasei corpora"  0        201.116.161.83                           mx       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  893
A     1073     0x0000000200004001  1022171701.889357  1022171701.889357  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.71                           jp       "asahi kasei corpora"  0        146.208.9.41                             us       "arin"                 0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1056
A     1181     0x0000000200004001  1022171701.956543  1022171701.956543  0.000000   1           3        eth:ipv4:icmp    0          201.118.86.105                           mx       "--"                   0        138.212.189.66                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          3_1                                      0x00000000  0                  1170
A     1208     0x0000000200004001  1022171701.980834  1022171701.980834  0.000000   1           3        eth:ipv4:icmp    0          138.213.40.91                            ff       "apnic"                0        138.212.189.66                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1184
A     1236     0x0000000200004001  1022171702.009674  1022171702.009674  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.71                           jp       "asahi kasei corpora"  0        36.237.77.156                            tw       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1226
A     1561     0x0000000200004001  1022171702.247453  1022171702.247453  0.000000   1           3        eth:ipv4:icmp    0          138.212.186.88                           jp       "asahi kasei corpora"  0        201.19.77.72                             br       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1559
A     1576     0x0000000200004001  1022171702.265015  1022171702.265015  0.000000   1           3        eth:ipv4:icmp    0          138.212.191.25                           jp       "asahi kasei corpora"  0        19.50.144.156                            us       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1574
A     1722     0x0000000200004001  1022171702.396273  1022171702.396273  0.000000   1           3        eth:ipv4:icmp    0          138.212.190.25                           jp       "asahi kasei corpora"  0        19.6.20.159                              us       "searched the apnic "  0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1715
A     1745     0x0000000200004001  1022171702.417049  1022171702.417049  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        65.171.40.80                             ff       "sprint"               0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1744
A     1754     0x0000000200004001  1022171702.423157  1022171702.423157  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        193.108.29.243                           lv       "ripencc"              0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1752
A     1824     0x0000000200004001  1022171702.510250  1022171702.510250  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        138.213.33.28                            ff       "apnic"                0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1819
A     1881     0x0000000200004001  1022171722.772690  1022171722.785414  0.012724   1           3        eth:ipv4:icmp    0          193.133.161.22                           gb       "--"                   0        138.212.191.75                           jp       "asahi kasei corpora"  0        1        0x00       0x01      9          3_3;3_3;3_3;3_3;3_3;3_3;3_3;3_3;3_3      0x00000000  0                  7889
B     1881     0x0000000200004001  1022171702.597916  1022171702.597916  0.000000   1           3        eth:ipv4:icmp    0          138.212.191.75                           jp       "asahi kasei corpora"  0        193.133.161.22                           gb       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1880
A     1909     0x0000000200004001  1022171702.623420  1022171702.623420  0.000000   1           3        eth:ipv4:icmp    0          201.74.106.234                           br       "--"                   0        138.212.187.11                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1876
A     1990     0x0000000200004001  1022171702.721365  1022171702.721365  0.000000   1           3        eth:ipv4:icmp    0          139.97.6.149                             fi       "elisa oyj"            0        138.212.189.66                           jp       "asahi kasei corpora"  0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1978
A     1999     0x0000000200004001  1022171702.739522  1022171702.739522  0.000000   1           3        eth:ipv4:icmp    0          138.212.189.88                           jp       "asahi kasei corpora"  0        216.218.79.22                            us       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  1998
A     2040     0x0000000200004001  1022171702.768754  1022171702.768754  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        201.108.14.212                           mx       "--"                   0        1        0x00       0x01      1          3_3                                      0x00000000  0                  2014
A     2066     0x0000000200004000  1022171702.799287  1022171702.799287  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.246                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          8_0                                      0x99bb0002  1                  0
B     2066     0x0000000200004001  1022171702.799877  1022171702.799877  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.246                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0_0                                      0x99bb0002  0                  0
A     2067     0x0000000200004000  1022171702.800596  1022171702.800596  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.247                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          8_0                                      0x99bc0002  1                  0
B     2067     0x0000000200004001  1022171702.800830  1022171702.800830  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.247                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0_0                                      0x99bc0002  0                  0
A     2069     0x0000000200004000  1022171702.801985  1022171702.801985  0.000000   1           3        eth:ipv4:icmp    0          70.101.52.210                            us       "--"                   0        138.212.184.244                          jp       "asahi kasei corpora"  0        1        0x00       0x01      1          8_0                                      0x99bd0002  1                  0
B     2069     0x0000000200004001  1022171702.803416  1022171702.803416  0.000000   1           3        eth:ipv4:icmp    0          138.212.184.244                          jp       "asahi kasei corpora"  0        70.101.52.210                            us       "--"                   0        1        0x00       0x01      1          0_0                                      0x99bd0002  0                  0
A     2095     0x0000000200004001  1022171702.842744  1022171702.845077  0.002333   1           3        eth:ipv4:icmp    0          138.212.187.74                           jp       "asahi kasei corpora"  0        19.229.57.104                            us       "--"                   0        1        0x00       0x01      2          3_3;3_3                                  0x00000000  0                  2093
A     2127     0x0000000200004001  1022171702.888068  1022171702.888068  0.000000   1           3        eth:ipv4:icmp    0          138.212.187.10                           jp       "asahi kasei corpora"  0        55.49.54.100                             ff       "dod network informa"  0        1        0x00       0x01      1          3_3                                      0x00000000  0                  2125

Add layer 2/4 information

$ t2build macRecorder portClassifier
...
BUILD SUCCESSFUL

$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
$

In the flow file below you will now see from the macRecorder plugin all mac addresses including packet counts per flow. If redundant routing is presents you will see minimum two mac pairs per flow. In that case the counts should be almost equal, if not then something is wrong. Moreover the manufacturer of the interface card is listed, so that the user does not need to look it up on the web. The portClassifier is somewhat misleading, it does not classifier, it transforms the port number into a human readable string, such as port 443 -> https in our case.

$ tcol annoloc2_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration    numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP            srcIPCC  srcIPWho            srcPort  dstIP            dstIPCC  dstIPWho            dstPort  l4Proto  macPairs  srcMac_dstMac_numP                         srcManuf_dstManuf  dstPortClassN  dstPortClass  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz ... 
A     6        0x0000000000004000  1440190879.558960  1440190889.687103  10.128143   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49499    212.179.154.238  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_10     IntelCor_D-LinkIn  443            https         10          12           499          4771          0         291      ...
B     6        0x0000000000004001  1440190879.568111  1440190889.698269  10.130158   1           3        eth:ipv4:tcp  0          212.179.154.238  il       "--"                443      10.0.0.4         00       "private_reserved"  49499    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_12     D-LinkIn_IntelCor  443            https         12          10           4771         499           0         1360     
...
A     5        0x0000000000004000  1440190879.556703  1440190889.687411  10.130708   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49498    212.179.154.238  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_8      IntelCor_D-LinkIn  443            https         8           11           499          4772          0         291     .... 
B     5        0x0000000000004001  1440190879.566954  1440190889.698269  10.131315   1           3        eth:ipv4:tcp  0          212.179.154.238  il       "--"                443      10.0.0.4         00       "private_reserved"  49498    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_11     D-LinkIn_IntelCor  443            https         11          8            4772         499           0         1360     ...
A     7        0x0000000000004000  1440190879.560857  1440190889.687649  10.126792   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49502    212.179.154.238  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_9      IntelCor_D-LinkIn  443            https         9           12           499          4772          0         291      ...
B     7        0x0000000000004001  1440190879.569496  1440190889.700475  10.130979   1           3        eth:ipv4:tcp  0          212.179.154.238  il       "--"                443      10.0.0.4         00       "private_reserved"  49502    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_12     D-LinkIn_IntelCor  443            https         12          9            4772         499           0         1360     ...
A     4        0x0000000000004000  1440190879.555095  1440190889.687856  10.132761   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49497    212.179.154.238  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_8      IntelCor_D-LinkIn  443            https         8           11           499          4773          0         291      ...
B     4        0x0000000000004001  1440190879.564047  1440190889.700475  10.136428   1           3        eth:ipv4:tcp  0          212.179.154.238  il       "--"                443      10.0.0.4         00       "private_reserved"  49497    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_11     D-LinkIn_IntelCor  443            https         11          8            4773         499           0         1360     ...
A     21       0x0000000000004000  1440190886.063492  1440190887.890521  1.827029    1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49518    74.125.9.136     us       "arin"              443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_9      IntelCor_D-LinkIn  443            https         9           15           1107         10638         0         756      ...
B     21       0x0000000000004001  1440190886.316484  1440190887.890432  1.573948    1           3        eth:ipv4:tcp  0          74.125.9.136     us       "arin"              443      10.0.0.4         00       "private_reserved"  49518    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_15     D-LinkIn_IntelCor  443            https         15          9            10638        1107          0         1360     ...
A     9        0x0000000000004000  1440190879.571118  1440190899.687757  20.116639   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49504    212.179.154.251  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_9      IntelCor_D-LinkIn  443            https         9           11           336          4736          0         210      ...
B     9        0x0000000000004001  1440190879.579572  1440190899.714085  20.134513   1           3        eth:ipv4:tcp  0          212.179.154.251  il       "--"                443      10.0.0.4         00       "private_reserved"  49504    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_11     D-LinkIn_IntelCor  443            https         11          9            4736         336           0         1360     ...
A     10       0x0000000000004000  1440190879.622028  1440190899.688092  20.066064   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49505    212.179.154.238  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_8      IntelCor_D-LinkIn  443            https         8           12           499          4772          0         291      ...
B     10       0x0000000000004001  1440190879.632873  1440190899.718657  20.085784   1           3        eth:ipv4:tcp  0          212.179.154.238  il       "--"                443      10.0.0.4         00       "private_reserved"  49505    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_12     D-LinkIn_IntelCor  443            https         12          8            4772         499           0         1360     ...
A     22       0x0000000000004000  1440190886.313335  1440190897.002695  10.689360   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49519    74.125.9.136     us       "arin"              443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_4      IntelCor_D-LinkIn  443            https         4           2            0            0             0         0        ...
B     22       0x0000000000004001  1440190886.660728  1440190897.002648  10.341920   1           3        eth:ipv4:tcp  0          74.125.9.136     us       "arin"              443      10.0.0.4         00       "private_reserved"  49519    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_2      D-LinkIn_IntelCor  443            https         2           4            0            0             0         0        ...
A     12       0x0000000000004000  1440190880.013416  1440190929.689786  49.676370   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49507    212.179.154.206  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_1874   IntelCor_D-LinkIn  443            https         1874        4269         9911         5775002       0         1289     ...
B     12       0x0000000000004001  1440190880.024211  1440190929.699327  49.675116   1           3        eth:ipv4:tcp  0          212.179.154.206  il       "--"                443      10.0.0.4         00       "private_reserved"  49507    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_4269   D-LinkIn_IntelCor  443            https         4269        1874         5775002      9911          0         1360     ...
A     13       0x0000000000004000  1440190880.013773  1440190949.690199  69.676426   1           3        eth:ipv4:tcp  0          10.0.0.4         00       "private_reserved"  49508    212.179.154.206  il       "--"                443      6        1         28:b2:bd:1c:51:6f_f8:e9:03:0e:57:13_4366   IntelCor_D-LinkIn  443            https         4366        12156        15090        16501679      0         1292     ...
B     13       0x0000000000004001  1440190880.024391  1440190949.698813  69.674422   1           3        eth:ipv4:tcp  0          212.179.154.206  il       "--"                443      10.0.0.4         00       "private_reserved"  49508    6        1         f8:e9:03:0e:57:13_28:b2:bd:1c:51:6f_12156  D-LinkIn_IntelCor  443            https         12156       4366         16501679     15090         0         1360    ....
... 

Now got a quick insight into basic plugins. You can now start using T2 on your own pcaps or look into other tutorials about specifics of traffic mining, or specific plugins. Have fun!!

Operational mode switching: ETH, IPv4/6, SCTP

T2 can operate in several operational modes. default is dual IP stack + L2 ETH flow production. In order to accelerate T2 it can be switched into IPv4 or IPv6 mode or only into a plain L2 flow/packet producer depending on your demands or your network.

Search for user defines in networkHeaders.h and have a look at the default settings:

Moreover SCTP to flow transformation is supported. Which is by default disabled, because it adds addition code, the standard admin does not need. The researcher or protocol expert might need that functionality, so set SCTP_ACTIVAT 1. The constant SCTP_STATFINDEX controls whether all SCTP streams sorted into several flows with the same flow Index or different incrementing flow indexes.

compile all plugins, as you may have plugins which implement the SCTP flow segregation, e.g. sctpDecode.

$ t2build -R
...
$

and run T2 with your sctp pcap. Unfortunately I do not have one for you here which is annonymized. But I’m working on it. You got now a quick insight into T2 functionality, basic plugin opertions and workflow. You can now start using T2 on your own pcaps or look into other tutorials about specifics of your interest. Have fun!!