Tutorial: All about global statistics

Introduction

Using tawk any statistics can be produced from the flow file. Nevertheless, T2 supplies several statistics in the end report, flow / packet file and extra protocol based files because it is faster and was useful to us in troubleshooting, security and forensic. The latter files will be discussed here, and links to further tutorials are supplied

Preparation

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile the following plugins:

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow basicStats icmpDecode igmpDecode ospfDecode nDPI vrrpDecode tcpStates txtSink
...
BUILD SUCCESSFUL

If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:

$ mkdir ~/data ~/results
$

The anonymized sample PCAP used in this tutorial can be downloaded here: annoloc2.pcap Please extract it under your data folder. Now you are all set for T2 IP label experiments.

Basic L3/4 packet/length statistics

Basic packet and packet length statistics is readily available in the end report as indicated below. Hence, a first impression of the relevance of a pcap without looking into flow files is possible. Besides basic L3/4 statistics, every plugin produces its own statistics in the end report between the “—-” lines.

$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.8 (Anteater), Tarantula. PID: 394742
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.8
    02: nDPI, 0.8.8
    03: basicStats, 0.8.8
    04: tcpStates, 0.8.8
    05: icmpDecode, 0.8.8
    06: ospfDecode, 0.8.8
    07: igmpDecode, 0.8.8
    08: vrrpDecode, 0.8.8
    09: txtSink, 0.8.8
[INF] IPv4 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 405499 (405.50 K)
[INF] IPv6 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 49666 (49.67 K)
Processing file: /home/user/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 0.808593 sec
Finished unloading flow memory. Time: 1.108053 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218608 (1.22 M) [99.97%]
Number of IPv6 packets: 160 [0.01%]
Number of A packets: 561591 (561.59 K) [46.07%]
Number of B packets: 657424 (657.42 K) [53.93%]
Number of A bytes: 29274086 (29.27 M) [45.68%]
Number of B bytes: 34808640 (34.81 M) [54.32%]
Average A packet load: 52.13
Average B packet load: 52.95
--------------------------------------------------------------------------------
nDPI: Number of flows classified: 4130 (4.13 K) [23.48%]
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
tcpStates: Aggregated tcpStates anomalies: 0xdf
icmpDecode: icmpStat: 0x21
icmpDecode: Number of ICMP echo request packets: 224 [7.32%]
icmpDecode: Number of ICMP echo reply packets: 191 [6.24%]
icmpDecode: ICMP echo reply / request ratio: 0.85
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 247 [0.02%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of TCP bytes: 52643546 (52.64 M) [82.15%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of UDP bytes: 11234272 (11.23 M) [17.53%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17589 (17.59 K)
Number of processed A flows: 9980 (9.98 K) [56.74%]
Number of processed B flows: 7609 (7.61 K) [43.26%]
Number of request     flows: 9452 (9.45 K) [53.74%]
Number of reply       flows: 8137 (8.14 K) [46.26%]
Total   A/B    flow asymmetry: 0.13
Total req/rply flow asymmetry: 0.07
Number of processed   packets/flows: 69.31
Number of processed A packets/flows: 56.27
Number of processed B packets/flows: 86.40
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B   packets/s: 48859.83 (48.86 K)
Number of processed A     packets/s: 22509.36 (22.51 K)
Number of processed   B   packets/s: 26350.48 (26.35 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 704.99
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270268576 b/s (270.27 Mb/s)
Max number of flows in memory: 15206 (15.21 K) [5.80%]
Memory usage: 0.11 GB [0.23%]
Aggregate flow status: 0x0c0098fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 payload length > framing length
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] Ethernet flows
[INF] IPv4 flows
[INF] IPv6 flows
[INF] ARP
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP
$

Oups, no VRRP and OSPF messages….

Anyway, here is a list of all plugins producing extra global statistics information:

plugin files
protoStats _flow.*, _proto.txt, end report
basicStats _flow.*, _end report
icmpDecode _flow.*, _icmp.txt, _end report
igmpDecode _flow.*, _igmp.txt, _end report
ospfDecode _flow.*, _ospfStats.txt, _ospfHello.txt, _end report
inDPI _flow.*, _nDPI.txt, _end report
vrrpDecode _flow.*, _vrrp.txt, _end report

protoStats

This plugin is discussed in detail in global statistical plugins.

Detailed statistics

In the following plugins are listed which produce a more detailed view of certain important plugins.

ICMP statistics

Internet Control Message Protocol (ICMP) diagnostic or control icmpDecode

IGMP statistics

Internet Group Management Protocol (IGMP) is a communications protocol to manage multicast group memberships e.g. for streaming services.

OSPF statistics

Open Shortest Path First (OSPF) is an interior routing protocol based on Dijkstra’s algorithm. It produces three extra summary files besides the flow file output.

As we currently cannot disseminate a pcap which contains relevant OSPF messsges only the results will be shown.

First a statistics is supplied of all OSPF messages in the *_ospfStat.txt* file:

$ t2 -r ~/data/nudel.cap -w ~/results/
...
$ tcol nudel_ospfStat.txt
Total # of OSPF packets: 952
OSPF / Total traffic [%]: 100.000
Link State Request / Update ratio [%]: 0.000
Link State Update / Acknowledgment ratio [%]: 566.667
Number of multicast packets: 952
Number of packets with null authentication: 0
Number of packets with password authentication: 0
Number of packets with cryptographic authentication: 952
Number of packets with unknown authentication: 0
Number of packets with bad TTL: 0
Number of packets with bad dest: 0
Number of packets with bad type: 0
OSPF Type                                                 # of packets  Relative Frequency [%]
Hello                                                     932           97.899
DBD                                                       0             0.000
LSReq                                                     0             0.000
LSUp                                                      17            1.786
LSAck                                                     3             0.315
$

Hello messages can be used to discover the network topology and are stored in the Hello file. Using the rospf under ospfDecode/scripts a nice network topology layout will be produced:

$ ./rospf ~/results/nudel_ospfHello.txt
./rospf ~/results/cattura1_ospfHello.txt
Name	Area	Network	Netmask
N0	0	192.168.10.0	0xfffffffc
N1	0	192.168.10.4	0xfffffffc
N2	0	192.168.10.8	0xfffffffc
N3	0	192.168.9.0	0xffffff00

Router	Interface_n	Network_n
R0	192.168.10.10	N2	192.168.10.6	N1	192.168.9.2	N3
R1	192.168.10.5	N1	192.168.10.1	N0
R2	192.168.10.9	N2	192.168.10.2	N0	192.168.9.1	N3

Router	Connected Routers
R0	R1	R2
R1	R0	R2
R2	R0	R1

The database update messages are enabled with the OSPF_OUTPUT_DBD switch. Apply the dbd script to the resulting *_ospfDBD.txt* file and you have your familiar Cisco output to the command: show ip ospf database

$ t2conf ospfDecode -D OSPF_OUTPUT_DBD=1
$ t2 -r ~/data/nudel.pcap -w results/nudel_ospfDBD.txt
...
$ ./dbd ~/results/cattura1_ospfDBD.txt
OSPF Router with ID (192.168.10.10)

Router Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.10	192.168.10.10	30	0x80000002	0x2d0b
192.168.10.5	192.168.10.5	18	0x80000003	0x8b47
192.168.10.9	192.168.10.9	18	0x80000002	0x92ae

Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.2	192.168.10.9	19	0x80000001	0x94d
192.168.10.5	192.168.10.5	31	0x80000001	0x2135

Summary Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
EMPTY


OSPF Router with ID (192.168.10.5)

Router Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.10	192.168.10.10	12	0x80000002	0x2d0b
192.168.10.5	192.168.10.5	12	0x80000002	0xf754

Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.5	192.168.10.5	12	0x80000001	0x2135

Summary Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
EMPTY


OSPF Router with ID (192.168.10.9)

Router Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.10	192.168.10.10	31	0x80000002	0x2d0b
192.168.10.5	192.168.10.5	17	0x80000003	0x8b47
192.168.10.9	192.168.10.9	17	0x80000002	0x92ae

Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
192.168.10.2	192.168.10.9	17	0x80000001	0x94d
192.168.10.5	192.168.10.5	30	0x80000001	0x2135

Summary Net Link States (Area 0)

Link ID	ADV Router	Age	Seq#	Checksum
EMPTY
$

If you switch on the OSPF_OUTPUT_MSG switch then all OSPF messages are extracted and stored in a _ospfMsg file:

$ t2conf ospfDecode -D OSPF_OUTPUT_MSG=1
$ t2 -r ~/data/nudel.pcap -w ~/results
$ tcol cattura1_ospfMsg.txt
Area  MsgType  LSType   SrcIP          LinkID         Netmask_RouterIP  ADVRouter      Metric  IfaceType   LinkType
0     LSReq    Router   192.168.10.6   192.168.10.5   192.168.10.5
0     LSReq    Router   192.168.10.5   192.168.10.10  192.168.10.10
0     LSUp     Router   192.168.10.5   192.168.10.4   0xfffffffc        192.168.10.5   1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.5   192.168.10.0   0xfffffffc        192.168.10.5   1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.5   192.168.8.0    0xffffff00        192.168.10.5   1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.6   192.168.10.8   0xfffffffc        192.168.10.10  1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.6   192.168.10.4   0xfffffffc        192.168.10.10  1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.6   192.168.9.0    0xffffff00        192.168.10.10  1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.5   192.168.10.5   192.168.10.5      192.168.10.5   1       > 100 Mbps  Transit
0     LSUp     Router   192.168.10.5   192.168.10.0   0xfffffffc        192.168.10.5   1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.5   192.168.8.0    0xffffff00        192.168.10.5   1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.6   192.168.10.8   0xfffffffc        192.168.10.10  1       > 100 Mbps  Stub
0     LSUp     Router   192.168.10.6   192.168.10.5   192.168.10.6      192.168.10.10  1       > 100 Mbps  Transit
0     LSUp     Router   192.168.10.6   192.168.9.0    0xffffff00        192.168.10.10  1       > 100 Mbps  Stub
0     LSUp     Network  192.168.10.5   192.168.10.5   0xfffffffc        192.168.10.5
0     LSAck    Router   192.168.10.6   192.168.10.5   192.168.10.5
...
$

nDPI statistics

nDPI is a wrapper for the open source nDPI project. Under the link the plugin is discussed in detail.

VRRP statistics

The Virtual Router Redundancy Protocol (VRRP) provides automatic assignment of routers to hosts. To have an overview about update processes is vital for troubleshooting purposes.

Download VRRP_preempt.cap and run t2 on it:

$ t2 -r ~/data/VRRP_preempt.cap -w ~/results
...
$ tcol VRRP_preempt_vrrp.txt
VirtualRtrID  Priority  SkewTime[s]  MasterDownInterval[s]  AddrCount  Addresses    Version  Type  AdverInt[s]  AuthType  AuthString  Checksum  CalcChecksum  flowIndex
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             100       0.609375     3.609375               1          192.168.0.1  2        1     1            0                     0xba52    0xba52        1
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
1             200       0.218750     3.218750               1          192.168.0.1  2        1     1            0                     0x5652    0x5652        2
$

You see a list of router messages. We plan to do an extra routing tutorial.

Mining based statistics

For traffic mining purposes a specific packet size and inter-distance (PIAT) are good invariants. In order to support researchers for signal approaches the plugin nFrstPkts was developed which served us well over the years.

For the pure statistical approach the pktSIATHisto is supplied which creates a 3-D distribution of PIAT. The plugin descriptiveStats calculates descriptive statistical moments from the PIAT distribution, thus requires pktSIATHisto as dependence.

All these plugins are discussed in detail under the links supplied above.