Tutorial: Flexible Flow and Packet Length Statistics

Introduction

In traffic mining or interface operations packet length and packets/flow are parameters with a high information gain. Thus, T2 provides the user with a user controlled packet length statistics and certain packets/flow indicators.

Preparation

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile the following plugins:

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow tcpStates basicStats nFrstPkts pktSIATHisto descriptiveStats txtSink
...
BUILD SUCCESSFUL

If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:

$ mkdir ~/data ~/results
$

The anonymized sample PCAP used in this tutorial can be downloaded here: annoloc2.pcap Please extract it under your data folder. Now you are all set for T2 IP label experiments.

Basic packet statistics and packet/flow measure

The end report contains information about packet, bytes and protocol statistics, including measures which gives a practitioner an indication about the stress on a flow engine like T2 when operated on an interface.

$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.8 (Anteater), Tarantula. PID: 46224
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.8
    02: basicStats, 0.8.8
    03: tcpStates, 0.8.8
    04: nFrstPkts, 0.8.8
    05: pktSIATHisto, 0.8.8
    06: descriptiveStats, 0.8.8
    07: txtSink, 0.8.8
[INF] IPv4 Ver: 5, Rev: 28062020, Range Mode: 0, subnet ranges loaded: 405633 (405.63 K)
[INF] IPv6 Ver: 5, Rev: 28062020, Range Mode: 0, subnet ranges loaded: 49666 (49.67 K)
Processing file: /home/user/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 1.136210 sec
Finished unloading flow memory. Time: 1.763557 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218608 (1.22 M) [99.97%]
Number of IPv6 packets: 160 [0.01%]
Number of A packets: 564232 (564.23 K) [46.29%]
Number of B packets: 654783 (654.78 K) [53.71%]
Number of A bytes: 29448132 (29.45 M) [45.95%]
Number of B bytes: 34634594 (34.63 M) [54.05%]
Average A packet load: 52.19
Average B packet load: 52.89
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
tcpStates: Aggregated tcpStates anomalies: 0xdf
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 247 [0.02%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of TCP bytes: 52643546 (52.64 M) [82.15%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of UDP bytes: 11234272 (11.23 M) [17.53%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17589 (17.59 K)
Number of processed A flows: 9980 (9.98 K) [56.74%]
Number of processed B flows: 7609 (7.61 K) [43.26%]
Number of request     flows: 9933 (9.93 K) [56.47%]
Number of reply       flows: 7656 (7.66 K) [43.53%]
Total   A/B    flow asymmetry: 0.13
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 69.31
Number of processed A packets/flows: 56.54
Number of processed B packets/flows: 86.05
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B   packets/s: 48859.83 (48.86 K)
Number of processed A     packets/s: 22615.21 (22.61 K)
Number of processed   B   packets/s: 26244.62 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 704.99
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270268576 b/s (270.27 Mb/s)
Max number of flows in memory: 15206 (15.21 K) [5.80%]
Memory usage: 0.26 GB [0.52%]
Aggregate flow status: 0x0c0098fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 payload length > framing length
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] Ethernet flows
[INF] IPv4 flows
[INF] IPv6 flows
[INF] ARP
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP
$

Prominent measures are the packets/flows mean and the bandwidth estimation useful to assess the limits of on an interface. As the bandwidth is very low, we have a bad example her. But if it were in the range of 10GBit then we should look at higher tatistical moment are computed by the script t2flowstat from the flow file:

For both flow directions, 100 samples, 1 packet/bin, just the summary

$ t2flowstat annoloc2_flows.txt -c numPktsSnt -s 1 -m 200 -n
----------------------------------
#Flows:  17589
Col Sum: 1219015
Max Col: 23601
Median:  1.889559
Mean:    69.305532
Std Dev: 55.179076
NP Skew: 1.221767
$

For the A direction only

$ t2flowstat annoloc2_flows.txt -c numPktsSnt -s 1 -m 200 -n -d A
----------------------------------
#Flows:  9933
Col Sum: 564232
Max Col: 12342
Median:  1.942525
Mean:    56.803785
Std Dev: 40.559900
NP Skew: 1.352599
$

For the B direction only

$  t2flowstat annoloc2_flows.txt -c numPktsSnt -s 1 -m 200 -n -d B
----------------------------------
#Flows:  7656
Col Sum: 654783
Max Col: 23601
Median:  1.800890
Mean:    85.525470
Std Dev: 38.597294
NP Skew: 2.169183
$

The overall median is around 1.9 pkts/flow, hence 50% of the flows contain two packets or less. So the flow engine has to create a flow every second to third packet in average, which brings every tool down when run at 10GBit++. For more information about this topic, refer to the Performance on interface et al tutorial. The mean and NP skew indicated a highly asymmetric but rapid declining distribution with increasing packets/flow.

We might integrate this statistics into the end report in a future version so that the generation of a flow file is not necessary.

Controlling the packet length

All the statistics available from the loaded plugin in the flow file is computed in the core, using different header information, hence the snap length does not alter the statistics. The content plugins use the snapped length information, there the information can be truncated with lower snap length.

The layer to be considered can be configured by the PACKETLENGTH variable residing in the core definition file packetCapture.h as shown below:

The constant FRGIPPKTLENVIEW defines for fragmented packets whether the length of the IP header for all following packets should be added or not. In Traffic Mining you like only to look at the content length, so all fragments following the first packet should be stripped of the IP header. For a troubleshooter or admin different story.

Let’s look at the default output of all relevant plugins first:

$ tawk '{ print wildcard("^(num)|Iat") }' ~/results/faf-exercise_flows.txt | head -n 10 | tcol
numHdrDesc  numHdrs  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  L2L3L4Pl_Iat  Ps_Iat_Cnt_PsCnt_IatCnt  dsMinIat  dsMaxIat  dsMeanIat  dsLowQuartileIat  dsMedianIat  dsUppQuartileIat  dsIqdIat  dsModeIat  dsRangeIat  dsStdIat  dsRobStdIat  dsSkewIat  dsExcIat
1           3        1           0            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           0            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            0            0             0_0.000000    0_0_1_1_1                0         0         0          0                 0            0                 0         0          0           0         0            0          0
$

You see the layer 7 length, e.g. the nFrstPkts output mostly starts with 0, so no content.

As a network person we are interested to see the full length of the packets, so set PACKETLENGTH=0 so that the full packet length including layer 2 is added to the length statistics.

$ t2conf tranalyzer2 -D PACKETLENGTH=0
$ t2build -R
...
$ t2 -r ~/data/faf-exercise.pcap -w ~/results
...
$

As the end report and packet/flow statistics does not change, therefore it is not printed. Now look into the flow file again.

$ tawk '{ print wildcard("^(num)|Iat") }' ~/results/annoloc2_flows.txt | head -n 10 | tcol
umHdrDesc  numHdrs  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  L2L3L4Pl_Iat  Ps_Iat_Cnt_PsCnt_IatCnt  dsMinIat  dsMaxIat  dsMeanIat  dsLowQuartileIat  dsMedianIat  dsUppQuartileIat  dsIqdIat  dsModeIat  dsRangeIat  dsStdIat  dsRobStdIat  dsSkewIat  dsExcIat
1           3        1           0            54           0             54_0.000000   54_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           0            54           0             54_0.000000   54_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            62           54            62_0.000000   62_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            54           62            54_0.000000   54_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            62           54            62_0.000000   62_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            54           62            54_0.000000   54_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            62           54            62_0.000000   62_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            54           62            54_0.000000   54_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
1           3        1           1            62           54            62_0.000000   62_0_1_1_1               0         0         0          0                 0            0                 0         0          0           0         0            0          0
$

You easily spot the change in the numPktsSnt, numPktsRcvd, L2L3L4Pl_Iat and Ps_Iat_Cnt_PsCnt_IatCnt columns, now the numbers increased. Try the 1 and 2 yourself.

Fragmentation option

If fragmentation is present the packet length can be seen as one big packet, so the 2nd fragmented packet does not include the IP header. Or you want to see the true packet statistics on the wire and all packets count the length of each subsequent header.

The end report states that fragmented packets are present. Let’s look at these with our last configuration PACKETLENGTH=0 and FRGIPPKTLENVIEW=1.

$ tawk 'bitsanyset($flowStat, 0x0000001000000000)' ~/results/annoloc2_flows.txt | head -n 10 | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP            srcIPCC  srcIPOrg                          srcPort  dstIP            dstIPCC  dstIPOrg                   dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT      stdIAT        pktps     bytps      pktAsm     bytAsm     tcpStates  nFpCnt  L2L3L4Pl_Iat                                                                                                                                                                                                                                                                             tCnt  Ps_Iat_Cnt_PsCnt_IatCnt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   dsMinPl  dsMaxPl  dsMeanPl  dsLowQuartilePl  dsMedianPl  dsUppQuartilePl  dsIqdPl  dsModePl  dsRangePl  dsStdPl   dsRobStdPl  dsSkewPl     dsExcPl    dsMinIat  dsMaxIat  dsMeanIat  dsLowQuartileIat  dsMedianIat  dsUppQuartileIat  dsIqdIat  dsModeIat  dsRangeIat  dsStdIat  dsRobStdIat  dsSkewIat  dsExcIat
A     1894     0x040008d200004000  1022171702.614414  1022171702.614414  0.000000   1           3        eth:ipv4:udp  00:80:48:b3:0e:ed  00:d0:02:6d:78:00  0x0800              138.212.188.118  jp       "ASAHI KASEI CORPORATION"         0        201.9.46.255     br       "Telemar Norte Leste SA"   0        17       1           0            66           0             66        66        66          0           0       0         0           0             0         0          1          1          0x00       1       66_0.000000                                                                                                                                                                                                                                                                              1     66_0_1_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                66       66       66        66               66          66               0        66        0          0         0           0            0          0         0         0          0                 0            0                 0         0          0           0         0            0          0
A     6282     0x0400085200004000  1022171713.796490  1022171713.796491  0.000001   1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:80:c8:8f:ca:5e  0x0800              16.46.171.138    us       "HEWLETT PACKARD ENTERPRISE COM"  0        138.212.189.231  jp       "ASAHI KASEI CORPORATION"  0        17       2           0            2405         0             911       1494      1202.5      206.1216    0       1e-06     5e-07       3.535534e-07  2000000   2.405e+09  1          1          0x00       2       1494_0.000000;911_0.000001                                                                                                                                                                                                                                                               2     911_0_1_1_2;1494_0_1_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  911      1494     1202.5    911              1202.5      1494             583      911       583        291.5     291.5       0            -2         0         0         0          0                 0            0                 0         0          0           0         0            0          0
A     984      0x0400081200004000  1022171701.848919  1022171726.366145  24.517226  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:e0:29:04:0a:18  0x0800              201.232.53.168   co       "EPM Telecomunicaciones SA ESP"   3289     138.212.189.228  jp       "ASAHI KASEI CORPORATION"  1533     17       203         0            207946       0             72        1514      1024.365    667.7642    0       0.391237  0.1207745   0.1718241     8.279893  8481.629   1          1          0x00       20      1514_0.000000;1494_0.000409;72_0.000000;1514_0.374473;1494_0.000018;72_0.000018;1514_0.375298;1494_0.000013;72_0.000007;1514_0.360322;1494_0.000009;72_0.000001;1514_0.374244;1494_0.000360;72_0.000001;1514_0.388981;1494_0.000444;72_0.000000;1514_0.360684;1494_0.000416              11    72_0_68_68_134;1494_0_65_67_134;1494_1_2_67_2;1514_0_1_68_134;1514_12_1_68_1;1514_355_3_68_3;1514_360_19_68_19;1514_370_32_68_32;1514_375_6_68_6;1514_385_5_68_5;1514_390_1_68_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          72       1514     1024.365  72               1494        1514             1442     72        1442       675.9622  675.9622    -0.6988212   -1.510996  0.5       392.5     121.5693   0.5               0.5          362.5             362       0.5        392         174.1365  174.1365     0.7409255  -1.44783
A     98       0x0400081200004000  1022171701.699706  1022171726.576813  24.877107  1           3        eth:ipv4:udp  00:80:48:b3:0e:ed  00:d0:02:6d:78:00  0x0800              138.212.188.118  jp       "ASAHI KASEI CORPORATION"         655      201.9.46.255     br       "Telemar Norte Leste SA"   655      17       1014        508          793668       70104         66        1514      782.7101    720.2989    0       0.833577  0.02453363  0.04786205    40.76036  31903.55   0.3324573  0.8376794  0x00       20      1514_0.000000;66_0.000002;1514_0.001272;66_0.000002;1514_0.089488;66_0.000001;1514_0.001204;66_0.000002;1514_0.074224;66_0.000001;174_0.000003;1514_0.083978;66_0.000001;1514_0.000027;66_0.000001;1514_0.093176;66_0.000000;1514_0.001097;66_0.000001;1514_0.106532                     64    66_0_501_501_601;174_0_12_12_601;1514_0_88_501_601;1514_1_120_501_120;1514_2_29_501_29;1514_3_5_501_5;1514_4_1_501_1;1514_54_1_501_1;1514_64_1_501_1;1514_65_1_501_1;1514_66_2_501_2;1514_67_2_501_2;1514_68_3_501_3;1514_69_1_501_1;1514_70_6_501_6;1514_71_2_501_2;1514_72_1_501_1;1514_73_3_501_3;1514_74_4_501_4;1514_75_1_501_1;1514_76_2_501_2;1514_77_3_501_3;1514_78_3_501_3;1514_79_3_501_3;1514_80_5_501_5;1514_81_1_501_1;1514_82_5_501_5;1514_83_6_501_6;1514_84_9_501_9;1514_85_2_501_2;1514_86_11_501_11;1514_87_8_501_8;1514_88_10_501_10;1514_89_8_501_8;1514_90_13_501_13;1514_91_12_501_12;1514_92_10_501_10;1514_93_15_501_15;1514_94_13_501_13;1514_95_9_501_9;1514_96_9_501_9;1514_97_11_501_11;1514_98_4_501_4;1514_99_6_501_6;1514_100_8_501_8;1514_101_8_501_8;1514_102_5_501_5;1514_103_2_501_2;1514_104_1_501_1;1514_105_3_501_3;1514_106_4_501_4;1514_107_1_501_1;1514_108_5_501_5;1514_109_2_501_2;1514_110_2_501_2;1514_112_1_501_1;1514_114_1_501_1;1514_115_2_501_2;1514_119_1_501_1;1514_148_1_501_1;1514_169_1_501_1;1514_170_1_501_1;1514_179_2_501_2;1514_830_1_501_1  66       1514     782.7101  66               174         1514             1448     66        1448       722.7795  722.7795    0.02293284   -1.998584  0.5       835       24.8075    0.5               0.5          67                66.5      0.5        834.5       47.92335  47.92335     5.5217     79.30751
A     1289     0x0400081200004000  1022171702.058266  1022171726.575284  24.517018  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:10:60:59:f1:4b  0x0800              16.103.245.128   us       "HEWLETT PACKARD ENTERPRISE COM"  1120     138.212.191.249  jp       "ASAHI KASEI CORPORATION"  1461     17       134         0            107664       0             72        1514      803.4627    707.2161    0       0.747904  0.1829628   0.1880735     5.465591  4391.399   1          1          0x00       20      1514_0.000000;72_0.000001;1514_0.373331;72_0.001235;1514_0.372348;72_0.000001;1514_0.379557;72_0.000001;1514_0.357597;72_0.000001;1514_0.374397;72_0.000001;1514_0.372138;72_0.000001;1514_0.373593;72_0.000001;1514_0.366913;72_0.000000;1514_0.371722;72_0.000001                      12    72_0_65_66_68;72_1_1_66_1;1494_0_2_2_68;1514_0_1_66_68;1514_355_2_66_2;1514_360_8_66_8;1514_365_11_66_11;1514_370_30_66_30;1514_375_9_66_9;1514_380_3_66_3;1514_390_1_66_1;1514_740_1_66_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                72       1514     803.4627  72               1494        1514             1442     72        1442       720.6296  720.6296    -0.02982101  -1.999067  0.5       745       184.5865   0.5               1            372.5             372       0.5        744.5       191.7651  191.7651     0.1838971  -1.55974
A     1290     0x040008f200004000  1022171702.058274  1022171726.575521  24.517247  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:10:60:59:f1:4b  0x0800              16.103.245.128   us       "HEWLETT PACKARD ENTERPRISE COM"  0        138.212.191.249  jp       "ASAHI KASEI CORPORATION"  0        17       64          0            95616        0             1494      1494      1494        0           0       0.747897  0.383082    0.08560037    2.610407  3899.948   1          1          0x00       20      1494_0.000000;1494_0.374560;1494_0.372355;1494_0.379555;1494_0.358766;1494_0.373236;1494_0.372266;1494_0.373462;1494_0.367221;1494_0.371415;1494_0.375320;1494_0.373893;1494_0.371685;1494_0.365597;1494_0.372201;1494_0.375167;1494_0.372508;1494_0.363928;1494_0.377676;1494_0.370959  11    1494_0_1_64_1;1494_355_1_64_1;1494_360_7_64_7;1494_365_10_64_10;1494_370_28_64_28;1494_375_10_64_10;1494_380_3_64_3;1494_390_1_64_1;1494_720_1_64_1;1494_730_1_64_1;1494_740_1_64_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       1494     1494     1494      1494             1494        1494             0        1494      0          0         0           0            0          357.5     745       389.2064   367.5             372.5        377.5             10        372.5      387.5       78.18096  7.413        4.213324   15.88362
A     126      0x0400083200004000  1022171701.700965  1022171726.594434  24.893469  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:80:c8:8f:ca:5e  0x0800              16.46.171.138    us       "HEWLETT PACKARD ENTERPRISE COM"  4623     138.212.189.231  jp       "ASAHI KASEI CORPORATION"  1490     17       742         0            969486       0             899       1514      1306.585    278.4473    0       0.202366  0.03354916  0.04754491    29.80701  38945.39   1          1          0x00       20      1514_0.000000;1494_0.000008;911_0.000002;1514_0.098384;1494_0.000011;911_0.000000;1514_0.093618;1494_0.000012;911_0.000002;1514_0.105108;1494_0.000225;911_0.000005;1514_0.095565;1494_0.000204;911_0.000001;1514_0.106943;1494_0.000007;911_0.000001;1514_0.094354;1494_0.000009        28    899_0_1_1_486;906_0_1_1_486;907_0_1_1_486;911_0_243_244_486;911_1_1_244_8;1494_0_239_247_486;1494_1_7_247_8;1494_2_1_247_1;1514_0_1_248_486;1514_91_2_248_2;1514_92_9_248_9;1514_93_9_248_9;1514_94_20_248_20;1514_95_46_248_46;1514_96_40_248_40;1514_97_12_248_12;1514_98_8_248_8;1514_99_1_248_1;1514_101_1_248_1;1514_103_5_248_5;1514_104_13_248_13;1514_105_24_248_24;1514_106_34_248_34;1514_107_14_248_14;1514_108_6_248_6;1514_109_1_248_1;1514_191_1_248_1;1514_200_1_248_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     899      1514     1306.585  911              1494        1514             603      1514      615        279.6179  279.6179    -0.7065479   -1.496943  0.5       202.5     33.87922   0.5               0.5          95.5              95        0.5        202         47.57575  47.57575     0.7782133  -1.190346
A     406      0x0400081200004000  1022171701.717743  1022171726.607895  24.890152  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:e0:29:04:0b:81  0x0800              18.14.224.62     us       "Massachusetts Institute of Tec"  4383     138.212.191.34   jp       "ASAHI KASEI CORPORATION"  2428     17       136         0            157964       0             809       1514      1161.5      346.0216    0       0.665421  0.1830158   0.1876362     5.464008  6346.446   1          1          0x00       20      1514_0.000000;809_0.000000;1514_0.359622;809_0.000001;1514_0.379067;809_0.000002;1514_0.372816;809_0.000001;1514_0.373867;809_0.000001;1514_0.362254;809_0.000001;1514_0.529715;809_0.000001;1514_0.219500;809_0.000001;1514_0.665421;809_0.000001;1514_0.203431;809_0.000001            12    809_0_68_68_69;1514_0_1_68_69;1514_87_1_68_1;1514_200_1_68_1;1514_215_1_68_1;1514_355_1_68_1;1514_360_12_68_12;1514_365_2_68_2;1514_370_36_68_36;1514_375_10_68_10;1514_520_2_68_2;1514_660_1_68_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        809      1514     1161.5    809              1161.5      1514             705      809       705        352.5     352.5       0            -2         0.5       665       184.3259   0.5               44           372.5             372       0.5        664.5       191.182   191.182      0.1759952  -1.692286
$

Now we like to see what happens if we switch off the IP header length suppression. Set FRGIPPKTLENVIEW=0, recompile and run t2 on the pcap.

$ t2conf tranalyzer2 -D FRGIPPKTLENVIEW=0
$ t2build -R
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
$
$ tawk '{ if (bitsanyset($flowStat, 0x0000001000000000)) print }' ~/results/annoloc2_flows.txt | head -n 10 | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP            srcIPCC  srcIPOrg                          srcPort  dstIP            dstIPCC  dstIPOrg                   dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT      stdIAT        pktps     bytps      pktAsm     bytAsm     tcpStates  nFpCnt  L2L3L4Pl_Iat                                                                                                                                                                                                                                                                             tCnt  Ps_Iat_Cnt_PsCnt_IatCnt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   dsMinPl  dsMaxPl  dsMeanPl  dsLowQuartilePl  dsMedianPl  dsUppQuartilePl  dsIqdPl  dsModePl  dsRangePl  dsStdPl   dsRobStdPl  dsSkewPl    dsExcPl    dsMinIat  dsMaxIat  dsMeanIat  dsLowQuartileIat  dsMedianIat  dsUppQuartileIat  dsIqdIat  dsModeIat  dsRangeIat  dsStdIat  dsRobStdIat  dsSkewIat  dsExcIat
A     1894     0x040008d200004000  1022171702.614414  1022171702.614414  0.000000   1           3        eth:ipv4:udp  00:80:48:b3:0e:ed  00:d0:02:6d:78:00  0x0800              138.212.188.118  jp       "ASAHI KASEI CORPORATION"         0        201.9.46.255     br       "Telemar Norte Leste SA"   0        17       1           0            86           0             86        86        86          0           0       0         0           0             0         0          1          1          0x00       1       86_0.000000                                                                                                                                                                                                                                                                              1     86_0_1_1_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                86       86       86        86               86          86               0        86        0          0         0           0           0          0         0         0          0                 0            0                 0         0          0           0         0            0          0
A     6282     0x0400085200004000  1022171713.796490  1022171713.796491  0.000001   1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:80:c8:8f:ca:5e  0x0800              16.46.171.138    us       "HEWLETT PACKARD ENTERPRISE COM"  0        138.212.189.231  jp       "ASAHI KASEI CORPORATION"  0        17       2           0            2445         0             931       1514      1222.5      206.1216    0       1e-06     5e-07       3.535534e-07  2000000   2.445e+09  1          1          0x00       2       1514_0.000000;931_0.000001                                                                                                                                                                                                                                                               2     931_0_1_1_2;1514_0_1_1_2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  931      1514     1222.5    931              1222.5      1514             583      931       583        291.5     291.5       0           -2         0         0         0          0                 0            0                 0         0          0           0         0            0          0
A     984      0x0400081200004000  1022171701.848919  1022171726.366145  24.517226  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:e0:29:04:0a:18  0x0800              201.232.53.168   co       "EPM Telecomunicaciones SA ESP"   3289     138.212.189.228  jp       "ASAHI KASEI CORPORATION"  1533     17       203         0            210646       0             92        1514      1037.665    663.0211    0       0.391237  0.1207745   0.1718241     8.279893  8591.755   1          1          0x00       20      1514_0.000000;1514_0.000409;92_0.000000;1514_0.374473;1514_0.000018;92_0.000018;1514_0.375298;1514_0.000013;92_0.000007;1514_0.360322;1514_0.000009;92_0.000001;1514_0.374244;1514_0.000360;92_0.000001;1514_0.388981;1514_0.000444;92_0.000000;1514_0.360684;1514_0.000416              10    92_0_68_68_134;1514_0_66_135_134;1514_1_2_135_2;1514_12_1_135_1;1514_355_3_135_3;1514_360_19_135_19;1514_370_32_135_32;1514_375_6_135_6;1514_385_5_135_5;1514_390_1_135_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 92       1514     1037.665  92               1514        1514             1422     1514      1422       671.1582  671.1582    -0.6992838  -1.511002  0.5       392.5     121.5693   0.5               0.5          362.5             362       0.5        392         174.1365  174.1365     0.7409255  -1.44783
A     98       0x0400081200004000  1022171701.699706  1022171726.576813  24.877107  1           3        eth:ipv4:udp  00:80:48:b3:0e:ed  00:d0:02:6d:78:00  0x0800              138.212.188.118  jp       "ASAHI KASEI CORPORATION"         655      201.9.46.255     br       "Telemar Norte Leste SA"   655      17       1014        508          803688       70104         86        1514      792.5917    710.5359    0       0.833577  0.02453363  0.04786205    40.76036  32306.33   0.3324573  0.8395408  0x00       20      1514_0.000000;86_0.000002;1514_0.001272;86_0.000002;1514_0.089488;86_0.000001;1514_0.001204;86_0.000002;1514_0.074224;86_0.000001;174_0.000003;1514_0.083978;86_0.000001;1514_0.000027;86_0.000001;1514_0.093176;86_0.000000;1514_0.001097;86_0.000001;1514_0.106532                     64    86_0_501_501_601;174_0_12_12_601;1514_0_88_501_601;1514_1_120_501_120;1514_2_29_501_29;1514_3_5_501_5;1514_4_1_501_1;1514_54_1_501_1;1514_64_1_501_1;1514_65_1_501_1;1514_66_2_501_2;1514_67_2_501_2;1514_68_3_501_3;1514_69_1_501_1;1514_70_6_501_6;1514_71_2_501_2;1514_72_1_501_1;1514_73_3_501_3;1514_74_4_501_4;1514_75_1_501_1;1514_76_2_501_2;1514_77_3_501_3;1514_78_3_501_3;1514_79_3_501_3;1514_80_5_501_5;1514_81_1_501_1;1514_82_5_501_5;1514_83_6_501_6;1514_84_9_501_9;1514_85_2_501_2;1514_86_11_501_11;1514_87_8_501_8;1514_88_10_501_10;1514_89_8_501_8;1514_90_13_501_13;1514_91_12_501_12;1514_92_10_501_10;1514_93_15_501_15;1514_94_13_501_13;1514_95_9_501_9;1514_96_9_501_9;1514_97_11_501_11;1514_98_4_501_4;1514_99_6_501_6;1514_100_8_501_8;1514_101_8_501_8;1514_102_5_501_5;1514_103_2_501_2;1514_104_1_501_1;1514_105_3_501_3;1514_106_4_501_4;1514_107_1_501_1;1514_108_5_501_5;1514_109_2_501_2;1514_110_2_501_2;1514_112_1_501_1;1514_114_1_501_1;1514_115_2_501_2;1514_119_1_501_1;1514_148_1_501_1;1514_169_1_501_1;1514_170_1_501_1;1514_179_2_501_2;1514_830_1_501_1  86       1514     792.5917  86               174         1514             1428     86        1428       712.9836  712.9836    0.02316274  -1.99884   0.5       835       24.8075    0.5               0.5          67                66.5      0.5        834.5       47.92335  47.92335     5.5217     79.30751
A     1289     0x0400081200004000  1022171702.058266  1022171726.575284  24.517018  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:10:60:59:f1:4b  0x0800              16.103.245.128   us       "HEWLETT PACKARD ENTERPRISE COM"  1120     138.212.191.249  jp       "ASAHI KASEI CORPORATION"  1461     17       134         0            109024       0             92        1514      813.6119    697.696     0       0.747904  0.1829628   0.1880735     5.465591  4446.871   1          1          0x00       20      1514_0.000000;92_0.000001;1514_0.373331;92_0.001235;1514_0.372348;92_0.000001;1514_0.379557;92_0.000001;1514_0.357597;92_0.000001;1514_0.374397;92_0.000001;1514_0.372138;92_0.000001;1514_0.373593;92_0.000001;1514_0.366913;92_0.000000;1514_0.371722;92_0.000001                      11    92_0_65_66_68;92_1_1_66_1;1514_0_3_68_68;1514_355_2_68_2;1514_360_8_68_8;1514_365_11_68_11;1514_370_30_68_30;1514_375_9_68_9;1514_380_3_68_3;1514_390_1_68_1;1514_740_1_68_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              92       1514     813.6119  92               1514        1514             1422     1514      1422       710.9208  710.9208    -0.0298541  -1.999109  0.5       745       184.5865   0.5               1            372.5             372       0.5        744.5       191.7651  191.7651     0.1838971  -1.55974
A     1290     0x040008f200004000  1022171702.058274  1022171726.575521  24.517247  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:10:60:59:f1:4b  0x0800              16.103.245.128   us       "HEWLETT PACKARD ENTERPRISE COM"  0        138.212.191.249  jp       "ASAHI KASEI CORPORATION"  0        17       64          0            96896        0             1514      1514      1514        0           0       0.747897  0.383082    0.08560037    2.610407  3952.157   1          1          0x00       20      1514_0.000000;1514_0.374560;1514_0.372355;1514_0.379555;1514_0.358766;1514_0.373236;1514_0.372266;1514_0.373462;1514_0.367221;1514_0.371415;1514_0.375320;1514_0.373893;1514_0.371685;1514_0.365597;1514_0.372201;1514_0.375167;1514_0.372508;1514_0.363928;1514_0.377676;1514_0.370959  11    1514_0_1_64_1;1514_355_1_64_1;1514_360_7_64_7;1514_365_10_64_10;1514_370_28_64_28;1514_375_10_64_10;1514_380_3_64_3;1514_390_1_64_1;1514_720_1_64_1;1514_730_1_64_1;1514_740_1_64_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       1514     1514     1514      1514             1514        1514             0        1514      0          0         0           0           0          357.5     745       389.2064   367.5             372.5        377.5             10        372.5      387.5       78.18096  7.413        4.213324   15.88362
A     126      0x0400083200004000  1022171701.700965  1022171726.594434  24.893469  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:80:c8:8f:ca:5e  0x0800              16.46.171.138    us       "HEWLETT PACKARD ENTERPRISE COM"  4623     138.212.189.231  jp       "ASAHI KASEI CORPORATION"  1490     17       742         0            979366       0             919       1514      1319.9      273.6272    0       0.202366  0.03354916  0.04754491    29.80701  39342.29   1          1          0x00       20      1514_0.000000;1514_0.000008;931_0.000002;1514_0.098384;1514_0.000011;931_0.000000;1514_0.093618;1514_0.000012;931_0.000002;1514_0.105108;1514_0.000225;931_0.000005;1514_0.095565;1514_0.000204;931_0.000001;1514_0.106943;1514_0.000007;931_0.000001;1514_0.094354;1514_0.000009        27    919_0_1_1_486;926_0_1_1_486;927_0_1_1_486;931_0_243_244_486;931_1_1_244_8;1514_0_240_495_486;1514_1_7_495_8;1514_2_1_495_1;1514_91_2_495_2;1514_92_9_495_9;1514_93_9_495_9;1514_94_20_495_20;1514_95_46_495_46;1514_96_40_495_40;1514_97_12_495_12;1514_98_8_495_8;1514_99_1_495_1;1514_101_1_495_1;1514_103_5_495_5;1514_104_13_495_13;1514_105_24_495_24;1514_106_34_495_34;1514_107_14_495_14;1514_108_6_495_6;1514_109_1_495_1;1514_191_1_495_1;1514_200_1_495_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      919      1514     1319.9    931              1514        1514             583      1514      595        274.7766  274.7766    -0.7092626  -1.496931  0.5       202.5     33.87922   0.5               0.5          95.5              95        0.5        202         47.57575  47.57575     0.7782133  -1.190346
A     406      0x0400081200004000  1022171701.717743  1022171726.607895  24.890152  1           3        eth:ipv4:udp  00:d0:02:6d:78:00  00:e0:29:04:0b:81  0x0800              18.14.224.62     us       "Massachusetts Institute of Tec"  4383     138.212.191.34   jp       "ASAHI KASEI CORPORATION"  2428     17       136         0            159324       0             829       1514      1171.5      336.2054    0       0.665421  0.1830158   0.1876362     5.464008  6401.086   1          1          0x00       20      1514_0.000000;829_0.000000;1514_0.359622;829_0.000001;1514_0.379067;829_0.000002;1514_0.372816;829_0.000001;1514_0.373867;829_0.000001;1514_0.362254;829_0.000001;1514_0.529715;829_0.000001;1514_0.219500;829_0.000001;1514_0.665421;829_0.000001;1514_0.203431;829_0.000001            12    829_0_68_68_69;1514_0_1_68_69;1514_87_1_68_1;1514_200_1_68_1;1514_215_1_68_1;1514_355_1_68_1;1514_360_12_68_12;1514_365_2_68_2;1514_370_36_68_36;1514_375_10_68_10;1514_520_2_68_2;1514_660_1_68_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        829      1514     1171.5    829              1171.5      1514             685      829       685        342.5     342.5       0           -2         0.5       665       184.3259   0.5               44           372.5             372       0.5        664.5       191.182   191.182      0.1759952  -1.692286
$

If you look now numPktsRcvd or numBytesSnt or the second entry at Ps_Iat_Cnt_PsCnt_IatCnt the numbers now increased by the header length(s).

Don’t forget to reset constants and recompile for the next tutorials:

$ t2conf tranalyzer2 -D PACKETLENGTH=3 FRGIPPKTLENVIEW=1
$ t2build -R
...
$

Have fun!