Tutorial: Flow Masking and Ranging Aggregation

Flow Masking and Ranging Aggregation, WTF?

Imagine you are interested in flow based statistics of traffic between networks or between certain port ranges, or you like to get rid of VLANs or protocols? You are not? Go to some place else and make yourself useful. If yes, keep on reading.

Note, I’m not talking about the different L3/4 operational modes already listed in the basics tutorial which discussed in operational aggregation modes.

In this tutorial we explore the masking and ranging flexibility to redefine flows.i So the option to remove one of these parameters or aggregate several IP’s in one flow, e.g. all /24 or according to country and organisation. The first works, the latter is currently in test, and not part with the current distribution.

Preparation

Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.

First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile the standard plugins. Just as a precaution if you have some old plugins or files there. Then build the plugins listed in the command line below:

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow basicStats connStat txtSink
...

$

If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

$ mkdir ~/data
$ mkdir ~/results
$ cd data

Download the sample pcaps if did not do it already: annoloc2.pcap and 802.1Q_tunneling.cap.

Now you’re all set.

T2 Flow Aggregation scheme

The constants controlling the flow aggregation are residing in tranalyzer.h. Open the file and search for Aggregation modes as being shown below:

// Aggregation modes
#define L4PROT  0x01
#define DSTPORT 0x02
#define SRCPORT 0x04
#define DSTIP   0x08
#define SRCIP   0x10
#define VLANID  0x20
#define SUBNET  0x80

// Flow Agreggation
#define AGGREGATIONFLAG  0x00 // each bit: 1 : aggregation activated, see aggregation modes #defines above
#define SRCIP4CMSK 24  // src IPv4 aggregation CIDR mask
#define DSTIP4CMSK 24  // dst IPv4 aggregation CIDR mask

#define SRCIP6CMSK 120 // src IPv6 aggregation CIDR mask
#define DSTIP6CMSK 120 // dst IPv6 aggregation CIDR mask

#define SRCPORTLW 53  // src port lower bound
#define SRCPORTHW 53  // src port upper bound
#define DSTPORTLW 53  // dst port lower bound
#define DSTPORTHW 53  // dst port upper bound
...

The aggregation modes define a specific bit in the eigth bit AGGREGATIONFLAG. By default it is `0x00, so normal six tuple aggregation. In the following the activation of each mode is discussed. Let’s start with the most powerful one, the network aggregation.

Network Flow Aggregation

Imagine you are interested in traffic only flowing between networks. E.g. 10.4.1.0/24 to 10.5.2.0/24 or from 10.4.5.0/24 to all outside networks. Or even better aggregate all traffic between universities of china and a specific corporation in usa, would that be neat? That will be possible in a later 0.8.x version. The practically tested and successful method of imposing a mask on the src or dst IP in the hash definition is currently the best alternative.

Switch on srcIP and dstIP, it does not make any sense to switch only one on, as we cannot know before the packets is sorted into a masked flow which network mask is to apply. Try the default /24 mask.

The following configs are equivalent

t2conf tranalyzer2 -D AGGREGATIONFLAG="(SRCIP | DSTIP)"

t2conf tranalyzer2 -D AGGREGATIONFLAG=0x18

Then compile and execute t2

$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x18
$ t2conf basicFlow -D BFO_MAC=0 -D BFO_ETHERTYPE=0
$ t2build tranalyzer2 basicFlow
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 28154
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.6
    02: basicStats, 0.8.6
    03: connStat, 0.8.6
    04: txtSink, 0.8.6
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.0: 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.0: 35005508 (35.01 M) [54.63%] bytes
connStat: Number of unique source IPs: 2587 (2.59 K)
connStat: Number of unique destination IPs: 1801 (1.80 K)
connStat: Number of unique source/destination IPs connections: 186
connStat: Max unique number of source IP / destination port connections: 407
connStat: IP prtcon/sdcon, prtcon/scon: 2.188172, 0.157325
connStat: Source IP with max connections: 138.212.189.0 (JP): 499 connections
connStat: Destination IP with max connections: 138.212.187.0 (JP): 690 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 16361 (16.36 K)
Number of processed A flows: 9154 (9.15 K) [55.95%]
Number of processed B flows: 7207 (7.21 K) [44.05%]
Number of request     flows: 8690 (8.69 K) [53.11%]
Number of reply       flows: 7671 (7.67 K) [46.89%]
Total   A/B    flow asymmetry: 0.12
Total req/rply flow asymmetry: 0.06
Number of processed   packets/flows: 74.51
Number of processed A packets/flows: 61.35
Number of processed B packets/flows: 91.21
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A   packets/s: 22511.12 (22.51 K)
Number of processed   B packets/s: 26348.71 (26.35 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
$

As you can see basicStats and connStat report now /24 networks. The resulting flow file is a bit shorter, because several IPv4/6 flows are aggregated into /24 or /120 flows respectively.

$ cd ~/results
$ tawk 't2sort(numPktsSnt, 10)' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP          srcIPCC  srcIPWho                       srcPort  dstIP          dstIPCC  dstIPWho                       dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT        stdIAT       pktps     bytps     pktAsm     bytAsm      connSip  connDip  connSipDip  connSipDprt  connF
B     91       0x0000000200004001  1022171701.699480  1022171726.636773  24.937293  1           3        eth:ipv4:tcp             138.212.189.0  jp       "ASAHI KASEI CORPORATION"      139      138.212.86.0   jp       "Asahi Kasei Networks Corpor"  3429     6        23601       12342        33733962     42462         103       1460      1429.344    188.7309    0       0.253336  0.001056625   0.003715082  946.4139  1352752   0.313246   0.9974856   1        1        1           1            1
A     91       0x0000000200004000  1022171701.699996  1022171726.637210  24.937214  1           3        eth:ipv4:tcp             138.212.86.0   jp       "Asahi Kasei Networks Corpor"  3429     138.212.189.0  jp       "ASAHI KASEI CORPORATION"      139      6        12342       23601        42462        33733962      0         63        3.440447    14.30862    0       0.36365   0.002020519   0.00532618   494.923   1702.756  -0.313246  -0.9974856  1        1        2           2            2
B     5934     0x0000000200004001  1022171714.045827  1022171722.457644  8.411817   1           3        eth:ipv4:tcp             139.45.174.0   ie       "Stripe Inc"                   56071    138.212.190.0  jp       "ASAHI KASEI CORPORATION"      3837     6        10159       5692         14821880     0             0         1460      1458.99     29.41481    0       1.465593  0.0008280156  0.01468998   1207.706  1762031   0.2818119  1           1        96       3           1            1
B     3462     0x0000000200004001  1022171705.686717  1022171714.043794  8.357077   1           3        eth:ipv4:tcp             139.45.174.0   ie       "Stripe Inc"                   56070    138.212.190.0  jp       "ASAHI KASEI CORPORATION"      3820     6        10048       5709         14656900     0             0         1460      1458.688    34.27719    0       1.39519   0.0008317156  0.0141066    1202.334  1753831   0.2753697  1           1        192      5           1            1
A     324      0x0000000200004000  1022171701.712093  1022171726.638722  24.926629  1           3        eth:ipv4:tcp             19.59.134.0    us       "Ford Motor Company"           65230    138.212.187.0  jp       "ASAHI KASEI CORPORATION"      58290    6        9459        5223         13696632     0             1448      1448      1448        0           0       0.067445  0.00263523    0.006627299  379.4737  549477.9  0.2885166  1           1        2        2           2            2
B     69       0x0000000200004001  1022171701.698940  1022171726.629403  24.930463  1           3        eth:ipv4:tcp             138.212.187.0  jp       "ASAHI KASEI CORPORATION"      139      138.212.36.0   jp       "Asahi Kasei Networks Corpor"  2860     6        8978        4413         12814184     25156         68        1460      1427.287    187.9357    0       0.070768  0.002776838   0.005427861  360.1217  513997    0.3409006  0.9960814   13       2        1           3            0.2307692
B     77       0x0000000200004001  1022171701.699040  1022171726.629407  24.930367  1           3        eth:ipv4:tcp             138.212.190.0  jp       "ASAHI KASEI CORPORATION"      139      138.212.36.0   jp       "Asahi Kasei Networks Corpor"  2861     6        7319        3622         10446036     20736         39        1460      1427.249    188.4492    0       0.131045  0.003406252   0.006901587  293.5777  419008.5  0.3379033  0.9960377   6        1        1           1            0.1666667
B     31       0x0000000200004001  1022171701.715914  1022171726.608383  24.892469  1           3        eth:ipv4:tcp             138.212.187.0  jp       "ASAHI KASEI CORPORATION"      139      138.212.77.0   jp       "Asahi Kasei Networks Corpor"  61340    6        7289        3811         10398387     10773         0         1460      1426.586    205.4521    0       0.204505  0.003415072   0.01080472   292.8195  417732.2  0.3133333  0.9979301   42       2        1           3            0.07142857
A     4585     0x0000000200004000  1022171709.260746  1022171716.327308  7.066562   1           3        eth:ipv4:tcp             133.26.75.0    jp       "Meiji University"             36237    138.212.185.0  jp       "ASAHI KASEI CORPORATION"      20       6        6865        4465         10008944     0             0         1460      1457.967    49.88806    0       0.404642  0.00102936    0.009408548  971.4766  1416381   0.211827   1           5        270      10          6            1.2
B     8091     0x0000000200004001  1022171722.458182  1022171726.637621  4.179439   1           3        eth:ipv4:tcp             139.45.174.0   ie       "Stripe Inc"                   56072    138.212.190.0  jp       "ASAHI KASEI CORPORATION"      3854     6        6044        3358         8817920      0             0         1460      1458.954    27.7065     0       0.145115  0.000691502   0.00191674   1446.127  2109833   0.2856839  1           1        1        1           1            1
$

Try to set the aggregation mask for SRCIP4CMSK and DSTIP4CMSK to 8 using t2conf, recompile and rerun t2 as home work. If you do reset it to /24 again for the next chapter.

Port Flow Aggregation

It serves a good purpose if you are interested in flow reduction with a specific statistically questions on your mind and not being interested in the actual flow content. So don’t load any L7 plugins in carving mode as the statemachines get corrupted, as different packets from different six-tuple flows are aggregated.

It can be useful for all plugins which are NOT initiated by ports, all statistical plugins, httpSniffer.

In order to enable the port aggregation mode the following configs are equivalent:

t2conf tranalyzer2 -D AGGREGATIONFLAG="(SRCPORT | DSTPORT)"

t2conf tranalyzer2 -D AGGREGATIONFLAG=0x06

Then the constants listed below define the range of ports which be aggregated into port class 1, the rest is shoved into port class 0. The default class 1 is defined by the standardized ports.

...
#define AGGREGATIONFLAG  0x00 // each bit: 1 : aggregation activated, see aggregation modes #defines above
...
#define SRCPORTLW 1     // src port lower bound
#define SRCPORTHW 1024  // src port upper bound
#define DSTPORTLW 1     // dst port lower bound
#define DSTPORTHW 1024  // dst port upper bound
...

Let’s change the upper range to 23, aka telnet. So we throw all services into class 1, which you should not appear in modern networks anymore. Then switch on the appropriate aggregation mode, compile and rerun t2.

$ t2conf tranalyzer2 -D SRCPORTHW=23 -D DSTPORTHW=23 -D AGGREGATIONFLAG=0x06
$ t2build tranalyzer2
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 139.45.174.202 (IE): 30343 (30.34 K) [2.49%] packets
basicStats: Biggest L3 Talker: 139.45.174.202 (IE): 45902872 (45.90 M) [71.63%] bytes
connStat: Number of unique source IPs: 4040 (4.04 K)
connStat: Number of unique destination IPs: 3181 (3.18 K)
connStat: Number of unique source/destination IPs connections: 4
connStat: Max unique number of source IP / destination port connections: 586
connStat: IP prtcon/sdcon, prtcon/scon: 146.500000, 0.145050
connStat: Source IP with max connections: 138.212.189.66 (JP): 369 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 400 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 12957 (12.96 K)
Number of processed A flows: 7503 (7.50 K) [57.91%]
Number of processed B flows: 5454 (5.45 K) [42.09%]
Number of request     flows: 7021 (7.02 K) [54.19%]
Number of reply       flows: 5936 (5.94 K) [45.81%]
Total   A/B    flow asymmetry: 0.16
Total req/rply flow asymmetry: 0.08
Number of processed   packets/flows: 94.08
Number of processed A packets/flows: 82.07
Number of processed B packets/flows: 110.61
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A   packets/s: 24680.56 (24.68 K)
Number of processed   B packets/s: 24179.27 (24.18 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Let’s find the firsst 10 biggest talkers using unencrypted legacy services, you do not want to use anymore in your corporate network:

$ cd ~/results
$ tawk '{if ($srcPort == 1)  print $srcIP, $numPktsSnt}' annoloc2_flows.txt | sort -nr -k2 | head -n 10
38.212.185.150	8696
138.212.190.31	5276
138.212.188.251	5107
138.212.185.102	4056
138.212.186.210	3259
138.212.190.31	3080
193.86.146.215	2885
138.212.188.139	2466
138.212.184.244	2273
138.212.187.170	2156
$

Oups, that is bad.

Protocol Flow Aggregation

The following configs are equivalent

t2conf tranalyzer2 -D AGGREGATIONFLAG=L4PROT

t2conf tranalyzer2 -D AGGREGATIONFLAG=0x01
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x01
$ t2build tranalyzer2
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
connStat: Number of unique source IPs: 4185 (4.18 K)
connStat: Number of unique destination IPs: 3071 (3.07 K)
connStat: Number of unique source/destination IPs connections: 182
connStat: Max unique number of source IP / destination port connections: 413
connStat: IP prtcon/sdcon, prtcon/scon: 2.269231, 0.098686
connStat: Source IP with max connections: 138.212.189.66 (JP): 368 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 402 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17088 (17.09 K)
Number of processed A flows: 9705 (9.71 K) [56.79%]
Number of processed B flows: 7383 (7.38 K) [43.21%]
Number of request     flows: 9662 (9.66 K) [56.54%]
Number of reply       flows: 7426 (7.43 K) [43.46%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 71.34
Number of processed A packets/flows: 58.14
Number of processed B packets/flows: 88.69
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A   packets/s: 22615.05 (22.61 K)
Number of processed   B packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ cd ~/results
$ tawk 't2sort(numPktsSnt, 10)' annoloc2_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       ethVlanID  srcIP            srcIPCC  srcIPWho                       srcPort  dstIP            dstIPCC  dstIPWho                       dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT        stdIAT       pktps     bytps     pktAsm     bytAsm      connSip  connDip  connSipDip  connSipDprt  connF
B     91       0x0000000200004001  1022171701.699480  1022171726.636773  24.937293  1           3        eth:ipv4:tcp             138.212.189.38   jp       "ASAHI KASEI CORPORATION"      139      138.212.86.201   jp       "Asahi Kasei Networks Corpor"  3429     0        23601       12342        33733962     42462         103       1460      1429.344    188.7309    0       0.253336  0.001056625   0.003715082  946.4139  1352752   0.313246   0.9974856   1        1        1           1            1
A     91       0x0000000200004000  1022171701.699996  1022171726.637210  24.937214  1           3        eth:ipv4:tcp             138.212.86.201   jp       "Asahi Kasei Networks Corpor"  3429     138.212.189.38   jp       "ASAHI KASEI CORPORATION"      139      0        12342       23601        42462        33733962      0         63        3.440447    14.30862    0       0.36365   0.002020519   0.00532618   494.923   1702.756  -0.313246  -0.9974856  1        1        2           2            2
B     6221     0x0000000200004001  1022171714.045827  1022171722.457644  8.411817   1           3        eth:ipv4:tcp             139.45.174.202   ie       "Stripe Inc"                   56071    138.212.190.117  jp       "ASAHI KASEI CORPORATION"      3837     0        10159       5692         14821880     0             0         1460      1458.99     29.41481    0       1.465593  0.0008280156  0.01468998   1207.706  1762031   0.2818119  1           1        2        3           1            1
B     3582     0x0000000200004001  1022171705.686717  1022171714.043794  8.357077   1           3        eth:ipv4:tcp             139.45.174.202   ie       "Stripe Inc"                   56070    138.212.190.117  jp       "ASAHI KASEI CORPORATION"      3820     0        10048       5709         14656900     0             0         1460      1458.688    34.27719    0       1.39519   0.0008317156  0.0141066    1202.334  1753831   0.2753697  1           1        2        5           1            1
A     327      0x0000000200004000  1022171701.712093  1022171726.638722  24.926629  1           3        eth:ipv4:tcp             19.59.134.250    us       "Ford Motor Company"           65230    138.212.187.240  jp       "ASAHI KASEI CORPORATION"      58290    0        9459        5223         13696632     0             1448      1448      1448        0           0       0.067445  0.00263523    0.006627299  379.4737  549477.9  0.2885166  1           1        1        2           2            2
B     69       0x0000000200004001  1022171701.698940  1022171726.629403  24.930463  1           3        eth:ipv4:tcp             138.212.187.219  jp       "ASAHI KASEI CORPORATION"      139      138.212.36.145   jp       "Asahi Kasei Networks Corpor"  2860     0        8978        4413         12814184     25156         68        1460      1427.287    187.9357    0       0.070768  0.002776838   0.005427861  360.1217  513997    0.3409006  0.9960814   1        2        1           3            3
B     77       0x0000000200004001  1022171701.699040  1022171726.629407  24.930367  1           3        eth:ipv4:tcp             138.212.190.224  jp       "ASAHI KASEI CORPORATION"      139      138.212.36.145   jp       "Asahi Kasei Networks Corpor"  2861     0        7319        3622         10446036     20736         39        1460      1427.249    188.4492    0       0.131045  0.003406252   0.006901587  293.5777  419008.5  0.3379033  0.9960377   1        1        1           1            1
B     31       0x0000000200004001  1022171701.715914  1022171726.608383  24.892469  1           3        eth:ipv4:tcp             138.212.187.109  jp       "ASAHI KASEI CORPORATION"      139      138.212.77.73    jp       "Asahi Kasei Networks Corpor"  61340    0        7289        3811         10398387     10773         0         1460      1426.586    205.4521    0       0.204505  0.003415072   0.01080472   292.8195  417732.2  0.3133333  0.9979301   4        2        1           3            0.75
A     4778     0x0000000200004000  1022171709.260746  1022171716.327308  7.066562   1           3        eth:ipv4:tcp             133.26.75.121    jp       "Meiji University"             36237    138.212.185.150  jp       "ASAHI KASEI CORPORATION"      20       0        6865        4465         10008944     0             0         1460      1457.967    49.88806    0       0.404642  0.00102936    0.009408548  971.4766  1416381   0.211827   1           1        3        6           6            6
B     8553     0x0000000200004001  1022171722.458182  1022171726.637621  4.179439   1           3        eth:ipv4:tcp             139.45.174.202   ie       "Stripe Inc"                   56072    138.212.190.117  jp       "ASAHI KASEI CORPORATION"      3854     0        6044        3358         8817920      0             0         1460      1458.954    27.7065     0       0.145115  0.000691502   0.00191674   1446.127  2109833   0.2856839  1           1        1        1           1            1
$

Vlan Flow Aggregation

If you acquire your traffic from an LNS or on a trunk port the VLANs must be integrated in the flow hash as different VLANS with the same five tuple should be separated. If your traffic is stripped of VLANs, then you can ignore the VlanID, as it will always be 0.

Notwithstanding, it might be interesting to aggregate all VLANs with the same five tuple in one flow. One interesting incident, where T2 came to the rescue was a case where egress/ingress traffic was separated in two different VLANs; an accidential misconfig.

So the Vlan mode produced perfect flows and when I switched VLAN aggegation off I had twice the amount of flows, and that made me go to the customer and ask a question like: WTF? And he answered: WTF! And then we found a lot of WTF stuff including illegal access ….

First reset the aggregation mode and feed the 802.1Q_tunneling.cap to T2 in oder to see how the normal flow output looks like.

$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x00
$ t2build tranalyzer2
...
$ t2 -r ~/data/802.1Q_tunneling.cap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 31522
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.6
    02: basicStats, 0.8.6
    03: connStat, 0.8.6
    04: txtSink, 0.8.6
[INF] basicFlow: IPv4 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 312983 (312.98 K)
[INF] basicFlow: IPv6 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 21495 (21.50 K)
Processing file: /home/wurst/data/802.1Q_tunneling.cap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1277840495.135052 sec (Tue 29 Jun 2010 19:41:35 GMT)
Dump stop : 1277840530.538713 sec (Tue 29 Jun 2010 19:42:10 GMT)
Total dump duration: 35.403661 sec
Finished processing. Elapsed time: 0.000137 sec
Finished unloading flow memory. Time: 0.000256 sec
Percentage completed: 100.00%
Number of processed packets: 26
Number of processed bytes: 4686 (4.69 K)
Number of raw bytes: 4686 (4.69 K)
Number of pcap bytes: 5126 (5.13 K)
Number of IPv4 packets: 20 [76.92%]
Number of A packets: 16 [61.54%]
Number of B packets: 10 [38.46%]
Number of A bytes: 3466 (3.47 K) [73.97%]
Number of B bytes: 1220 (1.22 K) [26.03%]
Average A packet load: 216.62
Average B packet load: 122.00
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 1 [3.85%] packets
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 375 [8.00%] bytes
basicStats: Biggest L3 Talker: 10.118.10.1: 5 [19.23%] packets
basicStats: Biggest L3 Talker: 10.118.10.1: 610 [13.02%] bytes
connStat: Number of unique source IPs: 2
connStat: Number of unique destination IPs: 2
connStat: Number of unique source/destination IPs connections: 2
connStat: Max unique number of source IP / destination port connections: 2
connStat: IP prtcon/sdcon, prtcon/scon: 1.000000, 1.000000
connStat: Source IP with max connections: 10.118.10.1: 1 connections
connStat: Destination IP with max connections: 10.118.10.2: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 5, average: 4.69
Max VLAN header count: 2
Number of LLC packets: 6 [23.08%]
Number of ICMP packets: 20 [76.92%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 10
Number of processed A flows: 8 [80.00%]
Number of processed B flows: 2 [20.00%]
Number of request     flows: 8 [80.00%]
Number of reply       flows: 2 [20.00%]
Total   A/B    flow asymmetry: 0.60
Total req/rply flow asymmetry: 0.60
Number of processed   packets/flows: 2.60
Number of processed A packets/flows: 2.00
Number of processed B packets/flows: 5.00
Number of processed total packets/s: 0.73
Number of processed A+B packets/s: 0.73
Number of processed A   packets/s: 0.45
Number of processed   B packets/s: 0.28
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.28
Average full raw bandwidth: 1059 b/s (1.06 Kb/s)
Average full bandwidth : 551 b/s
Max number of flows in memory: 10 [0.00%]
Memory usage: 0.07 GB [0.11%]
Aggregate flow status: 0x0000000000004104
[INF] IPv4
[INF] VLAN encapsulation
[INF] Ethernet flows
$

Max VLAN header count: 2 Hmmm, that means, the packets are encapsulated maximal in two VLANs. Right, let’s look at the flows:

$ tcol ~/results/802.1Q_tunneling_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc                ethVlanID  srcIP        srcIPCC  srcIPWho           srcPort  dstIP        dstIPCC  dstIPWho           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT     stdIAT        pktps     bytps     pktAsm  bytAsm  connSip  connDip  connSipDip  connSipDprt  connF
A     1        0x0000000000004100  1277840495.135052  1277840495.141708  0.006656  1           5        eth:vlan{2}:ipv4:icmp  118;10     10.118.10.1  04       "Private network"  0        10.118.10.2  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.00188   0.0013312  0.0004990491  751.2019  54086.54  0       0       1        1        2           2            2
B     1        0x0000000000004101  1277840495.135910  1277840495.142543  0.006633  1           5        eth:vlan{2}:ipv4:icmp  118;10     10.118.10.2  04       "Private network"  0        10.118.10.1  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001721  0.0013266  0.0005040318  753.8067  54274.09  0       0       1        1        1           1            1
A     2        0x0000000000004100  1277840503.708352  1277840503.714432  0.006080  1           5        eth:vlan{2}:ipv4:icmp  209;20     10.209.20.3  04       "Private network"  0        10.209.20.4  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001733  0.001216   0.0004763269  822.3684  59210.52  0       0       1        1        2           2            2
B     2        0x0000000000004101  1277840503.709181  1277840503.715133  0.005952  1           5        eth:vlan{2}:ipv4:icmp  209;20     10.209.20.4  04       "Private network"  0        10.209.20.3  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001666  0.0011904  0.0004731365  840.0538  60483.87  0       0       1        1        1           1            1
A     3        0x0000000000000104  1277840510.969363  1277840510.969363  0.000000  1           4        eth:vlan:llc:cdp       118        0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            349          0             349       349       349         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     4        0x0000000000000104  1277840511.384783  1277840511.384783  0.000000  1           4        eth:vlan:llc:cdp       209        0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            347          0             347       347       347         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     5        0x0000000000000004  1277840525.369320  1277840525.369320  0.000000  1           3        eth:llc:cdp                       0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            353          0             353       353       353         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     6        0x0000000000000004  1277840525.404193  1277840525.404193  0.000000  1           3        eth:llc:cdp                       0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            353          0             353       353       353         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     7        0x0000000000000104  1277840528.106320  1277840528.106320  0.000000  1           4        eth:vlan:llc:cdp       118        0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            349          0             349       349       349         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     8        0x0000000000000104  1277840530.538713  1277840530.538713  0.000000  1           4        eth:vlan:llc:cdp       209        0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            347          0             347       347       347         0           0       0         0          0             0         0         1       1       0        0        0           0            0
$

Now switch on the VLAN aggregation mode using either of the following configs, which are equivalent:

t2conf tranalyzer2 -D AGGREGATIONFLAG=VLANID

t2conf tranalyzer2 -D AGGREGATIONFLAG=0x20
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=VLANID
$ t2build tranalyzer2
...
$ t2 -r ~/data/802.1Q_tunneling.cap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 31830
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.6
    02: basicStats, 0.8.6
    03: connStat, 0.8.6
    04: txtSink, 0.8.6
[INF] basicFlow: IPv4 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 312983 (312.98 K)
[INF] basicFlow: IPv6 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 21495 (21.50 K)
Processing file: /home/wurst/data/802.1Q_tunneling.cap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1277840495.135052 sec (Tue 29 Jun 2010 19:41:35 GMT)
Dump stop : 1277840530.538713 sec (Tue 29 Jun 2010 19:42:10 GMT)
Total dump duration: 35.403661 sec
Finished processing. Elapsed time: 0.000134 sec
Finished unloading flow memory. Time: 0.000252 sec
Percentage completed: 100.00%
Number of processed packets: 26
Number of processed bytes: 4686 (4.69 K)
Number of raw bytes: 4686 (4.69 K)
Number of pcap bytes: 5126 (5.13 K)
Number of IPv4 packets: 20 [76.92%]
Number of A packets: 16 [61.54%]
Number of B packets: 10 [38.46%]
Number of A bytes: 3466 (3.47 K) [73.97%]
Number of B bytes: 1220 (1.22 K) [26.03%]
Average A packet load: 216.62
Average B packet load: 122.00
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 1 [3.85%] packets
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 375 [8.00%] bytes
basicStats: Biggest L3 Talker: 10.118.10.1: 5 [19.23%] packets
basicStats: Biggest L3 Talker: 10.118.10.1: 610 [13.02%] bytes
connStat: Number of unique source IPs: 2
connStat: Number of unique destination IPs: 2
connStat: Number of unique source/destination IPs connections: 2
connStat: Max unique number of source IP / destination port connections: 2
connStat: IP prtcon/sdcon, prtcon/scon: 1.000000, 1.000000
connStat: Source IP with max connections: 10.118.10.1: 1 connections
connStat: Destination IP with max connections: 10.118.10.2: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 5, average: 4.69
Max VLAN header count: 2
Number of LLC packets: 6 [23.08%]
Number of ICMP packets: 20 [76.92%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 10
Number of processed A flows: 8 [80.00%]
Number of processed B flows: 2 [20.00%]
Number of request     flows: 8 [80.00%]
Number of reply       flows: 2 [20.00%]
Total   A/B    flow asymmetry: 0.60
Total req/rply flow asymmetry: 0.60
Number of processed   packets/flows: 2.60
Number of processed A packets/flows: 2.00
Number of processed B packets/flows: 5.00
Number of processed total packets/s: 0.73
Number of processed A+B packets/s: 0.73
Number of processed A   packets/s: 0.45
Number of processed   B packets/s: 0.28
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.28
Average full raw bandwidth: 1059 b/s (1.06 Kb/s)
Average full bandwidth : 551 b/s
Max number of flows in memory: 10 [0.00%]
Memory usage: 0.07 GB [0.11%]
Aggregate flow status: 0x0000000000004104
[INF] IPv4
[INF] VLAN encapsulation
[INF] Ethernet flows
$

Damn! But why do we have the same amount of flows? Is T2 broken? Or did we take something not into account? Ahhh, the VLANs and IPs are always differnt. If you look now at the ethVlanID it is empty, aka not used in the hash.

$ tcol ~/results/802.1Q_tunneling_flows.txt
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x20
$ t2build tranalyzer2
...
$ tcol ~/results/802.1Q_tunneling_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc                ethVlanID  srcIP        srcIPCC  srcIPWho           srcPort  dstIP        dstIPCC  dstIPWho           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT     stdIAT        pktps     bytps     pktAsm  bytAsm  connSip  connDip  connSipDip  connSipDprt  connF
A     1        0x0000000000004100  1277840495.135052  1277840495.141708  0.006656  1           5        eth:vlan{2}:ipv4:icmp             10.118.10.1  04       "Private network"  0        10.118.10.2  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.00188   0.0013312  0.0004990491  751.2019  54086.54  0       0       1        1        2           2            2
B     1        0x0000000000004101  1277840495.135910  1277840495.142543  0.006633  1           5        eth:vlan{2}:ipv4:icmp             10.118.10.2  04       "Private network"  0        10.118.10.1  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001721  0.0013266  0.0005040318  753.8067  54274.09  0       0       1        1        1           1            1
A     2        0x0000000000004100  1277840503.708352  1277840503.714432  0.006080  1           5        eth:vlan{2}:ipv4:icmp             10.209.20.3  04       "Private network"  0        10.209.20.4  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001733  0.001216   0.0004763269  822.3684  59210.52  0       0       1        1        2           2            2
B     2        0x0000000000004101  1277840503.709181  1277840503.715133  0.005952  1           5        eth:vlan{2}:ipv4:icmp             10.209.20.4  04       "Private network"  0        10.209.20.3  04       "Private network"  0        1        5           5            360          360           72        72        72          0           0       0.001666  0.0011904  0.0004731365  840.0538  60483.87  0       0       1        1        1           1            1
A     3        0x0000000000000104  1277840510.969363  1277840510.969363  0.000000  1           4        eth:vlan:llc:cdp                  0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            349          0             349       349       349         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     4        0x0000000000000104  1277840511.384783  1277840511.384783  0.000000  1           4        eth:vlan:llc:cdp                  0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            347          0             347       347       347         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     5        0x0000000000000004  1277840525.369320  1277840525.369320  0.000000  1           3        eth:llc:cdp                       0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            353          0             353       353       353         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     6        0x0000000000000004  1277840525.404193  1277840525.404193  0.000000  1           3        eth:llc:cdp                       0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            353          0             353       353       353         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     7        0x0000000000000104  1277840528.106320  1277840528.106320  0.000000  1           4        eth:vlan:llc:cdp                  0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            349          0             349       349       349         0           0       0         0          0             0         0         1       1       0        0        0           0            0
A     8        0x0000000000000104  1277840530.538713  1277840530.538713  0.000000  1           4        eth:vlan:llc:cdp                  0.0.0.0      --       "--"               0        0.0.0.0      --       "--"               0        0        1           0            347          0             347       347       347         0           0       0         0          0             0         0         1       1       0        0        0           0            0
$

Try to add the network aggregation mode, how many flows do you expect?

So, that is enough for today and don’t forget to reset the aggregation configuration again for the next tutorials.

$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x00 -D SRCPORTLW=1 -D SRCPORTHW=1024 -D DSTPORTLW=1 -D DSTPORTHW=1024
$ t2conf basicFlow -D BFO_MAC=1 -D BFO_ETHERTYPE=1
$ t2build -R
...
$

Have fun!