Tutorial: Flow Masking and Ranging Aggregation
Contents
Flow Masking and Ranging Aggregation, WTF?
Imagine you are interested in flow based statistics of traffic between networks or between certain port ranges, or you like to get rid of VLANs or protocols? You are not? Go to some place else and make yourself useful. If yes, keep on reading.
Note, I’m not talking about the different L3/4 operational modes already listed in the basics tutorial which discussed in operational aggregation modes.
In this tutorial we explore the masking and ranging flexibility to redefine flows.i So the option to remove one of these parameters or aggregate several IP’s in one flow, e.g. all /24 or according to country and organisation. The first works, the latter is currently in test, and not part with the current distribution.
Preparation
Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.
First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile the standard plugins. Just as a precaution if you have some old plugins or files there. Then build the plugins listed in the command line below:
$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow basicStats connStat txtSink
...
$If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:
$ mkdir ~/data
$ mkdir ~/results
$ cd dataDownload the sample pcaps if did not do it already: annoloc2.pcap and 802.1Q_tunneling.cap.
Now you’re all set.
T2 Flow Aggregation scheme
The constants controlling the flow aggregation are residing in tranalyzer.h. Open the file and search for Aggregation modes as being shown below:
// Aggregation modes
#define L4PROT 0x01
#define DSTPORT 0x02
#define SRCPORT 0x04
#define DSTIP 0x08
#define SRCIP 0x10
#define VLANID 0x20
#define SUBNET 0x80
// Flow Agreggation
#define AGGREGATIONFLAG 0x00 // each bit: 1 : aggregation activated, see aggregation modes #defines above
#define SRCIP4CMSK 24 // src IPv4 aggregation CIDR mask
#define DSTIP4CMSK 24 // dst IPv4 aggregation CIDR mask
#define SRCIP6CMSK 120 // src IPv6 aggregation CIDR mask
#define DSTIP6CMSK 120 // dst IPv6 aggregation CIDR mask
#define SRCPORTLW 53 // src port lower bound
#define SRCPORTHW 53 // src port upper bound
#define DSTPORTLW 53 // dst port lower bound
#define DSTPORTHW 53 // dst port upper bound
...The aggregation modes define a specific bit in the eigth bit AGGREGATIONFLAG. By default it is `0x00, so normal six tuple aggregation. In the following the activation of each mode is discussed. Let’s start with the most powerful one, the network aggregation.
Network Flow Aggregation
Imagine you are interested in traffic only flowing between networks. E.g. 10.4.1.0/24 to 10.5.2.0/24 or from 10.4.5.0/24 to all outside networks. Or even better aggregate all traffic between universities of china and a specific corporation in usa, would that be neat? That will be possible in a later 0.8.x version. The practically tested and successful method of imposing a mask on the src or dst IP in the hash definition is currently the best alternative.
Switch on srcIP and dstIP, it does not make any sense to switch only one on, as we cannot know before the packets is sorted into a masked flow which network mask is to apply. Try the default /24 mask.
The following configs are equivalent
t2conf tranalyzer2 -D AGGREGATIONFLAG="(SRCIP | DSTIP)"
t2conf tranalyzer2 -D AGGREGATIONFLAG=0x18Then compile and execute t2
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x18
$ t2conf basicFlow -D BFO_MAC=0 -D BFO_ETHERTYPE=0
$ t2build tranalyzer2 basicFlow
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 28154
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: basicFlow, 0.8.6
02: basicStats, 0.8.6
03: connStat, 0.8.6
04: txtSink, 0.8.6
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.0: 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.0: 35005508 (35.01 M) [54.63%] bytes
connStat: Number of unique source IPs: 2587 (2.59 K)
connStat: Number of unique destination IPs: 1801 (1.80 K)
connStat: Number of unique source/destination IPs connections: 186
connStat: Max unique number of source IP / destination port connections: 407
connStat: IP prtcon/sdcon, prtcon/scon: 2.188172, 0.157325
connStat: Source IP with max connections: 138.212.189.0 (JP): 499 connections
connStat: Destination IP with max connections: 138.212.187.0 (JP): 690 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 16361 (16.36 K)
Number of processed A flows: 9154 (9.15 K) [55.95%]
Number of processed B flows: 7207 (7.21 K) [44.05%]
Number of request flows: 8690 (8.69 K) [53.11%]
Number of reply flows: 7671 (7.67 K) [46.89%]
Total A/B flow asymmetry: 0.12
Total req/rply flow asymmetry: 0.06
Number of processed packets/flows: 74.51
Number of processed A packets/flows: 61.35
Number of processed B packets/flows: 91.21
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 22511.12 (22.51 K)
Number of processed B packets/s: 26348.71 (26.35 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
$As you can see basicStats and connStat report now /24 networks. The resulting flow file is a bit shorter, because several IPv4/6 flows are aggregated into /24 or /120 flows respectively.
$ cd ~/results
$ tawk 't2sort(numPktsSnt, 10)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm connSip connDip connSipDip connSipDprt connF
B 91 0x0000000200004001 1022171701.699480 1022171726.636773 24.937293 1 3 eth:ipv4:tcp 138.212.189.0 jp "ASAHI KASEI CORPORATION" 139 138.212.86.0 jp "Asahi Kasei Networks Corpor" 3429 6 23601 12342 33733962 42462 103 1460 1429.344 188.7309 0 0.253336 0.001056625 0.003715082 946.4139 1352752 0.313246 0.9974856 1 1 1 1 1
A 91 0x0000000200004000 1022171701.699996 1022171726.637210 24.937214 1 3 eth:ipv4:tcp 138.212.86.0 jp "Asahi Kasei Networks Corpor" 3429 138.212.189.0 jp "ASAHI KASEI CORPORATION" 139 6 12342 23601 42462 33733962 0 63 3.440447 14.30862 0 0.36365 0.002020519 0.00532618 494.923 1702.756 -0.313246 -0.9974856 1 1 2 2 2
B 5934 0x0000000200004001 1022171714.045827 1022171722.457644 8.411817 1 3 eth:ipv4:tcp 139.45.174.0 ie "Stripe Inc" 56071 138.212.190.0 jp "ASAHI KASEI CORPORATION" 3837 6 10159 5692 14821880 0 0 1460 1458.99 29.41481 0 1.465593 0.0008280156 0.01468998 1207.706 1762031 0.2818119 1 1 96 3 1 1
B 3462 0x0000000200004001 1022171705.686717 1022171714.043794 8.357077 1 3 eth:ipv4:tcp 139.45.174.0 ie "Stripe Inc" 56070 138.212.190.0 jp "ASAHI KASEI CORPORATION" 3820 6 10048 5709 14656900 0 0 1460 1458.688 34.27719 0 1.39519 0.0008317156 0.0141066 1202.334 1753831 0.2753697 1 1 192 5 1 1
A 324 0x0000000200004000 1022171701.712093 1022171726.638722 24.926629 1 3 eth:ipv4:tcp 19.59.134.0 us "Ford Motor Company" 65230 138.212.187.0 jp "ASAHI KASEI CORPORATION" 58290 6 9459 5223 13696632 0 1448 1448 1448 0 0 0.067445 0.00263523 0.006627299 379.4737 549477.9 0.2885166 1 1 2 2 2 2
B 69 0x0000000200004001 1022171701.698940 1022171726.629403 24.930463 1 3 eth:ipv4:tcp 138.212.187.0 jp "ASAHI KASEI CORPORATION" 139 138.212.36.0 jp "Asahi Kasei Networks Corpor" 2860 6 8978 4413 12814184 25156 68 1460 1427.287 187.9357 0 0.070768 0.002776838 0.005427861 360.1217 513997 0.3409006 0.9960814 13 2 1 3 0.2307692
B 77 0x0000000200004001 1022171701.699040 1022171726.629407 24.930367 1 3 eth:ipv4:tcp 138.212.190.0 jp "ASAHI KASEI CORPORATION" 139 138.212.36.0 jp "Asahi Kasei Networks Corpor" 2861 6 7319 3622 10446036 20736 39 1460 1427.249 188.4492 0 0.131045 0.003406252 0.006901587 293.5777 419008.5 0.3379033 0.9960377 6 1 1 1 0.1666667
B 31 0x0000000200004001 1022171701.715914 1022171726.608383 24.892469 1 3 eth:ipv4:tcp 138.212.187.0 jp "ASAHI KASEI CORPORATION" 139 138.212.77.0 jp "Asahi Kasei Networks Corpor" 61340 6 7289 3811 10398387 10773 0 1460 1426.586 205.4521 0 0.204505 0.003415072 0.01080472 292.8195 417732.2 0.3133333 0.9979301 42 2 1 3 0.07142857
A 4585 0x0000000200004000 1022171709.260746 1022171716.327308 7.066562 1 3 eth:ipv4:tcp 133.26.75.0 jp "Meiji University" 36237 138.212.185.0 jp "ASAHI KASEI CORPORATION" 20 6 6865 4465 10008944 0 0 1460 1457.967 49.88806 0 0.404642 0.00102936 0.009408548 971.4766 1416381 0.211827 1 5 270 10 6 1.2
B 8091 0x0000000200004001 1022171722.458182 1022171726.637621 4.179439 1 3 eth:ipv4:tcp 139.45.174.0 ie "Stripe Inc" 56072 138.212.190.0 jp "ASAHI KASEI CORPORATION" 3854 6 6044 3358 8817920 0 0 1460 1458.954 27.7065 0 0.145115 0.000691502 0.00191674 1446.127 2109833 0.2856839 1 1 1 1 1 1
$Try to set the aggregation mask for SRCIP4CMSK and DSTIP4CMSK to 8 using t2conf, recompile and rerun t2 as home work. If you do reset it to /24 again for the next chapter.
Port Flow Aggregation
It serves a good purpose if you are interested in flow reduction with a specific statistically questions on your mind and not being interested in the actual flow content. So don’t load any L7 plugins in carving mode as the statemachines get corrupted, as different packets from different six-tuple flows are aggregated.
It can be useful for all plugins which are NOT initiated by ports, all statistical plugins, httpSniffer.
In order to enable the port aggregation mode the following configs are equivalent:
t2conf tranalyzer2 -D AGGREGATIONFLAG="(SRCPORT | DSTPORT)"
t2conf tranalyzer2 -D AGGREGATIONFLAG=0x06Then the constants listed below define the range of ports which be aggregated into port class 1, the rest is shoved into port class 0. The default class 1 is defined by the standardized ports.
...
#define AGGREGATIONFLAG 0x00 // each bit: 1 : aggregation activated, see aggregation modes #defines above
...
#define SRCPORTLW 1 // src port lower bound
#define SRCPORTHW 1024 // src port upper bound
#define DSTPORTLW 1 // dst port lower bound
#define DSTPORTHW 1024 // dst port upper bound
...Let’s change the upper range to 23, aka telnet. So we throw all services into class 1, which you should not appear in modern networks anymore. Then switch on the appropriate aggregation mode, compile and rerun t2.
$ t2conf tranalyzer2 -D SRCPORTHW=23 -D DSTPORTHW=23 -D AGGREGATIONFLAG=0x06
$ t2build tranalyzer2
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 139.45.174.202 (IE): 30343 (30.34 K) [2.49%] packets
basicStats: Biggest L3 Talker: 139.45.174.202 (IE): 45902872 (45.90 M) [71.63%] bytes
connStat: Number of unique source IPs: 4040 (4.04 K)
connStat: Number of unique destination IPs: 3181 (3.18 K)
connStat: Number of unique source/destination IPs connections: 4
connStat: Max unique number of source IP / destination port connections: 586
connStat: IP prtcon/sdcon, prtcon/scon: 146.500000, 0.145050
connStat: Source IP with max connections: 138.212.189.66 (JP): 369 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 400 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 12957 (12.96 K)
Number of processed A flows: 7503 (7.50 K) [57.91%]
Number of processed B flows: 5454 (5.45 K) [42.09%]
Number of request flows: 7021 (7.02 K) [54.19%]
Number of reply flows: 5936 (5.94 K) [45.81%]
Total A/B flow asymmetry: 0.16
Total req/rply flow asymmetry: 0.08
Number of processed packets/flows: 94.08
Number of processed A packets/flows: 82.07
Number of processed B packets/flows: 110.61
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 24680.56 (24.68 K)
Number of processed B packets/s: 24179.27 (24.18 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...Let’s find the firsst 10 biggest talkers using unencrypted legacy services, you do not want to use anymore in your corporate network:
$ cd ~/results
$ tawk '{if ($srcPort == 1) print $srcIP, $numPktsSnt}' annoloc2_flows.txt | sort -nr -k2 | head -n 10
38.212.185.150 8696
138.212.190.31 5276
138.212.188.251 5107
138.212.185.102 4056
138.212.186.210 3259
138.212.190.31 3080
193.86.146.215 2885
138.212.188.139 2466
138.212.184.244 2273
138.212.187.170 2156
$Oups, that is bad.
Protocol Flow Aggregation
The following configs are equivalent
t2conf tranalyzer2 -D AGGREGATIONFLAG=L4PROT
t2conf tranalyzer2 -D AGGREGATIONFLAG=0x01$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x01
$ t2build tranalyzer2
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results/
...
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 57 [0.00%] packets
basicStats: Biggest L2 Talker: 00:d0:02:6d:78:00: 3420 (3.42 K) [0.01%] bytes
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest L3 Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
connStat: Number of unique source IPs: 4185 (4.18 K)
connStat: Number of unique destination IPs: 3071 (3.07 K)
connStat: Number of unique source/destination IPs connections: 182
connStat: Max unique number of source IP / destination port connections: 413
connStat: IP prtcon/sdcon, prtcon/scon: 2.269231, 0.098686
connStat: Source IP with max connections: 138.212.189.66 (JP): 368 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 402 connections
--------------------------------------------------------------------------------
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 17088 (17.09 K)
Number of processed A flows: 9705 (9.71 K) [56.79%]
Number of processed B flows: 7383 (7.38 K) [43.21%]
Number of request flows: 9662 (9.66 K) [56.54%]
Number of reply flows: 7426 (7.43 K) [43.46%]
Total A/B flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed packets/flows: 71.34
Number of processed A packets/flows: 58.14
Number of processed B packets/flows: 88.69
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 22615.05 (22.61 K)
Number of processed B packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ cd ~/results
$ tawk 't2sort(numPktsSnt, 10)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm connSip connDip connSipDip connSipDprt connF
B 91 0x0000000200004001 1022171701.699480 1022171726.636773 24.937293 1 3 eth:ipv4:tcp 138.212.189.38 jp "ASAHI KASEI CORPORATION" 139 138.212.86.201 jp "Asahi Kasei Networks Corpor" 3429 0 23601 12342 33733962 42462 103 1460 1429.344 188.7309 0 0.253336 0.001056625 0.003715082 946.4139 1352752 0.313246 0.9974856 1 1 1 1 1
A 91 0x0000000200004000 1022171701.699996 1022171726.637210 24.937214 1 3 eth:ipv4:tcp 138.212.86.201 jp "Asahi Kasei Networks Corpor" 3429 138.212.189.38 jp "ASAHI KASEI CORPORATION" 139 0 12342 23601 42462 33733962 0 63 3.440447 14.30862 0 0.36365 0.002020519 0.00532618 494.923 1702.756 -0.313246 -0.9974856 1 1 2 2 2
B 6221 0x0000000200004001 1022171714.045827 1022171722.457644 8.411817 1 3 eth:ipv4:tcp 139.45.174.202 ie "Stripe Inc" 56071 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3837 0 10159 5692 14821880 0 0 1460 1458.99 29.41481 0 1.465593 0.0008280156 0.01468998 1207.706 1762031 0.2818119 1 1 2 3 1 1
B 3582 0x0000000200004001 1022171705.686717 1022171714.043794 8.357077 1 3 eth:ipv4:tcp 139.45.174.202 ie "Stripe Inc" 56070 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3820 0 10048 5709 14656900 0 0 1460 1458.688 34.27719 0 1.39519 0.0008317156 0.0141066 1202.334 1753831 0.2753697 1 1 2 5 1 1
A 327 0x0000000200004000 1022171701.712093 1022171726.638722 24.926629 1 3 eth:ipv4:tcp 19.59.134.250 us "Ford Motor Company" 65230 138.212.187.240 jp "ASAHI KASEI CORPORATION" 58290 0 9459 5223 13696632 0 1448 1448 1448 0 0 0.067445 0.00263523 0.006627299 379.4737 549477.9 0.2885166 1 1 1 2 2 2
B 69 0x0000000200004001 1022171701.698940 1022171726.629403 24.930463 1 3 eth:ipv4:tcp 138.212.187.219 jp "ASAHI KASEI CORPORATION" 139 138.212.36.145 jp "Asahi Kasei Networks Corpor" 2860 0 8978 4413 12814184 25156 68 1460 1427.287 187.9357 0 0.070768 0.002776838 0.005427861 360.1217 513997 0.3409006 0.9960814 1 2 1 3 3
B 77 0x0000000200004001 1022171701.699040 1022171726.629407 24.930367 1 3 eth:ipv4:tcp 138.212.190.224 jp "ASAHI KASEI CORPORATION" 139 138.212.36.145 jp "Asahi Kasei Networks Corpor" 2861 0 7319 3622 10446036 20736 39 1460 1427.249 188.4492 0 0.131045 0.003406252 0.006901587 293.5777 419008.5 0.3379033 0.9960377 1 1 1 1 1
B 31 0x0000000200004001 1022171701.715914 1022171726.608383 24.892469 1 3 eth:ipv4:tcp 138.212.187.109 jp "ASAHI KASEI CORPORATION" 139 138.212.77.73 jp "Asahi Kasei Networks Corpor" 61340 0 7289 3811 10398387 10773 0 1460 1426.586 205.4521 0 0.204505 0.003415072 0.01080472 292.8195 417732.2 0.3133333 0.9979301 4 2 1 3 0.75
A 4778 0x0000000200004000 1022171709.260746 1022171716.327308 7.066562 1 3 eth:ipv4:tcp 133.26.75.121 jp "Meiji University" 36237 138.212.185.150 jp "ASAHI KASEI CORPORATION" 20 0 6865 4465 10008944 0 0 1460 1457.967 49.88806 0 0.404642 0.00102936 0.009408548 971.4766 1416381 0.211827 1 1 3 6 6 6
B 8553 0x0000000200004001 1022171722.458182 1022171726.637621 4.179439 1 3 eth:ipv4:tcp 139.45.174.202 ie "Stripe Inc" 56072 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3854 0 6044 3358 8817920 0 0 1460 1458.954 27.7065 0 0.145115 0.000691502 0.00191674 1446.127 2109833 0.2856839 1 1 1 1 1 1
$Vlan Flow Aggregation
If you acquire your traffic from an LNS or on a trunk port the VLANs must be integrated in the flow hash as different VLANS with the same five tuple should be separated. If your traffic is stripped of VLANs, then you can ignore the VlanID, as it will always be 0.
Notwithstanding, it might be interesting to aggregate all VLANs with the same five tuple in one flow. One interesting incident, where T2 came to the rescue was a case where egress/ingress traffic was separated in two different VLANs; an accidential misconfig.
So the Vlan mode produced perfect flows and when I switched VLAN aggegation off I had twice the amount of flows, and that made me go to the customer and ask a question like: WTF? And he answered: WTF! And then we found a lot of WTF stuff including illegal access ….
First reset the aggregation mode and feed the 802.1Q_tunneling.cap to T2 in oder to see how the normal flow output looks like.
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x00
$ t2build tranalyzer2
...
$ t2 -r ~/data/802.1Q_tunneling.cap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 31522
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: basicFlow, 0.8.6
02: basicStats, 0.8.6
03: connStat, 0.8.6
04: txtSink, 0.8.6
[INF] basicFlow: IPv4 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 312983 (312.98 K)
[INF] basicFlow: IPv6 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 21495 (21.50 K)
Processing file: /home/wurst/data/802.1Q_tunneling.cap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1277840495.135052 sec (Tue 29 Jun 2010 19:41:35 GMT)
Dump stop : 1277840530.538713 sec (Tue 29 Jun 2010 19:42:10 GMT)
Total dump duration: 35.403661 sec
Finished processing. Elapsed time: 0.000137 sec
Finished unloading flow memory. Time: 0.000256 sec
Percentage completed: 100.00%
Number of processed packets: 26
Number of processed bytes: 4686 (4.69 K)
Number of raw bytes: 4686 (4.69 K)
Number of pcap bytes: 5126 (5.13 K)
Number of IPv4 packets: 20 [76.92%]
Number of A packets: 16 [61.54%]
Number of B packets: 10 [38.46%]
Number of A bytes: 3466 (3.47 K) [73.97%]
Number of B bytes: 1220 (1.22 K) [26.03%]
Average A packet load: 216.62
Average B packet load: 122.00
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 1 [3.85%] packets
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 375 [8.00%] bytes
basicStats: Biggest L3 Talker: 10.118.10.1: 5 [19.23%] packets
basicStats: Biggest L3 Talker: 10.118.10.1: 610 [13.02%] bytes
connStat: Number of unique source IPs: 2
connStat: Number of unique destination IPs: 2
connStat: Number of unique source/destination IPs connections: 2
connStat: Max unique number of source IP / destination port connections: 2
connStat: IP prtcon/sdcon, prtcon/scon: 1.000000, 1.000000
connStat: Source IP with max connections: 10.118.10.1: 1 connections
connStat: Destination IP with max connections: 10.118.10.2: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 5, average: 4.69
Max VLAN header count: 2
Number of LLC packets: 6 [23.08%]
Number of ICMP packets: 20 [76.92%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 10
Number of processed A flows: 8 [80.00%]
Number of processed B flows: 2 [20.00%]
Number of request flows: 8 [80.00%]
Number of reply flows: 2 [20.00%]
Total A/B flow asymmetry: 0.60
Total req/rply flow asymmetry: 0.60
Number of processed packets/flows: 2.60
Number of processed A packets/flows: 2.00
Number of processed B packets/flows: 5.00
Number of processed total packets/s: 0.73
Number of processed A+B packets/s: 0.73
Number of processed A packets/s: 0.45
Number of processed B packets/s: 0.28
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.28
Average full raw bandwidth: 1059 b/s (1.06 Kb/s)
Average full bandwidth : 551 b/s
Max number of flows in memory: 10 [0.00%]
Memory usage: 0.07 GB [0.11%]
Aggregate flow status: 0x0000000000004104
[INF] IPv4
[INF] VLAN encapsulation
[INF] Ethernet flows
$Max VLAN header count: 2 Hmmm, that means, the packets are encapsulated maximal in two VLANs. Right, let’s look at the flows:
$ tcol ~/results/802.1Q_tunneling_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm connSip connDip connSipDip connSipDprt connF
A 1 0x0000000000004100 1277840495.135052 1277840495.141708 0.006656 1 5 eth:vlan{2}:ipv4:icmp 118;10 10.118.10.1 04 "Private network" 0 10.118.10.2 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.00188 0.0013312 0.0004990491 751.2019 54086.54 0 0 1 1 2 2 2
B 1 0x0000000000004101 1277840495.135910 1277840495.142543 0.006633 1 5 eth:vlan{2}:ipv4:icmp 118;10 10.118.10.2 04 "Private network" 0 10.118.10.1 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001721 0.0013266 0.0005040318 753.8067 54274.09 0 0 1 1 1 1 1
A 2 0x0000000000004100 1277840503.708352 1277840503.714432 0.006080 1 5 eth:vlan{2}:ipv4:icmp 209;20 10.209.20.3 04 "Private network" 0 10.209.20.4 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001733 0.001216 0.0004763269 822.3684 59210.52 0 0 1 1 2 2 2
B 2 0x0000000000004101 1277840503.709181 1277840503.715133 0.005952 1 5 eth:vlan{2}:ipv4:icmp 209;20 10.209.20.4 04 "Private network" 0 10.209.20.3 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001666 0.0011904 0.0004731365 840.0538 60483.87 0 0 1 1 1 1 1
A 3 0x0000000000000104 1277840510.969363 1277840510.969363 0.000000 1 4 eth:vlan:llc:cdp 118 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 349 0 349 349 349 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 4 0x0000000000000104 1277840511.384783 1277840511.384783 0.000000 1 4 eth:vlan:llc:cdp 209 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 347 0 347 347 347 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 5 0x0000000000000004 1277840525.369320 1277840525.369320 0.000000 1 3 eth:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 353 0 353 353 353 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 6 0x0000000000000004 1277840525.404193 1277840525.404193 0.000000 1 3 eth:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 353 0 353 353 353 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 7 0x0000000000000104 1277840528.106320 1277840528.106320 0.000000 1 4 eth:vlan:llc:cdp 118 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 349 0 349 349 349 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 8 0x0000000000000104 1277840530.538713 1277840530.538713 0.000000 1 4 eth:vlan:llc:cdp 209 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 347 0 347 347 347 0 0 0 0 0 0 0 1 1 0 0 0 0 0
$Now switch on the VLAN aggregation mode using either of the following configs, which are equivalent:
t2conf tranalyzer2 -D AGGREGATIONFLAG=VLANID
t2conf tranalyzer2 -D AGGREGATIONFLAG=0x20$ t2conf tranalyzer2 -D AGGREGATIONFLAG=VLANID
$ t2build tranalyzer2
...
$ t2 -r ~/data/802.1Q_tunneling.cap -w ~/results/
================================================================================
Tranalyzer 0.8.6 (Anteater), Tarantula. PID: 31830
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: basicFlow, 0.8.6
02: basicStats, 0.8.6
03: connStat, 0.8.6
04: txtSink, 0.8.6
[INF] basicFlow: IPv4 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 312983 (312.98 K)
[INF] basicFlow: IPv6 Ver: 4, Rev: 01102019, Range Mode: 0, subnet ranges loaded: 21495 (21.50 K)
Processing file: /home/wurst/data/802.1Q_tunneling.cap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1277840495.135052 sec (Tue 29 Jun 2010 19:41:35 GMT)
Dump stop : 1277840530.538713 sec (Tue 29 Jun 2010 19:42:10 GMT)
Total dump duration: 35.403661 sec
Finished processing. Elapsed time: 0.000134 sec
Finished unloading flow memory. Time: 0.000252 sec
Percentage completed: 100.00%
Number of processed packets: 26
Number of processed bytes: 4686 (4.69 K)
Number of raw bytes: 4686 (4.69 K)
Number of pcap bytes: 5126 (5.13 K)
Number of IPv4 packets: 20 [76.92%]
Number of A packets: 16 [61.54%]
Number of B packets: 10 [38.46%]
Number of A bytes: 3466 (3.47 K) [73.97%]
Number of B bytes: 1220 (1.22 K) [26.03%]
Average A packet load: 216.62
Average B packet load: 122.00
--------------------------------------------------------------------------------
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 1 [3.85%] packets
basicStats: Biggest L2 Talker: 00:13:c3:df:ae:18: 375 [8.00%] bytes
basicStats: Biggest L3 Talker: 10.118.10.1: 5 [19.23%] packets
basicStats: Biggest L3 Talker: 10.118.10.1: 610 [13.02%] bytes
connStat: Number of unique source IPs: 2
connStat: Number of unique destination IPs: 2
connStat: Number of unique source/destination IPs connections: 2
connStat: Max unique number of source IP / destination port connections: 2
connStat: IP prtcon/sdcon, prtcon/scon: 1.000000, 1.000000
connStat: Source IP with max connections: 10.118.10.1: 1 connections
connStat: Destination IP with max connections: 10.118.10.2: 1 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 5, average: 4.69
Max VLAN header count: 2
Number of LLC packets: 6 [23.08%]
Number of ICMP packets: 20 [76.92%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 10
Number of processed A flows: 8 [80.00%]
Number of processed B flows: 2 [20.00%]
Number of request flows: 8 [80.00%]
Number of reply flows: 2 [20.00%]
Total A/B flow asymmetry: 0.60
Total req/rply flow asymmetry: 0.60
Number of processed packets/flows: 2.60
Number of processed A packets/flows: 2.00
Number of processed B packets/flows: 5.00
Number of processed total packets/s: 0.73
Number of processed A+B packets/s: 0.73
Number of processed A packets/s: 0.45
Number of processed B packets/s: 0.28
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.28
Average full raw bandwidth: 1059 b/s (1.06 Kb/s)
Average full bandwidth : 551 b/s
Max number of flows in memory: 10 [0.00%]
Memory usage: 0.07 GB [0.11%]
Aggregate flow status: 0x0000000000004104
[INF] IPv4
[INF] VLAN encapsulation
[INF] Ethernet flows
$Damn! But why do we have the same amount of flows? Is T2 broken? Or did we take something not into account? Ahhh, the VLANs and IPs are always differnt. If you look now at the ethVlanID it is empty, aka not used in the hash.
$ tcol ~/results/802.1Q_tunneling_flows.txt
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x20
$ t2build tranalyzer2
...
$ tcol ~/results/802.1Q_tunneling_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm connSip connDip connSipDip connSipDprt connF
A 1 0x0000000000004100 1277840495.135052 1277840495.141708 0.006656 1 5 eth:vlan{2}:ipv4:icmp 10.118.10.1 04 "Private network" 0 10.118.10.2 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.00188 0.0013312 0.0004990491 751.2019 54086.54 0 0 1 1 2 2 2
B 1 0x0000000000004101 1277840495.135910 1277840495.142543 0.006633 1 5 eth:vlan{2}:ipv4:icmp 10.118.10.2 04 "Private network" 0 10.118.10.1 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001721 0.0013266 0.0005040318 753.8067 54274.09 0 0 1 1 1 1 1
A 2 0x0000000000004100 1277840503.708352 1277840503.714432 0.006080 1 5 eth:vlan{2}:ipv4:icmp 10.209.20.3 04 "Private network" 0 10.209.20.4 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001733 0.001216 0.0004763269 822.3684 59210.52 0 0 1 1 2 2 2
B 2 0x0000000000004101 1277840503.709181 1277840503.715133 0.005952 1 5 eth:vlan{2}:ipv4:icmp 10.209.20.4 04 "Private network" 0 10.209.20.3 04 "Private network" 0 1 5 5 360 360 72 72 72 0 0 0.001666 0.0011904 0.0004731365 840.0538 60483.87 0 0 1 1 1 1 1
A 3 0x0000000000000104 1277840510.969363 1277840510.969363 0.000000 1 4 eth:vlan:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 349 0 349 349 349 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 4 0x0000000000000104 1277840511.384783 1277840511.384783 0.000000 1 4 eth:vlan:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 347 0 347 347 347 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 5 0x0000000000000004 1277840525.369320 1277840525.369320 0.000000 1 3 eth:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 353 0 353 353 353 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 6 0x0000000000000004 1277840525.404193 1277840525.404193 0.000000 1 3 eth:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 353 0 353 353 353 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 7 0x0000000000000104 1277840528.106320 1277840528.106320 0.000000 1 4 eth:vlan:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 349 0 349 349 349 0 0 0 0 0 0 0 1 1 0 0 0 0 0
A 8 0x0000000000000104 1277840530.538713 1277840530.538713 0.000000 1 4 eth:vlan:llc:cdp 0.0.0.0 -- "--" 0 0.0.0.0 -- "--" 0 0 1 0 347 0 347 347 347 0 0 0 0 0 0 0 1 1 0 0 0 0 0
$Try to add the network aggregation mode, how many flows do you expect?
So, that is enough for today and don’t forget to reset the aggregation configuration again for the next tutorials.
$ t2conf tranalyzer2 -D AGGREGATIONFLAG=0x00 -D SRCPORTLW=1 -D SRCPORTHW=1024 -D DSTPORTLW=1 -D DSTPORTHW=1024
$ t2conf basicFlow -D BFO_MAC=1 -D BFO_ETHERTYPE=1
$ t2build -R
...
$Have fun!