Tutorial: Flow Mode

Description

This tutorial details the use of the Flow mode being the default operational technique of the Anteater. As we discuss the enhanced aggregation and masking features in a different tutorial, we like to concentrate here on the flow timeout features often used in anomaly research, such as botnet detection and practical network monitoring applications.

Preparation

In order to do so, we need to prepare T2. If you did not complete the tutorials before, just follow the procedure described below.

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins and compile only basicFlow, basicStats, and txtSink.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow basicStats connStat txtSink
...
BUILD SUCCESSFUL
$

If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

$ mkdir ~/data ~/results
$

The anonymized sample PCAP being used here, can be downloaded here: faf-exercise.pcap. Please extract it under your data folder. Now you are all set for T2 flow timeout experiments.

Flow timeout control

The Flow timeout controls of the Anteater is a core function, hence residing in the tranalyzer.h file:

RELTIME defines the internal time base to be absolute or relative to the pcap or interface ops start; default is absolute. For graphical applications, it is sometimes convenient to have smaller time values relative to 0. We used it in Basic traffic volume and connection analysis to produce a anomaly plot for the connStat plugin.

FLOW_TIMEOUT controls the timeout and releases flows from memory if no more packets are seen since FLOW_TIMEOUT seconds.

It is chosen a bit larger than three minutes, as most L7 protocols have lower timeouts, so we capture most of them into one flow. Nevertheless, a large FLOW_TIMEOUT saves flow allocation and output operations, but also increases the flow liftime in memory. So you need to optimize this value in high speed applications.

FDURLIMIT controls independent of FLOW_TIMEOUT the forced release of each flow in memory, thus being a simulated Cisco flow mode. It is activated with a non zero value in seconds. Hence, a value of 10 would terminate an active flow after 10 seconds if a new packet is encountered, and start a new sub-flow with the next packet having the same flow hash. If there is no new packet with the right hash value, then there is no flow release, until an appropriate packet arrives.

This is very different from FLOW_TIMEOUT because any packet can trigger the flow termination.

It is a built in FORCE MODE as it does not use the T2_RM_FLOW(flowP) function, see forcemode

If FDURLIMIT is enabled FDLSFINDEX controls the enumeration of the sub-flows. Set it to 1 and all sub-flows keep the findex of the original flow. Otherwise the findex increments normally. This feature facilitates to extract sub-flows belonging to a flow.

$ t2 -r ~/data/faf-exercise.pcap -w ~/results/
================================================================================
Tranalyzer 0.8.8 (Anteater), Tarantula. PID: 15929
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.8
    02: basicStats, 0.8.8
    03: txtSink, 0.8.8
[INF] IPv4 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 405533 (405.53 K)
[INF] IPv6 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 49626 (49.63 K)
Processing file: /home/stefan/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.391358 sec
Finished unloading flow memory. Time: 0.391372 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 73
Number of processed A flows: 37 [50.68%]
Number of processed B flows: 36 [49.32%]
Number of request     flows: 36 [49.32%]
Number of reply       flows: 37 [50.68%]
Total   A/B    flow asymmetry: 0.01
Total req/rply flow asymmetry: -0.01
Number of processed   packets/flows: 80.85
Number of processed A packets/flows: 53.68
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B packets/s: 0.12
Number of processed A   packets/s: 0.04
Number of processed   B packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregate flow status: 0x0400000000004000
[INF] IPv4 flows
$

The end report states that there are 73 flows in total and 1986 A flow packets and 3916 B flow packets. Now select a flow which is long enough to be early terminated, such as the one with port 1379 involved.

$ tawk 'port(1379)' faf-exercise3_flows.txt | tcol
dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT    stdIAT    pktps      bytps     pktAsm  bytAsm
A    10       0x0400000000004000  1258562467.749142  1258562509.633370  41.884228  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        14          18           1801         15606         0         455       128.6429    183.7502    0       22.97809  2.991731  6.053773  0.3342547  42.99948  -0.125  -0.7930717
B    10       0x0400000000004001  1258562467.754689  1258562509.653962  41.899273  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        18          14           15606        1801          0         1380      867         568.1608    0       22.97829  2.327737  5.515566  0.4296017  372.4647  0.125   0.7930717
$

Now let us forcefully terminate flows above a certain length in the next chapter.

Flow duration limit: FDURLIMIT

Set FDURLIMIT to 3, recompile and rerun T2 on the pcap. Then t2 will terminate every flow with the timeframe >= 3 seconds and restart a new flow with the same hash but different findex.

$ t2conf tranalyzer2 -D FDURLIMIT=3
$ t2build tranalyzer2
...
$ t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3
================================================================================
Tranalyzer 0.8.8 (Anteater), Tarantula. PID: 62658
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.8
    02: basicStats, 0.8.8
    03: txtSink, 0.8.8
[INF] IPv4 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 405533 (405.53 K)
[INF] IPv6 Ver: 5, Rev: 01052020, Range Mode: 0, subnet ranges loaded: 49626 (49.63 K)
Processing file: /home/stefan/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.138637 sec
Finished unloading flow memory. Time: 0.138651 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 558 [9.45%] packets
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 799321 (799.32 K) [16.01%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of GRE packets: 10 [0.17%]
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 104
Number of processed A flows: 53 [50.96%]
Number of processed B flows: 51 [49.04%]
Number of request     flows: 52 [50.00%]
Number of reply       flows: 52 [50.00%]
Total   A/B    flow asymmetry: 0.02
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 56.75
Number of processed A packets/flows: 37.47
Number of processed B packets/flows: 76.78
Number of processed total packets/s: 0.12
Number of processed A+B packets/s: 0.12
Number of processed A   packets/s: 0.04
Number of processed   B packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregate flow status: 0x0400020000004001
$

As expected we have now more flows, but the same amount of packets, otherwise something would be very wrong. The reporting of basicStats changed as the packets of an specific IP are now scattered over many flows. And the end report has the RMFLOW bit set to indicate that flows are forcefully terminated.

If you extract now the same flow as above you realize that the flows have a smaller length as three seconds. Note the timedifference between timeLast of flow 10 and timeFirst of flow 11 of 10 seconds. Hence, after 10 seconds a new packet came in and flow 10 terminated with its last packet at 1258562467.900050 and restarted B-flow 11 with this very packet at 1258562477.772691. This happens when you have bursty traffic. If the traffic would be regular, all flows would have a duration of three seconds. The RMFLOW bit indicates the flow where the FDURLIMIT timeout occurred.

$ tawk 'port(1379)' faf-exercise3_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A    10       0x0400020000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            449          444           0         449       112.25      159.8441    0       0.144417  0.037727     0.05336916   26.50622  2975.323  0.1428571   0.005599104
B    10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            444          449           0         444       148         170.8957    0       0.006751  0.002334333  0.002289841  428.3878  63401.4   -0.1428571  -0.005599104
A    11       0x0400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1352         15162         0         455       169         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  1825.717  -0.2380952  -0.8362601
B    11       0x0400020000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15162        1352          10        1380      1166.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22253.67  0.2380952   0.8362601
A    13       0x0400020000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    13       0x0400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
A    16       0x0400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    16       0x0400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
$

Yes I picked this traffic on purpose to demonstrate that behaviour of t2. As you can see our flow 10 is now divided into four flows 10,11,13,16. But if you want to select all the sub flows you need to select the port or the full 6 tuple if there are many more flows this the same port or ip pair. In order to facilitate that operation, since 0.8.8lmw2 the FDLSFINEX switch was added to label all subflows with the same findex.

Flow duration, same index: FDLSFINDEX

In order to label all subflows with the same findex the constant FDLSFINDEX has to be enabled in conjunction with a non-zero value of FDURLIMIT in tranalyzer.h or just use tconf as indicated below.

$ t2conf tranalyzer2 -D FDLSFINDEX=1
$ t2build tranalyzer2
$ t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3f
...
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregate flow status: 0x4400020000004001
$

The endreport looks the same as above with the exception that now the FDLSIDX bit is set, indicating that all subflows belonging to the original, unterminated flow have now the same findex. Now select flow 10 again and you discover that the flows 10.11.13.and 16 have now all findex=10 which is much easier to extract as indicated below.

$ tawk 'flow(10)' faf-exercise3f_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A    10       0x0400020000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            449          444           0         449       112.25      159.8441    0       0.144417  0.037727     0.05336916   26.50622  2975.323  0.1428571   0.005599104
B    10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            444          449           0         444       148         170.8957    0       0.006751  0.002334333  0.002289841  428.3878  63401.4   -0.1428571  -0.005599104
A    10       0x4400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1352         15162         0         455       169         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  1825.717  -0.2380952  -0.8362601
B    10       0x4400020000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15162        1352          10        1380      1166.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22253.67  0.2380952   0.8362601
A    10       0x4400020000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    10       0x4400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
A    10       0x4400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    10       0x4400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
$

Now let’s explore the difference to the flow timeout.

No packets seen since: FLOW_TIMEOUT

As explained above the constant FLOW_TIMEOUT controls the lifetime of a flow if a packet was not seen. Which is very different from the FDURLIMIT which terminates a flow and starts a new flow after an internal flow life time.

The timeout of flows can be very useful if you want to follow protocols and change the timeout in dependence of a protocol state, as implemented in tcpStates plugin.

FLOW_TIMEOUT is actually a float, so you can set a timeout Π seconds if you like. We choose 3 as we like to compare it to the previous experiments. So switch FDURLIMIT off, set FLOW_TIMEOUT to 3, recompile and rerun t2 on the same pcap.

$ t2conf tranalyzer2 -D FDURLIMIT=0 -D FLOW_TIMEOUT=3
$ t2build tranalyzer2
$ t2 -r ~/data/faf-exercise.pcap -w ~/results/faf-exercise3t
...
--------------------------------------------------------------------------------
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 Talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 88
Number of processed A flows: 45 [51.14%]
Number of processed B flows: 43 [48.86%]
Number of request     flows: 44 [50.00%]
Number of reply       flows: 44 [50.00%]
Total   A/B    flow asymmetry: 0.02
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 67.07
Number of processed A packets/flows: 44.13
Number of processed B packets/flows: 91.07
Number of processed total packets/s: 0.12
Number of processed A+B packets/s: 0.12
Number of processed A   packets/s: 0.04
Number of processed   B packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.02%]
Aggregate flow status: 0x0400000000004000
[INF] IPv4 flows
$

Now we have 88 flows, not 73 as with 182 sec timeout or 104 with FDURLIMIT. What is the reason? If you select the same port 1379 as before, you discover that flow 10 shows the same behaviour as with FDURLIMIT set to 3 sec, because all flows did not see a packet for 3 seconds, hence they terminate and the following packet landed in the next flow. Oups, bad example to show the difference? Actually not, because it illustrates the need to be observant, because the circumstances of the traffic produce for the this very flow the same flow chunks.

$ tawk 'port(1379)' faf-exercise3t_flows.txt | tcol
dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT       pktps     bytps     pktAsm      bytAsm
A    10       0x0400000000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        4           3            449          444           0         449       112.25      159.8441    0       0.144417  0.037727     0.05336916   26.50622  2975.323  0.1428571   0.005599104
B    10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        3           4            444          449           0         444       148         170.8957    0       0.006751  0.002334333  0.002289841  428.3878  63401.4   -0.1428571  -0.005599104
A    11       0x0400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        8           13           1352         15162         0         455       169         185.5386    0       0.380024  0.09256638   0.09487616   10.80306  1825.717  -0.2380952  -0.8362601
B    11       0x0400000000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        13          8            15162        1352          10        1380      1166.308    353.6241    0       0.426916  0.05240969   0.0970576    19.08044  22253.67  0.2380952   0.8362601
A    13       0x0400000000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    13       0x0400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
A    16       0x0400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
B    16       0x0400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        1           1            0            0             0         0         0           0           0       0         0            0            0         0         0           0
$

Now look for port 49330, you will see two flows, one single A with one packet. The findex numbers differ from each other, therefore we select the port.

$ tawk '' faf-exercise3t_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps     bytps     pktAsm      bytAsm
A     42       0x0400000000004000  1258594163.408285  1258594185.427968  22.019683  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1513        3101         0            4268858       0         0         0           0           0       0.671332  0.01455367   0.0370894   68.71125  0         -0.3441699  -1
B     42       0x0400000000004001  1258594163.487027  1258594185.427506  21.940479  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        3101        1513         4268858      0             0         1380      1376.607    60.23097    0       0.67109   0.007075286  0.02745561  141.3369  194565.4  0.3441699   1
A     44       0x0400000000004000  1258594191.015208  1258594191.015208  0.000000   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1           0            0            0             0         0         0           0           0       0         0            0           0         0         1           0
$

Compare it to the same flow from the previous chapter.

$ tawk 'port(49330)' faf-exercise3f_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT       stdIAT      pktps     bytps     pktAsm      bytAsm
A     36       0x0400000000004000  1258594163.408285  1258594165.927703  2.519418  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        64          103          0            138002        0         0         0           0           0       0.671332  0.03936591   0.1132588   25.40269  0         -0.2335329  -1
B     36       0x0400020000004001  1258594163.487027  1258594165.927248  2.440221  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        103         64           138002       0             0         1380      1339.825    216.4253    0       0.67109   0.02369147   0.09572505  42.20929  56553.07  0.2335329   1
A     36       0x4400000000004000  1258594166.005912  1258594168.988792  2.982880  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        223         444          0            611340        0         0         0           0           0       0.079202  0.01337614   0.02821018  74.75996  0         -0.3313343  -1
B     36       0x4400020000004001  1258594166.005251  1258594168.988590  2.983339  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        444         223          611340       0             80        1380      1376.892    59.41463    0       0.079166  0.006719232  0.02111784  148.8265  204918    0.3313343   1
A     36       0x4400000000004000  1258594169.065239  1258594171.945675  2.880436  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        194         402          0            553903        0         0         0           0           0       0.128673  0.01484761   0.03038867  67.35091  0         -0.3489933  -1
B     36       0x4400020000004001  1258594169.064789  1258594171.944501  2.879712  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        402         194          553903       0             523       1380      1377.868    42.63219    0       0.128665  0.007163462  0.02259158  139.5973  192346.7  0.3489933   1
A     36       0x4400000000004000  1258594172.021878  1258594174.978021  2.956143  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        226         436          0            601495        0         0         0           0           0       0.124433  0.01308028   0.02880465  76.45097  0         -0.3172205  -1
B     36       0x4400020000004001  1258594172.021456  1258594174.977086  2.955630  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        436         226          601495       0             1195      1380      1379.576    8.828124    0       0.124927  0.006778968  0.02202431  147.5151  203508.2  0.3172205   1
A     36       0x4400000000004000  1258594175.055730  1258594177.968394  2.912664  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        233         527          0            726502        0         0         0           0           0       0.079708  0.0125007    0.02766376  79.9955   0         -0.3868421  -1
B     36       0x4400020000004001  1258594175.055030  1258594177.966989  2.911959  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        527         233          726502       0             696       1380      1378.562    29.66002    0       0.07942   0.005525537  0.01959059  180.9778  249489.1  0.3868421   1
A     36       0x4400000000004000  1258594178.040853  1258594180.952029  2.911176  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        261         558          0            769189        0         0         0           0           0       0.078208  0.01115393   0.02653477  89.65449  0         -0.3626374  -1
B     36       0x4400020000004001  1258594178.040152  1258594180.951598  2.911446  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        558         261          769189       0             529       1380      1378.475    35.87016    0       0.078343  0.005217645  0.01904833  191.6573  264194.8  0.3626374   1
A     36       0x4400000000004000  1258594181.028734  1258594183.937406  2.908672  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        221         476          0            656762        0         0         0           0           0       0.079713  0.01316141   0.02855756  75.9797   0         -0.3658537  -1
B     36       0x4400020000004001  1258594181.028052  1258594183.937122  2.909070  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        476         221          656762       0             1262      1380      1379.752    5.360048    0       0.079432  0.006111494  0.02068904  163.6262  225763.6  0.3658537   1
A     36       0x4400020000004000  1258594184.015600  1258594185.427968  1.412368  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        91          155          0            211665        0         0         0           0           0       0.079954  0.01552053   0.03033226  64.43079  0         -0.2601626  -1
B     36       0x4400000000004001  1258594184.015154  1258594185.427506  1.412352  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        155         91           211665       0             0         1380      1365.581    129.1754    0       0.079416  0.009111947  0.02447955  109.746   149867    0.2601626   1
A     36       0x4400000000004000  1258594191.015208  1258594191.015208  0.000000  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        1           0            0            0             0         0         0           0           0       0         0            0           0         0         1           0
$

The FLOW_TIMEOUT produces one more flow 44, a packet which comes after the timeout. Most of the packets are coming very regularly, in contrast to flow 10. The same flow from the previous chapter shows nine flows mostly evenly 3 seconds long. See the differences between those modes?

Do not forget to reset the config to default for the other tutorials.

$ t2conf tranalyzer2 -D FDURLIMIT=0 -D FDLSFINDEX=0 -D FLOW_TIMEOUT=182
$ t2build tranalyzer2
$

Have fun experimenting!