Tutorial: Flow timeout programming

This tutorial introduces the flexible timeout programming of T2. As outlined in the Flow Mode tutorial, the flow timeouts can be altered in a diverse way changing constants in tranalyzer.h preferably via t2conf. Sometimes it is preferable to change the timeout depending on a state of a protocol, like in e.g. tcpStates, which follows the TCP state machine of the sender and receiver. Nevertheless, you can change timeouts of flows for any reason you like.

To facilitate the understanding of the subject we will implement a rudimentary version of tcpStates. But before we start, all unnecessary plugins should be deleted from the plugin folder ~/.tranalyzer/plugins. Make sure that the internal timeout controls are set to default:

$ t2conf tranalyzer2 --reset
$

Then, the core, basicFlow and txtSink must be freshly compiled, so remove all installations in the plugins directory:

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)?y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow txtSink
...
$

I prepared a plugin for this tutorial if you do not want to code yourself using tcpWin: tcpWinTMT

If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

$ mkdir ~/data ~/results
$

The anonymized sample pcap can be downloaded here: faf-exercise.pcap. Please extract it under your data folder: ~/data, if you haven’t already. Now you are all set for first own timeout plugin.

Adding Timeout handlers in packet interrupt

Extract the tar ball of tcpWinTMT under your t2 plugins directory:

$ tranpl
$ tar -xf tcpWinTMT.tar.gz
$

tcpWinTMT.c in an editor. It builds on the original window size counter, used in the previous plugin coding tutorials. I will talk you through all important elements.

First you need to implement a state machine following the protocol TCP. Sure, you could just use the flags from the TCP header which are copied into the variable tcpFlags below and program the state machine depending on that information. But then you need to add tcpFlags as a dependent plugin, and that add unnecessary complexity.

To make it simpler we define internal rudimentary states stored in tcpWinFlowP->state which is an extract of the actual flags. Have a look at the state machine in the L4 packet interrupt below.

At the beginning of a new flow we are in state NEW, if we do not see a SYN flag we will directly move to the established EST state with a timeout TMT_EST. TH_SYN is defined in the core: networkheaders.h. Otherwise a BGN state is entered with a different flow timeout. If a flow is terminated via FIN or RST we move to state ABRT but leave the timeout at TMT_EST to capture packets coming after the RST.

So this is all about handling timeouts with TCP. Now we want to terminate a flow if he exceeds a certain count of window size below threshold counts.

So if the windows zero count hits TCPWIN_THRES in the EST state the flow timeout is greatly reduced in the flow and its reverse flow, hence leading to a new flow for the next packet.

If you do not change the timeout of the reverse flow, it could be that the flow does not get removed, because always both flows have to meet the timeout criteria.

The code is listed below:

...
void claimLayer4Information(packet_t *packet, unsigned long flowIndex) {
        flow_t * const flowP = &flows[flowIndex];
        if (flowP->layer4Protocol != L3_TCP) return; // process only TCP

        // only 1. frag packet will be processed
        if (!t2_is_first_fragment(packet)) return;

        tcpWinFlow_t *tcpWinFlowP = &tcpWinFlows[flowIndex];

        flow_t *revFlowP;
        tcpWinFlow_t *revtcpWinFlowP;

        if (FLOW_HAS_OPPOSITE(flowP)) {
                revFlowP = &flows[flowP->oppositeFlowIndex];
                revtcpWinFlowP = &tcpWinFlows[flowP->oppositeFlowIndex];
        } else {
                revFlowP = NULL;
                revtcpWinFlowP = NULL;
        }

        const tcpHeader_t * const tcpHeader = (tcpHeader_t*)packet->layer4Header;
        const uint32_t tcpWin = ntohs(tcpHeader->window);

        pktTcpCnt++; // count all TCP packets

        if (tcpWin < TCPWIN_THRES) {                    // is windowsize below threshold?
                tcpWinFlowP->winThCnt++;                // count the packet
                tcpWinFlowP->stat |= TCPWIN_THU;        // set the status bit
        }

        const uint8_t tcpFlags = *((uint8_t*)tcpHeader + 13); // copy all tcp flags
        tcpWinFlowP->stat |= tcpFlags & 0x1f; // extract only the main tcp flags (SYN,FIN,RST,ACK,PSH)

        switch (tcpWinFlowP->state) { // internal simplified TCP connection statemachine
                case NEW: // new flow
                        if (tcpFlags & TH_SYN) {
                                flowP->timeout = TMT_BGN;
                                tcpWinFlowP->state = BGN;
                                break;
                        } else if (!(tcpFlags & TH_ACK)) break;
                        tcpWinFlowP->state = EST;
                /*FALLTHRU*/
                case BGN: // begin state after 1. packet
                        flowP->timeout = TMT_EST;
                /*FALLTHRU*/
                case EST: // or established connection
                        if (tcpWinFlowP->stat & (TH_FIN | TH_RST)) {
                                tcpWinFlowP->state = ABRT;
                                flowP->timeout = TMT_EST;
                                break;
                        }
                        if (tcpWinFlowP->winThCnt < TCPWIN_FLWPRG) break; // is wincount above threshold?

                        // if win count thres met, set abort conditions for flow and rev flow
                        tcpWinFlowP->stat |= TCPWIN_ABNRM; // set abort state
                        tcpWinFlowP->state = ABRT;         // set abort status flag
                        flowP->timeout = TMT_ABRT;         // set abort timeout
                        if (revFlowP) {                    // do the same for the reverse flow if it exists
                                revtcpWinFlowP->state = ABRT;
                                revFlowP->timeout = TMT_ABRT;
                        }
                /*FALLTHRU*/
                case ABRT: // abort either FIN or RST
                        flowP->timeout = TMT_ABRT;
                        break;
                default: // anything else do nothing
                        break;

        }

        //output for pkt mode
        if (sPktFile) fprintf(sPktFile, "0x%02"PRIx8"\t0x%02"PRIx8"\t%"PRIu32"\t%"PRIu32"\t%f\t", tcpWinFlowP->stat, tcpWinFlowP->state, tcpWin, tcpWinFlowP->winThCnt, flowP->timeout);
}
...

Note the assignment of the reverse flowP, revtcpWinFlowP. Always use the FLOW_HAS_OPPOSITE(flowP) macro and assign only values to a reverse flow if it actually exists.

To allow better understanding of the core’s operation, I also added some packet mode output for certain variables.

Defining flags, states and timeouts

Open tcpWiniTMT.h in an editor now. If the window size hits zero for two times the timeout is set to TMT_ABRT = 1ms, which mostly timeouts flows in our pcap. The other states have timeouts above 1s, which might affect one flow as we already know from earlier experiments in the Flow Mode tutorial is bursty with interruptions for 3 seconds.

Register timeouts with the core

At startup your timeouts have to be registered with the core, so that every flow is tested at each packet being received from an interface or the pcap. I also added some output for the packet mode that you can watch the timeouts controlling the flows. The code is shown below:

So that is it, define timeouts, add timeout handlers, program your state machine and set flow timeouts appropriately. The rest is flow and packet mode output.

Adding flow output and plugin Reports

Now I’m adding some global variables allowing us to assess the pcap from the end report and the flow file.

At the beginning of tcpWinTMT.c the static definition of the global vars must be present, otherwise the compiler complains massively.

Finally a line for the internal aggregated state is added in the end report, see <---.

Now you are all set, compile tcpWin and invoke T2, also in packet mode.

$ t2build tcpWinTMT
...
$ t2 -r ~/data/faf-exercise.pcap -w ~/results -s

$ tcol ~/results/faf-exercise_flows.txt
================================================================================
Tranalyzer 0.8.11 (Anteater), Tarantula. PID: 9770
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.11
    02: tcpWinTMT, 0.8.11
    03: txtSink, 0.8.11
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406208 (406.21 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51196 (51.20 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.144254 sec
Finished unloading flow memory. Time: 0.144265 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 3534 (3.53 K) [59.88%]
Number of B packets: 2368 (2.37 K) [40.12%]
Number of A bytes: 4423242 (4.42 M) [88.58%]
Number of B bytes: 570172 (570.17 K) [11.42%]
Average A packet load: 1251.62 (1.25 K)
Average B packet load: 240.78
--------------------------------------------------------------------------------
tcpWinTMT: Aggregated tcpWinStat=0xdf
tcpWinTMT: Aggregated tcpWinState=0x07
tcpWinTMT: Number of TCP winsize packets below threshold 1: 4 [0.07%]
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 94
Number of processed A flows: 48 [51.06%]
Number of processed B flows: 46 [48.94%]
Number of request     flows: 47 [50.00%]
Number of reply       flows: 47 [50.00%]
Total   A/B    flow asymmetry: 0.02
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 62.79
Number of processed A packets/flows: 73.62
Number of processed B packets/flows: 51.48
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.07
Number of processed   B   packets/s: 0.05
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 12 [0.00%]
Memory usage: 0.01 GB [0.02%]
Aggregated flowStat=0x0400000000004000
[INF] IPv4 flows

94 flows, normally the pcap produces 73 flows, so we see some effect of our dynamic timeout changes.

We have 4 events of window size hitting 0, and tcpWinStat set to that effect: TCPWIN_ABNRM & TCPWIN_THU both set, so count hit the threshold 2. Flow 44 was aborted with timeout value 1ms. flow 45 was terminated because he passed the timeout=1.2s. And flow 47 contains the rest. The flow listing below shows you the split of the original flow into three sub flows:

$ tawk 'port(49330)' faf-exercise_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg           dstPort  l4Proto  tcpWinStat  tcpWinState  tcpWinThCnt
A     44       0x0400000000004000  1258594163.408285  1258594164.647755  1.239470   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        0xd2        0x04         2
B     44       0x0400000000004001  1258594163.487027  1258594164.647545  1.160518   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        0x1a        0x04         0
A     45       0x0400000000004000  1258594165.318635  1258594185.427506  20.108871  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800              143.166.11.10  us       "Dell"             64334    192.168.1.105  07       "Private network"  49330    6        0x19        0x04         0
B     45       0x0400000000004001  1258594165.319087  1258594185.427968  20.108881  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        0x90        0x02         1
A     47       0x0400000000004000  1258594191.015208  1258594191.015208  0.000000   1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800              192.168.1.105  07       "Private network"  49330    143.166.11.10  us       "Dell"             64334    6        0x94        0x04         1
$

Extract the packets from flow 44 around the win count hit 2. In packet 1422 the tcpWinThCnt reaches two, the timeout is set to 1ms and both A/B flows are removed. The next packet starts with findex 45 and detects another tcpWin zero event.

$ tawk 'packet("1418-1426")' faf-exercise_packets.txt | tcol
%pktNo  flowInd  flowStat            time               pktIAT    flowDuration  numHdrs  hdrDesc       ethVlanID  srcMac             dstMac             ethType  srcIP          srcIPCC  srcIPOrg         srcPort  dstIP          dstIPCC  dstIPOrg         dstPort  l4Proto  tcpWinStat  tcpWinState  tcpWin  tcpWinThCnt  flowTmout  l7Content
1418    44       0x0400000000004001  1258594164.283567  0.000007  0.796540      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             64334    192.168.1.105  07       Private network  49330    6        0x1a        0x01         64860   0            1.000000   ...;......u...!a<.R..\n........nu.._.B..1...............c#.@#.N2.}\...pQZ..e>:p.../.....2rA...H.....F>3.....n.....l.. ...Sz.A+5.......$.l.....S.)...^K5c....Avq?..V=/3...y1.L2.}...8....#....?.&f..9_k..1.!.S|...7......."..F.N.......2........s\n......1.1....p.w.iv.4.H/9)0..Z.:...p_jIk b.w..R.WY.p\r.K....\n98.n:.....W.=.o.9..........F....X3.0tz.k...O.15p...".:.......F.v\t....@...P.............,h.(Z.N1K>.!../~.<n.>u..&.b...i.l..WIS|..W2TONX.N..{..0^.H...)/.S^%....1G\t.:....:..\r.D.@b.G1.1.d....0.f....L+S...[....V.!\. E.7$.6B.....E....c...V?)@?.M....^.EW.p......2.E.M.4=k`.>.p.k....d.....{.....9.Y.:It.D..L&F\nLgtG....tN.m....Sb..s._..J.Z...S=9t+.....D...d....x.b'\n.......eh&...;q.1.W@g(.\...'......_65....)..L..Cf.....c.d#..1@.o.4..........Tg...1......N.......7...t.... T....a.L.....".L.\rt*q1..JU;..t....c\r+].K(.Hl.A.%.u..r.._..c. ,...U....).x.!..9lza.....\l.-_.^_(....[w..N..N.!.v9..:(Z....9.....B.b.0...ue..a....{4Idr.?..n.SP.*.6..b...;erJ......x..3.....i#......Dw8......F......_..J....$.?v...kH. L........v...M.b~.h.s...8c..M/j`.........)4.}i.s. rd..].....86..v....@#\t..v\..A..&n.m..}7P9. 'C."!a$...a...0?.IXg|.u9.K".\t.D..Ero...L.\nCPU.....|i0j....V.\rC8....p\@.Xx.0N5...E..S..{9:#K.."-...A>UMo..qs..z.a.h..b... F....x........x|U.166....~o.j..........)..P..P.v\t.....x..*.k..hc.@@......\r*9=G.....A......9p.W....kw.....B..$...z*u8.:*...Y..]`..........>.....6....P....o...\n..Cf-..\rYwe.ub..
1419    44       0x0400000000004000  1258594164.283986  0.010241  0.875701      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0x12        0x01         2760    0            1.000000
1420    44       0x0400000000004000  1258594164.283988  0.000002  0.875703      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0x92        0x01         0       1            1.000000
1421    44       0x0400000000004001  1258594164.647545  0.363978  1.160518      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             64334    192.168.1.105  07       Private network  49330    6        0x1a        0x01         64860   0            1.000000   .
1422    44       0x0400000000004000  1258594164.647755  0.363767  1.239470      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0xd2        0x04         0       2            0.001000
1423    45       0x0400000000004000  1258594165.318635  0.000000  0.000000      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:08:74:38:01:b4  0x0800   143.166.11.10  us       Dell             64334    192.168.1.105  07       Private network  49330    6        0x10        0x02         64860   0            1.000000   .
1424    45       0x0400000000004001  1258594165.319087  0.000000  0.000000      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0x90        0x02         0       1            1.000000
1425    45       0x0400000000004001  1258594165.848253  0.529166  0.529166      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0x90        0x02         22080   1            1.000000
1426    45       0x0400000000004001  1258594165.848753  0.000500  0.529666      3        eth:ipv4:tcp             00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800   192.168.1.105  07       Private network  49330    143.166.11.10  us       Dell             64334    6        0x90        0x02         64860   1            1.000000

Flow 10 with default timeout, see: flow mode tutorial got split into four flows 10, 11, 13 and 16 because the EST state timeout=1.2s was exceeded.

$ tawk 'port(1379)' faf-exercise_flows.txt | tcol
%dir  flowInd  flowStat            timeFirst          timeLast           duration  numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg               srcPort  dstIP          dstIPCC  dstIPOrg               dstPort  l4Proto  tcpWinStat  tcpWinState  tcpWinThCnt
A     10       0x0400000000004000  1258562467.749142  1258562467.900050  0.150908  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        0x1a        0x01         0
B     10       0x0400000000004001  1258562467.754689  1258562467.761692  0.007003  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        0x1a        0x01         0
A     11       0x0400000000004000  1258562477.713894  1258562478.454425  0.740531  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        0x18        0x02         0
B     11       0x0400000000004001  1258562477.772691  1258562478.454017  0.681326  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        0x18        0x02         0
A     13       0x0400000000004000  1258562501.432514  1258562501.432514  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        0x10        0x02         0
B     13       0x0400000000004001  1258562501.432305  1258562501.432305  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        0x11        0x04         0
A     16       0x0400000000004000  1258562509.633370  1258562509.633370  0.000000  1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              192.168.1.104  07       "Private network"      1379     63.245.221.11  us       "Mozilla Corporation"  80       6        0x11        0x04         0
B     16       0x0400000000004001  1258562509.653962  1258562509.653962  0.000000  1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              63.245.221.11  us       "Mozilla Corporation"  80       192.168.1.104  07       "Private network"      1379     6        0x10        0x02         0
$

As an exercise, change the TMT_EST 4.0, what do you expect? Then change the TCPWIN_FLWPRG to 3. What do you expect?

The next tutorial will teach you how to add the alarm mode

Have fun experimenting!

See Also