Tranalyzer - Plugin summary files

Tutorial: Plugin summary files

Summary files
See Also

When all data is processed or T2 is interrupted by ^C or a signal, then the end report is printed. At the same time certain global information can be printed by your plugin. Before we start doing this, clean out your plugin .so directory if this is the first tutorial you follow. Then all unnecessary plugins should be deleted from the plugin folder ~/.tranalyzer/plugins and compile the core and basicFlow.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow
...

I prepared a plugin which we will discuss in the following: tcpWinSD Copy it to the plugins folder of your t2 installation and unpack it:

$ cd tranayzer2-0.8.9/plugins
$ tar -xf tcpWinSD.tar.gz
$ source ../scripts/t2_aliases
$

The anonymized sample pcap can be downloaded here: annoloc2.pcap. Please extract it under your data folder: ~/data, if you haven’t already. Now you are all set for summary file programming.

Summary files

Every plugin can produce a summary of traffic content. Let’s generate a file which lists all IPs with their maximal winThCnts, so we could find the IP having the largest L4 trouble. In order to do so, we first of all need to define all necessary variables being global to the plugin.

So move to the tcpWinSD plugin and open tcpWinSD.c

$ source 
$ tcpWinSD
$ vi src/tcpWin.c

In order to produce just summary files no flowTermination callback is needed, you can do everything in a packet callback, but that is inefficient, as you go over the code in every packet. If it is possible, it is always good practice to move heavy processing to flowGeneration/Termination, as it is only called once per flow.

For the matter of simplicity we just have arrays and a global index as defined in the global section of the plugin.

...
/*
 * Plugin variables that may be used by other plugins (MUST be declared in
 * the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
 */
tcpWinFlow_t *tcpWinFlows;

...

// window size counts
gwz_t gwz;          // <-- global window size structure 
gwz_t *gwzP = &gwz; // <-- Pointer for later dependency experiments
...

Now go to the onFlowTerminate(...) callback. First we need the pointer to the flow structure. To show the principal we just do a linear search whether we already have the IP stored and check whether its counter is greater than the potential existing, then we store the new data. Yes, it can be more elegant, using a hash or a tree. We will discuss T2 support for that matter in tutorial block t2kunfu. Note that the following code is only for IPv4, as we want to start simple.

void onFlowTerminate(unsigned long flowIndex) {
    const flow_t * const flowP = &flows[flowIndex];
    tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];

    if (tcpWinFlowP->winThCnt && tcpWinFlowP->pktTcpCnt >= TCPWIN_MINPKTS) {
        const int wzi = gwz.wzi;     // store element count in const local variable, makes the compiler happy

        if (wzi < TCPWIN_MAXWSCNT) { // If array full, stop saving
            int i; // 
            for (i = 0; i < wzi; i++) if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP already exist?

    	    const float f = (float)tcpWinFlowP->winThCnt/(float)tcpWinFlowP->pktTcpCnt; // produce a useful relative number

            if (f > gwz.wzCnt[i]) {                     // only update if rel count is greater than the previous one
                gwz.tcpCnt[i] = tcpWinFlowP->pktTcpCnt; // update tcp packet count
                gwz.wzCnt[i] = f;                       // update relative count
                if (i == wzi) {				// new one?
                    gwz.wzip[i] = flowP->srcIP;         // save new IP
                    gwz.wzi++;                          // increment global window size counter
                }
            }
        }
    }

}

The onApplicationTerminate() callback not only serves to free flow memory but also to produce all kinds of global output into specialized files. The code below stores the IP and its count. Just add it after the free(...) command.

void onApplicationTerminate() {

    free(tcpWinFlows); // free the tcpWin Flows

    // open TCPWIN statistics file
    FILE *fp;
    int i;
    char srcIP[INET6_ADDRSTRLEN];

    fp = t2_open_file(baseFileName, TCPWIN_FNSUP, "w");
    if (UNLIKELY(!fp)) { // if file cannot be opened print warning and return;
        T2_PWRN("tcpWin", "Failed to allocate memory for: %s", TCPWIN_FNSUP);
        return;
    }

    fprintf(fp, "# IP\tpktTcpCnt\twinRelThCnt\n"); // print header

    for (i = 0; i < gwz.wzi; i++) {
        T2_IP_TO_STR(gwz.wzip[i], 4, srcIP, INET6_ADDRSTRLEN);                  // transfer IP to string
        fprintf(fp, "%s\t%"PRIu32"\t%f\n", srcIP, gwz.tcpCnt[i], gwz.wzCnt[i]); // print in file
    }

    fclose(fp);
}

Now open tcpWin.h and add the maximal size of the global IP/window size count array and the appendix of the global file. Just add the lines marked by <-- (including the typedef gwz_t)

// plugin defines
#define TCPWIN_THRES    1           // tcp Window threshold for packet counts
#define TCPWIN_MINPKTS  50          // <-- Summary file: minimal tcp packets seen to start saving process
#define TCPWIN_MAXWSCNT 100         // <-- Summary file: maximal number of window size threshold count array elements
#define TCPWIN_FNSUP "_tcpwin.txt"  // <-- Summary file: file name supplement

// Status variable: stat
#define TCPWIN_THU 0x1 // TCPWIN threshold undershoot

// plugin structures
typedef struct {                      // <-- 
    ipAddr_t wzip[TCPWIN_MAXWSCNT];   // <-- IP Address array
    uint32_t tcpCnt[TCPWIN_MAXWSCNT]; // <-- tcp packet count
    float wzCnt[TCPWIN_MAXWSCNT];     // <-- window size count
    int wzi;                          // <-- window size index
} gwz_t;                              // <--

// plugin struct pointer for potential dependencies
extern tcpWinFlow_t *tcpWinFlows;
extern gwz_t *gwzP; // dependency pointer to

So you are all set. Note that we only select flows which contain more than 50 packets, to avoid small flow clutter. Compile and run t2:

$ t2build tcpWinSD
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 17086 (17.09 K)
Number of processed A flows: 9704 (9.70 K) [56.80%]
Number of processed B flows: 7382 (7.38 K) [43.20%]
Number of request     flows: 9661 (9.66 K) [56.54%]
Number of reply       flows: 7425 (7.42 K) [43.46%]
Total   A/B    flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed   packets/flows: 71.35
Number of processed A packets/flows: 58.14
Number of processed B packets/flows: 88.70
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B   packets/s: 48859.83 (48.86 K)
Number of processed A     packets/s: 22615.01 (22.61 K)
Number of processed   B   packets/s: 26244.82 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
$

Open your results cmd window and look at what t2 produced

$ ls
annoloc2_tcpwin.txt
$

So only the *_tcpwin.txt* file contains the summary of all IP and their counts being below the winSZ threshold. If you are interested in the IP with the worst connection sort the tcpwin file according to the 3rd column.

$ tawk -s '#' 't2sort(winRelThCnt)' annoloc2_tcpwin.txt
# IP	pktTcpCnt	winRelThCnt
138.212.187.203	72	1.000000
36.152.156.46	79	0.962025
216.237.125.166	429	0.489510
216.217.165.245	52	0.269231
138.212.186.191	251	0.247012
138.212.185.150	195	0.246154
138.212.186.160	205	0.229268
138.212.186.52	162	0.203704
200.44.192.225	315	0.196825
201.98.31.61	388	0.164948
216.56.159.22	95	0.157895
138.212.186.60	140	0.150000
201.9.136.60	197	0.142132
193.87.5.62	235	0.140426
201.9.140.14	194	0.123711
19.112.1.129	78	0.115385
138.212.185.186	74	0.094595
212.88.230.156	131	0.045802
201.123.124.98	340	0.041176
193.86.108.236	847	0.035419
138.212.185.98	133	0.030075
200.50.55.138	87	0.022989
200.50.132.98	111	0.018018
193.87.239.57	365	0.016438
83.42.68.176	62	0.016129
200.32.26.254	250	0.016000
19.123.222.7	72	0.013889
138.212.188.204	73	0.013699
193.87.97.162	843	0.011862
138.212.191.84	92	0.010870
133.26.84.187	2079	0.009620
209.132.7.75	114	0.008772
193.104.31.16	133	0.007519
193.87.112.223	957	0.005225
209.147.223.37	244	0.004098
138.212.188.66	249	0.004016
192.224.45.42	719	0.002782
70.5.118.83	415	0.002410
83.128.136.224	977	0.002047
36.89.79.225	538	0.001859
138.212.184.48	1081	0.001850
201.98.215.67	618	0.001618
219.127.165.87	670	0.001493
138.212.188.99	1134	0.000882
201.9.148.42	1332	0.000751
138.212.190.117	2309	0.000433
$

So in all packets the host 138.212.187.203 asked the opposite IP to stop sending packets. If you increase CPWIN_MINPKTS what will happen?

Then add the destination address, does that change much? Look at the flow asymmetry. Why not improve the measure by adding the TCP WS option. These are your jobs now. Let yourself be inspired by the tcpFlags plugin of T2.

Have fun writing plugins!

The next tutorial will teach you geo-whois-labeling

Tutorial: Plugin summary files

Contents

Summary files

See Also