Tutorial: Plugin summary files
Contents
When all data is processed or T2 is interrupted by ^C
or a signal, then the end report is printed. At the same time certain global information can be printed by your plugin. Before we start doing this, clean out your plugin .so directory if this is the first tutorial you follow. Then all unnecessary plugins should be deleted from the plugin folder ~/.tranalyzer/plugins
and compile the core and basicFlow.
$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$ t2build tranalyzer2 basicFlow
...
I prepared a plugin which we will discuss in the following: tcpWinSD Copy it to the plugins folder of your t2 installation and unpack it:
$ cd tranayzer2-0.8.10/plugins
$ tar -xf tcpWinSD.tar.gz
$ source ../scripts/t2_aliases
$
The anonymized sample pcap can be downloaded here: annoloc2.pcap. Please extract it under your data folder: ~/data
, if you haven’t already. Now you are all set for summary file programming.
Summary files
Every plugin can produce a summary of traffic content. Let’s generate a file which lists all IPs with their maximal winThCnts, so we could find the IP having the largest L4 trouble. In order to do so, we first of all need to define all necessary variables being global to the plugin.
So move to the tcpWinSD plugin and open tcpWinSD.c
$ source
$ tcpWinSD
$ vi src/tcpWin.c
In order to produce just summary files no onFlowTerminate(...)
callback is needed, you can do everything in a packet callback, but that is inefficient, as you go over the code in every packet. If it is possible, it is always good practice to move heavy processing to onFlowGenerated(...)
/onFlowTerminate(...)
callbacks, as it is only called once per flow.
For the matter of simplicity we just have arrays and a global index as defined in the global section of the plugin.
...
/*
* Plugin variables that may be used by other plugins (MUST be declared in
* the header file as 'extern tcpWinFlow_t *tcpWinFlows;'
*/
tcpWinFlow_t *tcpWinFlows;
...
// window size counts
gwz_t gwz; // <-- global window size structure
gwz_t *gwzP = &gwz; // <-- Pointer for later dependency experiments
...
Now go to the onFlowTerminate(...)
callback. First we need the pointer to the flow structure. To show the principal we just do a linear search whether we already have the IP stored and check whether its counter is greater than the potential existing, then we store the new data. Yes, it can be more elegant, using a hash or a tree. We will discuss T2 support for that matter in tutorial block t2kunfu. Note that the following code is only for IPv4, as we want to start simple.
void onFlowTerminate(unsigned long flowIndex) {
const flow_t * const flowP = &flows[flowIndex];
tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
if (tcpWinFlowP->winThCnt && tcpWinFlowP->pktTcpCnt >= TCPWIN_MINPKTS) {
const int wzi = gwz.wzi; // store element count in const local variable, makes the compiler happy
if (wzi < TCPWIN_MAXWSCNT) { // If array full, stop saving
int i; //
for (i = 0; i < wzi; i++) if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP already exist?
const float f = (float)tcpWinFlowP->winThCnt/(float)tcpWinFlowP->pktTcpCnt; // produce a useful relative number
if (f > gwz.wzCnt[i]) { // only update if rel count is greater than the previous one
gwz.tcpCnt[i] = tcpWinFlowP->pktTcpCnt; // update tcp packet count
gwz.wzCnt[i] = f; // update relative count
if (i == wzi) { // new one?
gwz.wzip[i] = flowP->srcIP; // save new IP
gwz.wzi++; // increment global window size counter
}
}
}
}
}
The onApplicationTerminate()
callback not only serves to free flow memory but also to produce all kinds of global output into specialized files. The code below stores the IP and its count. Just add it after the free(...)
command.
void onApplicationTerminate() {
free(tcpWinFlows); // free the tcpWin Flows
// open TCPWIN statistics file
FILE *fp;
int i;
char srcIP[INET6_ADDRSTRLEN];
fp = t2_open_file(baseFileName, TCPWIN_FNSUP, "w");
if (UNLIKELY(!fp)) { // if file cannot be opened print warning and return;
T2_PWRN("tcpWin", "Failed to allocate memory for: %s", TCPWIN_FNSUP);
return;
}
fprintf(fp, "# IP\tpktTcpCnt\twinRelThCnt\n"); // print header
for (i = 0; i < gwz.wzi; i++) {
T2_IP_TO_STR(gwz.wzip[i], 4, srcIP, INET6_ADDRSTRLEN); // transfer IP to string
fprintf(fp, "%s\t%"PRIu32"\t%f\n", srcIP, gwz.tcpCnt[i], gwz.wzCnt[i]); // print in file
}
fclose(fp);
}
Now open tcpWin.h and add the maximal size of the global IP/window size count array and the appendix of the global file. Just add the lines marked by <--
(including the typedef gwz_t
)
// plugin defines
#define TCPWIN_THRES 1 // tcp Window threshold for packet counts
#define TCPWIN_MINPKTS 50 // <-- Summary file: minimal tcp packets seen to start saving process
#define TCPWIN_MAXWSCNT 100 // <-- Summary file: maximal number of window size threshold count array elements
#define TCPWIN_FNSUP "_tcpwin.txt" // <-- Summary file: file name supplement
// Status variable: stat
#define TCPWIN_THU 0x1 // TCPWIN threshold undershoot
// plugin structures
typedef struct { // <--
ipAddr_t wzip[TCPWIN_MAXWSCNT]; // <-- IP Address array
uint32_t tcpCnt[TCPWIN_MAXWSCNT]; // <-- tcp packet count
float wzCnt[TCPWIN_MAXWSCNT]; // <-- window size count
int wzi; // <-- window size index
} gwz_t; // <--
// plugin struct pointer for potential dependencies
extern tcpWinFlow_t *tcpWinFlows;
extern gwz_t *gwzP; // dependency pointer to
So you are all set. Note that we only select flows which contain more than 50 packets, to avoid small flow clutter. Compile and run t2:
$ t2build tcpWinSD
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 17086 (17.09 K)
Number of processed A flows: 9704 (9.70 K) [56.80%]
Number of processed B flows: 7382 (7.38 K) [43.20%]
Number of request flows: 9661 (9.66 K) [56.54%]
Number of reply flows: 7425 (7.42 K) [43.46%]
Total A/B flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed packets/flows: 71.35
Number of processed A packets/flows: 58.14
Number of processed B packets/flows: 88.70
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 22615.01 (22.61 K)
Number of processed B packets/s: 26244.82 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
$
Open your results cmd window and look at what t2 produced
$ ls
annoloc2_tcpwin.txt
$
So only the *_tcpwin.txt* file contains the summary of all IP and their counts being below the winSZ threshold. If you are interested in the IP with the worst connection sort the tcpwin file according to the 3rd column.
$ tawk -s '#' 't2sort(winRelThCnt)' annoloc2_tcpwin.txt
# IP pktTcpCnt winRelThCnt
138.212.187.203 72 1.000000
36.152.156.46 79 0.962025
216.237.125.166 429 0.489510
216.217.165.245 52 0.269231
138.212.186.191 251 0.247012
138.212.185.150 195 0.246154
138.212.186.160 205 0.229268
138.212.186.52 162 0.203704
200.44.192.225 315 0.196825
201.98.31.61 388 0.164948
216.56.159.22 95 0.157895
138.212.186.60 140 0.150000
201.9.136.60 197 0.142132
193.87.5.62 235 0.140426
201.9.140.14 194 0.123711
19.112.1.129 78 0.115385
138.212.185.186 74 0.094595
212.88.230.156 131 0.045802
201.123.124.98 340 0.041176
193.86.108.236 847 0.035419
138.212.185.98 133 0.030075
200.50.55.138 87 0.022989
200.50.132.98 111 0.018018
193.87.239.57 365 0.016438
83.42.68.176 62 0.016129
200.32.26.254 250 0.016000
19.123.222.7 72 0.013889
138.212.188.204 73 0.013699
193.87.97.162 843 0.011862
138.212.191.84 92 0.010870
133.26.84.187 2079 0.009620
209.132.7.75 114 0.008772
193.104.31.16 133 0.007519
193.87.112.223 957 0.005225
209.147.223.37 244 0.004098
138.212.188.66 249 0.004016
192.224.45.42 719 0.002782
70.5.118.83 415 0.002410
83.128.136.224 977 0.002047
36.89.79.225 538 0.001859
138.212.184.48 1081 0.001850
201.98.215.67 618 0.001618
219.127.165.87 670 0.001493
138.212.188.99 1134 0.000882
201.9.148.42 1332 0.000751
138.212.190.117 2309 0.000433
$
So in all packets the host 138.212.187.203 asked the opposite IP to stop sending packets. If you increase CPWIN_MINPKTS
what will happen?
Then add the destination address, does that change much? Look at the flow asymmetry. Why not improve the measure by adding the TCP WS option. These are your jobs now. Let yourself be inspired by the tcpFlags
plugin of T2.
Have fun writing plugins!
The next tutorial will teach you geo-whois-labeling
See Also
- The basics: your first flow plugin
- Adding plugin end report
- Adding plugin monitoring output
- Adding plugin packet output
- geo-whois-labeling
- All about plugin dependencies
- Plugin sinks
- Manipulating flow timeouts
- Alarm mode
- Force mode
- Pcap extraction
- Developing Tranalyzer plugins in C++
- Developing Tranalyzer plugins in Rust