Tutorial: Plugin Sinks
Contents
If any of our sink plugins do not match your needs, you can either tweak them or develop your own. In this tutorial you will learn about the t2 functions supporting you in that task.
Preparation
You must have completed the plugin summary file and plugin dependencies tutorials because these prepares you for the comming material. If you are just interested in the sink part here is the necessary tcpWinSD.tar.gz.
$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$
If you use the tcpWin developed in the last two tutorials you need the basicFlow dependency.
$ t2build tranalyzer2 basicFlow tcpStates tcpGeoWin
...
$
The anonymized sample pcap can be downloaded here: annoloc2.pcap. Please extract it under your data folder: ~/data
, if you haven’t already. Now you are all set for sink stuff.
Sink your own
If you like to send something via a socket just use the socketSink plugin, do not reinvent the wheel, or let yourself be inspired by it. Here we’d like to address a real case which is not covered by the flow buffer or our sink plugins.
Let’s build a customized summary sink which pushes info via a socket, like netflowSink, only simpler. So a customer tells you to produce from the summary file a periodic update, and send it to 127.0.0.1 on port 5555 via UDP.
I already prepared a tcpWinSink for you ready to be downloaded. So you can have it open in another bash window and follow the explanations below.
As sink plugins start with plugin numbers 9xx, I picked 950. We will also need a dependency: tcpGeoWin, which we already prepared in the previous tutorial plugingeolabel.
For meson you need to add the tcpGeoWIn in the include_directories
function
$ tran
$ source scripts/t2_aliases
$ tcpWinSink
$ vi meson.build
...
inc = include_directories(
join_paths('..', '..', 'utils'),
join_paths('..', '..', 'tranalyzer2', 'src'),
join_paths('..', 'tcpGeoWin', 'src'),
)
...
If you only want meson, you are good, but if you want fallback e.g. cmake, add the same to CMakeLists.txt
$ vi CMakeLists.txt
...
target_include_directories(${_plugin_name}
PRIVATE
../../utils
../../tranalyzer2/src
../../tcpGeoWin/src
)
...
and for the generality of all maybe older systems autotools as an ultimate fallback, add it to the CFLAGS constant:
$ cd src
$ vi Makefile.am
...
libtcpWinSink_la_SOURCES = tcpWinSink.c tcpWinSink.h
libtcpWinSink_la_CFLAGS = \
-I$(top_srcdir)/../../utils \
-I$(top_srcdir)/../../tranalyzer2/src \
-I$(top_srcdir)/../tcpGeoWin/src
...
Now the compiler is happy for all cases, it will always find your dependend plugin whatever version of OS.
Now open tcpWinSink.c and look for bufferToSink()
$ vi tcpWinSink.c
First we need to define the pointer to the window size counts tcpGeoWin produces (refer to the Plugin dependencies tutorial if you forgot). You have to add the tcpWin.h dependency and the socket also has to be defined.
...
#include "tcpWinSink.h"
#include "tcpGeoWin.h"
/*
* Variables from dependencies, i.e., other plugins, MUST be declared weak,
* in order to prevent dlopen() from trying to resolve them. If the symbols
* are missing, it means the required dependency was not loaded. The error
* will be reported by loadPlugins.c when checking for the dependencies
* listed in the get_dependencies() or T2_PLUGIN_INIT_WITH_DEPS() function.
*/
extern gwz_t *gwzP __attribute__((weak)); // define pointer to tcpWin window size threshold count structure
// local for this plugin
static int sfd; // socket handle
static struct sockaddr_in server; // struct for socket to connect to server
static struct hostent *host; // host address structure
static char sBuf[MAXSBUF]; // send buffer
static struct timeval tmrk; // time marker
...
As in the Plugin dependencies tutorial, we need a T2_PLUGIN_INIT_WITH_DEPS(...)
macro, this time with tcpWin as a dependency.
The initialize()
function mostly contains standard UDP/TCP socket code, and at the end, the initialization of the time marker. actTime denotes the actual pcap time of the core defined in global.h. So if you sniff from an interface, it will be the local time. The code is copied from the socketSink and adapted to our problem at hand.
void initialize() {
#if SOCKTYPE == 1 // tcp socket
if (!(sfd = socket(AF_INET, SOCK_STREAM, 0))) {
#else // SOCKTYPE == 0 // udp socket
if (!(sfd = socket(AF_INET, SOCK_DGRAM, 0))) {
#endif // SOCKTYPE
T2_PERR("tcpWinSink", "Could not create socket: %s", strerror(errno));
exit(-1);
}
memset(&server, '\0', sizeof(server));
host = gethostbyname(SERVADD);
if (UNLIKELY(!host)) {
T2_PERR("tcpWinSink", "gethostbyname() failed for '%s'", SERVADD);
if (sfd) close(sfd);
exit(1);
}
server.sin_addr = *(struct in_addr*)host->h_addr;
server.sin_family = AF_INET;
server.sin_port = htons(DPORT);
#if SOCKTYPE == 1 // for a tcp socket you need to have a three way handshake first
if (connect(sfd, (struct sockaddr*)&server, sizeof(server)) < 0) {
T2_PERR("tcpWinSink", "Could not connect to socket: %s, check whether the server side is listening at %s on port %d", strerror(errno), SERVADD, DPORT);
exit(-1);
}
#endif // SOCKTYPE == 1
tmrk = actTime; // and set our internal send buffer timer
}
Now we come to the important part of the job. As we are not using the flow buffer of T2, the bufferToSink(...)
callback with the __attribute__((unused))
tag has to be appended to your tcpWinSink plugin. If you are using the tcpGeoWin I provided it does not contain any output buffer on flow terminated, as we are not using any other sink anyway. Now the compiler has no reason to complain about the fact that you do not use the buffer.
Implementing the bufferToSink(outputBuffer_t *buffer)
callback
We like to print a winsize list reporting reporting every n seconds.
In bufferToSink(...)
your code defines what the sink is doing. It consist normally of four main parts.
- A timer check when to send data. If not present, every flow terminating triggers your
bufferToSink(...)
, that is inefficient. But you may try it without a timer. - Data preparation and recoding section
- Buffer population as intended, so that the socket just needs to push it out.
- The end buffer part. You may reuse it in all your future applications, because it is back-pressure safe.
void bufferToSink(outputBuffer_t *buffer __attribute__((unused))) {
// check core timer
if (actTime.tv_sec - tmrk.tv_sec < TMINTVL || gwzP->wzi == 0) return; // timer expired and is something to print?
tmrk = actTime; // reset timer
// Data preparation part
int i, sBufLen = 0;
char srcIP[INET6_ADDRSTRLEN];
char time[MAX_TM_BUF];
strftime(time, MAX_TM_BUF, "%a %d %b %Y %X", localtime(&tmrk.tv_sec)); // make a nice time string
// shovel output of tcpWin into sBuf
sBufLen = snprintf(sBuf, MAXSBUF-sBufLen, "# %s\n# IP\tpktTcpCnt\twinRelThCnt\n", time); // send header
for (i = 0; i < gwzP->wzi; i++) {
T2_IP_TO_STR(gwzP->wzip[i].addr, gwzP->wzip[i].ver, srcIP, INET6_ADDRSTRLEN); // transfer IP to string
sBufLen += snprintf(&sBuf[sBufLen], MAXSBUF-sBufLen, "%s\t%"PRIu32"\t%f\n", srcIP, gwzP->tcpCnt[i], gwzP->wzCnt[i]); // push into sBuf
}
sBuf[sBufLen++] = '\n'; // final separation
sBuf[sBufLen++] = '\0'; // just make sure that string is always properly terminated
// Send sBuf via socket
int32_t written = 0, act_written; // buffer positional indexes
while (written < sBufLen) { // make sure that the socket sends the whole buffer regardless of flow control socket backpressure
#if SOCKTYPE == 1
act_written = write(sfd, sBuf + written, sBufLen - written);
#else // SOCKTYPE == 0
act_written = sendto(sfd, sBuf + written, sBufLen - written, 0, (struct sockaddr*)&server, sizeof(server));
#endif // SOCKTYPE
if (UNLIKELY(act_written <= 0)) {
T2_PERR("tcpWinSink", "Could not send message to socket: %s", strerror(errno));
if (sfd) close(sfd);
exit(-1);
}
written += act_written;
}
}
Note that bufferToSink
is called every time a flow timeouts. Therefore we either need to reduce our flow timeout, or add tcpStates, as we did above, assuring that TCP flows timeout immediately after a RST or FIN.
onApplicationTerminate(...)
is easy, just close the socket handle. The if
is good defensive programming, although we always exit in case of an error.
Now open tcpWinSink.h, there you already see the constants we need, the server address, port, the socket type, we choose a TCP socket for the start. The time interval is set to 5 seconds and the maximal buffer length is chosen large enough for simplicity, otherwise we need to malloc
and stuff, which makes the tutorial unnecessary complicated.
So you can rewrite the buffer which is properly allocated according to the data to be sent as home work. Hint: It depends on the maximal number of entries in the gwz
structure and the output format. You will only need a buffer pointer and malloc
. And don’t forget to free the buffer at the end.
tcpWinSink.h holds now all necessary constants. I copied them from the socketSink, and adapted them to our task. The MAXBUF constant should suffice for our application. Nevertheless, if you intent to add more info into sBuf check whether you need to increase its size.
...
// local includes
#include "t2Plugin.h"
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define SERVADD "127.0.0.1" // destination address
#define DPORT 5555 // hex destination port host order
#define SOCKTYPE 1 // 0: UDP; 1: TCP
#define TMINTVL 5 // Interval to send a report
#define MAXSBUF 4096 // maximal size of sBuf
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...
Now open a netcat
TCP socket in another bash window: nc -l 127.0.0.1 -p 5555
Go back to your original window and invoke T2.
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
And you will see in the netcat
window every five seconds a report
$ nc -l 127.0.0.1 -p 5555
# Thu 23 May 2002 18:35:05
# IP pktTcpCnt winRelThCnt
193.104.31.16 133 0.007519
# Thu 23 May 2002 18:35:10
# IP pktTcpCnt winRelThCnt
193.104.31.16 133 0.007519
19.112.1.129 94 0.063830
# Thu 23 May 2002 18:35:15
# IP pktTcpCnt winRelThCnt
193.104.31.16 133 0.007519
19.112.1.129 94 0.063830
138.212.191.84 92 0.010870
36.89.79.225 538 0.001859
# Thu 23 May 2002 18:35:20
# IP pktTcpCnt winRelThCnt
193.104.31.16 133 0.007519
19.112.1.129 94 0.063830
138.212.191.84 92 0.010870
36.89.79.225 538 0.001859
201.9.148.42 1332 0.000751
200.32.26.254 142 0.014085
83.42.68.176 62 0.016129
# Thu 23 May 2002 18:35:25
# IP pktTcpCnt winRelThCnt
193.104.31.16 133 0.007519
19.112.1.129 94 0.063830
138.212.191.84 92 0.010870
36.89.79.225 538 0.001859
201.9.148.42 1332 0.000751
200.32.26.254 250 0.016000
83.42.68.176 62 0.016129
19.123.222.7 72 0.013889
138.212.187.203 72 1.000000
193.87.239.57 365 0.016438
70.5.118.83 415 0.002410
$
Try t2 on your interface, but restart the netcat first.
$ st2 -i INTERFACE
[sudo] password for wurst:
...
Make sure you produce a lot of TCP traffic so that one connection produces back pressure situations or you have to lower TCPWIN_MINPKTS
or increase TCPWIN_THRES
in tcpGeoWin.h listed below. Use t2conf
for that and don’t forget to t2build
:
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define TCPWIN_THRES 1 // tcp Window threshold for packet counts
#define TCPWIN_MINPKTS 50 // <-- Summary file: minimal tcp packets seen to start saving process
#define TCPWIN_MAXWSCNT 100 // <-- Summary file: maximal number of window size threshold count array elements
#define TCPWIN_FNSUP "_tcpwin.txt" // <-- Summary file: file name supplement
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...
If you used the tcpGeoWin I provided, that is about it. Besides sending something over the socket you should see the summary report in the results directory from tcpGeoWin.
How about building a round robin list, where old IP’s are thrown out and new come in, so the amount of entries would be constant after some time and tcpWinSD would not just stop working when the maximum entries are reached.
Have fun writing plugins!
The next tutorial will teach you how to implement timeout handlers
See Also
- Plugin Programming Cheatsheet
- The basics: your first flow plugin
- Adding plugin end report
- Adding plugin monitoring output
- Adding plugin packet output
- Producing summary files
- geo-whois-labeling
- All about plugin dependencies
- Manipulating flow timeouts
- Alarm mode
- Force mode
- Pcap extraction
- Developing Tranalyzer plugins in C++
- Developing Tranalyzer plugins in Rust