Plugin sink
Contents
Introduction
If any of our sink plugins do not match your needs, you can either tweak them or develop your own. In this tutorial you will learn about the t2 functions supporting you in that task.
Prerequisites
Tutorials
You must have completed the following tutorials:
Getting started
Create folders for your data and results
If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:
mkdir ~/data ~/results
Reset tranalyzer2 and the plugins configuration
If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:
t2conf -a --reset
You can also clean all build files:
t2build -a -c
Empty the plugin folder
To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Download the PCAP file
The PCAP file used in this tutorial can be downloaded here:
Please save it in your ~/data folder:
wget --no-check-certificate -P ~/data https://tranalyzer.com/download/data/annoloc2.pcap
Build tranalyzer2 and the required plugins
For this tutorial, we will need to build the core (tranalyzer2) and the following plugins:
- basicFlow
- basicStats
- tcpStates
- tcpWin (follow the instructions in the Source code section below to install this plugin)
- txtSink
As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.
...
BUILDING SUCCESSFUL
Source code
If you are impatient you can download the initial and
final
versions of the tcpWinSink plugin we will develop in this tutorial.
If you want to learn, keep on reading and you will create everything yourself
(and learn a lot more in the process!).
The initial version is automatically created by t2plugin,
so don’t worry, you won’t have too much to type!
To use one of those plugins, just unpack it in the plugins folder of your T2 installation.
tranpl
tar -xf ~/Downloads/tcpWinSink1.tar.gz
And let t2_aliases know about it:
source "$T2HOME/scripts/t2_aliases"
Sink your own
If you like to send something via a socket just use the socketSink plugin, do not reinvent the wheel, or let yourself be inspired by it. Here we’d like to address a real case which is not covered by the flow buffer or our sink plugins.
Let’s build a customized summary sink which pushes info via a socket, like netflowSink, only simpler. So a customer tells you to produce from the summary file a periodic update, and send it to 127.0.0.1 on port 5555 via UDP.
First, we create a new plugin using the t2plugin script.
Let’s call it tcpWinSink and give it the plugin number 950.
As we will be adding callbacks as we go, we want to start with a minimal sink plugin (-s):
C plugin 'tcpWinSink' created Make sure to update your aliases by calling 'source "$T2HOME/scripts/t2_aliases"'
That’s it! You should now have a working mininal sink plugin (which does nothing!)! Make sure to update the aliases by calling:
source "$T2HOME/scripts/t2_aliases"
Now, let’s head straight to our plugin:
tcpWinSink
First, we need a dependency to tcpWin, which we already prepared in the previous tutorials.
For meson you need to add the tcpWin in the include_directories function:
tcpWinSink
vi meson.build
...
inc = include_directories(
join_paths('..', '..', 'utils'),
join_paths('..', '..', 'tranalyzer2', 'src'),
join_paths('..', 'tcpWin', 'src'), # <--
)
...If you only want meson, you are good, but if you want fallback e.g. cmake, add the same to CMakeLists.txt:
vi CMakeLists.txt
...
target_include_directories(${_plugin_name}
PRIVATE
../../utils
../../tranalyzer2/src
../../tcpWin/src # <--
)
...and for the generality of all maybe older systems autotools as an ultimate fallback,
add it to the libtcpWinSink_la_CFLAGS constant:
vi src/Makefile.am
...
libtcpWinSink_la_SOURCES = tcpWinSink.c tcpWinSink.h
libtcpWinSink_la_CFLAGS = \
-I$(top_srcdir)/../../utils \
-I$(top_srcdir)/../../tranalyzer2/src \
-I$(top_srcdir)/../tcpWin/src # <-- do not forget the backslash in the previous line!
...Now the compiler is happy for all cases, it will always find your dependent plugin whatever version of OS.
First we need to define the pointer to the window size counts tcpWin produces (refer to the Plugin dependencies tutorial if you forgot). You have to add the tcpWin.h dependency and the socket also has to be defined.
vi src/tcpWinSink.c
...
#include "tcpWinSink.h"
#include "tcpWin.h" // <--
#include <unistd.h> // <-- for close
/*
* Variables from dependencies, i.e., other plugins, MUST be declared weak,
* in order to prevent dlopen() from trying to resolve them. If the symbols
* are missing, it means the required dependency was not loaded. The error
* will be reported by loadPlugins.c when checking for the dependencies
* listed in the get_dependencies() or T2_PLUGIN_INIT_WITH_DEPS() function.
*/
extern gwz_t *gwzP __attribute__((weak)); // <-- define pointer to tcpWin window size threshold count structure
/*
* Static variables are only visible in this file
*/
static int sfd; // <-- socket handle
static struct sockaddr_in server; // <-- struct for socket to connect to server
static struct timeval tmrk; // <-- time marker
static char sBuf[TWS_MAXSBUF]; // <-- send buffer
/*
* Static functions prototypes
*/
// Tranalyzer functions
/*
* This describes the plugin name, version, major and minor version of
* Tranalyzer required and dependencies
*/
T2_PLUGIN_INIT_WITH_DEPS("tcpWinSink", "0.9.0", 0, 9, "tcpWin"); // <--
...As in the Plugin dependencies tutorial,
we need a T2_PLUGIN_INIT_WITH_DEPS(...) macro, this time
with tcpWin as a dependency.
The t2Init() function mostly contains standard UDP/TCP socket code, and at the end,
the initialization of the time marker. actTime denotes the actual pcap time of the core
defined in global.h. So if you sniff from an interface, it will be the local time.
The code is copied from the socketSink and adapted to our problem at hand.
vi src/tcpWinSink.c
...
void t2Init() {
const struct hostent * const host = gethostbyname(TWS_SERVADD);
if (UNLIKELY(!host)) {
T2_PFATAL(plugin_name, "gethostbyname() failed for '%s'", TWS_SERVADD);
}
server.sin_addr = *(struct in_addr*)host->h_addr;
server.sin_family = AF_INET;
server.sin_port = htons(TWS_DPORT);
#if TWS_SOCKTYPE == 1 // TCP socket
if (UNLIKELY(!(sfd = socket(AF_INET, SOCK_STREAM, 0)))) {
#else // TWS_SOCKTYPE == 0 // UDP socket
if (UNLIKELY(!(sfd = socket(AF_INET, SOCK_DGRAM, 0)))) {
#endif // TWS_SOCKTYPE
T2_PFATAL(plugin_name, "Could not create socket: %s", strerror(errno));
}
#if TWS_SOCKTYPE == 1 // for a TCP socket you need to have a three way handshake first
if (UNLIKELY(connect(sfd, (struct sockaddr*)&server, sizeof(server)) < 0)) {
T2_PFATAL(plugin_name, "Could not connect to socket: %s, check whether the server side is listening at %s on port %d", strerror(errno), TWS_SERVADD, TWS_DPORT);
}
#endif // TWS_SOCKTYPE == 1
tmrk = actTime; // and set our internal send buffer timer
}
...Now we come to the important part of the job.
Implementing the t2BufferToSink() callback
As we are not using the flow buffer of T2, we can add the UNUSED tag to the parameters of the t2BufferToSink(...) callback.
Now the compiler has no reason to complain about the fact that you do not use the buffer.
We like to print a winsize list reporting reporting every n seconds.
In t2BufferToSink(...) your code defines what the sink is doing. It consists normally of four main parts.
- A timer check when to send data. If not present, every flow terminating triggers your
t2BufferToSink(...), that is inefficient. But you may try it without a timer. - Data preparation and recoding section
- Buffer population as intended, so that the socket just needs to push it out.
- The end buffer part. You may reuse it in all your future applications, because it is back-pressure safe.
vi src/tcpWinSink.c
...
/*
* This callback is only required for sink plugins
* Refer to parse_binary2text() in utils/bin2txt.c for an example
*/
void t2BufferToSink(outputBuffer_t *buf UNUSED, binary_value_t *bv UNUSED) {
// check core timer
if (actTime.tv_sec - tmrk.tv_sec < TWS_TMINTVL || gwzP->wzi == 0) return; // timer expired and is something to print?
tmrk = actTime; // reset timer
// Data preparation part
char srcIP[INET_ADDRSTRLEN];
char time[MAX_TM_BUF];
strftime(time, MAX_TM_BUF, "%a %d %b %Y %X", localtime(&tmrk.tv_sec)); // make a nice time string
// shovel output of tcpWin into sBuf
int sBufLen = snprintf(sBuf, TWS_MAXSBUF-sBufLen, "# %s\n# IP\tpktTcpCnt\twinRelThCnt\n", time); // send header
for (int i = 0; i < gwzP->wzi; i++) {
T2_IPV4_TO_STR(gwzP->wzip[i], srcIP, sizeof(srcIP)); // convert IP to string
sBufLen += snprintf(&sBuf[sBufLen], TWS_MAXSBUF-sBufLen, // push into sBuf
"%s\t%" PRIu32 "\t%f\n",
srcIP, gwzP->tcpCnt[i], gwzP->wzCnt[i]);
}
sBuf[sBufLen++] = '\n'; // final separation
sBuf[sBufLen++] = '\0'; // just make sure that string is always properly terminated
// Send sBuf via socket
int32_t written = 0, act_written; // buffer positional indexes
while (written < sBufLen) { // make sure that the socket sends the whole buffer regardless of flow control socket back pressure
#if TWS_SOCKTYPE == 1
act_written = write(sfd, sBuf + written, sBufLen - written);
#else // TWS_SOCKTYPE == 0
act_written = sendto(sfd, sBuf + written, sBufLen - written, 0, (struct sockaddr*)&server, sizeof(server));
#endif // TWS_SOCKTYPE
if (UNLIKELY(act_written <= 0)) {
T2_PERR(plugin_name, "Could not send message to socket: %s", strerror(errno));
if (sfd) close(sfd);
exit(EXIT_FAILURE);
}
written += act_written;
}
}
...Note that t2BufferToSink() is called every time a flow timeouts. Therefore we either need to reduce our flow timeout,
or add tcpStates, as we did above, assuring that TCP flows timeout immediately after a RST or FIN.
t2Finalize(...) is easy, just close the socket handle.
The if is good defensive programming, although we always exit in case of an error.
void t2Finalize() {
if (sfd) close(sfd);
}Now open tcpWinSink.h, there you already see the constants we need, the server address, port, the socket type,
we choose a TCP socket for the start. The time interval is set to 5 seconds and the maximal buffer length is chosen large
enough for simplicity, otherwise we need to malloc and stuff, which makes the tutorial unnecessary complicated.
So you can rewrite the buffer which is properly allocated according to the data to be sent as home work.
Hint: It depends on the maximal number of entries in the gwz structure and the output format.
You will only need a buffer pointer and malloc. And don’t forget to free the buffer at the end.
tcpWinSink.h holds now all necessary constants. I copied them from the socketSink,
and adapted them to our task. The TWS_MAXBUF constant should suffice for our application.
Nevertheless, if you intent to add more info into sBuf check whether you need to increase its size.
vi src/tcpWinSink.h
...
// local includes
#include "t2Plugin.h"
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define TWS_SERVADD "127.0.0.1" // destination address
#define TWS_DPORT 5555 // destination port host order
#define TWS_SOCKTYPE 1 // 0: UDP; 1: TCP
#define TWS_TMINTVL 5 // Interval to send a report
#define TWS_MAXSBUF 4096 // maximal size of sBuf
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...After you edited the skeleton code you should compare your implementation with tcpWinSink1.tar.gz.
Now open a netcat TCP socket in another bash window:
nc -l 127.0.0.1 -p 5555
Go back to your original window and invoke T2.
t2 -r ~/data/annoloc2.pcap -w ~/results
And you will see in the netcat window every five seconds a report
nc -l 127.0.0.1 -p 5555
# Thu 23 May 2002 18:35:02
# IP pktTcpCnt winRelThCnt
133.26.84.187 105 0.019048
# IP pktTcpCnt winRelThCnt
133.26.84.187 105 0.019048
193.86.108.236 65 0.030769
193.87.97.162 91 0.021978
193.87.112.223 56 0.035714
193.104.31.16 133 0.007519
138.212.190.117 2309 0.000433
# IP pktTcpCnt winRelThCnt
133.26.84.187 61 0.032787
193.86.108.236 65 0.030769
193.87.97.162 91 0.021978
193.87.112.223 56 0.035714
193.104.31.16 133 0.007519
138.212.190.117 2309 0.000433
19.112.1.129 92 0.043478
138.212.191.84 96 0.010417
138.212.185.150 1927 0.001038
36.89.79.225 538 0.001859
201.123.124.98 52 0.038462
# IP pktTcpCnt winRelThCnt
133.26.84.187 55 0.036364
193.86.108.236 65 0.030769
193.87.97.162 91 0.021978
193.87.112.223 56 0.035714
193.104.31.16 133 0.007519
138.212.190.117 2309 0.000433
19.112.1.129 92 0.043478
138.212.191.84 92 0.010870
138.212.185.150 549 0.003643
36.89.79.225 538 0.001859
201.123.124.98 52 0.038462
138.212.185.98 66 0.030303
201.9.148.42 1332 0.000751
193.87.239.57 233 0.008584
# IP pktTcpCnt winRelThCnt
133.26.84.187 55 0.036364
193.86.108.236 65 0.030769
193.87.97.162 65 0.030769
193.87.112.223 56 0.035714
193.104.31.16 133 0.007519
138.212.190.117 2309 0.000433
19.112.1.129 92 0.043478
138.212.191.84 92 0.010870
138.212.185.150 165 0.006061
36.89.79.225 538 0.001859
201.123.124.98 52 0.038462
138.212.185.98 66 0.030303
201.9.148.42 1332 0.000751
193.87.239.57 233 0.008584
200.32.26.254 142 0.014085
83.42.68.176 62 0.016129
192.224.45.42 542 0.003690
83.128.136.224 749 0.002670
138.212.184.48 862 0.002320
212.88.230.156 76 0.039474Try t2 on your interface, but restart netcat first.
st2 -i INTERFACE
[sudo] password for wurst:
...Make sure you produce a lot of TCP traffic so that one connection produces back pressure situations
or you have to lower TCPWIN_MINPKTS or increase TCPWIN_THRES in tcpWin.h listed below.
Use t2conf for that and don’t forget to t2build:
tcpWin
vi src/tcpWin.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define TCPWIN_THRES 1 // TCP window size threshold undershoot flag
#define TCPWIN_FLWPRG 2 // Threshold to abort a flow
#define TCPWIN_MINPKTS 50 // Summary file: minimal TCP packets seen to start saving process
#define TCPWIN_MAXWSCNT 100 // Summary file: maximal number of window size threshold count array elements
#define TCPWIN_SUFFIX "_tcpwin.txt" // Summary file: file name suffix
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...If you used the tcpWin I provided, that is about it. Besides sending something over the socket you should see the summary report in the results directory from tcpWin.
How about building a round robin list, where old IP’s are thrown out and new come in, so the amount of entries would be constant after some time and tcpWin would not just stop working when the maximum entries are reached.
Conclusion
You can download the final version of the tcpWinSink plugin.
The next tutorial will teach you how to develop a plugin in C++.
Have fun writing plugins!
See also
- Plugin programming cheatsheet
- The basics: your first flow plugin
- Plugin end report
- Plugin monitoring
- Plugin packet mode
- Plugin summary files
- Plugin geo labeling
- Plugin dependencies
- Plugin alarm mode
- Plugin force mode
- Plugin pcap extraction
- Plugin flow timeout
- Developing Tranalyzer plugins in C++
- Developing Tranalyzer plugins in Rust