Tutorial: Netflow Export

Contents

This example is prepared for unix systems (linux and osx) only. In order to assure that no old or unnecessary plugins are being loaded please clean your plugin directory by

> t2build -e
> ls
>

The netflowSink plugin requires basicFlow, basicStats, tcpStates and tcpFlags plugins. In addition, the macRecorder is recommended, but optional.

then recompile T2 by

> t2build basicFlow basicStats tcpStates tcpFlags macRecorder

As you can see we do not have a txtSink, so no Flow file is generated. If you wish to do that, you can add any sink you deem necessary. Nevertheless, any additional medium adds delays, which is crucial if you sniff from an interface.

To illustrate the configuration and application of the data carving mode lets have a look at netflowSink and log into nfcap, so that the well known nfdump tool can read it. Nevertheless any tool which implements netflow9 or 10 (IPFIX) will be able to process the flow information.

If you want to benefit from tranalyzers extended capabillities you can also use the socketSink plugin which sends flows to any location you deem appropriate.

First move to directory

> cd netflowSink/src 
> vi netflowSink.h

Lets have a look at the configuration:

The default address to log is the local interface, if you want to log remotely change the address in NF_SERVADD.

The destination port is set to the nfcap default, you can choose any convenient port. Socket type is UDP, but you may change it, if you have a tool which requires it. Choose the netflow version you want to export and set the maximum IPv4/6 flows bunches which will be transport in one netflow message to the receiving server. The default values work fine, but you may change them to optimize performance.

Recompile netflowSink

> t2build netflowSink

or

> cd ..;./autogen.sh

To collect T2 flow data with nfcapd in another window of your local machine or on the remote server, use the following command:

nfcapd -T all -B 1000000 -n sourcename,127.0.0.1,.

or

nfcapd -T all -B 1000000 -n sourcename,serveraddress,.

We use increased buffering, so that nfcapd can keep up with tranalyzer. The sourcename identifies the sensor associated by the serveraddress. The “.” denotes the base directory where to save the compressed flows.

Now start tranalyzer with a pcap file on your local machine:

> t2 -r yourpcap

or from an interface

>
> t2 -i interface

The file which is then produced by nfcapd: nfcapd.2018xxxxxxxx can be interpreted by nfdump. Here is a sample of an output from nfdump. Just run you own traffic and play around with it.

>nfdump -r nfcapd.2018xxxxxxxx -o extended -s srcip -s ip/flows -s dstport/pps/packets/bytes -s record/bytes
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2012-04-02 23:52:23.393 336936.797 170         10.70.1.7       ->        10.70.1.7:0     ...... 255   286589    3.4 G        0    80150  11778     1
2012-04-06 00:35:52.988  8012.525 TCP          10.3.10.6:3260  ->        10.3.16.5:22609   0xff 255   242709  721.7 M       30   720620   2973   411
2012-04-05 23:33:36.242  5946.051 TCP          10.3.10.6:3260  ->        10.3.16.5:20557   0xff   0   220118  674.7 M       37   907713   3065   484
2012-04-03 00:20:37.610 334303.283 TCP         10.3.10.6:0     ->        10.3.16.5:0       0xff   0   158775  467.8 M        0    11195   2946   415
2012-04-06 00:35:53.642  8011.736 TCP          10.3.1.5:22609  ->        10.3.58.6:3260    0xff 255   123984  235.2 M       15   234822   1896   411
2012-04-05 23:33:36.193  3737.387 TCP          10.3.1.5:20557  ->        10.3.58.6:3260    0xff  37   110605  206.4 M       29   441793   1866   480
...
Top 10 Src IP Addr ordered by -:
Date first seen          Duration Proto       Src IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2012-04-03 02:07:23.036     0.000 any    c77d:e1..:d34b::        1( 0.0)        1( 0.0)     6971( 0.0)        0        0  6971
2012-04-06 15:07:41.411     0.000 any    9a88:44..ff:ffff        1( 0.0)        1( 0.0)    35092( 0.0)        0        0 35092
2012-04-06 00:02:17.677     0.000 any    77a1:31..ff:ffff        1( 0.0)        1( 0.0)    48246( 0.0)        0        0 48246
2012-04-05 23:36:37.196     0.000 any    67ee:c4..ff:ffff        1( 0.0)        1( 0.0)    21774( 0.0)        0        0 21774
...
Top 10 IP Addr ordered by flows:
Date first seen          Duration Proto           IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2012-04-02 23:52:22.893 336938.342 any           10.3.10.6  627373(25.0)    2.7 M(46.0)    6.9 G(40.4)        7   163201  2578
2012-04-03 00:20:16.838 335265.140 any           10.3.1.5   543617(21.6)    2.5 M(43.6)    6.5 G(38.3)        7   155255  2574
2012-04-02 23:52:28.949 336674.488 any           10.3.1.7   281054(11.2)   499233( 8.6)    1.2 G( 6.8)        1    27362  2306
...
Top 10 Dst Port ordered by bytes:
Date first seen          Duration Proto          Dst Port    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2012-04-02 23:52:22.657 336939.471 any                   0   663918(26.4)    1.8 M(30.5)    7.0 G(41.0)        5   165463  3938
2012-04-05 23:41:39.177 61919.898 any               22609   147028( 5.9)   627038(10.8)    1.9 G(11.2)       10   246135  3038
2012-04-03 18:12:56.992 254021.561 any              20557   131832( 5.3)   572821( 9.9)    1.8 G(10.5)        2    56114  3110
...
Top 10 Dst Port ordered by pps:
Date first seen          Duration Proto          Dst Port    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2012-04-03 07:31:53.253     0.001 any               63888        4( 0.0)        4( 0.0)     4292( 0.0)     4000   34.3 M  1073
2012-04-03 08:57:33.141     0.001 any               64274        4( 0.0)        4( 0.0)    31278( 0.0)     4000  250.2 M  7819
2012-04-03 13:17:53.681     0.001 any               65372        4( 0.0)        4( 0.0)    12406( 0.0)     4000   99.2 M  3101

For more about nfdump, here is a tutorial:

Tutorial nfdump

Note, that tranalyzer in combination with TAWK provides much more flexibility, especially when it comes to non standard questions, as in troubleshooting or traffic mining. nfcapd can be emulated by the socketSink plugin and netcat in an easy way.