Plugin pcap extraction

plugin development


As detailed in the pcap extraction tutorial, T2 is capable to extract flows into pcaps using the pcapd plugin. The regex_pcre is an example for this operation regex_pcre flow extraction. In order to implement this feature in your own plugin, specialized macros are supplied, which will be detailed in this tutorial. You do not need to enable the ALARM_MODE it also works in the normal flow mode.

Getting started

Create folders for your data and results

If you have not created a separate data and results directory yet, please do it now. This will greatly facilitate your workflow:

mkdir ~/data ~/results

Reset tranalyzer2 and the plugins configuration

If you have followed the other tutorials, you may have modified some of the core and plugins configuration. To ensure your results match those in this tutorial, make sure to reset everything:

t2conf -a --reset

You can also clean all build files:

t2build -a -c

Empty the plugin folder

To ensure we are not left with some unneeded plugins or plugins which were built using different core configuration, it is safer to empty the plugins folder:

t2build -e -y

Are you sure you want to empty the plugin folder '/home/user/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied

Download the PCAP file

The PCAP file used in this tutorial can be downloaded here:

Please save it in your ~/data folder:

wget --no-check-certificate -P ~/data

Build tranalyzer2 and the required plugins

For this tutorial, we will need to build the core (tranalyzer2) and the following plugins:

As you may have modified some of the automatically generated files, it is safer to use the -r and -f options.

t2build -r -f tranalyzer2 basicFlow basicStats tcpStates pcapd txtSink



Source code

In this tutorial, we will extend tcpWin12.tar.gz, the final version of the previous tutorial (Plugin force mode).

If you are impatient, you can download the final version of the tcpWin plugin we will develop in this tutorial.

To use one of those plugins, just unpack it in the plugins folder of your T2 installation.


tar -xf ~/Downloads/tcpWin13.tar.gz

And let t2_aliases know about it:

source "$T2HOME/scripts/t2_aliases"

Implementing the FL_ALARM capability

Open tcpWin.c in an editor and move to the t2OnLayer4() callback. The T2_SET_STATUS() line was already added in the Plugin Alarm Mode tutorial, so we do not need to add it again. Now, we want the PCAP extraction to work independently of the ALARM_MODE. So we remove the #if ALARM_MODE == 1 and #endif // ALARM_MODE == 1 pragmas before and after T2_SET_STATUS() (line marked by // <--).


vi src/tcpWin.c

void t2OnLayer4(packet_t *packet, unsigned long flowIndex) {
    flow_t * const flowP = &flows[flowIndex];

    if (flowP->l4Proto != L3_TCP) {             // process only TCP
        TCPWIN_SPKTMD_PRI_NONE();               // Packet mode
        return;                                 // go back to core

    // only 1. frag packet will be processed
    if (!t2_is_first_fragment(packet)) {
        TCPWIN_SPKTMD_PRI_NONE();               // Packet mode
        return;                                 // go back to core

    tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
    const tcpHeader_t * const tcpHeader = TCP_HEADER(packet);
    const uint32_t tcpWin = ntohs(tcpHeader->window);

    if (tcpWin < TCPWIN_THRES) {              // is the window size below the threshold?
        tcpWinFlowP->winThCnt++;              // count the packet / flow
        winThCntG++;                          // increment global packet counter
        tcpWinFlowP->stat |= TCPWIN_STAT_THU; // set the status bit
        tcpWinStat |= tcpWinFlowP->stat;      // Aggregate all packet flags

        T2_SET_STATUS(flowP, FL_ALARM);       // <-- Set the alarm bit in flow and global

#if FORCE_MODE == 1
        if (tcpWinFlowP->winThCnt > 1) T2_RM_FLOW(flowP);
#endif // FORCE_MODE == 1

    // Packet mode
    if (sPktFile) {
                "%" PRIu32 /* tcpWinSize     */ SEP_CHR
                "%" PRIu32 /* tcpWinThPktCnt */ SEP_CHR
                , tcpWin, tcpWinFlowP->winThCnt);


Now pcapd will extract all packets of a flow direction where the FL_ALARM is set.

If you want to have the global FL_ALARM status including flow and alarm count shown as a warning in the end report, then move the T2_REPORT_ALARMS() line in the t2OnFlowTerminate() callback outside the #if ALARM_MODE == 1 and #endif // ALARM_MODE == 1 pragmas.

vi src/tcpWin.c

void t2OnFlowTerminate(unsigned long flowIndex, outputBuffer_t *buf) {
    const tcpWinFlow_t * const tcpWinFlowP = &tcpWinFlows[flowIndex];
    bSFlow_t * const bSFlowP = &bSFlow[flowIndex];

    float f = 0.0;
    if (bSFlowP->numTPkts) {
        f = (float)tcpWinFlowP->winThCnt/(float)bSFlowP->numTPkts;

    OUTBUF_APPEND_U8(buf, tcpWinFlowP->stat);
    OUTBUF_APPEND_U8(buf, tcpWinFlowP->ttl);
    OUTBUF_APPEND_U32(buf, tcpWinFlowP->tcpWinInit);
    OUTBUF_APPEND_U32(buf, tcpWinFlowP->winThCnt);
    OUTBUF_APPEND_FLT(buf, f);

    flow_t * const flowP = &flows[flowIndex];
    if (!FLOW_IS_IPV4(flowP) || !PROTO_IS_TCP(flowP)) return;  // IPv4 and TCP only

    if (tcpWinFlowP->winThCnt == 0 || bSFlowP->numTPkts < TCPWIN_MINPKTS) return;

#if ALARM_MODE == 1
    winAlarms += tcpWinFlowP->winThCnt;      // global alarm count
    winThFlows++;                            // count number of alarms flows
#endif // ALARM_MODE == 1

    T2_REPORT_ALARMS(tcpWinFlowP->winThCnt); // <-- report alarm to the core

    const int wzi = gwz.wzi; // store element count in const local variable, makes the compiler happy
    if (wzi >= TCPWIN_MAXWSCNT) return;  // If array full, stop saving

    int i;
    for (i = 0; i < wzi; i++) {
        if (gwz.wzip[i].IPv4.s_addr == flowP->srcIP.IPv4.s_addr) break; // does IP exist?

    if (f > gwz.wzCnt[i]) {                     // only update if count is greater than the previous one
        gwz.tcpCnt[i] = bSFlowP->numTPkts;      // update packet count
        gwz.wzCnt[i] = f;                       // update relative count
        if (i == wzi) {                         // new one?
            gwz.wzip[i] = flowP->srcIP;         // save new IP
            gwz.sID[i] = flowP->subnetNrSrc;    // save subnetID from core
#endif // SUBNET_ON != 0 && (AGGREGATIONFLAG & SUBNET) == 0
            gwz.wzi++;                          // increment global window size counter



After you edited the skeleton code you should compare your implementation with tcpWin13.tar.gz.

Recompile and run t2 on the pcap.

t2build tcpWin

t2 -r ~/data/faf-exercise.pcap -w ~/results/

Tranalyzer 0.9.0 (Anteater), Cobra. PID: 91536, SID: 666
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.9.0
    02: basicStats, 0.9.0
    03: tcpStates, 0.9.0
    04: tcpWin, 0.9.0
    05: txtSink, 0.9.0
    06: pcapd, 0.9.0
[INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481663 (481.66 K)
[INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41533 (41.53 K)
Processing file: /home/wurst/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Snapshot length: 65535
Dump start: 1258544215.37210000 sec (Wed 18 Nov 2009 11:36:55 UTC)
Dump stop : 1258594491.683288000 sec (Thu 19 Nov 2009 01:34:51 UTC)
Total dump duration: 50276.646078000 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.009388000 sec
Finished unloading flow memory. Time: 0.009429000 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
basicStats: Biggest L3 flow talker: (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 flow talker: (US): 4268858 (4.27 M) [85.49%] bytes
tcpStates: Aggregated tcpStatesAFlags=0x4a
tcpWin: Aggregated tcpWinStat=0x01
tcpWin: Number of TCP winsize packets below threshold 1: 4 [0.07%]
tcpWin: IP:, country: 07, org: Private network
pcapd: number of packets extracted: 1467 (1.47 K) [24.86%]
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
Number of processed      flows: 72
Number of processed IPv4 flows: 72 [100.00%]
Number of processed A    flows: 36 [50.00%]
Number of processed B    flows: 36 [50.00%]
Number of request        flows: 36 [50.00%]
Number of reply          flows: 36 [50.00%]
Total   A/B    flow asymmetry: 0.00
Total req/rply flow asymmetry: 0.00
Number of processed   packets/flows: 81.97
Number of processed A packets/flows: 55.17
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.01 GB [0.12%]
Aggregated flowStat=0x0402000000004000
[WRN] 4 alarms in 1 flows [1.39%]
[INF] IPv4 flows
[INF] IPAlarm

The plugin report states that 1467 packets were extracted of one flow. Note the [WRN] which reports the number of alarms. 0x0402000000004000 describes the global aggregate flow status which T2_SET_STATUS() also sets. Try a tawk -V to decode the flow status:

tawk -V flowStat=0x0402000000004000

The flowStat column with value 0x0402000000004000 is to be interpreted as follows:

   bit | flowStat              | Description
    14 | 0x0000 0000 0000 4000 | IPv4 flow
    49 | 0x0002 0000 0000 0000 | Alarm mode & pcapd dumps packets from this flow to new pcap if not -e option
    58 | 0x0400 0000 0000 0000 | IPv4 packet


Now look into the results directory:

ls ~/results

faf-exercise_flows.txt  faf-exercise_headers.txt  faf-exercise_pcapd.pcap

Note the file faf-exercise_pcapd.pcap, which represents the extracted packets belonging to the flow where the FL_ALARM bit is set. To verify, select these flows using tawk.

tawk 'bitsanyset($flowStat, 0x0002000000000000)' ~/results/faf-exercise_flows.txt | tcol

%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcIP          srcIPCC  srcIPOrg           srcPort  dstIP          dstIPCC  dstIPOrg  dstPort  l4Proto  tcpStates  tcpWinStat  tcpWinThCnt
A     36       0x0402000000004000  1258594163.408285  1258594191.015208  27.606923  1           3        eth:ipv4:tcp  00:08:74:38:01:b4  00:19:e3:e7:5d:23  0x0800      07       "Private network"  49330  us       "Dell"    64334    6        0x42       0x01        4

Only one flow? Why? Look at the config of pcapd:

t2conf -G PD_OPP pcapd

PD_OPP = 0

So set it to 1 and see whether you get the other flow, as homework. If you are uneasy working with pcapd refer to the PCAP extraction and upscaling tutorial.

If you are just interested in pcap extraction, then you do not need to load txtSink. It is advisable to use the -w option, otherwise t2 stores the pcap in the data folder, and you start desperately looking for your extracted pcap. It happened to people ;-)


Have fun extracting pcaps!

You can download the final version of the tcpWin plugin.

The next tutorial will teach you how to implement timeout handlers.

See also