Monitoring/reporting
Contents
- Introduction
- Preparation
- Signalling and basic remote control of the Anteater
- Controls on large pcaps
- Controls on interfaces
- Activation of machine monitoring/reporting mode
- Timebase, pcaps and stuff
- Adding/removing protocols
- Time base on an interface and stuff
- Logging into remote tools
- Logging into RRD Tool
- Conclusion
Introduction
This tutorial details the use of the monitoring mode of the Anteater, which works independently of the flow or packet information production.
It was originally used to give the user an intermittent report and thus a decision base if large pcaps in the TByte regime were processed. Then it evolved into remote controlled and autonomous multi-sensor reporting and monitoring where an admin once configures T2 and deploys it on a machine listening on an interface. In this tutorial we will discuss all aspects of T2 reporting.
Preparation
First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Then compile the following plugins:
t2build tranalyzer2 basicFlow basicStats tcpStates ftpDecode txtSink
...
BUILD SUCCESSFUL
If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:
mkdir ~/data ~/results
The anonymized sample PCAPs used in this tutorial can be downloaded here:
Please save them in your ~/data folder.
In addition, sniffing directly from the interface will be required, so have your sudo password ready.
Now you are all set for T2 monitoring experiments.
Signalling and basic remote control of the Anteater
The Anteater can be remote controlled, using UNIX signals. The basic ones are listed below:
| Signal | Description |
|---|---|
SIGUSR1 |
Invoke one intermittent report |
SIGUSR2 |
Start/stop periodical reports: Monitoring Mode |
SIGINT |
1. Stop flow creation (like ^C in the shell) |
| 2. Terminate T2 unload all flows | |
SIGTERM |
Terminate T2 immediately (Like 2 * ^C), unload flows |
SIGSTOP |
Stop T2 processing (Like ^Z) |
SIGCONT |
Continue T2 processing (Like fg or bg) |
You can send these signals to t2 via a kill command such as:
kill -USR1 PID
The T2 report supplies its PID. In order to facilitate the control of T2, we developed t2stat,
which we will be discussed in the following chapter.
Controls on large pcaps
Imagine you have really large pcaps and you want to check on the anteater and get an intermittent report.
Have two bash windows open. One where you start t2, and another one where control signals can be sent from.
To facilitate the signal commands for the user we supply a t2stat command:
Usage:
t2stat [OPTION...]
Optional arguments:
INTERVAL Send a signal to Tranalyzer every INTERVAL seconds
-SIGNAME Send SIGNAME signal instead of USR1
-s Run the command as root (with sudo)
-p Print Tranalyzer PID(s) and exit
-l List Tranalyzer PID(s), commands, running time and exit
-i Interactively cycle through all Tranalyzer processes
-h Show help options and exit
If there is no Anteater running t2stat will tell you:
No running instance of Tranalyzer found
Type the t2stat command in the signal window but don’t send it yet.
Now lets start t2 on a larger pcap and produce a packet file so that we have time
to send a signal. When t2 starts, hit return in the signal window and you will see in
the t2 window an Andy looking over a wall at you with a report:
================================================================================ Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73743 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.8.14 02: basicStats, 0.8.14 03: tcpStates, 0.8.14 04: ftpDecode, 0.8.14 05: txtSink, 0.8.14 [INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K) [INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K) Processing file: /home/wurst/data/annoloc2.pcap Link layer type: Ethernet [EN10MB/1] Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT) [WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500 @ @ | | ===============================vVv==(a a)==vVv=============================== =====================================\ /===================================== ======================================\ /====================================== oo USR1 A type report: Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73743 PCAP time: 1022171712.313520 sec (Thu 23 May 2002 16:35:12 GMT) PCAP duration: 10.622348 sec Time: 1643656805.288080 sec (Mon 31 Jan 2022 20:20:05 CET) Elapsed time: 2.983330 sec Processing file: /home/wurst/data/annoloc2.pcap Total bytes to process: 83586990 (83.59 M) Percentage completed: 42.40% Total bytes processed so far: 35443192 (35.44 M) Remaining time: 4.052367 sec ETF: 1643656809.340447 sec (Mon 31 Jan 2022 20:20:09 CET) Number of processed packets: 517029 (517.03 K) Number of processed bytes: 27170706 (27.17 M) Number of raw bytes: 356395429 (356.40 M) Number of pad bytes: 4295705926 (4.30 G) Number of IPv4 packets: 516830 (516.83 K) [99.96%] Number of IPv6 packets: 77 [0.01%] Number of A packets: 240498 (240.50 K) [46.52%] Number of B packets: 276531 (276.53 K) [53.48%] Number of A bytes: 12552308 (12.55 M) [46.20%] Number of B bytes: 14618398 (14.62 M) [53.80%] Average A packet load: 52.19 Average B packet load: 52.86 -------------------------------------------------------------------------------- ftpDecode: Number of FTP control packets: 1032 (1.03 K) [0.20%] -------------------------------------------------------------------------------- Headers count: min: 2, max: 4, average: 3.01 Number of GRE packets: 13 [0.00%] Number of ICMP packets: 1321 (1.32 K) [0.26%] Number of ICMPv6 packets: 6 [0.00%] Number of TCP packets: 401196 (401.20 K) [77.60%] Number of TCP bytes: 22267024 (22.27 M) [81.95%] Number of UDP packets: 114363 (114.36 K) [22.12%] Number of UDP bytes: 4814726 (4.81 M) [17.72%] Number of IPv4 fragmented packets: 957 [0.19%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 10355 (10.36 K) Number of processed A flows: 5844 (5.84 K) [56.44%] Number of processed B flows: 4511 (4.51 K) [43.56%] Number of request flows: 5842 (5.84 K) [56.42%] Number of reply flows: 4513 (4.51 K) [43.58%] Total A/B flow asymmetry: 0.13 Total req/rply flow asymmetry: 0.13 Number of processed packets/flows: 49.93 Number of processed A packets/flows: 41.15 Number of processed B packets/flows: 61.30 Number of processed total packets/s: 48673.70 (48.67 K) Number of processed A+B packets/s: 48673.70 (48.67 K) Number of processed A packets/s: 22640.76 (22.64 K) Number of processed B packets/s: 26032.95 (26.03 K) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of average processed flows/s: 974.83 Average full raw bandwidth: 268411776 b/s (268.41 Mb/s) Average snapped bandwidth : 20463050 b/s (20.46 Mb/s) Average full bandwidth : 267848912 b/s (267.85 Mb/s) Fill size of main hash map: 9724 [3.71%] Max number of flows in memory: 9724 (9.72 K) [3.71%] Memory usage: 0.07 GB [0.10%] Aggregated flowStat=0x0c00987a0202d044 [WRN] L3 SnapLength < Length in IP header [WRN] L4 header snapped [WRN] Consecutive duplicate IP ID [WRN] IPv4/6 payload length > framing length [WRN] IPv4/6 fragmentation header packet missing [INF] Stop dissecting: Clipped packet, unhandled protocol or subsequent fragment [INF] Layer 2 flows [INF] IPv4 flows [INF] IPv6 flows [INF] ARP [INF] IPv4/6 fragmentation [INF] IPv4/6 in IPv4/6 [INF] GRE encapsulation [INF] SSDP/UPnP ================================================================================ ...
Note at the beginning USR1 A type denotes that there was a signal received and
all reporting is aggregated from the beginning of T2 operation, almost like the end report.
Note, that the reaction to signals depends to internal core configuration, discussed in
a later chapter below. For the time being we stick with the default.
If you require now a report every 30s then type t2stat 30. Then a signal is
sent to T2 every 30s, hence a remote control. Nevertheless, he can do it also
by himself, more later.
Note that the report does not only tell you all about packet statistics but also the
- pcap duration
- Elapsed time
- Total bytes to process
- Percentage completed
- Total bytes processed so far
- Remaining time
- Estimated Time Finish (ETF)
Especially the ETF is very valuable for large or multiple large pcap operations, e.g. multiple 10TB files. So the Anteater tells you when to come back from your coffee break, how cool is that?
Actually the Anteater forced us to implement that function because he does not like to be watched all the time. But that is a secret, so don’t tell him that I told you.
Now let’s have a look how the interface ops work.
Controls on interfaces
Have again two bash windows open. One for T2 on an interface with some traffic, and another one where control signals can be sent from.
Note, sudo is required to gain access to the interface. Since version 0.8.9 the st2 command is available.
It also takes care of the location of your home plugin directory, hence the parameter: -p /home/wurst/.tranalyzer/plugins/
can be omitted. To ascertain the name of your interface which is connected to your network use ifconfig or ip link and swap it
with yourinterfacename below.
[sudo] password for wurst: ================================================================================ Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73880 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.8.14 02: basicStats, 0.8.14 03: tcpStates, 0.8.14 04: ftpDecode, 0.8.14 05: txtSink, 0.8.14 [INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K) [INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K) Live capture on interface: yourinterfacename Link layer type: Ethernet [EN10MB/1] Dump start: 1643657018.182595 sec (Mon 31 Jan 2022 19:23:38 GMT) ...
Now switch to the other bash window and invoke t2stat -s.
As now you need root privileges to send signals to a rooted t2.
[sudo] password for wurst:
You will then see the following output in the t2 window.
...
@ @
| |
===============================vVv==(a a)==vVv===============================
=====================================\ /=====================================
======================================\ /======================================
oo
USR1 A type report: Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73880
PCAP time: 1643657064.193359 sec (Mon 31 Jan 2022 19:24:24 GMT)
PCAP duration: 46.010764 sec
Time: 1643657064.907929 sec (Mon 31 Jan 2022 20:24:24 CET)
Elapsed time: 46.800136 sec
Live capture on interface: yourinterfacename
Number of packets received: 126 [100.00%]
Number of processed packets: 125
Number of processed bytes: 10180 (10.18 K)
Number of raw bytes: 10180 (10.18 K)
Number of pad bytes: 161
Number of IPv4 packets: 45 [36.00%]
Number of IPv6 packets: 57 [45.60%]
Number of A packets: 125 [100.00%]
Number of A bytes: 10180 (10.18 K) [100.00%]
Average A packet load: 81.44
Average B packet load: 0.00
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Headers count: min: 3, max: 4, average: 3.06
Number of LLC packets: 23 [18.40%]
Number of ICMPv6 packets: 12 [9.60%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 6
Number of processed A flows: 6 [100.00%]
Number of request flows: 6 [100.00%]
Total A/B flow asymmetry: 1.00
Total req/rply flow asymmetry: 1.00
Number of processed packets/flows: 20.83
Number of processed A packets/flows: 20.83
Number of processed total packets/s: 2.72
Number of processed A+B packets/s: 2.72
Number of processed A packets/s: 2.72
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.13
Average full raw bandwidth: 1770 b/s (1.77 Kb/s)
Average full bandwidth : 1530 b/s (1.53 Kb/s)
Fill size of main hash map: 6 [0.00%]
Max number of flows in memory: 6 [0.00%]
Memory usage: 0.02 GB [0.03%]
Aggregated flowStat=0x0c0000000000c004
[INF] Layer 2 flows
[INF] IPv4 flows
[INF] IPv6 flows
================================================================================
Why there are no FTP plugin reports? Because we have no FTP in our network, and will never have.
t2stat -s -USR2 produces by himself a USR1 A type every second an intermittent anteater report
until you send another USR2 signal. If you want another period e.g. 30s then the core configuration
has to be changed, s.below.
There is also a machine mode, which is better suited for scripting and post-processing tools. It will be discussed in the next section.
Activation of machine monitoring/reporting mode
In order to configure the more script friendly machine representation, open the file tranalyzer.h under
~/tranalyzer2-0.9.3/tranalyzer2/src and skip to the // End report / monitoring section as shown below:
tranalyzer2
vi src/tranalyzer.h
...
// End report / monitoring: USR1/2 interrupts
#define PLUGIN_REPORT 1 // enable plugins to contribute to the tranalyzer command line end report
#define DIFF_REPORT 0 // 1: differential, 0: Absolute tranalyzer command line USR1 report
#define MACHINE_REPORT 0 // 1: machine compliant; 0: human compliant
...To switch from human to machine compliant reporting set MACHINE_REPORT=1.
The monitoring output is by default aggregated over time, like the end report.
Switch also to the differential output mode, which is mostly used by reporting
tools, such as RRD Tool.
Then, the counts are sampled in a certain time period, by default 1s, defined in main.h.
So set DIFF_REPORT=1. Note that you need to recompile with t2build -R, as the
plugins need to change code for that function.
The PLUGIN_REPORT enables the monitoring output in the plugins. We leave it on.
Use t2conf to change the config and recompile with -R.
Tranalyzer is currently running...
Proceed anyway (y/N)? N
Oups! Your t2 is still running in the other window. So either send him a SIGTERM.
t2stat -s -TERM
Or stop him by invoking ^C^C in the t2 window.
Now restart t2 in the t2 window:
================================================================================ Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73880 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.8.14 02: basicStats, 0.8.14 03: tcpStates, 0.8.14 04: ftpDecode, 0.8.14 05: txtSink, 0.8.14 [INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K) [INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K) Live capture on interface: yourinterfacename Link layer type: Ethernet [EN10MB/1] %repTyp time dur pktsRec pktsDrp ifDrp memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn 0x0042Pkts 0x0042Bytes 0x00fePkts 0x00feBytes 0x0806Pkts 0x0806Bytes 0x8035Pkts 0x8035Bytes 0x0800Pkts 0x0800Bytes 0x86ddPkts 0x86ddBytes ICMPPkts ICMPBytes IGMPPkts IGMPBytes TCPPkts TCPBytes UDPPkts UDPBytes GREPkts GREBytes ICMPv6Pkts ICMPv6Bytes SCTPPkts SCTPBytes ftpPkts ftpDataPkts Dump start: 1643657018.182595 sec (Mon 31 Jan 2022 19:23:38 GMT) ...
And invoke the following sequence in your control window:
t2stat -s -USR2 # trigger periodic monitoring
Wait a few seconds…
t2stat -s -USR2 # switch periodic monitoring off
t2stat -s -SIGINT # stop new flow creation
t2stat -s -SIGINT # terminate t2, create end report
Then you will see the following output, which looks a more script friendly.
... ================================================================================ Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 73880 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.8.14 02: basicStats, 0.8.14 03: tcpStates, 0.8.14 04: ftpDecode, 0.8.14 05: txtSink, 0.8.14 [INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K) [INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K) Live capture on interface: yourinterfacename Link layer type: Ethernet [EN10MB/1] %repTyp time dur pktsRec pktsDrp ifDrp memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn 0x0042Pkts 0x0042Bytes 0x00fePkts 0x00feBytes 0x0806Pkts 0x0806Bytes 0x8035Pkts 0x8035Bytes 0x0800Pkts 0x0800Bytes 0x86ddPkts 0x86ddBytes ICMPPkts ICMPBytes IGMPPkts IGMPBytes TCPPkts TCPBytes UDPPkts UDPBytes GREPkts GREBytes ICMPv6Pkts ICMPv6Bytes SCTPPkts SCTPBytes ftpPkts ftpDataPkts Dump start: 1613666096.469516 sec (Thu 18 Feb 2021 16:34:56 GMT) USR1MR_D 1613666103.342685 6.873169 28 0 0 20508 11 11 10 1 27 22 5 10 13 0 2959 2492 467 0 0 0 3.444 0x0c0000000000c004 0 0 0 00 0 0 0 10 1441 13 1278 0 0 0 0 8 1301 0 0 0 0 11 1098 0 0 0 0 USR1MR_D 1613666104.342536 0.999851 28 0 0 0 0 0 0 0 1 1 0 0 1 0 86 86 0 0 0 0 0.688 0x0c0000000000c004 0 0 0 00 0 0 0 0 0 1 86 0 0 0 0 0 0 0 0 0 0 1 86 0 0 0 0 USR1MR_D 1613666105.342200 0.999664 32 0 0 0 1 1 1 0 2 2 0 1 0 0 269 269 0 0 0 0 2.153 0x0c0000000200c004 0 0 0 00 0 0 0 1 209 0 0 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0 USR1MR_D 1613666106.342418 1.000218 33 0 0 0 2 2 1 1 3 2 1 3 0 0 341 275 66 0 0 0 2.727 0x0c0000000200c004 0 0 0 00 0 0 0 3 341 0 0 0 0 0 0 2 132 1 209 0 0 0 0 0 0 0 0 USR1MR_D 1613666107.342459 1.000041 37 0 0 0 0 0 0 0 4 4 0 2 1 0 429 429 0 0 0 0 3.432 0x0c0010000200c004 0 0 0 00 0 0 0 2 279 1 90 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0 USR1MR_D 1613666108.342960 1.000501 38 0 0 0 0 0 0 0 1 1 0 1 0 0 209 209 0 0 0 0 1.671 0x0c0010000200c004 0 0 0 00 0 0 0 1 209 0 0 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0 USR1MR_D 1613666109.342512 0.999552 39 0 0 0 0 0 0 0 1 1 0 0 0 0 60 60 0 0 0 0 0.480 0x0c0010000200c004 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 USR1MR_D 1613666110.343233 1.000721 39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.000 0x0c0010000200c004 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 USR1MR_D 1613666111.342791 0.999558 75 0 0 684 4 4 2 2 36 22 14 35 0 0 9414 2879 6535 0 0 0 75.345 0x0c0010000200c004 0 0 0 00 0 0 0 35 9354 0 0 0 0 0 0 33 9164 2 190 0 0 0 0 0 0 0 0 USR1MR_D 1613666112.343044 1.000253 624 0 0 0 0 0 0 0 476 166 310 475 1 0 481622 12282 469340 0 0 0 3852.001 0x0c0010000200c004 0 0 00 0 0 0 0 475 481532 1 90 0 0 0 0 474 481462 0 0 0 0 0 0 0 0 0 0 [INF] SIGINT: Stop flow creation: 0x0002 [INF] SIGINT: Stop flow creation: 0x0001 Dump stop : 1643657116.890205 sec (Mon 31 Jan 2022 19:25:16 GMT) Total dump duration: 25.999741 sec Finished processing. Elapsed time: 27.574739 sec Finished unloading flow memory. Time: 27.575033 sec Number of packets received: 643 [100.00%] Number of processed packets: 643 Number of processed bytes: 566099 (566.10 K) Number of raw bytes: 566099 (566.10 K) Number of pad bytes: 98 Number of IPv4 packets: 601 [93.47%] Number of IPv6 packets: 28 [4.35%] Number of A packets: 268 [41.68%] Number of B packets: 375 [58.32%] Number of A bytes: 23019 (23.02 K) [4.07%] Number of B bytes: 543080 (543.08 K) [95.93%] Average A packet load: 85.89 Average B packet load: 1448.21 (1.45 K) -------------------------------------------------------------------------------- basicStats: Biggest L2 talker: xx:xx:xx:xx:xx:xx: 14 [2.18%] packets basicStats: Biggest L2 talker: yy:yy:yy:yy:yy:yy: 840 [0.15%] bytes basicStats: Biggest L3 talker: a.b.c.d (CH): 356 [55.37%] packets basicStats: Biggest L3 talker: u.v.w.x (CH): 536078 (536.08 K) [94.70%] bytes tcpStates: Aggregated tcpStatesAFlags=0x03 -------------------------------------------------------------------------------- Headers count: min: 3, max: 4, average: 3.03 Number of LLC packets: 14 [2.18%] Number of GRE packets: 14 [2.18%] Number of ICMPv6 packets: 22 [3.42%] Number of TCP packets: 585 [90.98%] Number of TCP bytes: 560249 (560.25 K) [98.97%] Number of UDP packets: 10 [1.56%] Number of UDP bytes: 1862 (1.86 K) [0.33%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 19 Number of processed A flows: 15 [78.95%] Number of processed B flows: 4 [21.05%] Number of request flows: 15 [78.95%] Number of reply flows: 4 [21.05%] Total A/B flow asymmetry: 0.58 Total req/rply flow asymmetry: 0.58 Number of processed packets/flows: 33.84 Number of processed A packets/flows: 17.87 Number of processed B packets/flows: 93.75 Number of processed total packets/s: 24.73 Number of processed A+B packets/s: 24.73 Number of processed A packets/s: 10.31 Number of processed B packets/s: 14.42 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of average processed flows/s: 0.73 Average full raw bandwidth: 174186 b/s (174.19 Kb/s) Average full bandwidth : 173928 b/s (173.93 Kb/s) Max number of flows in memory: 19 [0.01%] Memory usage: 0.02 GB [0.03%] Aggregated flowStat=0x0c0010000200c004 [WRN] Consecutive duplicate IP ID [INF] Ethernet flows [INF] IPv4 flows [INF] IPv6 flows [INF] SSDP/UPnP
USR1MR_D denotes signal received, differential reporting.
Nevertheless, the output of the header and End report including all warnings are not beneficial if we want to post-process the
stream. So add the -l option. It reroutes all the end reporting into the results directory with the filename yourinterface_log.txt.
Execute the st2 command line again and execute the same t2stat commands as above.
Then you will see the following output, which omits all the unnecessary stuff
which is not column oriented.
st2 -i yourinterfacename -w ~/results/ -l
%repTyp time dur pktsRec pktsDrp ifDrp memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn 0x0042Pkts 0x0042Bytes 0x00fePkts 0x00feBytes 0x0806Pkts 0x0806Bytes 0x8035Pkts 0x8035Bytes 0x0800Pkts 0x0800Bytes 0x86ddPkts 0x86ddBytes ICMPPkts ICMPBytes IGMPPkts IGMPBytes TCPPkts TCPBytes UDPPkts UDPBytes GREPkts GREBytes ICMPv6Pkts ICMPv6Bytes SCTPPkts SCTPBytes ftpPkts ftpDataPkts
USR1MR_D 1613666471.195360 12.732418 37 0 0 20068 13 13 11 2 36 30 6 17 13 0 4113 3580 533 0 0 0 2.584 0x0c0010000200c004 0 0 0 00 0 0 0 17 2479 13 1274 0 0 0 0 10 1433 4 836 0 0 10 1004 0 0 0 0
USR1MR_D 1613666472.196159 1.000799 37 0 0 0 0 0 0 0 1 1 0 0 0 0 60 60 0 0 0 0 0.480 0x0c0010000200c004 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
USR1MR_D 1613666473.195578 0.999419 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.000 0x0c0010000200c004 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
USR1MR_D 1613666474.195928 1.000350 48 0 0 0 0 0 0 0 3 2 1 2 0 0 192 126 66 0 0 0 1.535 0x0c0010000200c004 0 0 0 00 0 0 0 2 132 0 0 0 0 0 0 2 132 0 0 0 0 0 0 0 0 0 0
USR1MR_D 1613666475.196087 1.000159 58 0 0 0 1 1 1 0 8 8 0 0 8 0 832 832 0 0 0 0 6.655 0x0c0010000200c004 0 0 0 00 0 0 0 0 0 8 832 0 0 0 0 0 0 0 0 0 0 8 832 0 0 0 0
USR1MR_D 1613666476.195855 0.999768 60 0 0 1080 3 3 2 1 10 8 2 5 3 0 786 623 163 0 0 0 6.289 0x0c0010000200c044 0 0 0 01 60 0 0 5 400 3 266 0 0 0 0 4 330 0 0 0 0 2 176 0 0 0 0
USR1MR_D 1613666477.196219 1.000364 62 0 0 0 0 0 0 0 2 2 0 0 2 0 176 176 0 0 0 0 1.407 0x0c0010000200c044 0 0 0 00 0 0 0 0 0 2 176 0 0 0 0 0 0 0 0 0 0 2 176 0 0 0 0
USR1MR_D 1613666478.196418 1.000199 63 0 0 0 1 1 1 0 2 2 0 1 0 0 269 269 0 0 0 0 2.152 0x0c0010000200c044 0 0 0 00 0 0 0 1 209 0 0 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0
USR1MR_D 1613666479.196230 0.999812 65 0 0 0 0 0 0 0 1 1 0 1 0 0 209 209 0 0 0 0 1.672 0x0c0010000200c044 0 0 0 00 0 0 0 1 209 0 0 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0
USR1MR_D 1613666480.196109 0.999879 66 0 0 0 0 0 0 0 2 2 0 1 0 0 269 269 0 0 0 0 2.152 0x0c0010000200c044 0 0 0 00 0 0 0 1 209 0 0 0 0 0 0 0 0 1 209 0 0 0 0 0 0 0 0So if you look in your ~/results folder:
ls ~/results
yourinterfacename.txt yourinterfacename_headers.txt yourinterfacename_log.txt
Now all end reports from the core and the plugins are routed into the log file. T2 still produces flow and other plugin
related files to your ~/results directory. If you are only interested in the monitoring output unload
txtSink and T2 becomes a more performant monitoring tool,
because the flow write process to the disk is omitted. And you do not need to supply the -w option anymore.
Plugin 'txtSink' UNLOADING SUCCESSFUL
You can also use the BLOCK_BUF switch in tranalyzer.h, disabling all flow output generation, making T2 even faster, which we will discuss later.
For the time being we are good so far.
If you were only interested in remote controlling the T2 monitoring process, here you are. Nevertheless, in case of pcap processing, or time series generation for research purposes we need to talk about time bases and other nitty-gritty functionality of the Anteater.
Timebase, pcaps and stuff
Open tranalyzer2/src/main.h and skip to the Monitoring mode section. It looks a bit
confusing, but it combines several different operations. Don’t worry I’ll talk you through.
tranalyzer2
vi src/main.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
// Packet mode (-s option)
#define SPKTMD_PKTNO 1 // Whether or not to print the packet number
#define SPKTMD_PCNTC 1 // Whether or not to print L7 content as characters
#define SPKTMD_PCNTH 0 // Whether or not to print L7 content as hex
// Monitoring mode
#define MONINTTHRD 1 // Monitoring: Threaded interrupt handling.
#define MONINTBLK 0 // Monitoring: Block interrupts in main loop during packet processing, disables MONINTTHRD.
#define MONINTPSYNC 1 // Monitoring: 1: pcap main loop synchronized printing. 0: Interrupt printing.
#define MONINTTMPCP 0 // Monitoring: 1: pcap time base, 0: real time base.
#define MONINTTMPCP_ON 0 // Monitoring: Startup monitoring 1: on 0: off; if (MONINTTMPCP == 0)
#define MONINTV 1 // Monitoring: GI_ALRM: MONINTV >= 1 sec interval of monitoring output.
// Monitoring mode protocol stat
#define MONPROTMD 1 // Monitoring: 0: Protocol numbers; 1: Protocol names (L3 only)
#define MONPROTL3 L3_ICMP,L3_IGMP,L3_TCP,L3_UDP,L3_GRE,L3_ICMP6,L3_SCTP
#define MONPROTL2 0x0042,0x00fe,ETHERTYPE_ARP,ETHERTYPE_RARP,ETHERTYPE_IP,ETHERTYPE_IPV6
#define MONPROTFL "proto.txt"
// statistics summary min max
#define MIN_MAX_ESTIMATE 0 // min max bandwidth statistics
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...The first two constants MONINTTHRD and MONINTBLK control the monitoring and reporting process,
either via a thread or integrated into the cores main loop.
These modes were introduced, because the libpcap showed instabilities on certain platforms in the past
if Unix signals are involved, when operating on an interface. These bugs are fixed by the libpcap crew,
so the MONINTBLK mode is not needed anymore. If you do not like threading, you can switch to MONINTBLK mode.
The rest of Earth’s population might like to stick with the faster default setting.
MONINTPSYNC controls the location of the print operation, either in the signal interrupt routine or synchronized
to the packet processing at main loop level. If you sniff on an interface which has almost no traffic, but you
require every 1 sec a report, then set MONINTPSYNC=0. Otherwise, leave it at 1 because then the packet processing
is always prioritized and thus more performant. So default setting is good for us.
MONINTTMPCP defines the time base of the monitoring process; either pcap time, or real time.
If you read from a pcap, which may consist of many days of packet capture, you can simulate a monitoring signal,
if MONINTTMPCP=1. Then T2 will generate a pcap time dependent stream of reports.
On an interface T2 will also produce a stream of monitoring output synchronized to packet arrival. Meaning if
there are no packets, there will be no monitoring output.
So let’s set MONINTTMPCP to 1, pcap time rules.
If in real-time base MONINTTMPCP_ON defines whether T2 starts monitoring right at startup or waits for a
USR1 or USR2 signal. Will be discussed later.
MONINTV denotes the default monitoring interval in seconds. Let’s set it to 10, so that we do not produce so
much output from our test pcap: faf-exercise.pcap.
The Monitoring mode protocol stat section denotes the protocols T2 outputs. MONPROTMD defines the type of header output,
ether protocol numbers or human readable names, which is the default. If you like to write your own post-processing,
protocol numbers might be a better choice. MONPROTL2 and MONPROTL3 define the protocols which should be monitored.
Later we will add some more. For the time being we leave everything as default.
Note, that every plugin can contribute to the monitoring mode, as in the end report. So loading new plugins might produce new columns.
Summarizing the config we need to make: We switch to pcap based, and set the timing interval to 10s. So it has to look like this.
...
// Monitoring mode
...
#define MONINTTMPCP 1 // Monitoring: 1: pcap time base, 0: real time base.
...
#define MONINTV 10 // Monitoring: GI_ALRM: MONINTV >= 1 sec interval of monitoring output.
...Now close the file and recompile T2 or use t2conf below and apply T2 to the pcap.
As we produce no flows anymore, the -w option can be omitted, and we do not
need to be root anymore because we operate now on our test pcap.
t2conf tranalyzer2 -D MONINTTMPCP=1 -D MONINTV=10 && t2build tranalyzer2
t2 -r ~/data/faf-exercise.pcap -l
Note that the timestamp and the duration are not correlating with the configured monitoring interval. What is the reason?
Invoke the packet mode using the -s option:
t2 -r ~/data/faf-exercise.pcap -w ~/results/ -l -s | tcol
%repTyp time dur memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn 0x0042 0x00fe 0x0806 0x8035 0x0800 0x86dd ICMP IGMP TCP UDP GRE ICMPv6 SCTP ftpPkts
USR1MR_D 1258562467.749142 18252.711932 413264 1 19 10 9 200 66 134 200 0 0 153213 7121 146092 0 0 0 0.067 0x0000000000004000 0 0 0 0 200 0 0 0 200 0 0 0 0 0
USR1MR_D 1258562477.713894 9.964752 0 1 1 0 1 7 4 3 7 0 0 1762 1148 614 0 0 0 1.415 0x0000000000004000 0 0 0 0 7 0 0 0 7 0 0 0 0 0
USR1MR_D 1258562501.432305 23.718411 0 2 2 1 1 32 12 20 32 0 0 22559 2125 20434 0 0 0 7.609 0x0000000000004000 0 0 0 0 32 0 0 0 32 0 0 0 0 0
...Change to the bash window where you are in the ~/results directory and look into the packet file:
tcol ~/results/faf-exercise_packets.txt
%pktNo flowInd flowStat time pktIAT flowDuration numHdrs hdrDesc ethVlanID srcMac dstMac ethType srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto srcManuf dstManuf pktLen l7Len l7Content
1 1 0x0000000000004000 1258544215.037210 0.000000 0.000000 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1258 77.67.44.206 es akamai technologies 80 6 Dell Apple 66 0
2 1 0x0000000000004001 1258544215.202900 0.000000 0.000000 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1258 6 Apple Dell 62 0
3 1 0x0000000000004000 1258544215.203358 0.166148 0.166148 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1258 77.67.44.206 es akamai technologies 80 6 Dell Apple 64 0
4 1 0x0000000000004000 1258544215.203850 0.000492 0.166640 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1258 77.67.44.206 es akamai technologies 80 6 Dell Apple 425 367 GET /softw/90/update/avg9infoavi.ctf HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: backup.avg.cz\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nIf-Modified-Since: Tue, 17 Nov 2009 20:41:10 GMT\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
5 1 0x0000000000004001 1258544215.370055 0.167155 0.167155 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1258 6 Apple Dell 1434 1380 HTTP/1.1 200 OK\r\nDate: Wed, 18 Nov 2009 11:39:48 GMT\r\nServer: Apache\r\nLast-Modified: Wed, 18 Nov 2009 09:04:15 GMT\r\nETag: "15c007-cea-478a186e401c0"\r\nAccept-Ranges: bytes\r\nContent-Length: 3306\r\nConnection: close\r\nContent-Type: text/plain\r\n\r\nAVG CTF Index File;ver(10)\r\nbin(u7avi1777ff.bin)grp(avi:1777)tm(0911180855)pri(2)len(6637487)\r\nbin(u7avi1777u1323ff.bin)grp(avi:1777)dep(avi:1323)tm(0911180855)pri(2)len(590030)\r\nbin(u7avi1777u1705ff.bin)grp(avi:1777)dep(avi:1705)tm(0911180855)pri(2)len(95323)\r\nbin(u7iavi2511ff.bin)grp(iavi:2511)tm(0911180855)pri(2)len(45641416)\r\nbin(u7iavi2511u2251fo.bin)grp(iavi:2511)dep(iavi:2251)tm(0911180855)pri(2)len(6175342)\r\nbin(u7iavi2511u2353fn.bin)grp(iavi:2511)dep(iavi:2353)tm(0911180855)pri(2)len(4506824)\r\nbin(u7iavi2511u2404fm.bin)grp(iavi:2511)dep(iavi:2404)tm(0911180855)pri(2)len(3342546)\r\nbin(u7iavi2511u2431fl.bin)grp(iavi:2511)dep(iavi:2431)tm(0911180855)pri(2)len(2609643)\r\nbin(u7iavi2511u2458fk.bin)grp(iavi:2511)dep(iavi:2458)tm(0911180855)pri(2)len(1380674)\r\nbin(u7iavi2511u2480fj.bin)grp(iavi:2511)dep(iavi:2480)tm(0911180855)pri(2)len(681743)\r\nbin(u7iavi2511u2490fi.bin)grp(iavi:2511)dep(iavi:2490)tm(0911180855)pri(2)len(555496)\r\nbin(u7iavi2511u2500fh.bin)grp(iavi:2511)dep(iavi:2500)tm(0911180855)pri(2)len(332906)\r\nbin(u7iavi2511u2505fh.bin)grp(iavi:2511)dep(iavi:2505)tm(0911180855)pri(2)len(204942)\r\nbin(u7iavi2511u2508
6 1 0x0000000000004001 1258544215.370067 0.000012 0.167167 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1258 6 Apple Dell 375 321 fg.bin)grp(iavi:2511)dep(iavi:2508)tm(0911180855)pri(2)len(72274)\r\nbin(u7iavi2511u2510ff.bin)grp(iavi:2511)dep(iavi:2510)tm(0911180855)pri(2)len(30540)\r\nbin(u9ichjw4qt.bin)grp(ichjw:4)pri(2)tm(0907131859)len(113488)\r\nbin(u9ichjw4u2ia.bin)grp(ichjw:4)dep(ichjw:2)pri(2)tm(0907131859)len(94470)\r\nbin(u9ichjw4u3gq.bin)grp(ic
7 1 0x0000000000004000 1258544215.370501 0.166651 0.333291 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1258 77.67.44.206 es akamai technologies 80 6 Dell Apple 64 0
8 1 0x0000000000004001 1258544215.370560 0.000493 0.167660 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1258 6 Apple Dell 1434 1380 hjw:4)dep(ichjw:3)pri(2)tm(0907131859)len(13313)\r\nbin(u9idat249b243tc.bin)grp(idat:249)dif(idat:243)pri(2)tm(0911051401)len(621430)\r\nbin(u9idat249b245bp.bin)grp(idat:249)dif(idat:245)pri(2)tm(0911051401)len(197648)\r\nbin(u9idat249b246mj.bin)grp(idat:249)dif(idat:246)pri(2)tm(0911051400)len(159289)\r\nbin(u9idat249b247uh.bin)grp(idat:249)dif(idat:247)pri(2)tm(0911051400)len(95680)\r\nbin(u9idat249le.bin)grp(idat:249)pri(2)tm(0911051400)len(1927074)\r\nbin(u9ifw42jx.bin)grp(ifw:42)pri(2)tm(0911131400)len(541853)\r\nbin(u9ifw42u38we.bin)grp(ifw:42)dep(ifw:38)pri(2)tm(0911131400)len(123575)\r\nbin(u9ifw42u39ke.bin)grp(ifw:42)dep(ifw:39)pri(2)tm(0911131400)len(92745)\r\nbin(u9ifw42u40dc.bin)grp(ifw:42)dep(ifw:40)pri(2)tm(0911131400)len(67292)\r\nbin(u9ifw42u41tz.bin)grp(ifw:42)dep(ifw:41)pri(2)tm(0911131400)len(18393)\r\nbin(x8all234yc.bin)grp(xplph:0012;xplsb:0099;xplsb2:0118;xplsc:0149)tm(0911180700)pri(2)len(1146352)\r\nbin(x8xplph_12gj.bin)grp(xplph:12)tm(0910280700)pri(2)len(4551)\r\nbin(x8xplsb2_118c8.bin)grp(xplsb2:118)tm(0911180700)pri(2)len(4989)\r\nbin(x8xplsb_9946.bin)grp(xplsb:99)tm(0911160700)pri(2)len(985460)\r\nbin(x8xplsb_99d9546.bin)grp(xplsb:99)dif(xplsb:95)tm(0911160700)pri(2)len(32267)\r\nbin(x8xplsb_99d9646.bin)grp(xplsb:99)dif(xplsb:96)tm(0911160700)pri(2)len(3505)\r\nbin(x8xplsb_99d9746.bin)grp(xplsb:99)dif(xplsb:97)tm(0911160700)pri(2)len(644)\r\nbin(x8xplsb_99d9846.bin
9 1 0x0000000000004001 1258544215.370571 0.000011 0.167671 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1258 6 Apple Dell 520 466 )grp(xplsb:99)dif(xplsb:98)tm(0911160700)pri(2)len(581)\r\nbin(x8xplsc_149c8.bin)grp(xplsc:149)tm(0911180700)pri(2)len(151922)\r\nbin(x8xplsc_149d145c8.bin)grp(xplsc:149)dif(xplsc:145)tm(0911180700)pri(2)len(2561)\r\nbin(x8xplsc_149d146c8.bin)grp(xplsc:149)dif(xplsc:146)tm(0911180700)pri(2)len(1921)\r\nbin(x8xplsc_149d147c8.bin)grp(xplsc:149)dif(xplsc:147)tm(0911180700)pri(2)len(1767)\r\nbin(x8xplsc_149d148c8.bin)grp(xplsc:149)dif(xplsc:148)tm(0911180700)pri(2)len(1411)\r\n
...
197 9 0x0000000000004001 1258544218.165782 0.009740 0.028239 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us -- 80 192.168.1.104 02 private_reserved 1266 6 Apple Dell 54 0
198 9 0x0000000000004000 1258544218.166240 0.007500 0.035982 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1266 198.189.255.75 us -- 80 6 Dell Apple 64 0
199 8 0x0000000000004001 1258544218.294696 0.167378 0.342534 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 es akamai technologies 80 192.168.1.104 02 private_reserved 1265 6 Apple Dell 54 0
200 10 0x0000000000004000 1258562467.749142 0.000000 0.000000 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1379 63.245.221.11 ff mozilla corporation 80 6 Dell Apple 66 0
201 10 0x0000000000004001 1258562467.754689 0.000000 0.000000 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 ff mozilla corporation 80 192.168.1.104 02 private_reserved 1379 6 Apple Dell 62 0
202 10 0x0000000000004000 1258562467.754896 0.005754 0.005754 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1379 63.245.221.11 ff mozilla corporation 80 6 Dell Apple 64 0
203 10 0x0000000000004000 1258562467.755633 0.000737 0.006491 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1379 63.245.221.11 ff mozilla corporation 80 6 Dell Apple 507 449 GET /thunderbird/2.0.0.23/start/ HTTP/1.1\r\nHost: www.mozillamessaging.com\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.23) Gecko/20090812 Thunderbird/2.0.0.23\r\nAccept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip,deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nKeep-Alive: 300\r\nConnection: keep-alive\r\n\r\n
204 10 0x0000000000004001 1258562467.761440 0.006751 0.006751 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 ff mozilla corporation 80 192.168.1.104 02 private_reserved 1379 6 Apple Dell 54 0
205 10 0x0000000000004001 1258562467.761692 0.000252 0.007003 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 ff mozilla corporation 80 192.168.1.104 02 private_reserved 1379 6 Apple Dell 498 444 HTTP/1.1 302 Found\r\nServer: Apache/2.2.3 (CentOS)\r\nX-Powered-By: PHP/5.1.6\r\nLocation: http://www.mozillamessaging.com/en-US/thunderbird/2.0.0.23/start/\r\nVary: Accept-Language,Accept-Encoding\r\nContent-Encoding: gzip\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 20\r\nX-Varnish-IP: 10.200.74.11\r\nDate: Wed, 18 Nov 2009 16:44:02 GMT\r\nX-Varnish: 62432869 62419064\r\nAge: 17\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\n\r\n....................
206 10 0x0000000000004000 1258562467.900050 0.144417 0.150908 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 02 private_reserved 1379 63.245.221.11 ff mozilla corporation 80 6 Dell Apple 64 0
...Look at the duration column, you see several packets coming on a 0.3s-1s basis until packet number 200. There the duration interval jumps by 18252.71sec.
Hence, there was no packet in between. Since we are in pcap synchronization mode: MONINTTMPCP=1, the monitoring is synchronized to the packet dissection,
so you will not see samples every 10s, which you could expect at an interface in real-time MONINTTMPCP=0.
If you need to sample the monitoring output for later decent signal processing, a small awk script will do the trick; have a look at the encrypted traffic mining
tutorial. With a pcap you have to live with these gaps in the output. No packets, no monitoring in pcap time. If you have a pcap with
regular packet flow, all is fine. So the reader may try skypeu.pcap and see a nicer spacing. We will need
that pcap in the next chapter anyway, so download it. Switching to real time wont help only for large pcaps, which run longer than
MONINTV. Try MONINTV=1 with annoloc2.pcap.
Adding/removing protocols
Let’s look into controlling the output of the monitoring/reporting mode. As indicated above the L3/L4 protocols being displayed are dependent on the plugins being loaded and the constants in main.h:
MONPROTL3MONPROTL2
So either you use the constants defined in the file defined by MONPROTFL or just use the plain number
of the protocol. So remove all IPv6 protocols, IGMP, GRE, SCTP and every L2 protocol as displayed below,
comment out the default configuration:
...
// Monitoring mode protocol stat
#define MONPROTMD 1 // Monitoring: 0: Protocol numbers; 1: Protocol names (L3 only)
//#define MONPROTL3 L3_ICMP,L3_IGMP,L3_TCP,L3_UDP,L3_GRE,L3_ICMP6,L3_SCTP
//#define MONPROTL2 0x0042,0x00fe,ETHERTYPE_ARP,ETHERTYPE_RARP,ETHERTYPE_IP,ETHERTYPE_IPV6
#define MONPROTL3 L3_ICMP,L3_TCP,L3_UDP
#define MONPROTL2
#define MONPROTFL "proto.txt"
...The constant MONPROTFL denotes the list where protocol names and numbers are correlated.
Don’t change that, unless you have a better one. If you do, please tell us so that we
can supply it to the public.
Recompile T2, restart T2:
t2build tranalyzer2
t2 -r ~/data/skypeu.pcap -l
%repTyp time dur memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn ICMP TCP UDP ftpPkts
USR1MR_D 1146661318.015009 9.272231 413004 2 2 1 1 176 90 86 176 0 0 11802 6034 5768 0 0 0 14.034 0x0000000200004000 0 176 0 0
USR1MR_D 1146661328.015302 10.000293 0 0 0 0 0 408 204 204 408 0 0 27338 13670 13668 0 0 0 25.180 0x0000000200004000 0 408 0 0
USR1MR_D 1146661338.031531 10.016229 0 0 0 0 0 391 200 191 391 0 0 26206 13400 12806 0 0 0 24.145 0x0000000200004000 0 391 0 0
USR1MR_D 1146661348.089037 10.057506 0 0 0 0 0 396 199 197 396 0 0 26538 13334 13204 0 0 0 24.326 0x0000000200004000 0 396 0 0
USR1MR_D 1146661358.000057 9.911020 0 0 0 0 0 395 198 197 395 0 0 26468 13268 13200 0 0 0 24.596 0x0000000200004000 0 395 0 0
USR1MR_D 1146661368.000377 10.000320 0 0 0 0 0 396 197 199 396 0 0 26536 13202 13334 0 0 0 24.447 0x0000000200004000 0 396 0 0
USR1MR_D 1146661378.043820 10.043443 0 0 0 0 0 402 200 202 402 0 0 26934 13400 13534 0 0 0 24.675 0x0000000200004000 0 402 0 0
USR1MR_D 1146661388.020862 9.977042 0 0 0 0 0 391 196 195 391 0 0 26204 13136 13068 0 0 0 24.225 0x0000000200004000 0 391 0 0
USR1MR_D 1146661398.029126 10.008264 0 0 0 0 0 397 200 197 397 0 0 26602 13400 13202 0 0 0 24.484 0x0000000200004000 0 397 0 0
USR1MR_D 1146661408.001388 9.972262 0 0 0 0 0 395 200 195 395 0 0 26470 13400 13070 0 0 0 24.463 0x0000000200004000 0 395 0 0
USR1MR_D 1146661418.013728 10.012340 0 0 0 0 0 392 195 197 392 0 0 26272 13070 13202 0 0 0 24.207 0x0000000200004000 0 392 0 0
USR1MR_D 1146661428.581968 10.568240 0 0 0 0 0 194 96 98 194 0 0 13002 6436 6566 0 0 0 11.315 0x0000000200004000 0 194 0 0
USR1MR_D 1146690149.215032 28720.633064 0 -1 1 1 0 8 4 4 8 0 0 536 268 268 0 0 0 0.000 0x0000000200004000 0 8 0 0
USR1MR_D 1146690159.054412 9.839380 0 1 1 0 1 197 99 98 197 0 0 13208 6636 6572 0 0 0 14.542 0x0000000200004000 0 197 0 0
USR1MR_D 1146690169.051267 9.996855 0 0 0 0 0 408 204 204 408 0 0 27338 13670 13668 0 0 0 25.189 0x0000000200004000 0 408 0 0
USR1MR_D 1146690179.016154 9.964887 0 0 0 0 0 390 200 190 390 0 0 26140 13400 12740 0 0 0 24.216 0x0000000200004000 0 390 0 0
USR1MR_D 1146690189.012973 9.996819 0 0 0 0 0 395 199 196 395 0 0 26470 13334 13136 0 0 0 24.403 0x0000000200004000 0 395 0 0
USR1MR_D 1146690199.000000 9.987027 0 0 0 0 0 395 198 197 395 0 0 26468 13268 13200 0 0 0 24.409 0x0000000200004000 0 395 0 0
USR1MR_D 1146690209.022780 10.022780 0 0 0 0 0 398 197 201 398 0 0 26670 13202 13468 0 0 0 24.515 0x0000000200004000 0 398 0 0
USR1MR_D 1146690219.039590 10.016810 0 0 0 0 0 399 200 199 399 0 0 26734 13400 13334 0 0 0 24.571 0x0000000200004000 0 399 0 0
USR1MR_D 1146690229.040467 10.000877 0 0 0 0 0 394 196 198 394 0 0 26404 13136 13268 0 0 0 24.340 0x0000000200004000 0 394 0 0
USR1MR_D 1146690239.049341 10.008874 0 0 0 0 0 397 200 197 397 0 0 26602 13400 13202 0 0 0 24.479 0x0000000200004000 0 397 0 0
USR1MR_D 1146690249.030201 9.980860 0 0 0 0 0 395 200 195 395 0 0 26470 13400 13070 0 0 0 24.442 0x0000000200004000 0 395 0 0
USR1MR_D 1146690259.015097 9.984896 0 0 0 0 0 390 193 197 390 0 0 26140 12938 13202 0 0 0 24.168 0x0000000200004000 0 390 0 0
...The removed protocols are now excluded, as configured. The jump in the middle is caused by the pause between two consecutive Skype conversations in the pcap. Now reset the protocol config to default, by removing your constants and un-commenting the default ones:
...
// Monitoring mode protocol stat
#define MONPROTMD 1 // Monitoring: 0: Protocol numbers; 1: Protocol names (L3 only)
#define MONPROTL3 L3_ICMP,L3_IGMP,L3_TCP,L3_UDP,L3_GRE,L3_ICMP6,L3_SCTP
#define MONPROTL2 0x0042,0x00fe,ETHERTYPE_ARP,ETHERTYPE_RARP,ETHERTYPE_IP,ETHERTYPE_IPV6
#define MONPROTFL "proto.txt"
...Recompile T2 and test. We will now look a bit deeper into time bases and interface operations.
Time base on an interface and stuff
Again, on an interface it normally does not matter whether a pcap or real time base is chosen, unless the average packet rate is very
erratic and sometimes is 0. Then it is beneficial to use the default real time base: MONINTTMPCP=0.
Hmmm, so a pcap which processes 1 minute would then indeed generate monitoring output, but only on your time base uncorrelated to the pcap time.
The same will happen on an interface if there are traffic gaps. The solution is to switch to real time, MONINTTMPCP=0, and interrupt based printing: MONINTPSYNC=0,
so printing is not synchronized to the core main loop containing the blocking pcap functions. In most cases MONINTPSYNC=1 is ok. Moreover, we like T2 to start immediately after
invocation with the monitoring processes, so set MONINTTMPCP_ON=1 or use the appropriate t2conf command.
So your config should look like this:
...
// Monitoring mode
...
#define MONINTPSYNC 0 // Monitoring: 1: pcap main loop synchronized printing. 0: Interrupt printing.
#define MONINTTMPCP 0 // Monitoring: 1: pcap time base, 0: real time base.
#define MONINTTMPCP_ON 1 // Monitoring: Startup monitoring 1: on 0: off; if (MONINTTMPCP == 0)
#define MONINTV 10 // Monitoring: GI_ALRM: MONINTV >= 1 sec interval of monitoring output.
...Recompile tranalyzer: t2build tranalyzer2, invoke T2 on an interface, bind him just for fun to CPU 1 -c1 option, so he won’t jump to others directed
by the operating system.
Now T2 starts right away with monitoring output and you will see almost an equal spacing of 10sec. And lets bind t2 to CPU 1 to avoid CPU hopping.
t2conf tranalyzer2 -D MONINTTMPCP_ON=1 -D MONINTPSYNC=0 && t2build tranalyzer2
st2 -i yourinterface -l -c1
%repTyp time dur pktsRec pktsDrp ifDrp memUsageKB fillSzHashMap numFlows numAFlows numBFlows numPkts numAPkts numBPkts numV4Pkts numV6Pkts numVxPkts numBytes numABytes numBBytes numFrgV4Pkts numFrgV6Pkts numAlarms rawBandwidth globalWarn 0x0042Pkts 0x0042Bytes 0x00fePkts 0x00feBytes 0x0806Pkts 0x0806Bytes 0x8035Pkts 0x8035Bytes 0x0800Pkts 0x0800Bytes 0x86ddPkts 0x86ddBytes ICMPPkts ICMPBytes IGMPPkts IGMPBytes TCPPkts TCPBytes UDPPkts UDPBytes GREPkts GREBytes ICMPv6Pkts ICMPv6Bytes SCTPPkts SCTPBytes ftpPkts ftpDataPkts
USR1MR_D 1613671353.151080 9.311684 455 0 0 20192 11 11 10 1 455 178 277 437 13 0 433614 14998 418616 0 0 0 372.533 0x0c0000000000c004 0 0 0 00 0 0 0 437 432040 13 1274 0 0 0 0 435 431900 0 0 0 0 11 1094 0 0 0 0
USR1MR_D 1613671363.151299 10.000219 816 0 0 0 0 0 0 0 358 142 216 344 9 0 336833 12382 324451 0 0 0 269.460 0x0c0010000000c004 0 0 0 00 0 0 0 344 335631 9 902 0 0 0 0 342 335491 0 0 0 0 7 722 0 0 0 0
USR1MR_D 1613671373.151488 10.000189 1240 0 0 0 6 6 3 3 427 186 241 415 7 0 347149 17533 329616 0 0 0 277.714 0x0c0010000000c004 0 0 0 00 0 0 0 415 346207 7 642 0 0 0 0 411 345832 2 235 0 0 5 462 0 0 0 0
USR1MR_D 1613671383.151655 10.000167 1430 0 0 1232 2 2 1 1 188 94 94 161 22 0 143961 10308 133653 0 0 0 115.167 0x0c0010000000c004 0 0 0 00 0 0 0 161 141469 22 2192 0 0 0 0 159 141329 0 0 0 0 20 2012 0 0 0 0
USR1MR_D 1613671393.151837 10.000182 1452 0 0 0 1 1 1 0 22 21 1 8 9 0 2306 2240 66 0 0 0 1.845 0x0c0010000200c004 0 0 0 00 0 0 0 8 1108 9 898 0 0 0 0 2 132 4 836 0 0 7 718 0 0 0 0
USR1MR_D 1613671403.152035 10.000198 1490 0 0 0 2 2 2 0 36 32 4 15 13 0 4676 4297 379 0 0 0 3.741 0x0c0010000200c044 0 0 0 03 180 0 0 15 2922 13 1274 0 0 0 0 9 1946 4 836 0 0 11 1094 0 0 0 0
USR1MR_D 1613671413.152182 10.000147 1516 0 0 0 0 0 0 0 30 29 1 4 18 0 2488 2422 66 0 0 0 1.990 0x0c0010000200c044 0 0 0 03 180 0 0 4 272 18 1736 0 0 0 0 2 132 0 0 0 0 16 1556 0 0 0 0
USR1MR_D 1613671423.152374 10.000192 1547 0 0 0 0 0 0 0 30 24 6 12 13 0 3171 2638 533 0 0 0 2.537 0x0c0010000200c044 0 0 0 00 0 0 0 12 1597 13 1274 0 0 0 0 10 1457 0 0 0 0 11 1094 0 0 0 0
USR1MR_D 1613671433.152542 10.000168 1572 0 0 0 0 0 0 0 26 23 3 8 13 0 2188 1951 237 0 0 0 1.750 0x0c0010000200c044 0 0 0 00 0 0 0 8 614 13 1274 0 0 0 0 6 474 0 0 0 0 11 1094 0 0 0 0
USR1MR_D 1613671443.152728 10.000186 1595 0 0 0 0 0 0 0 23 22 1 4 14 0 1936 1870 66 0 0 0 1.549 0x0c0010000200c044 0 0 0 00 0 0 0 4 272 14 1364 0 0 0 0 2 132 0 0 0 0 12 1184 0 0 0 0
USR1MR_D 1613671453.152897 10.000169 1616 0 0 0 0 0 0 0 20 19 1 4 11 0 1670 1604 66 0 0 0 1.336 0x0c0010000200c044 0 0 0 00 0 0 0 4 272 11 1098 0 0 0 0 2 132 0 0 0 0 9 918 0 0 0 0
...You may now start another Anteater and bind him to CPU 2, and maybe with other plugins, which you have to copy into e.g. ~/.tranalyzer/pluginsOther.
st2 -i yourinterface -l -p /home/wurst/.tranalyzer/pluginsOther/ -c2
Now control the two independently by the kill command. If you use t2stat -s -USR1, it will send the same signal to both
and you can trigger additional reports. If you want to control the start of the reporting via signals as before,
then set MONINTTMPCP_ON=0 and recompile. These exercises are left to the reader.
Let us now concentrate on the practitioner who likes to send logs to centralized machines processing them with his preferred tool. So how can he log into tools residing on a remote machine?
Logging into remote tools
An easy way to log into remote machines is to use netcat. Open a bash window and start a netcat listener on port 5555.
We use the local interface to make it easier for the user, who have no access to another machine. If you want
to send it anyway to another machine, just swap 127.0.0.1 with the IP address of your remote host. And
mind your firewall policies.
nc -l 127.0.0.1 -p 5555
Change the interval to MONINTV=1 so that we receive enough samples to see some reaction in time:
Compile T2 and start t2 on an interface and pipe it into a netcat sender to port 5555:
t2conf tranalyzer2 -D MONINTV=1 && t2build tranalyzer2
st2 -i yourinterface -l | nc 127.0.0.1 5555
In the listener bash window, lines should now continually appear. These can be stored into a file or piped into a logging tool such as RRD Tool. Let’s try that.
Logging into RRD Tool
Download RRD Tool and install it.
To convert T2 output into the RRD DB language the t2rrd script is needed:
Usage:
t2rrd [OPTION...] [FIELD1] [FIELD2]
Monitoring arguments:
-m create and populate a database [default: monitoring.rrd]
-k append data to existing database [default: create a new DB]
Plotting arguments:
-p name name of the PNG to generate [default: field1[_field2].png]
-r WxH size of the generated PNG (width x height) [default: 640x480]
-s start start time of the plot [default: end-10m]
(use special value 'first' to plot from the first record)
-e end end time of the plot [default: last]
(use special value 'last' to plot until the last record)
-i int interval to plot [default: 10m (last 10min)]
([s]econds, [m]inutes, [h]ours, [d]ays, [w]eeks, [mo]nth, [y]ear)
-a plot the whole database
-U int update the plot every 'int' seconds [default: 1]
-n generate a fixed plot for the given interval and exit
-R real-time: plot regular intervals (see -U option)
[default: plot to the last update]
-f display Holt-Winters confidence bands
-t win display the 'win' hour trend
-x use logarithmic axis
-u max crop values to 'max'
-l min crop values to 'min'
-b do not display the anteater banner
Optional arguments:
-d name name of the database [default: monitoring.rrd]
Help and documentation arguments:
-L List the fields available in the database and exit
-I display info about the database and exit
-h, --help show this help, then exit
First, we need to pipe T2 output into t2rrd -m to create and populate a RRD database.
Let’s do it for an interface, so we can see constant updates:
st2 -i yourInterface -l | t2rrd -m
Nice, now the database is getting populated! Let’s have a look at what we can plot in another bash window:
t2rrd -L
memUsageKB fillSzHashMap numFlows
numAFlows numBFlows numPkts
numAPkts numBPkts numV4Pkts
numV6Pkts numVxPkts numBytes
numABytes numBBytes numFrgV4Pkts
numFrgV6Pkts numAlarms rawBandwidth
0x0042Pkts 0x0042Bytes 0x00fePkts
0x00feBytes 0x0806Pkts 0x0806Bytes
0x8035Pkts 0x8035Bytes 0x0800Pkts
0x0800Bytes 0x86ddPkts 0x86ddBytes
ICMPPkts ICMPBytes IGMPPkts
IGMPBytes TCPPkts TCPBytes
UDPPkts UDPBytes GREPkts
GREBytes ICMPv6Pkts ICMPv6Bytes
SCTPPkts SCTPBytes ftpPkts
ftpDataPkts
t2rrd -i 1m numAFlows numBFlows
You can watch now while you are working, add more t2rrd processes to display other columns.
Now try it on a pcap annoloc2.pcap!
But, first, make sure to stop your t2rrd -m command, so you do not end up mixing statistics from various sources.
t2 -r ~/data/annoloc2.pcap -l | t2rrd -m
t2rrd -n -e last -i 30s numAFlows numBFlows
If you do not like the nice Anteater, then use the -b option. The default is number of flows.
If we want to see numAPkts and numBPkts:
t2rrd -n -e last -i 30s numAPkts numBPkts
Conclusion
Don’t forget to reset all constants for the next tutorials:
t2conf tranalyzer2 -D MONINTTMPCP_ON=0 -D MONINTTMPCP=0 -D MONINTV=1 -D DIFF_REPORT=0 -D MACHINE_REPORT=0 -D MONINTPSYNC=1
t2build -R
Or use the new command:
t2conf --reset -a && t2build -R
Have fun experimenting.