Tutorial: Basic Analysis
Contents
WTF is the Anteater?
Tranalyzer2 (T2) was created at a Swiss operator out of the need that standard Cisco NetFlow did not supply the fields we needed for our troubleshooting and security work. We needed all kinds of encapsulated protocols, content info, advanced statistics and an easy way to extract information for traffic mining. And most important a tool which can digest really large pcaps and runs stable on an interface. Therefore, only code and functionality which is needed by the user is added. That should explain, why a lot of T2 is controlled by compiler switches, making it adaptable and lightweight. But no worries, we made compiling on different infrastructure easy for you.
Having also students with us, we saw they always reinventing the wheel when it came to traffic analysis, so in 2008 T2 became open source. Since then practical ideas from people working in the field and in research inspired the path of the Anteater.
This tutorial will teach you about the basic configuration, usage, basic plugins and post-processing philosophy. So, let’s first look at the basic protocol and output modes.
T2 operational modes
By default T2 operates in the following basic protocol modes:
- Layer2
- IPv4
- IPv6
- SCTP
By default since the 0.8.0 version T2 operates concurrently in all protocol modes and feeds output into the same files. If you are only interested in IPv4 and decapsulation of protocols such as L2TP, GRE, IPvx-in-IPvy, etc is not relevant, T2 can easily be configured to do only this. Moreover L4 protocols support is supplied e.g. SCTP which transforms all streams into extra flows, if enabled in networkHeaders.h. We will discuss at the end of the tutorial and in the T2 Kungfu (coming soon).
T2 is capable to produce the following concurrent jobs.
- Flow
- Packet
- Monitoring
- Alarm
- Force
Lets have a quick look at these.
Flow
The most prominent one is flow, where traffic is aggregated into so called flows to process large amount of traffic. A flow is defined in T2 as A and opposite B Flow which are linked by a unique flowIndex, a 64 bit number. The default aggregation of T2 flows is
(vlan, srcIP, srcPort, dstIP, dstPort, L3protocol)which covers most cases in corporate networks, as VLANs are very common. It can be extended to
(srcEther, dstEther, ethertype, vlan, srcIP, srcPort, dstIP, dstPort, sctpChannel, L3protocol)or reduced to aggregating all traffic into a few flows, defining only several networks without VLANs, ports and protocols. The advanced flow aggregation modes will be discussed when introducing T2 core operations in tutorial t2 core kungfu (coming soon)
Each plugin added to T2 will produce additional columns in the flow file, producing an output easy to process for any script language or standard tools, such as Excel or SPSS. All this is discussed under
Packet
The packet mode’s output format is as scripting friendly as the flow output and thought as a drill down instrument, which links back to flows and L7 content via the flowIndex. We will discuss it in detail in the tutorial packet mode.
Monitoring
Network managers often need certain time sampled parameters, such as number of packets or bandwidth. T2 reports into standard tools. All aspects will be discussed in the tutorial monitoring mode.
Alarm
Sometimes L3-4 or content driven rules or even a custom build AI classifier defines what is interesting for the user. Hence, the alarm mode enables each plugin to control flow processing and release to output. This mode is discussed in detail under section alarm mode
Force
When operating on an interface sometimes the timeout of a flow is too long for appropriate reaction, e.g. when Malware is detected. So notification when a certain packet is seen is required. The force mode enables any plugin to control flow termination at any point in time. All following packets after flow release will be send to a new flow. This mode is discussed in detail under section force mode
How to Anteater
To get started download Tranalyzer and unpack the tar ball: (BTW: lm means Linux and Mac tested) and for the uninitiated bash user, the $ in front of each command denotes the bash command line prompt. Do NOT copy it into your command shell!
$ tar -xf tranalyzer2-0.8.4lm2.tar.gz
$ cd ~/tranalyzer2-0.8.4
$ ls
autogen.sh ChangeLog doc plugins README.md scripts setup.sh tests tranalyzer2 utils
$You see the doc folder, the README (compilation, dependencies for different OS), the setup.sh script. Since we moved to git the original trunk folder was separated into the core, tranalyzer2 and the plugins folder. So if you move to the plugins folder all the plugins are there, and the scripts, such as autogen.sh work as before. So don’t worry.
$ cd plugins
$ ls
arpDecode binSink dhcpDecode fnameLabel icmpDecode lldpDecode mysqlSink nFrstPkts pcapd protoStats regex_pcre snmpDecode sslDecode tcpFlags tp0f wavelet
autogen.sh cdpDecode dnsDecode ftpDecode igmpDecode macRecorder natNudel ntpDecode pktSIATHisto psqlSink sctpDecode socketSink stpDecode tcpStates txtSink
basicFlow connStat entropy geoip ircDecode modbus nDPI ospfDecode popDecode pwX smbDecode sqliteSink syslogDecode telnetDecode voipDetector
basicStats descriptiveStats findexer httpSniffer jsonSink mongoSink netflowSink p0f portClassifier radiusDecode smtpDecode sshDecode t2PSkel tftpDecode vrrpDecode
$If you are a rookie to T2, use the setup.sh script under the tranalyzer root directory, it will install all tools, links and environment variables for you and compiles T2 with the standard basic plugins.
$ ./setup.sh
...If the setup finished successfully you are all set. A other good and old fashion way without ./setup.sh is to invoke ./autogen.sh (note however that this method will NOT install t2_aliases)
$ ./autogen.sh
...If you want to use the t2 aliases (t2build, t2conf, …) in your current bash window, you have to run the following command:
$ source scripts/t2_aliases
$If a new bash window is opened all environmental variables will be automatically set. Now try to use the autocompletion: t2 tab-tab
t2 t2build t2caplist t2conf t2dmon t2doc t2edit t2fm t2plot t2PSkel t2stat t2timeline t2wizardt2 always points to the newest tranalyzer compiled under ~/tranalyzer2-0.8.4/tranalyzer2/src, so you do not need to move to this directory and type ./tranalyzer.
If compilation fails, it will tell you what is missing, then refer to the README or copy the appropriate dependencies from here. If nothing works, look in the FAQ. if that does not solve your problem write to the Anteater. He will definitely help you.
If setup is successful then you may start t2 with the help option for a quick test:
$ t2 -h
Tranalyzer 0.8.4 (Anteater), Tarantula - High performance flow based network traffic analyzer
Usage:
tranalyzer [OPTION...] <INPUT$
Input:
-i IFACE Listen on interface IFACE
-r PCAP Read packets from PCAP file or from stdin if PCAP is "-"
-R FILE Process every PCAP file listed in FILE
-D EXPR[:SCHR][,STOP]
Process every PCAP file whose name matches EXPR, up to an
optional last index STOP. If STOP is omitted, then Tranalyzer
never stops. EXPR can be a filename, e.g., file.pcap0, or an
expression, such as "dump*.pcap00", where the star matches
anything (note the quotes to prevent the shell from
interpreting the expression). SCHR can be used to specify the
the last character before the index (default: 'p')
Output:
-w PREFIX Append PREFIX to any output file produced. If omitted, then
output is diverted to stdout
-W PREFIX[:SIZE][,START]
Like -w, but fragment flow files according to SIZE, producing
files starting with index START. SIZE can be specified in bytes
(default), KB ('K'), MB ('M') or GB ('G'). Scientific notation,
i.e., 1e5 or 1E5 (=100000), can be used as well. If a 'f' is
appended, e.g., 10Kf, then SIZE denotes the number of flows.
-l Print end report in PREFIX_log.txt instead of stdout
-s Packet forensics mode
Optional arguments:
-p PATH Load plugins from path PATH instead of ~/.tranalyzer/plugins
-b FILE Use plugin list FILE instead of plugin_folder/plugins.txt
-e FILE Creates a PCAP file by extracting all packets belonging to
flow indexes listed in FILE
-f FACTOR Sets hash multiplication factor
-x ID Sensor ID
-c CPU Bind tranalyzer to one core. If CPU is 0 then OS selects the
core to bind
-F FILE Read BPF filter from FILE
-v Show the version of the program and exit
-h Show help options and exit
Remaining arguments:
BPF Berkeley Packet Filter command, as in tcpdumpIf you cannot wait and would like to try it now on an interface, go ahead and use the -i option. Here we will read from pcaps, so the -r, -R or -D options are relevant. While the latter two are only being used if more than one pcap is to be analysed, for this tutorial -r is the option of choice. The -w option defines where the flow files will be written to. If you omit the -w option, T2 writes to the folder of the pcap. The rest is currently not important.
A good practice for analysis and mining jobs is to create a separate data and results directory as follows:
$ mkdir ~/data
$ mkdir ~/results
$ cd dataDownload the pcap annoloc2.pcap into your data folder
$ cd ~/data
$ wget https://tranalyzer.com/download/data/annoloc2.pcapNow feed the pcap to the Anteater as follows:
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.4 (Anteater), Tarantula. PID: 20074
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: protoStats, 0.8.4
02: basicFlow, 0.8.4
03: macRecorder, 0.8.4
04: portClassifier, 0.8.4
05: basicStats, 0.8.4
06: tcpFlags, 0.8.4
07: tcpStates, 0.8.4
08: icmpDecode, 0.8.4
09: connStat, 0.8.4
10: txtSink, 0.8.4
[INF] basicFlow: IPv4 Ver: 3, Rev: 21062019, Range Mode: 0, subnet ranges loaded: 269266 (269.27 K)
[INF] basicFlow: IPv6 Ver: 3, Rev: 21062019, Range Mode: 0, subnet ranges loaded: 10602 (10.60 K)
Processing file: /home/stefan/tranalyzer-website/tranalyzer/download/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 0.742266 sec
Finished unloading flow memory. Time: 1.111429 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218588 (1.22 M) [99.96%]
Number of IPv6 packets: 180 [0.01%]
Number of A packets: 561592 (561.59 K) [46.07%]
Number of B packets: 657423 (657.42 K) [53.93%]
Number of A bytes: 29274120 (29.27 M) [45.68%]
Number of B bytes: 34808606 (34.81 M) [54.32%]
Average A packet load: 52.13
Average B packet load: 52.95
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
tcpFlags: Aggregated ipFlags: 0x3966
tcpFlags: Aggregated tcpAnomaly: 0xff47
tcpFlags: Number of TCP scans, succ scans, retries: 238, 671, 933
tcpFlags: Number WinSz below 1: 2415 (2.42 K) [0.25%]
tcpStates: Aggregated anomaly flags: 0xdf
icmpDecode: Number of ICMP echo request packets: 224 [7.32%]
icmpDecode: Number of ICMP echo reply packets: 191 [6.24%]
icmpDecode: ICMP echo reply / request ratio: 0.85
connStat: Number of unique source IPs: 4309 (4.31 K)
connStat: Number of unique destination IPs: 2919 (2.92 K)
connStat: Number of unique source/destination IPs connections: 182
connStat: Max unique number of source IP / destination port connections: 413
connStat: IP prtcon/sdcon, prtcon/scon: 2.269231, 0.095846
connStat: Source IP with max connections: 138.212.189.66 (JP): 369 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 403 connections
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 20 [0.00%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of TCP bytes: 52643546 (52.64 M) [82.15%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of UDP bytes: 11234272 (11.23 M) [17.53%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 17605 (17.61 K)
Number of processed A flows: 9997 (10.00 K) [56.79%]
Number of processed B flows: 7608 (7.61 K) [43.21%]
Number of request flows: 9469 (9.47 K) [53.79%]
Number of reply flows: 8136 (8.14 K) [46.21%]
Total A/B flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.08
Number of processed packets/flows: 69.24
Number of processed A packets/flows: 56.18
Number of processed B packets/flows: 86.41
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 22509.40 (22.51 K)
Number of processed B packets/s: 26350.44 (26.35 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 705.63
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270445248 b/s (270.45 Mb/s)
Max number of flows in memory: 15222 (15.22 K) [5.81%]
Memory usage: 0.17 GB [0.25%]
Aggregate flow status: 0x000018fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] IPv4
[INF] IPv6
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP flows
[INF] Ethernet flows
[INF] ARP flowsT2 produces an end report, which serves as an initial assessment of the pcap content and anomalies. After basic packet, byte statistics each plugin adds some statistical or hex coded info between the ------ lines which will be discussed later. Moreover flow based statistics are reported to assess the traffic seen on the wire. At the end certain protocol based info and warnings about traffic content are reported to alert the user. Thus, an initial assessment is possible without even looking into flows or packets which is essential when dealing with large quantities of traffic.
All plugins reside in the plugins folder and own a src (.h, .c), a doc (.tex, .pdf) and a test (autotesting) directory. Important for now is the doc folder, where you will find a pdf describing the plugin (the complete documentation of Tranalyzer2, all the plugins and scripts can be found under doc/documentation.pdf). The rest will be discussed later.
To give you a basic introduction to the traffic mining art using Tranalyzer is the primary goal of this tutorial, so lets start with the very basics; have fun!
Basic Flow based Plugins
For beginners let’s start with the very basic flow plugins and only use flow based text output, aka the extended NetFlow7 flow output:
- tranalyzer2: Anteater’s core
- basicFlow: Flow output definition + geo labeling + encapsulation info
- basicStats: Basic statistics including Traffic Mining extensions
- txtSink: Produces a tab separated text file:
_flows.txt
so to unload unnecessary compiled plugins invoke:
$ t2build -u protoStats macRecorder portClassifier tcpFlags tcpStates icmpDecode connStat
Plugin 'protoStats'
Plugin 'macRecorder'
Plugin 'portClassifier'
Plugin 'tcpFlags'
Plugin 'tcpStates'
Plugin 'icmpDecode'
Plugin 'connStat'
BUILD SUCCESSFUL
$Now restart the Anteater and have a look at what changed in the end report:
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.4 (Anteater), Tarantula. PID: 23478
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: basicFlow, 0.8.4
02: basicStats, 0.8.4
03: txtSink, 0.8.4
[INF] basicFlow: IPv4 Ver: 3, Rev: 21062019, Range Mode: 0, subnet ranges loaded: 269266 (269.27 K)
[INF] basicFlow: IPv6 Ver: 3, Rev: 21062019, Range Mode: 0, subnet ranges loaded: 10602 (10.60 K)
Processing file: /home/stefan/tranalyzer-website/tranalyzer/download/data/annoloc2.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1022171701.691172 sec (Thu 23 May 2002 16:35:01 GMT)
[WRN] snapL2Length: 54 - snapL3Length: 40 - IP length in header: 1500
Dump stop : 1022171726.640398 sec (Thu 23 May 2002 16:35:26 GMT)
Total dump duration: 24.949226 sec
Finished processing. Elapsed time: 0.439250 sec
Finished unloading flow memory. Time: 0.636931 sec
Percentage completed: 100.00%
Number of processed packets: 1219015 (1.22 M)
Number of processed bytes: 64082726 (64.08 M)
Number of raw bytes: 844642686 (844.64 M)
Number of pcap bytes: 83586990 (83.59 M)
Number of IPv4 packets: 1218588 (1.22 M) [99.96%]
Number of IPv6 packets: 180 [0.01%]
Number of A packets: 564228 (564.23 K) [46.29%]
Number of B packets: 654787 (654.79 K) [53.71%]
Number of A bytes: 29447896 (29.45 M) [45.95%]
Number of B bytes: 34634830 (34.63 M) [54.05%]
Average A packet load: 52.19
Average B packet load: 52.89
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
--------------------------------------------------------------------------------
Headers count: min: 2, max: 4, average: 3.01
Number of GRE packets: 20 [0.00%]
Number of IGMP packets: 12 [0.00%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
Number of TCP packets: 948743 (948.74 K) [77.83%]
Number of TCP bytes: 52643546 (52.64 M) [82.15%]
Number of UDP packets: 266900 (266.90 K) [21.89%]
Number of UDP bytes: 11234272 (11.23 M) [17.53%]
Number of IPv4 fragmented packets: 2284 (2.28 K) [0.19%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed flows: 17102 (17.10 K)
Number of processed A flows: 9721 (9.72 K) [56.84%]
Number of processed B flows: 7381 (7.38 K) [43.16%]
Number of request flows: 9678 (9.68 K) [56.59%]
Number of reply flows: 7424 (7.42 K) [43.41%]
Total A/B flow asymmetry: 0.14
Total req/rply flow asymmetry: 0.13
Number of processed packets/flows: 71.28
Number of processed A packets/flows: 58.04
Number of processed B packets/flows: 88.71
Number of processed total packets/s: 48859.83 (48.86 K)
Number of processed A+B packets/s: 48859.83 (48.86 K)
Number of processed A packets/s: 22615.05 (22.61 K)
Number of processed B packets/s: 26244.78 (26.24 K)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 685.47
Average full raw bandwidth: 270835712 b/s (270.84 Mb/s)
Average snapped bandwidth : 20548206 b/s (20.55 Mb/s)
Average full bandwidth : 270445248 b/s (270.45 Mb/s)
Max number of flows in memory: 17102 (17.10 K) [6.52%]
Memory usage: 0.07 GB [0.10%]
Aggregate flow status: 0x000018fa0202d044
[WRN] L3 SnapLength < Length in IP header
[WRN] L4 header snapped
[WRN] Consecutive duplicate IP ID
[WRN] IPv4/6 fragmentation header packet missing
[WRN] IPv4/6 packet fragmentation sequence not finished
[INF] IPv4
[INF] IPv6
[INF] IPv4/6 fragmentation
[INF] IPv4/6 in IPv4/6
[INF] GRE encapsulation
[INF] SSDP/UPnP flows
[INF] Ethernet flows
[INF] ARP flowsT2 produces an end report, which serves as an initial assessment of the pcap content and anomalies. Each plugin adds some info between the ------ lines. basicStats shows the biggest talker regarding traffic volume, and country of origin. It is one of the first features relevant to understand large traffic pcaps. There are also biggest talkers in regard to number of connections. This will be discussed in chapter Basic Traffic volume and connection analysis.
So it is an old pcap from 2002 in the afternoon, IPv4/6 + Ethernet traffic and the payload is snapped. At the bottom, you see warnings ([WRN]) and information ([INF]). It is decoded from the Aggregated flow status, which denotes the OR info from all flows status registers.
There are packets snapped even down to the L2 header, fragments without header or end. The difference between the snapped bandwidth and the full raw bandwidth denotes that either the snaplength was small, maybe the default, or somebody actually mangled with the packet content. The average packet load is symmetric for A and B flow, very odd. The protocols used indicate that the traffic is either corporate or the wild. So if you want good traffic with content for your job, I wouldn’t trust that pcap. and send it right back to the customer. Sure, you can extract way more info, which we will do in the tutorial T2 Kungfu, coming soon.
T2 produces also the following files
$ ls
annoloc2_flows.txt annoloc2_headers.txtThe header file contains information about the columns of the flow file, such as time, column positions loaded, T2 config, the name of the pcap file, vital interface information, etc. This information makes it easier to reproduce results from different experiments and it is good doc.
# Date: 1541443480.086865 sec (Mon 05 Nov 2018 19:44:40 CET)
# Tranalyzer 0.8.4 (Anteater), Tarantula.
# sensorID: 666
# PID: 17031
# Command line: ./tranalyzer -r /home/yourname/data/annoloc2.pcap -w /home/yourname/result
# HW Info: eierfeile;Linux;4.18.16-arch1-1-ARCH;#1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018;x86_64
#
# Plugins loaded:
# 00: basicFlow, version 0.8.4
# 01: basicStats, version 0.8.4
# 03: txtSink, version 0.8.4
#
# Col No. Type Name Description
1 C dir Flow direction
2 U64 flowInd Flow index
3 H64 flowStat Flow status and warnings
4 U64.U32 timeFirst Date time of first packet
5 U64.U32 timeLast Date time of last packet
6 U64.U32 duration Flow duration
7 U8 numHdrDesc Number of different headers descriptions
8 U8:R numHdrs Number of headers (depth) in hdrDesc
9 SC:R hdrDesc Headers description
10 MAC:R srcMac Mac source
11 MAC:R dstMac Mac destination
12 H16 ethType Ethernet type
13 U16:R ethVlanID VLAN IDs
14 IPX srcIP Source IP address
15 SC srcIPCC Source IP country code
16 S srcIPWho Source IP who
17 U16 srcPort Source port
18 IPX dstIP Destination IP address
...Now compare it with the flow file, the columns flowInd to l4Proto originate from basicFlow. After that and until bytAsym, the columns are produced by the basicStats plugin. I picked some interesting flows which demonstrate T2 ops when traffic is mangled with, so all flows which are damaged due to a limited snapLength in the acquisition process. Let’s sort them out: .
$ tawk -V flowStat=0x0000080f00000000
The flowStat column with value 0x0000080f00000000 is to be interpreted as follows:
bit | flowStat | Description
=============================================================================
32 | 0x0000 0001 0000 0000 | Acquired packet length < minimal L2 datagram
33 | 0x0000 0002 0000 0000 | Acquired packet length < packet length in L3 header
34 | 0x0000 0004 0000 0000 | Acquired packet length < minimal L3 Header
35 | 0x0000 0008 0000 0000 | Acquired packet length < minimal L4 Header
43 | 0x0000 0800 0000 0000 | Stop dissectingLet’s say we are only interested to weed out the ones where even the Layer2 header is damaged and T2 gave up further dissecting, then use the following tawk command:
$ tawk 'bitsanyset($flowStat, 0x0000080100000000)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto macPairs srcMac_dstMac_numP srcManuf_dstManuf dstPortClassN dstPortClass numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates icmpStat icmpTCcnt icmpBFTypH_TypL_Code icmpTmGtw icmpEchoSuccRatio icmpPFindex connSip connDip connSipDip connSipDprt connF
A 262 0x0000080200028000 1022171701.707777 1022171701.707777 0.000000 1 4 eth:ipv4:ipv6:UNK(168) 00:d0:02:6d:78:00 00:60:08:2c:ca:8e 0x86dd cfb6:1c18:5010:faf0:7f66:0:101:80a -- "--" 0 6c2:6a7f:1:384b::c100 -- "--" 0 168 1 00:d0:02:6d:78:00_00:60:08:2c:ca:8e_1 Ditech_3com 0 unknown 1 0 41630 0 41630 41630 41630 0 0 0 0 0 0 0 1 1 0x0000 65535 0 192 192 0 0x46 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 888 0x0000080200028000 1022171701.810764 1022171701.810764 0.000000 1 4 eth:ipv4:ipv6:UNK(133) 00:d0:02:6d:78:00 00:60:08:2c:ca:8e 0x86dd e499:578c:5090:81d0:891b:0:101:80a -- "--" 0 514:2343:2e3c:512::c100 -- "--" 0 133 1 00:d0:02:6d:78:00_00:60:08:2c:ca:8e_1 Ditech_3com 0 unknown 1 0 55304 0 55304 55304 55304 0 0 0 0 0 0 0 1 1 0x0000 65535 0 121 121 0 0x08 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 3068 0x0000080200028000 1022171704.484515 1022171704.484515 0.000000 1 4 eth:ipv4:ipv6:UNK(147) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd baea:e860:8090:4470:cf67:0:101:50a -- "--" 0 baea:ee14:baeb:2ac::c100 -- "--" 0 147 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 18922 0 18922 18922 18922 0 0 0 0 0 0 0 1 1 0x0000 65535 0 202 202 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 3481 0x0000080200028000 1022171705.349871 1022171705.349871 0.000000 1 4 eth:ipv4:ipv6:UNK(126) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd d973:ceac:5010:fa94:12db:0:101:80a -- "--" 0 12ea:b284:3:ceeb::c100 -- "--" 0 126 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 38816 0 38816 38816 38816 0 0 0 0 0 0 0 1 1 0x0000 65535 0 154 154 0 0x23 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 4045 0x0000080200005000 1022171706.878547 1022171707.000313 0.121766 2 4;4 eth:ipv4:gre:UNK(0x98c0);eth:ipv4:gre:UNK(0xb30d) 00:d0:02:6d:78:00 00:60:08:2c:ca:8e 0x0800 200.134.37.255 br "Associação Rede Nacio" 0 138.212.185.216 jp "ASAHI KASEI CORPORATION" 0 47 1 00:d0:02:6d:78:00_00:60:08:2c:ca:8e_2 Ditech_3com 0 unknown 2 2 170 210 68 102 85 12.02081 0 0.121766 0.06088299 0.04305077 16.42495 1396.12 0 -0.1052632 0x0100 0 0 54 54 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.03764701 0.03764701 0.0188235 0.01331023 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 2 2 2 2
B 4045 0x0000080200005001 1022171706.931865 1022171706.962666 0.030801 2 4;4 eth:ipv4:gre:UNK(0x9604);eth:ipv4:gre:UNK(0x386d) 00:60:08:2c:ca:8e 00:d0:02:6d:78:00 0x0800 138.212.185.216 jp "ASAHI KASEI CORPORATION" 0 200.134.37.255 br "Associação Rede Nacio" 0 47 1 00:60:08:2c:ca:8e_00:d0:02:6d:78:00_2 3com_Ditech 0 unknown 2 2 210 170 76 134 105 20.5061 0 0.030801 0.0154005 0.0108898 64.93295 6817.96 0 0.1052632 0x0100 0 0 62 62 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.053318 0.084119 0.0687185 0.0108898 0.087542 0.01719738 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 2 1 1 1 0.5
A 4603 0x0000080200028000 1022171708.439927 1022171708.439927 0.000000 1 4 eth:ipv4:ipv6:UNK(95) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 48d5:5fad:50d0:16d0:4a16:0:101:80a -- "--" 0 53eb:c1d7:86:4d37::c100 -- "--" 0 95 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 28079 0 28079 28079 28079 0 0 0 0 0 0 0 1 1 0x0000 65535 0 138 138 0 0xb0 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 4914 0x0000180200005000 1022171709.440443 1022171709.552021 0.111578 4 4;4;4;4 eth:ipv4:gre:UNK(0xa808);eth:ipv4:gre:UNK(0x7d69);eth:ipv4:gre:UNK(0x6f6d);eth:ipv4:gre:UNK(0x0104) 00:d0:02:6d:78:00 00:60:08:2d:05:66 0x0800 19.228.184.27 us "Ford Motor Company" 0 138.212.186.191 jp "ASAHI KASEI CORPORATION" 0 47 1 00:d0:02:6d:78:00_00:60:08:2d:05:66_4 Ditech_3com 0 unknown 4 3 162 166 36 46 40.5 2.534484 0 0.111571 0.0278945 0.0363073 35.84936 1451.899 0.1428571 -0.01219512 0x0100 0 0 241 241 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.109328 0.109335 0.08199875 0.03557979 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 2 2 2
B 4914 0x0000180200005001 1022171709.441758 1022171709.552128 0.110370 3 4;4;4 eth:ipv4:gre:UNK(0x5100);eth:ipv4:gre:UNK(0xc309);eth:ipv4:gre:UNK(0x1fa1) 00:60:08:2d:05:66 00:d0:02:6d:78:00 0x0800 138.212.186.191 jp "ASAHI KASEI CORPORATION" 0 19.228.184.27 us "Ford Motor Company" 0 47 1 00:60:08:2d:05:66_00:d0:02:6d:78:00_3 3com_Ditech 0 unknown 3 4 166 162 48 66 55.33333 5.541092 0 0.109442 0.03679 0.0419465 27.1813 1504.032 -0.1428571 0.01219512 0x0100 0 0 255 255 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.000107 0.002243 0.001221667 0.0006970839 0.08322041 0.03558662 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 6136 0x0000080200028000 1022171713.252296 1022171713.252296 0.000000 1 4 eth:ipv4:ipv6:UNK(28) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 6f:1256:5050:4402:331e:0:101:50a -- "--" 0 223:fd3e:223:ff56::c100 -- "--" 0 28 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 2543 0 2543 2543 2543 0 0 0 0 0 0 0 1 1 0x0000 65535 0 5 5 0 0x08 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 6180 0x0000080200028000 1022171713.395746 1022171713.395746 0.000000 1 4 eth:ipv4:ipv6:UNK(229) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd e29f:74ec:50d0:f53c:98e3:0:101:80a -- "--" 0 12ea:b5a9:3:cf3b::c100 -- "--" 0 229 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 22844 0 22844 22844 22844 0 0 0 0 0 0 0 1 1 0x0000 65535 0 14 14 0 0x78 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 6209 0x0000080200028000 1022171713.450109 1022171713.450109 0.000000 1 4 eth:ipv4:ipv6:UNK(64) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 392:fc19:5050:20e2:a7c3:0:101:80a -- "--" 0 be1:fcff:6ca:1b09::c100 -- "--" 0 64 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 174 0 174 174 174 0 0 0 0 0 0 0 1 1 0x0000 65535 0 15 15 0 0x4b 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 7060 0x0000080200028000 1022171716.432198 1022171716.432198 0.000000 1 4 eth:ipv4:ipv6:UNK(223) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 75a4:9abd:80d0:4470:b5e2:317b:101:50a -- "--" 0 75a4:a071:75a4:be39::c100 -- "--" 0 223 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 9 0 9 9 9 0 0 0 0 0 0 0 1 1 0x0000 65535 0 74 74 0 0xbd 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 7201 0x0000080200028000 1022171716.967592 1022171716.967592 0.000000 1 4 eth:ipv4:ipv6:UNK(228) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 4b9:f24c:8090:4470:4b24:0:101:50a -- "--" 0 4b9:f800:4ba:c98::c100 -- "--" 0 228 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 54359 0 54359 54359 54359 0 0 0 0 0 0 0 1 1 0x0000 65535 0 177 177 0 0x8a 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 7634 0x0000080200028000 1022171718.639159 1022171718.639159 0.000000 1 4 eth:ipv4:ipv6:UNK(23) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd 169e:bdd8:5090:4470:e911:0:101:50a -- "--" 0 5dc9:ed57:5dc9:f687::c100 -- "--" 0 23 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 47306 0 47306 47306 47306 0 0 0 0 0 0 0 1 1 0x0000 65535 0 242 242 0 0xc7 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 8015 0x0000080200028000 1022171719.869360 1022171719.869360 0.000000 1 4 eth:ipv4:ipv6:UNK(79) 00:d0:02:6d:78:00 00:50:04:56:32:a7 0x86dd 1439:5c49:5050:4470:c74a:0:101:80a -- "--" 0 53eb:c64f:86:4da9::c100 -- "--" 0 79 1 00:d0:02:6d:78:00_00:50:04:56:32:a7_1 Ditech_3com3c90 0 unknown 1 0 17563 0 17563 17563 17563 0 0 0 0 0 0 0 1 1 0x0000 65535 0 129 129 0 0xc1 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 8864 0x0000080200028000 1022171722.741406 1022171722.741406 0.000000 1 4 eth:ipv4:ipv6:UNK(131) 00:20:18:80:4a:b6 00:d0:02:6d:78:00 0x86dd 153c:303:5090:1920:4005:0:101:50a -- "--" 0 b9bd:772f:b9bd:94f7::c100 -- "--" 0 131 1 00:20:18:80:4a:b6_00:d0:02:6d:78:00_1 CisTechn_Ditech 0 unknown 1 0 15927 0 15927 15927 15927 0 0 0 0 0 0 0 1 1 0x0000 65535 0 44 44 0 0x05 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 8892 0x0000080200028000 1022171722.847259 1022171722.847259 0.000000 1 4 eth:ipv4:ipv6:UNK(22) 00:d0:02:6d:78:00 00:20:18:80:4a:b6 0x86dd b2e:42d1:5010:faf0:96fa:0:101:50a -- "--" 0 e4ba:2af3:e4ba:365b::c100 -- "--" 0 22 1 00:d0:02:6d:78:00_00:20:18:80:4a:b6_1 Ditech_CisTechn 0 unknown 1 0 44888 0 44888 44888 44888 0 0 0 0 0 0 0 1 1 0x0000 65535 0 109 109 0 0x01 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 9130 0x0000080200028000 1022171723.637810 1022171723.637810 0.000000 1 4 eth:ipv4:ipv6:UNK(231) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd d9d3:fbc:5090:4470:9b77:0:101:80a -- "--" 0 63c:854c:15b:8f49::c100 -- "--" 0 231 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 58513 0 58513 58513 58513 0 0 0 0 0 0 0 1 1 0x0000 65535 0 131 131 0 0xd6 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 990 0x000018f200004000 1022171701.849328 1022171726.366145 24.516817 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:e0:29:04:0a:18 0x0800 201.232.53.168 co "EPM Telecomunicaciones " 0 138.212.189.228 jp "ASAHI KASEI CORPORATION" 0 17 1 00:d0:02:6d:78:00_00:e0:29:04:0a:18_135 Ditech_SmcEther 0 unknown 135 0 103104 0 58 1480 763.7333 697.8352 0 0.391261 0.181606 0.1824253 5.506424 4205.44 1 1 0x0000 0 140 119 119 0 0x00 0x2320 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 3943 0x0000080a00005000 1022171706.645144 1022171726.589552 19.944408 4 4;4;4;4 eth:ipv4:gre:UNK(0xaa18);eth:ipv4:gre:UNK(0xe805);eth:ipv4:gre:UNK(0x1d11);eth:ipv4:gre:UNK(0x0e0f) 00:d0:02:6d:78:00 00:50:fc:20:5d:67 0x0800 201.9.4.49 br "Telemar Norte Leste S.A" 0 138.212.191.213 jp "ASAHI KASEI CORPORATION" 0 47 1 00:d0:02:6d:78:00_00:50:fc:20:5d:67_4 Ditech_EdimaxTe 0 unknown 4 5 137 256 0 86 34.25 26.7212 0 11.01577 4.986102 3.825814 0.2005575 6.869093 -0.1111111 -0.302799 0x0100 4 114 118 118 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.038939 8.783088 2.241058 3.104003 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 4 2 2
B 3943 0x0001080a00005001 1022171706.645835 1022171726.447349 19.801514 4 4;4;4;4 eth:ipv4:gre:UNK(0x8b00);eth:ipv4:gre:UNK(0x1400);eth:ipv4:gre:UNK(0x900a);eth:ipv4:gre:UNK(0xe6ef) 00:50:fc:20:5d:67 00:d0:02:6d:78:00 0x0800 138.212.191.213 jp "ASAHI KASEI CORPORATION" 0 201.9.4.49 br "Telemar Norte Leste S.A" 0 47 1 00:50:fc:20:5d:67_00:d0:02:6d:78:00_5 EdimaxTe_Ditech 0 unknown 5 4 256 137 0 234 51.2 81.84808 0 10.97614 3.960303 3.838957 0.252506 12.9283 0.1111111 0.302799 0x0100 1 1 64 64 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.000691 10.97683 2.196429 3.342015 4.437487 4.56113 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 3 1 1
A 1295 0x000018f200004000 1022171702.058267 1022171726.575521 24.517254 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:10:60:59:f1:4b 0x0800 16.103.245.128 us "HEWLETT PACKARD ENTERPR" 0 138.212.191.249 jp "ASAHI KASEI CORPORATION" 0 17 1 00:d0:02:6d:78:00_00:10:60:59:f1:4b_132 Ditech_Billingt 0 unknown 132 0 101508 0 58 1480 769 697.6147 0 0.747895 0.1857367 0.188037 5.383964 4140.268 1 1 0x0000 0 4008 111 111 0 0x00 0x0324 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 99 0x000008f200004000 1022171701.699708 1022171726.576813 24.877105 1 3 eth:ipv4:udp 00:80:48:b3:0e:ed 00:d0:02:6d:78:00 0x0800 138.212.188.118 jp "ASAHI KASEI CORPORATION" 0 201.9.46.255 br "Telemar Norte Leste S.A" 0 17 1 00:80:48:b3:0e:ed_00:d0:02:6d:78:00_502 CompexUs_Ditech 0 unknown 502 0 26104 0 52 52 52 0 0 0.833578 0.04955598 0.05810622 20.1792 1049.318 1 1 0x0000 1 3 64 64 0 0x00 0x0300 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 9987 0x0000080200028000 1022171726.587705 1022171726.587705 0.000000 1 4 eth:ipv4:ipv6:UNK(114) 00:d0:02:6d:78:00 00:a0:c9:07:e0:73 0x86dd e8:3dce:50d0:2180:f660:0:101:80a -- "--" 0 514:2cf1:2e3c:ec2::c100 -- "--" 0 114 1 00:d0:02:6d:78:00_00:a0:c9:07:e0:73_1 Ditech_IntelPro 0 unknown 1 0 47 0 47 47 47 0 0 0 0 0 0 0 1 1 0x0000 65535 0 253 253 0 0x97 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 132 0x000018f200004000 1022171701.700973 1022171726.594434 24.893461 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:80:c8:8f:ca:5e 0x0800 16.46.171.138 us "HEWLETT PACKARD ENTERPR" 0 138.212.189.231 jp "ASAHI KASEI CORPORATION" 0 17 1 00:d0:02:6d:78:00_00:80:c8:8f:ca:5e_496 Ditech_D-LinkAl 0 unknown 496 0 589475 0 885 1480 1188.458 289.7024 0 0.201927 0.05018844 0.0506012 19.92491 23679.91 1 1 0x0000 0 595 114 114 0 0x00 0x2324 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1
A 409 0x000008f200004000 1022171701.717743 1022171726.607895 24.890152 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:e0:29:04:0b:81 0x0800 18.14.224.62 us "Massachusetts Institute" 0 138.212.191.34 jp "ASAHI KASEI CORPORATION" 0 17 1 00:d0:02:6d:78:00_00:e0:29:04:0b:81_68 Ditech_SmcEther 0 unknown 68 0 54060 0 795 795 795 0 0 0.665422 0.3660316 0.06958047 2.732004 2171.943 1 1 0x0000 2315 65036 116 116 0 0x00 0x0302 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00 0x00 0 0x00000000_0x00000000_0x0000 0x00000000 0 0 1 1 1 1 1If you don’t like tabs as a separator, change SEP_CHR from "\t" to any character(s) you like in utils/bin2txt.h and recompile txtSink (note that some scripts and tools may require additional options if you change the default separator):
Nevertheless, tabs are standard separators in most of the bash tools.
We use a lot of hex coded status variables because each info in the flow has to be multiplied by the number of flows T2 has to hold in memory and you will experience that selecting flows will be way easier with hex coding. Each bit has a meaning, please refer to basicFlow.pdf under doc/, e.g., by running t2doc basicFlow or type
$ tawk -V flowStat=0x0001080a00005001
The flowStat column with value 0x0001080a00005001 is to be interpreted as follows:
bit | flowStat | Description
=============================================================================
0 | 0x00000000 00000001 | Inverted flow, did not initiate connection
12 | 0x00000000 00001000 | GRE v1/2
14 | 0x00000000 00004000 | IPv4
33 | 0x00000002 00000000 | Acquired packet length < packet length in L3 header
35 | 0x00000008 00000000 | Acquired packet length < minimal L4 Header
43 | 0x00000800 00000000 | Stop dissecting
48 | 0x00010000 00000000 | Header description overrunA single A Flow can be also the answering flow if the flowStat bit 0 is set. T2 sets this bit according to L4/7 info to the best of his knowledge. We will come back to that topic when discussing ICMP flows.
Now try to select flows yourself, lets say all flows of source port 443 and having an acquired packet length issue and where T2 stopped dissecting to prevent overrunning the pcap memory. A bitwise AND of flowStat and a mask is required and a selection of srcPort 443:
$ tawk 'bitsanyset($flowStat, 0x0000080f00000000) && sport(443)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm
B 4075 0x0000000200004001 1022171707.227811 1022171708.640243 1.412432 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:fc:2c:09:2a 0x0800 70.128.234.203 us "AT&T Corp." 443 138.212.190.164 jp "ASAHI KASEI CORPORATION" 1328 6 13 12 3907 917 0 536 300.5385 198.701 0 0.506266 0.1086486 0.118678 9.203983 2766.151 0.04 0.6198176A port 443 response from USA, AT&T to Japan the Asahi Kasei Corp. Interesting, somebody browsing? If you are really interested what the person is doing, you need to add the sslDecode plugin or look into the traffic with httpSniffer, maybe they have a config error and everything is plain text. This will be discussed elsewhere, not here. We start with the basics now.
But play around a bit and you will discover how easy it is to select specific flows.
Sometimes admins are only interested in standard 5 tuples (srcIP, srcPort, dstIP, dstPort, L4Proto), just plain NetFlow5 output. To configure that move to the plugins directory or use tran, a bash alias. Then move to basicFlow/src and open basicFlow.h
$ tran
$ cd basicFlow/src
$ vi basicFlow.hChange the following constants to 0 as indicated below:
...
BFO_MAC 0 // 1: Enables / 0: Disables MAC addresses output
BFO_ETHERTYPE 0 // 1: Enables / 0: Disables Ethertype header output
BFO_VLAN 0 // 0: Do not output VLAN information
...
BFO_SUBNET_TEST 0 // 1: Enables / 0: Disables subnet test
...
// Maximum number of values to store
BFO_MAX_HDRDESC 0 // Maximum number of headers descriptions to store
...Then move to basicStats, open basicStats.h
$ tran
$ cd basicStats/src
$ vi basicStats.hAnd change these constants to 0
BS_REV_CNT 0 // 1: add reverse counts from opposite flow, 0: native send counts
BS_STATS 0 // 1: basic statistics, 0: only countsRecompile the two plugins and invoke T2
$ t2build basicFlow basicStats
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...And here is your NetFlow5 output.
$ tawk 'bitsanyset($flowStat, 0x0000080100000000)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration srcIP srcPort dstIP dstPort l4Proto numPktsSnt numBytesSnt
A 262 0x0000080200028000 1022171701.707777 1022171701.707777 0.000000 cfb6:1c18:5010:faf0:7f66:0:101:80a 0 6c2:6a7f:1:384b::c100 0 168 1 41630
A 888 0x0000080200028000 1022171701.810764 1022171701.810764 0.000000 e499:578c:5090:81d0:891b:0:101:80a 0 514:2343:2e3c:512::c100 0 133 1 55304
A 3050 0x0000080200028000 1022171704.484515 1022171704.484515 0.000000 baea:e860:8090:4470:cf67:0:101:50a 0 baea:ee14:baeb:2ac::c100 0 147 1 18922
A 3455 0x0000080200028000 1022171705.349871 1022171705.349871 0.000000 d973:ceac:5010:fa94:12db:0:101:80a 0 12ea:b284:3:ceeb::c100 0 126 1 38816
A 3995 0x0000080200005000 1022171706.878547 1022171707.000313 0.121766 200.134.37.255 0 138.212.185.216 0 47 2 170
B 3995 0x0000080200005001 1022171706.931865 1022171706.962666 0.030801 138.212.185.216 0 200.134.37.255 0 47 2 210
A 4532 0x0000080200028000 1022171708.439927 1022171708.439927 0.000000 48d5:5fad:50d0:16d0:4a16:0:101:80a 0 53eb:c1d7:86:4d37::c100 0 95 1 28079
A 4829 0x0000180200005000 1022171709.440443 1022171709.552021 0.111578 19.228.184.27 0 138.212.186.191 0 47 4 162
B 4829 0x0000180200005001 1022171709.441758 1022171709.552128 0.110370 138.212.186.191 0 19.228.184.27 0 47 3 166
A 6026 0x0000080200028000 1022171713.252296 1022171713.252296 0.000000 6f:1256:5050:4402:331e:0:101:50a 0 223:fd3e:223:ff56::c100 0 28 1 2543
A 6069 0x0000080200028000 1022171713.395746 1022171713.395746 0.000000 e29f:74ec:50d0:f53c:98e3:0:101:80a 0 12ea:b5a9:3:cf3b::c100 0 229 1 22844
A 6097 0x0000080200028000 1022171713.450109 1022171713.450109 0.000000 392:fc19:5050:20e2:a7c3:0:101:80a 0 be1:fcff:6ca:1b09::c100 0 64 1 174
A 6915 0x0000080200028000 1022171716.432198 1022171716.432198 0.000000 75a4:9abd:80d0:4470:b5e2:317b:101:50a 0 75a4:a071:75a4:be39::c100 0 223 1 9
A 7047 0x0000080200028000 1022171716.967592 1022171716.967592 0.000000 4b9:f24c:8090:4470:4b24:0:101:50a 0 4b9:f800:4ba:c98::c100 0 228 1 54359
A 7464 0x0000080200028000 1022171718.639159 1022171718.639159 0.000000 169e:bdd8:5090:4470:e911:0:101:50a 0 5dc9:ed57:5dc9:f687::c100 0 23 1 47306
A 7828 0x0000080200028000 1022171719.869360 1022171719.869360 0.000000 1439:5c49:5050:4470:c74a:0:101:80a 0 53eb:c64f:86:4da9::c100 0 79 1 17563
A 8642 0x0000080200028000 1022171722.741406 1022171722.741406 0.000000 153c:303:5090:1920:4005:0:101:50a 0 b9bd:772f:b9bd:94f7::c100 0 131 1 15927
A 8667 0x0000080200028000 1022171722.847259 1022171722.847259 0.000000 b2e:42d1:5010:faf0:96fa:0:101:50a 0 e4ba:2af3:e4ba:365b::c100 0 22 1 44888
A 8893 0x0000080200028000 1022171723.637810 1022171723.637810 0.000000 d9d3:fbc:5090:4470:9b77:0:101:80a 0 63c:854c:15b:8f49::c100 0 231 1 58513
A 990 0x000018f200004000 1022171701.849328 1022171726.366145 24.516817 201.232.53.168 0 138.212.189.228 0 17 135 103104
A 3898 0x0000080a00005000 1022171706.645144 1022171726.589552 19.944408 201.9.4.49 0 138.212.191.213 0 47 4 137
B 3898 0x0000080a00005001 1022171706.645835 1022171726.447349 19.801514 138.212.191.213 0 201.9.4.49 0 47 5 256
A 1295 0x000018f200004000 1022171702.058267 1022171726.575521 24.517254 16.103.245.128 0 138.212.191.249 0 17 132 101508
A 99 0x000008f200004000 1022171701.699708 1022171726.576813 24.877105 138.212.188.118 0 201.9.46.255 0 17 502 26104
A 9711 0x0000080200028000 1022171726.587705 1022171726.587705 0.000000 e8:3dce:50d0:2180:f660:0:101:80a 0 514:2cf1:2e3c:ec2::c100 0 114 1 47
A 132 0x000018f200004000 1022171701.700973 1022171726.594434 24.893461 16.46.171.138 0 138.212.189.231 0 17 496 589475
A 409 0x000008f200004000 1022171701.717743 1022171726.607895 24.890152 18.14.224.62 0 138.212.191.34 0 17 68 54060flowInd and flowStat enables you to identify or select flows or connects packets to their flows. Sometimes people are just into simple boring NetFlow5 output, so use the following cut command:
$ tawk 'bitsanyset($flowStat, 0x0000080100000000)' annoloc2_flows.txt | cut -f 1,3- | tcol
%dir flowStat timeFirst timeLast duration srcIP srcPort dstIP dstPort l4Proto numPktsSnt numBytesSnt
A 0x0000080200028000 1022171701.707777 1022171701.707777 0.000000 cfb6:1c18:5010:faf0:7f66:0:101:80a 0 6c2:6a7f:1:384b::c100 0 168 1 41630
A 0x0000080200028000 1022171701.810764 1022171701.810764 0.000000 e499:578c:5090:81d0:891b:0:101:80a 0 514:2343:2e3c:512::c100 0 133 1 55304
A 0x0000080200028000 1022171704.484515 1022171704.484515 0.000000 baea:e860:8090:4470:cf67:0:101:50a 0 baea:ee14:baeb:2ac::c100 0 147 1 18922
A 0x0000080200028000 1022171705.349871 1022171705.349871 0.000000 d973:ceac:5010:fa94:12db:0:101:80a 0 12ea:b284:3:ceeb::c100 0 126 1 38816
A 0x0000080200005000 1022171706.878547 1022171707.000313 0.121766 200.134.37.255 0 138.212.185.216 0 47 2 170
B 0x0000080200005001 1022171706.931865 1022171706.962666 0.030801 138.212.185.216 0 200.134.37.255 0 47 2 210
A 0x0000080200028000 1022171708.439927 1022171708.439927 0.000000 48d5:5fad:50d0:16d0:4a16:0:101:80a 0 53eb:c1d7:86:4d37::c100 0 95 1 28079
A 0x0000180200005000 1022171709.440443 1022171709.552021 0.111578 19.228.184.27 0 138.212.186.191 0 47 4 162
B 0x0000180200005001 1022171709.441758 1022171709.552128 0.110370 138.212.186.191 0 19.228.184.27 0 47 3 166
A 0x0000080200028000 1022171713.252296 1022171713.252296 0.000000 6f:1256:5050:4402:331e:0:101:50a 0 223:fd3e:223:ff56::c100 0 28 1 2543
A 0x0000080200028000 1022171713.395746 1022171713.395746 0.000000 e29f:74ec:50d0:f53c:98e3:0:101:80a 0 12ea:b5a9:3:cf3b::c100 0 229 1 22844
A 0x0000080200028000 1022171713.450109 1022171713.450109 0.000000 392:fc19:5050:20e2:a7c3:0:101:80a 0 be1:fcff:6ca:1b09::c100 0 64 1 174
A 0x0000080200028000 1022171716.432198 1022171716.432198 0.000000 75a4:9abd:80d0:4470:b5e2:317b:101:50a 0 75a4:a071:75a4:be39::c100 0 223 1 9
A 0x0000080200028000 1022171716.967592 1022171716.967592 0.000000 4b9:f24c:8090:4470:4b24:0:101:50a 0 4b9:f800:4ba:c98::c100 0 228 1 54359
A 0x0000080200028000 1022171718.639159 1022171718.639159 0.000000 169e:bdd8:5090:4470:e911:0:101:50a 0 5dc9:ed57:5dc9:f687::c100 0 23 1 47306
A 0x0000080200028000 1022171719.869360 1022171719.869360 0.000000 1439:5c49:5050:4470:c74a:0:101:80a 0 53eb:c64f:86:4da9::c100 0 79 1 17563
A 0x0000080200028000 1022171722.741406 1022171722.741406 0.000000 153c:303:5090:1920:4005:0:101:50a 0 b9bd:772f:b9bd:94f7::c100 0 131 1 15927
A 0x0000080200028000 1022171722.847259 1022171722.847259 0.000000 b2e:42d1:5010:faf0:96fa:0:101:50a 0 e4ba:2af3:e4ba:365b::c100 0 22 1 44888
A 0x0000080200028000 1022171723.637810 1022171723.637810 0.000000 d9d3:fbc:5090:4470:9b77:0:101:80a 0 63c:854c:15b:8f49::c100 0 231 1 58513
A 0x000018f200004000 1022171701.849328 1022171726.366145 24.516817 201.232.53.168 0 138.212.189.228 0 17 135 103104
A 0x0000080a00005000 1022171706.645144 1022171726.589552 19.944408 201.9.4.49 0 138.212.191.213 0 47 4 137
B 0x0000080a00005001 1022171706.645835 1022171726.447349 19.801514 138.212.191.213 0 201.9.4.49 0 47 5 256
A 0x000018f200004000 1022171702.058267 1022171726.575521 24.517254 16.103.245.128 0 138.212.191.249 0 17 132 101508
A 0x000008f200004000 1022171701.699708 1022171726.576813 24.877105 138.212.188.118 0 201.9.46.255 0 17 502 26104
A 0x0000080200028000 1022171726.587705 1022171726.587705 0.000000 e8:3dce:50d0:2180:f660:0:101:80a 0 514:2cf1:2e3c:ec2::c100 0 114 1 47
A 0x000018f200004000 1022171701.700973 1022171726.594434 24.893461 16.46.171.138 0 138.212.189.231 0 17 496 589475
A 0x000008f200004000 1022171701.717743 1022171726.607895 24.890152 18.14.224.62 0 138.212.191.34 0 17 68 54060
...As you can see, flowInd is now gone. There are more tricks with tawk, being discussed in the Post processing with TAWK tutorial.
Now to reset basicFlow and basicStats to the default configuration, we need to flip the changed bits back to 1 and recompile the plugins with t2build. This time, we will use t2conf to reconfigure the plugin:
$ t2conf basicFlow -D BFO_MAC=1 -D BFO_ETHERTYPE=1 -D BFO_VLAN=1 -D BFO_SUBNET_TEST=1 -D BFO_MAX_HDRDESC=4
$ t2conf basicStats -D BS_REV_CNT=1 -D BS_STATS=1
$ t2build basicFlow basicStats
...
$Now we are adding L4 information which does the following jobs:
- tcpFlags: IP, UDP, TCP aggregated flags and anomaly status
- tcpStates: TCP state-machine and RFC check, it also terminates TCP flows after a RST or FIN
Compile them and run T2
$ t2build tcpFlags tcpStates
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
--------------------------------------------------------------------------------
basicStats: Biggest Talker: 138.212.189.38 (JP): 23601 (23.60 K) [1.94%] packets
basicStats: Biggest Talker: 138.212.189.38 (JP): 35005508 (35.01 M) [54.63%] bytes
tcpFlags: Aggregated ipFlags: 0x3966
tcpFlags: Aggregated tcpAnomaly: 0xff47
tcpFlags: Number of TCP scans, succ scans, retries: 238, 671, 933
tcpFlags: Number WinSz below 1: 2415 (2.42 K) [0.25%]
tcpStates: Aggregated anomaly flags: 0xdf
--------------------------------------------------------------------------------
...Note that between the lines of the end report now additional aggregated fields appear:
- tcpFlags: ipFlags, tcpAnomaly, TCP scans, successful scans and retries and tcpWinSzMin: all kinds of info for troubleshooting and security purposes
- tcpStates: aggregated anomaly flags, denoting deviations from RFC
The hex numbers denote aggregated anomaly output, where each bit has a specific meaning. Note that there are many flows where the TCP window size drops below 1 and there several retries and there are scans detected.
All bit fields are documented under each plugin folder or under doc/documents.pdf Or just wait for a new tutorial in future if you do not like to read PDFs.
Now you have NetFlow9/10 + a bit more, look at all the anomaly bits in the end report. That is too much for the beginning. Let’s do something more interesting, let’s say your your manager imposed the rule that nobody should communicate with China, because he thinks that there are no business ties with this country.
So he comes to you and demands the answer to the following question: Is there any traffic initiating connection egress to China?
$ tawk 'bitsanyset($flowStat, 1) == 0 && ($srcIPCC == "cn" || $dstIPCC == "cn") { print $srcIP, $srcIPCC, $dstIP, $dstIPCC }' annoloc2_flows.txt | sort -V -u | tcol
138.212.184.34 jp 36.192.216.3 cn
138.212.184.42 jp 36.214.21.116 cn
138.212.184.71 jp 36.16.139.73 cn
138.212.184.71 jp 36.218.130.149 cn
...so you get all the IP’s communicating to China, the lowest bit in flowStat denotes the initiation of the connection, a 0 means the srcIP started the flow.
But you are interested in the flow details (the hdr() command adds the header to the resulting output, which otherwise would be filtered):
$ tawk 'hdr() || ($dstIPCC == "cn" && tcp())' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates
B 2211 0x0000000200004001 1022171703.077130 1022171703.077130 0.000000 1 3 eth:ipv4:tcp 00:05:02:a7:59:98 00:d0:02:6d:78:00 0x0800 138.212.184.140 jp "ASAHI KASEI CORPORATION" 5500 36.204.73.10 cn "China TieTong Telecommu" 57019 6 1 1 11 0 11 11 11 0 0 0 0 0 0 0 0 1 0x0164 65535 0 255 255 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0x54 0x0004 0 0 0x00000000 0 0 0 0 0 0.000000 0.03941 0.03941 0.03941 0.03941 0 0.03941 0 0x43
A 3103 0x0000000000004000 1022171704.554485 1022171704.554485 0.000000 1 3 eth:ipv4:tcp 00:60:08:78:1b:63 00:d0:02:6d:78:00 0x0800 138.212.187.203 jp "ASAHI KASEI CORPORATION" 1825 36.176.200.106 cn "China Mobile Communicat" 4567 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x0144 65535 0 128 128 0 0x00 0x0840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 16384 16384 16384 16384 0 0 0 0 0x02 0x0000 1 4 0x00000016 1460 0 0 0 0 0.000000 0 65535 0 0 0 0.108845 -1 0x03
B 3354 0x0000000000004001 1022171705.071644 1022171705.071644 0.000000 1 3 eth:ipv4:tcp 00:50:bf:59:85:48 00:d0:02:6d:78:00 0x0800 138.212.188.67 jp "ASAHI KASEI CORPORATION" 1214 58.204.250.125 cn "China Education and Res" 4120 6 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x0164 65535 0 128 128 0 0x00 0x0800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0xd4 0x0004 0 0 0x00000000 0 0 0 0 0 0.000000 0.008885 0.008885 0.008885 0.008885 0 0.008885 0 0x43
...So there is no business ties to China, right?! Or is it legal traffic… There is seriously something fishy.
Let’s ask more about anomalies, such as broken fragmentation, aka fragmentation positional errors
$ tawk -V ipFlags
The ipFlags column is to be interpreted as follows:
bit | ipFlags | Description
=============================================================================
0 | 0x0001 | IP options corrupt
1 | 0x0002 | IPv4 packets out of order
2 | 0x0004 | IPv4 ID roll over
3 | 0x0008 | IP fragment below minimum
4 | 0x0010 | IP fragment out of range
5 | 0x0020 | More Fragment bit
6 | 0x0040 | IPv4: Don't Fragment bit, IPv6: reserve bit
7 | 0x0080 | Reserve bit
8 | 0x0100 | Fragmentation position error
9 | 0x0200 | Fragmentation sequence error
10 | 0x0400 | L3 checksum error
11 | 0x0800 | L4 checksum error
12 | 0x1000 | L3 header length snapped
13 | 0x2000 | Packet interdistance = 0
14 | 0x4000 | Packet interdistance < 0
15 | 0x8000 | TCP SYN flag with L7 content
So let us test frag bits:
$ tawk 'bitsanyset($ipFlags, 0x0318)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates
A 990 0x000018f200004000 1022171701.849328 1022171726.366145 24.516817 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:e0:29:04:0a:18 0x0800 201.232.53.168 co "EPM Telecomunicaciones " 0 138.212.189.228 jp "ASAHI KASEI CORPORATION" 0 17 135 0 103104 0 58 1480 763.7333 697.8352 0 0.391261 0.181606 0.1824253 5.506424 4205.44 1 1 0x0000 0 140 119 119 0 0x00 0x2320 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00
A 1295 0x000018f200004000 1022171702.058267 1022171726.575521 24.517254 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:10:60:59:f1:4b 0x0800 16.103.245.128 us "HEWLETT PACKARD ENTERPR" 0 138.212.191.249 jp "ASAHI KASEI CORPORATION" 0 17 132 0 101508 0 58 1480 769 697.6147 0 0.747895 0.1857367 0.188037 5.383964 4140.268 1 1 0x0000 0 4008 111 111 0 0x00 0x0324 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00
A 99 0x000008f200004000 1022171701.699708 1022171726.576813 24.877105 1 3 eth:ipv4:udp 00:80:48:b3:0e:ed 00:d0:02:6d:78:00 0x0800 138.212.188.118 jp "ASAHI KASEI CORPORATION" 0 201.9.46.255 br "Telemar Norte Leste S.A" 0 17 502 0 26104 0 52 52 52 0 0 0.833578 0.04955598 0.05810622 20.1792 1049.318 1 1 0x0000 1 3 64 64 0 0x00 0x0300 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00
A 132 0x000018f200004000 1022171701.700973 1022171726.594434 24.893461 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:80:c8:8f:ca:5e 0x0800 16.46.171.138 us "HEWLETT PACKARD ENTERPR" 0 138.212.189.231 jp "ASAHI KASEI CORPORATION" 0 17 496 0 589475 0 885 1480 1188.458 289.7024 0 0.201927 0.05018844 0.0506012 19.92491 23679.91 1 1 0x0000 0 595 114 114 0 0x00 0x2324 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00
A 409 0x000008f200004000 1022171701.717743 1022171726.607895 24.890152 1 3 eth:ipv4:udp 00:d0:02:6d:78:00 00:e0:29:04:0b:81 0x0800 18.14.224.62 us "Massachusetts Institute" 0 138.212.191.34 jp "ASAHI KASEI CORPORATION" 0 17 68 0 54060 0 795 795 795 0 0 0.665422 0.3660316 0.06958047 2.732004 2171.943 1 1 0x0000 2315 65036 116 116 0 0x00 0x0302 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0x00 0x0000 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x00Now let’s look for TCP anomalies, such as abnormal flag combinations appearing in packets. The column tcpAnomaly contains flags for combination of flags and abnormalities about sequence numbers.
$ tawk -V tcpAnomaly
The tcpAnomaly column is to be interpreted as follows:
bit | tcpAnomaly | Description
=============================================================================
0 | 0x0001 | FIN-ACK flag
1 | 0x0002 | SYN-ACK flag
2 | 0x0004 | RST-ACK flag
3 | 0x0008 | SYN-FIN flag, scan or malicious packet
4 | 0x0010 | SYN-FIN-RST flag, potential malicious scan packet or channel
5 | 0x0020 | FIN-RST flag, abnormal flow termination
6 | 0x0040 | Null flag, potential NULL scan packet, or malicious channel
7 | 0x0080 | XMas flag, potential Xmas scan packet, or malicious channel
8 | 0x0100 | L4 option field corrupt or not acquired
9 | 0x0200 | SYN retransmission
10 | 0x0400 | Sequence Number retry
11 | 0x0800 | Sequence Number out of order
12 | 0x1000 | Sequence mess in flow order due to pcap packet loss
13 | 0x2000 | Sequence number jump forward
14 | 0x4000 | ACK number out of order
15 | 0x8000 | Duplicate ACK
So select the following bit mask:
$ tawk 'bitsanyset($tcpAnomaly, 0x00f8)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates
A 890 0x0000000200008000 1022171701.811168 1022171701.811168 0.000000 1 3 eth:ipv6:tcp 00:60:08:2c:ca:8e 00:40:05:56:05:f0 0x86dd 3ffe:7c9b:e2:4ca6:4c::b0 -- "--" 48458 3ffe:7c9b:f5:8b05::2f50 -- "--" 6667 6 1 0 20 0 20 20 20 0 0 0 0 0 0 0 1 1 0x0142 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x83
A 855 0x0000000200024000 1022171701.803251 1022171702.336097 0.532846 1 4 eth:ipv4:ipv4:tcp 00:d0:02:6d:78:00 00:00:21:ef:64:92 0x0800 201.98.74.248 mx "Uninet S.A. de C.V." 5642 19.54.248.131 us "Ford Motor Company" 997 6 2 2 141 165 32 109 70.5 27.22361 0 0.532846 0.266423 0.1883895 3.75343 264.6168 0 -0.07843138 0x0144 3493 3493 57 57 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.110085 0.110085 0.0550425 0.03892093 0 -1 0x87
B 855 0x0000000200024001 1022171701.903727 1022171702.226012 0.322285 1 4 eth:ipv4:ipv4:tcp 00:00:21:ef:64:92 00:d0:02:6d:78:00 0x0800 19.54.248.131 us "Ford Motor Company" 997 201.98.74.248 mx "Uninet S.A. de C.V." 5642 6 2 2 165 141 32 133 82.5 35.70889 0 0.322285 0.1611425 0.113945 6.205688 511.9692 0 0.07843138 0x0144 56884 56884 64 64 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.100476 0.422761 0.2616185 0.113945 0.316661 0.1204089 0x87
A 2664 0x0000000200008000 1022171703.710373 1022171703.859739 0.149366 1 3 eth:ipv6:tcp 00:01:02:af:4a:b4 00:80:48:cd:88:83 0x86dd 2001:70e8:d3ce:e200:de45:6dff:c7ad:c251 -- "--" 32798 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 6667 6 2 2 57 104 20 37 28.5 6.010407 0 0.149366 0.074683 0.05280886 13.38993 381.6129 0 -0.2919255 0x0144 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.008073 0.008073 0.0040365 0.002854237 0 -1 0x87
B 2664 0x0000000200008001 1022171703.742177 1022171703.851666 0.109489 1 3 eth:ipv6:tcp 00:80:48:cd:88:83 00:01:02:af:4a:b4 0x86dd 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 6667 2001:70e8:d3ce:e200:de45:6dff:c7ad:c251 -- "--" 32798 6 2 2 104 57 20 84 52 22.62742 0 0.109489 0.0547445 0.03871021 18.26667 949.8671 0 0.2919255 0x0144 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.031804 0.141293 0.0865485 0.03871021 0.090585 0.03881529 0x87
A 4049 0x0000000200008000 1022171706.884790 1022171707.007311 0.122521 1 3 eth:ipv6:tcp 00:60:08:2c:ca:8e 00:80:48:cd:88:83 0x86dd 2001:70e8:d3ce:e202::30:26 -- "--" 2128 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 6668 6 2 2 70 106 20 50 35 10.6066 0 0.122521 0.06126049 0.04331771 16.32373 571.3306 0 -0.2045455 0x0144 65535 0 63 63 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.052077 0.052077 0.0260385 0.018412 0 -1 0x87
B 4049 0x0000000200008001 1022171706.922668 1022171706.955234 0.032566 1 3 eth:ipv6:tcp 00:40:05:56:05:f0 00:60:08:2c:ca:8e 0x86dd 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 6668 2001:70e8:d3ce:e202::30:26 -- "--" 2128 6 2 2 106 70 20 86 53 23.33452 0 0.032566 0.016283 0.01151382 61.41375 3254.928 0 0.2045455 0x0144 65535 0 63 63 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.037878 0.070444 0.054161 0.01151382 0.08019949 0.02171566 0x87
A 4573 0x0000000200024000 1022171708.338404 1022171708.552551 0.214147 1 4 eth:ipv4:ipv4:tcp 00:d0:02:6d:78:00 00:00:21:ef:64:92 0x0800 83.45.68.186 es "Telefonica de Espana SA" 33790 19.54.248.131 us "Ford Motor Company" 997 6 2 1 117 101 32 85 58.5 18.73833 0 0.214147 0.1070735 0.0757124 9.339379 546.3537 0.3333333 0.07339449 0x0144 5 5 48 48 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.203134 0.203134 0.101567 0.07181872 0 -1 0x87
B 4573 0x0000000200024001 1022171708.349417 1022171708.349417 0.000000 1 4 eth:ipv4:ipv4:tcp 00:00:21:ef:64:92 00:d0:02:6d:78:00 0x0800 19.54.248.131 us "Ford Motor Company" 997 83.45.68.186 es "Telefonica de Espana SA" 33790 6 1 2 101 117 101 101 101 0 0 0 0 0 0 0 -0.3333333 -0.07339449 0x0144 65535 0 64 64 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.011013 0.011013 0.011013 0 0.11258 0.07181872 0x87
A 5360 0x0000000200008000 1022171710.810212 1022171710.810212 0.000000 1 3 eth:ipv6:tcp 00:40:05:56:05:f0 00:50:fc:20:90:a5 0x86dd 3ffe:7c9b:f5:8b05::2f50 -- "--" 6667 2001:70e8:d3ce:e200:de29:6aff:91cc:d9a -- "--" 2912 6 1 1 90 20 90 90 90 0 0 0 0 0 0 0 0 0.6363636 0x0144 65535 0 56 56 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x83
B 5360 0x0000000200008001 1022171710.910120 1022171710.910120 0.000000 1 3 eth:ipv6:tcp 00:50:fc:20:90:a5 00:40:05:56:05:f0 0x86dd 2001:70e8:d3ce:e200:de29:6aff:91cc:d9a -- "--" 2912 3ffe:7c9b:f5:8b05::2f50 -- "--" 6667 6 1 1 20 90 20 20 20 0 0 0 0 0 0 0 0 -0.6363636 0x0144 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.099908 0.099908 0.099908 0 0.099908 0 0x83
A 5610 0x0000000200024000 1022171711.507332 1022171711.632627 0.125295 1 4 eth:ipv4:ipv4:tcp 00:00:21:ef:64:92 00:d0:02:6d:78:00 0x0800 19.54.248.131 us "Ford Motor Company" 52912 19.54.241.75 us "Ford Motor Company" 6667 6 2 2 97 153 32 65 48.5 11.66726 0 0.125295 0.0626475 0.04429847 15.96233 774.173 0 -0.224 0x0144 7946 7946 64 64 0 0x00 0x1844 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.097721 0.097721 0.0488605 0.03454959 0 -1 0x87
B 5610 0x0000000200024001 1022171711.526090 1022171711.534906 0.008816 1 4 eth:ipv4:ipv4:tcp 00:d0:02:6d:78:00 00:00:21:ef:64:92 0x0800 19.54.241.75 us "Ford Motor Company" 6667 19.54.248.131 us "Ford Motor Company" 52912 6 2 2 153 97 32 121 76.5 31.46625 0 0.008816 0.004408 0.003116927 226.8602 17354.81 0 0.224 0x0144 14 14 62 62 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.018758 0.027574 0.023166 0.003116927 0.07202651 0.0346899 0x87
A 6271 0x0000000200008000 1022171713.686548 1022171714.050885 0.364337 1 3 eth:ipv6:tcp 00:50:04:56:32:a7 00:80:48:cd:88:83 0x86dd 2001:70e8:d3ce:e200:de29:8cff:c040:71a9 -- "--" 53731 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 6669 6 2 2 70 106 20 50 35 10.6066 0 0.364337 0.1821685 0.1288126 5.489423 192.1298 0 -0.2045455 0x0144 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 0.09502 0.09502 0.04751 0.03359464 0 -1 0x87
A 3070 0x0000000200008000 1022171704.489648 1022171726.598057 22.108409 1 3 eth:ipv6:tcp 00:a0:c9:07:e0:73 00:40:05:56:05:f0 0x86dd 3ffe:7e26:7a00:e100::95 -- "--" 1944 2001:700f:d917:0:889b:e6ff:2d80:bab8 -- "--" 6667 6 6 0 257 0 20 93 42.83333 26.88194 0 9.723398 3.684735 3.382878 0.2713899 11.62454 1 1 0x0140 65535 0 64 64 0 0x00 0x1800 0 0x00_0x00000000 0_0 0x00000000_0x00000000 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0x00 0x0040 0 0 0x00000000 0 0 0 0 0 0.000000 0 65535 0 0 0 0 -1 0x87
...Now you’re talking…
$ tawk -V tcpStates
The tcpStates column is to be interpreted as follows:
bit | tcpStates | Description
=============================================================================
0 | 0x01 | Malformed connection establishment
1 | 0x02 | Malformed teardown
2 | 0x04 | Malformed flags during established connection
3 | 0x08 | Packets detected after teardown
4 | 0x10 | Packets detected after reset
6 | 0x40 | Reset from sender
7 | 0x80 | Potential evil behavior (scan)Note, that even normal applications can produce such malformed flag combinations, especially from a specific OS, which a lot of people are using. Horrible. Look a bit for yourself.
The tcpFlags plugin is built for traffic forensics and troubleshooting. It contains information about L3/4 headers and issues, such as fragmentation L4 error/flow control and Bandwidth/Round Trip Times (RTT) and some nitty-gritty tricks for security guys. Any section can be disabled in tcpFlow.h
For you currently the following extraction of tcpFlags.h is relevant for the beginning:
...
RTT_ESTIMATE 1 // 1: Round trip time estimation
IPCHECKSUM 2 // 1: Calculation of L3 (IP) header checksum,
// 2: L3 + L4 (TCP,UDP) checksum
WINDOWSIZE 1 // 1: Calculation of TCP window size parameters
WINMIN 1 // Minimal window size threshold defining a healthy communication, below packets are counted
SEQ_ACK_NUM 1 // 1: SEQ/ACK number feature analysis
FRAG_ANALYZE 1 // 1: Fragmentation analysis
...You can switch off the RTT estimation, calculation of checksums, the TCP window size features or the tricks with TCP seq/ack numbers. Although fragmentation in IPv4 today is mostly fishy, if you are not interested in it, switch it off. So the code becomes smaller and faster.
Let’s go over the most important fields you need to understand for a start.
There are still OS which increment the IPID by 1. This is a formidable feature to detect the load of a machine. Hence, T2 provides ipMindIPID and ipMaxdIPID column which denotes the min/max difference of IPIDs between packets. If the differences are large and we are sure of the 1 increment, several connections from that IP distribute packets. So, every connection will have jumps / flow. The ipMinTTL and ipMaxTTL give you an indication of how far your sniffing tap is from the senders IP address and whether several routing paths are involved.
ipFlags contains information about packet abnormalities and fragmentation mishaps. To see the meaning of the bits invoke:
$ tawk -V ipFlags
The ipFlags column is to be interpreted as follows:
bit | ipFlags | Description
=============================================================================
0 | 0x0001 | IP options corrupt
1 | 0x0002 | IPv4 packets out of order
2 | 0x0004 | IPv4 ID roll over
3 | 0x0008 | IP fragment below minimum
4 | 0x0010 | IP fragment out of range
5 | 0x0020 | More Fragment bit
6 | 0x0040 | IPv4: Don't Fragment bit, IPv6: reserve bit
7 | 0x0080 | Reserve bit
8 | 0x0100 | Fragmentation position error
9 | 0x0200 | Fragmentation sequence error
10 | 0x0400 | L3 checksum error
11 | 0x0800 | L4 checksum error
12 | 0x1000 | L3 header length snapped
13 | 0x2000 | Packet interdistance = 0
14 | 0x4000 | Packet interdistance < 0
15 | 0x8000 | TCP SYN flag with L7 contenttcpFlags is the standard NetFlow aggregation of the flags in the TCP header. So you can assess the communication state of the flow during observation.
A standard feature in NetFlow9 is the aggregation of all TCP flags occurring in a flow:
$ tawk -V tcpFlags
The tcpFlags column is to be interpreted as follows:
bit | tcpFlags | Description
=============================================================================
0 | 0x01 | FIN: No more data, finish connection
1 | 0x02 | SYN: Synchronize sequence numbers
2 | 0x04 | RST: Reset connection
3 | 0x08 | PSH: Push data
4 | 0x10 | ACK: Acknowledgement field value valid
5 | 0x20 | URG: Urgent pointer valid
6 | 0x40 | ECE: ECN-Echo
7 | 0x80 | CWR: Congestion Window Reduced flag is setIt represents valuable information about the state of a connection, and you can readily detect if a flow is complete, or the guy who acquired the data clipped it.
Basic Traffic volume and connection analysis
To acquire an overview about networks and their communication a graphical output can be helpful. graphviz is a wonderful program to produce al kinds of graphs. T2 supplies a conversion example script grphvz which you may expand for your own purposes.
One basic approach is to look into the connection matrix or simpler the connections between nodes. In the script the graph edges are tagged with
- flowStat direction bit, land of origin, tcpAnomaly, srcPort-dstPort, numPktsSnt, numBytesSnt.
- Initiating flow: green, Response Flow: red
- Width: numBytesSnt
So apply the already generated flow file to grphvz, convert the resulting .dot file to JPG and display it with eog or better feh. You may also use the interactive program xdot or dotty.
$ head -n 43 annoloc2_flows.txt > a_flows.txt
$ grphvz a_flows.txt
$ dotty a_flows_graph.dotOr if you like a picture, use dot:
$ dot -Tjpg a_flows_graph.dot -o a_flows_graph.jpg
$ feh a_flows_graph.jpgOr with the new scripts since the 0.8.3 version:
$ head -n 43 annoloc2_flows.txt > a_flows.txt
$ t2viz a_flows.txt
If we had the full traffic plotted then you could identify large or biggest talkers, just by looking for the arrow with the largest width. But, note that with larger number of flows the performance of graphviz degrades rapidly. We produced a netgrapher which can handle very large connection matrices. Unfortunately this is not open source. If you are interested contact us here.
Another method to find biggest talkers is to reverse sort with tawk. Note that the number 4 in the tawk statement below denotes the number of lines to display. If you omit it, all lines will be displayed.
$ tawk 't2sort(numPktsSnt, 4)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates
B 91 0x0000000200004001 1022171701.699480 1022171726.636773 24.937293 1 3 eth:ipv4:tcp 00:00:21:d2:cc:72 00:d0:02:6d:78:00 0x0800 138.212.189.38 jp "ASAHI KASEI CORPORATION" 139 138.212.86.201 jp "Asahi Kasei Networks Co" 3429 6 23601 12342 33733962 42462 103 1460 1429.344 188.7309 0 0.253336 0.001056625 0.003715082 946.4139 1352752 0.313246 0.9974856 0x0140 1 39 64 64 0 0x10 0x3840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 23576 34471811 24 677 42651 0 33232 33232 33232 33232 0 0 0 0 0x98 0xa800 0 0 0x00000000 0 0 0 0 0 0.000000 0 0 0.253317 0.001994585 0.004210955 0.002293067 0.004360984 0x03
A 91 0x0000000200004000 1022171701.699996 1022171726.637210 24.937214 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:00:21:d2:cc:72 0x0800 138.212.86.201 jp "Asahi Kasei Networks Co" 3429 138.212.189.38 jp "ASAHI KASEI CORPORATION" 139 6 12342 23601 42462 33733962 0 63 3.440447 14.30862 0 0.36365 0.002020519 0.00532618 494.923 1702.756 -0.313246 -0.9974856 0x01c0 1 21 127 127 0 0x00 0x1840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 12342 42651 0 11833 34124331 190 17520 17518.91 16201 17520 355 355 709 0 0x98 0xa100 169 507 0x00000022 0 0 0 0 0 0.000000 0 0 0.110333 0.0002984816 0.001134035 0 -1 0x03
B 6346 0x0000000200004001 1022171714.045827 1022171722.457644 8.411817 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:10:5a:c5:96:1a 0x0800 139.45.174.202 ie "Stripe Inc" 56071 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3837 6 10159 5692 14821880 0 0 1460 1458.99 29.41481 0 1.465593 0.0008280156 0.01468998 1207.706 1762031 0.2818119 1 0x0140 1 33832 59 250 1 0x18 0x3842 0 0x00_0x00000000 0_0 0x00000000_0x00000000 10159 14999999 0 1 0 2 5840 5840 5840 5840 0 0 0 0 0xdb 0xa003 1 4 0x00000016 1460 0 0 0 0 0.000000 0.005066 0 0.219088 0.001750757 0.003134686 0.002583663 0.01960833 0x02
B 3613 0x0000000200004001 1022171705.686717 1022171714.043794 8.357077 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:10:5a:c5:96:1a 0x0800 139.45.174.202 ie "Stripe Inc" 56070 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3820 6 10048 5709 14656900 0 0 1460 1458.688 34.27719 0 1.39519 0.0008317156 0.0141066 1202.334 1753831 0.2753697 1 0x0140 1 44772 59 250 1 0x18 0x3842 0 0x00_0x00000000 0_0 0x00000000_0x00000000 10048 14999999 0 1 0 2 5840 5840 5840 5840 0 0 0 0 0xdb 0xa003 1 4 0x00000016 1460 0 0 0 0 0.000000 0.006276 0 0.240063 0.001985467 0.003686316 0.002785216 0.01874814 0x02Or:
$ tawk 't2sort(numBytesSnt, 4)' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto numPktsSnt numPktsRcvd numBytesSnt numBytesRcvd minPktSz maxPktSz avePktSize stdPktSize minIAT maxIAT aveIAT stdIAT pktps bytps pktAsm bytAsm tcpFStat ipMindIPID ipMaxdIPID ipMinTTL ipMaxTTL ipTTLChg ipTOS ipFlags ipOptCnt ipOptCpCl_Num ip6OptCntHH_D ip6OptHH_D tcpPSeqCnt tcpSeqSntBytes tcpSeqFaultCnt tcpPAckCnt tcpFlwLssAckRcvdBytes tcpAckFaultCnt tcpInitWinSz tcpAveWinSz tcpMinWinSz tcpMaxWinSz tcpWinSzDwnCnt tcpWinSzUpCnt tcpWinSzChgDirCnt tcpWinSzThRt tcpFlags tcpAnomaly tcpOptPktCnt tcpOptCnt tcpOptions tcpMSS tcpWS tcpTmS tcpTmER tcpEcI tcpBtm tcpSSASAATrip tcpRTTAckTripMin tcpRTTAckTripMax tcpRTTAckTripAve tcpRTTAckTripJitAve tcpRTTSseqAA tcpRTTAckJitAve tcpStates
B 91 0x0000000200004001 1022171701.699480 1022171726.636773 24.937293 1 3 eth:ipv4:tcp 00:00:21:d2:cc:72 00:d0:02:6d:78:00 0x0800 138.212.189.38 jp "ASAHI KASEI CORPORATION" 139 138.212.86.201 jp "Asahi Kasei Networks Co" 3429 6 23601 12342 33733962 42462 103 1460 1429.344 188.7309 0 0.253336 0.001056625 0.003715082 946.4139 1352752 0.313246 0.9974856 0x0140 1 39 64 64 0 0x10 0x3840 0 0x00_0x00000000 0_0 0x00000000_0x00000000 23576 34471811 24 677 42651 0 33232 33232 33232 33232 0 0 0 0 0x98 0xa800 0 0 0x00000000 0 0 0 0 0 0.000000 0 0 0.253317 0.001994585 0.004210955 0.002293067 0.004360984 0x03
B 6346 0x0000000200004001 1022171714.045827 1022171722.457644 8.411817 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:10:5a:c5:96:1a 0x0800 139.45.174.202 ie "Stripe Inc" 56071 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3837 6 10159 5692 14821880 0 0 1460 1458.99 29.41481 0 1.465593 0.0008280156 0.01468998 1207.706 1762031 0.2818119 1 0x0140 1 33832 59 250 1 0x18 0x3842 0 0x00_0x00000000 0_0 0x00000000_0x00000000 10159 14999999 0 1 0 2 5840 5840 5840 5840 0 0 0 0 0xdb 0xa003 1 4 0x00000016 1460 0 0 0 0 0.000000 0.005066 0 0.219088 0.001750757 0.003134686 0.002583663 0.01960833 0x02
B 3613 0x0000000200004001 1022171705.686717 1022171714.043794 8.357077 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:10:5a:c5:96:1a 0x0800 139.45.174.202 ie "Stripe Inc" 56070 138.212.190.117 jp "ASAHI KASEI CORPORATION" 3820 6 10048 5709 14656900 0 0 1460 1458.688 34.27719 0 1.39519 0.0008317156 0.0141066 1202.334 1753831 0.2753697 1 0x0140 1 44772 59 250 1 0x18 0x3842 0 0x00_0x00000000 0_0 0x00000000_0x00000000 10048 14999999 0 1 0 2 5840 5840 5840 5840 0 0 0 0 0xdb 0xa003 1 4 0x00000016 1460 0 0 0 0 0.000000 0.006276 0 0.240063 0.001985467 0.003686316 0.002785216 0.01874814 0x02
A 327 0x0000000200004000 1022171701.712093 1022171726.638722 24.926629 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:bf:08:44:81 0x0800 19.59.134.250 us "Ford Motor Company" 65230 138.212.187.240 jp "ASAHI KASEI CORPORATION" 58290 6 9459 5223 13696632 0 1448 1448 1448 0 0 0.067445 0.00263523 0.006627299 379.4737 549477.9 0.2885166 1 0x0158 1 387 53 53 0 0x08 0x3844 0 0x00_0x00000000 0_0 0x00000000_0x00000000 9415 15002728 44 0 0 0 33304 33304 33304 33304 0 0 0 0 0xd0 0xa000 9459 28377 0x00000102 0 0 199361062 113909808 0.01 1020178116.059917 0 0 0.066065 0.008232069 0.01009581 0 -1 0x03The best method to spot connection anomalies is to visualize time, connecting IPs and connection counts. The connStat plugin produces the appropriate numbers for this task.
It adds four columns connections src IP, dst IP, connections between srcIP and dstIP and the number of unique destination port connections of a certain srcIP. Moreover an experimental feature connF = connSipDprt / connSip is added which describes the ratio of port connections of a srcIP and the total connection count of the very srcIP during the lifetime of this flow.
$ t2build connStat
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.4 (Anteater), Tarantula. PID: 10282
================================================================================
...
--------------------------------------------------------------------------------
...
connStat: Number of unique source IPs: 4413 (4.41 K)
connStat: Number of unique destination IPs: 3209 (3.21 K)
connStat: Number of unique source/destination IPs connections: 182
connStat: Max unique number of source IP / destination port connections: 413
connStat: IP prtcon/sdcon, prtcon/scon: 2.269231, 0.093587
connStat: Source IP with max connections: 138.212.189.66 (JP): 366 connections
connStat: Destination IP with max connections: 138.212.184.235 (JP): 402 connections
--------------------------------------------------------------------------------
...
$The end report contribution of connStat provides you with connection oriented facts of your traffic. So you should record these numbers at certain dates and times to establish a normal baseline.
If the sending/receiving IPs in your network, exceeds the maximum of your recordings and the unique local addresses exceed the amount of your machines in the network, something is definitely wrong.
Same for the number of receiving IPs and the ratio of src and dst IPs. The 5th line consists of experimental numbers which served well in finding malware, so if the prtcon/sdcon and prtcon/scon are >> 1 you should look a bit closer at this traffic. In a future tutorial T2 Kungfu I’ll try to elaborate more on that matter.
The biggest connection initiator or connector and the biggest endpoint connector at the end of the connStat report gives you an indication where to look first, when inspecting a flow file. Note here also an information about the country.
An example tawk command is shown below extracting only the initiation flows from the biggest connection initiator printing only connection relevant features (the not command is used instead of ! to prevent tawk from filtering out the header (note that the hdr() function could also have been used)).
$ tawk 'not(bitsanyset($flowStat, 1)) && shost("138.212.189.66") { print $timeFirst, $timeLast, $srcIP, $srcIPCC, $connSip, $connSipDip, connSipDprt, $connF }' annoloc2_flows.txt | LC_ALL=C sort -t$'\t' -n -k3,3 | tcol
timeFirst timeLast srcIP srcIPCC connSip connSipDip 87 connF
1022171701.715552 1022171701.715552 138.212.189.66 jp 366 2 87 1.128415
1022171701.748589 1022171724.156283 138.212.189.66 jp 84 2 87 0.02380952
1022171701.748589 1022171725.854193 138.212.189.66 jp 40 2 87 0.05
1022171701.748591 1022171725.224952 138.212.189.66 jp 62 2 87 0.03225806
1022171701.748593 1022171701.748593 138.212.189.66 jp 365 1 87 0.002739726
1022171701.748593 1022171725.983912 138.212.189.66 jp 30 2 87 0.06666667
1022171701.748603 1022171726.344313 138.212.189.66 jp 23 2 87 0.08695652
1022171701.748605 1022171726.559487 138.212.189.66 jp 5 2 87 0.8
1022171701.833674 1022171707.884734 138.212.189.66 jp 291 1 87 1.085911
1022171701.834407 1022171701.834407 138.212.189.66 jp 364 2 87 1.129121
1022171701.845499 1022171701.845499 138.212.189.66 jp 363 2 87 1.126722
1022171701.847836 1022171725.854167 138.212.189.66 jp 41 2 87 0.04878049
1022171701.847851 1022171725.858100 138.212.189.66 jp 37 2 87 0.05405406
1022171701.847851 1022171726.446242 138.212.189.66 jp 17 2 87 0.1176471
1022171701.847852 1022171726.417256 138.212.189.66 jp 9 2 87 0.2222222
1022171701.847853 1022171701.847853 138.212.189.66 jp 362 1 87 0.002762431
1022171701.847854 1022171726.546581 138.212.189.66 jp 10 2 87 0.2
1022171701.868853 1022171701.868853 138.212.189.66 jp 361 1 87 1.127424
1022171701.878890 1022171701.878890 138.212.189.66 jp 360 2 87 1.127778
1022171701.922395 1022171701.922395 138.212.189.66 jp 359 2 87 1.125348
1022171701.947721 1022171726.546595 138.212.189.66 jp 7 1 87 0.1428571
1022171701.960091 1022171701.960091 138.212.189.66 jp 358 1 87 1.122905
1022171702.048266 1022171726.446250 138.212.189.66 jp 20 2 87 0.35
1022171702.089215 1022171702.089215 138.212.189.66 jp 357 2 87 1.123249
1022171702.188444 1022171702.188444 138.212.189.66 jp 356 2 87 1.120787
1022171702.198206 1022171708.413842 138.212.189.66 jp 287 1 87 1.076655
...Up to now we used absolute time stamps, but for static plots the relative time to the beginning of the pcap is easier to grasp. So first move to tranalyzer/src and open the file tranalyzer.h
$ cd tranalyzer/src
$ vi tranalyzer.hAnd look for RELTIME. Change it to 1 as shown below
And recompile all plugins used so far, because certain plugins such as basicFlow are using the RELTIME switch; then rerun T2.
$ t2build -R
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.4 (Anteater), Tarantula. PID: 10282
================================================================================
...
$For visualization we only need to extract the said three features and pipe it into t2plot to create a nice 3D graphics in logarithmic scale of the z-axis.
$ tawk '{ print $timeFirst, $srcIP, $connSip }' annoloc2_flows.txt | t2plot -t "Simple connStat anomaly graph" -sy 0:250 -sx 0:40 -v 60,75 -r 1 -lz
You can now instantly identify the time based evolution of all IP addresses and spot the biggest connecter, get the count range and select him with a simple if clause in an awk or tawk script.
If you use the gpq3x script you can produce an online waterfall plot with the same characteristics. Together with t2 rrd monitoring you then have an efficient online graphical anomaly detection.
Timeline flow analysis
Often typical patterns emerge from the time based flow production. So if IPs stand out in the connStat end report a flow connection diagram can be useful. Just extract the biggest connector from above 138.212.189.66, store the extracted flows in a new file and run the t2timeline script as indicated below.
$ tawk 'host("138.212.189.66")' annoloc2_flows.txt > annoloc2_ip.txt
$ t2timeline -r -ws 700,400 annoloc2_ip.txt
Greens are requesting flows, reds are response flows. The z-axis denotes the flowInd number. If you point the mouse on the beginning of a flow several flow parameters are displayed helping you to identify flows. Maybe there are still too many flows to see something, but you could now select certain protocols, such as TCP or ports, such as port 80. Write a short tawk and rerun the timeline script yourself.
Moreover the timeline graph is very useful to assess the creation of training data for AI. For example, if you have a two class problem, the time lines of all pcaps of the two classes should look similar, if and only if the requirement is that the flows are created by the same equipment and relative timing, certain encrypted content classification task, have these requirements to produce a reasonable classifier. If then the timeline plots differ drastically, you caught somebody producing garbage training data. Because if you use it your classifier will find features, which do not correlate with the problem at hand.
And don’t forget to reset RELTIME to 0 if you intend to do more tutorials, as most of them base on absolute time.
Global statistical plugins
After inspecting the T2 end report, we have a good overview about the pcap state, certain abnormalities and statistics. As each network has its specific protocol statistics, T2 provides several global plugins which produce specific protocols statistics.
protoStat and icmpDecode are standard to be scrutinized after inspecting the end report. protoStat generates annoloc2_protocols.txt which is sorted according to Layer 2-4 protocol numbers.
I unloaded plugins not needed here to reduce the amount of confusing output and loaded icmpDecode and protoStat.
$ t2build -u tcpStates tcpFlags connStat basicStats
$ t2build icmpDecode protoStat
...
$ t2 -r ~/data/annoloc2.pcap -w ~/results
================================================================================
Tranalyzer 0.8.4 (Anteater), Tarantula. PID: 7096
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
01: protoStats, 0.8.4
02: basicFlow, 0.8.4
03: icmpDecode, 0.8.4
04: txtSink, 0.8.4
...
--------------------------------------------------------------------------------
icmpDecode: Number of ICMP echo request packets: 224 [7.32%]
icmpDecode: Number of ICMP echo reply packets: 191 [6.24%]
icmpDecode: ICMP echo reply / request ratio: 0.85
--------------------------------------------------------------------------------
...As you can see icmpDecode produces the important measure of reply / request ratio, for a rapid assessment of malicious activity, also the relative amount of request and reply packets are a valuable indication.
More detailed information about the general picture is provided in the protocols file from protoStat:
$ less annoloc2_protocols.txt
# Total Ether packets captured: 1219015
# L2/3 Protocol Packets Percentage Description
0x0800 1218588 99.965 Internet Protocol version 4 (IPv4)
0x0806 247 0.020 Address Resolution Protocol (ARP)
0x86dd 180 0.015 Internet Protocol version 6 (IPv6)
# Total IPv4 packets captured: 1218588
# Total IPv6 packets captured: 180
# L4 Protocol Packets Percentage Description
1 3059 0.251 Internet Control Message Protocol
2 12 0.001 Internet Group Management Protocol
6 948743 77.844 Transmission Control Protocol
17 266900 21.899 User Datagram Protocol
22 1 0.000 XEROX NS IDP
23 1 0.000 Trunk-1
28 1 0.000 Internet Reliable Transaction
47 20 0.002 General Routing Encapsulation
48 1 0.000 Mobile Host Routing Protocol
58 11 0.001 ICMP for IPv6
59 1 0.000 No Next Header for IPv6
64 1 0.000 SATNET and Backroom EXPAK
...
# Port Packets Percentage Description
13 2 0.000 Daytime (RFC 867)
20 120418 12.692 File Transfer [Default Data]
21 2082 0.219 File Transfer [Control]
22 3793 0.400 The Secure Shell (SSH) Protocol
23 309 0.033 Telnet
25 134 0.014 Simple Mail Transfer
49 175 0.018 Login Host Protocol (TACACS)
53 8 0.001 Domain Name Server
65 13 0.001 TACACS-Database ServiceHere as well the biggest protocol talker is interesting to begin an analysis. The script protStat sorts the protocols file according to number of packets. The -p option defines the lower limit of probability to display, we selected 1%.
$ protStat -p=1 annoloc2_protocols.txt
L2/3 Protocol Packets Probability[%] Description
0x0800 1218588 99.965 Internet Protocol version 4 (IPv4)
L4 Protocol Packets Probability[%] Description
6 948743 77.844 Transmission Control Protocol
17 266900 21.899 User Datagram Protocol
TCP Port Packets Probability[%] Description
139 203627 21.463 NETBIOS Session Service
20 120418 12.692 File Transfer [Default Data]
80 73283 7.724 World Wide Web HTTP
445 27611 2.910 Microsoft-DS
4662 26586 2.802 OrbitNet Message Service
1214 20708 2.183 KAZAA
56071 15851 1.671
56070 15757 1.661
58290 14682 1.548
6699 13711 1.445
81 10937 1.153 Cobalt cube web access or trojan
UDP Port Packets Probability[%] Description
27005 34284 12.845 FLEX LM (1-10)
27960 24798 9.291
7777 15241 5.710 cbt
28920 14301 5.358
10007 11847 4.439 MVS Capacity
27115 11220 4.204
12203 10654 3.992
27963 8591 3.219
28015 8458 3.169
27016 7948 2.978
27116 7508 2.813
27025 7347 2.753
1111 7312 2.740 LM Social Server
28910 6865 2.572
27035 6511 2.439
27961 4869 1.824
7000 3879 1.453 file server itself
28901 3619 1.356
1028 3570 1.338
62626 3364 1.260
61996 3324 1.245
28001 2984 1.118
53 2928 1.097 Domain Name Server
UDP-Lite Port Packets Probability[%] Description
SCTP Port Packets Probability[%] DescriptionWe have 0.25% ICMP traffic, which is not abnormal for that type of traffic. Often it is necessary to look at the ICMP messages in detail because some may indicate problems or even malicious behaviour. For that icmpDecode provides a detailed statistical overview:
$ lsx 22 annoloc2_icmpStats.txt
Total number of ICMP messages: 3070 (3.07 K) [0.25%]
Number of ICMP packets: 3059 (3.06 K) [0.25%]
Number of ICMPv6 packets: 11 [0.00%]
ICMP echo reply / request ratio: 0.853
# ICMP Type Code Packets Percentage
ICMP_ECHOREQUEST - 224 7.323
ICMP_ECHOREPLY - 191 6.244
ICMP_SOURCE_QUENCH - 0 0.000
ICMP_TRACEROUTE - 0 0.000
ICMP_DEST_UNREACH ICMP_NET_UNREACH 0 0.000
ICMP_DEST_UNREACH ICMP_HOST_UNREACH 25 0.817
ICMP_DEST_UNREACH ICMP_PROT_UNREACH 0 0.000
ICMP_DEST_UNREACH ICMP_PORT_UNREACH 2603 85.093
ICMP_DEST_UNREACH ICMP_FRAG_NEEDED 0 0.000
ICMP_DEST_UNREACH ICMP_SR_FAILED 0 0.000
ICMP_DEST_UNREACH ICMP_NET_UNKNOWN 0 0.000
ICMP_DEST_UNREACH ICMP_HOST_UNKNOWN 0 0.000
ICMP_DEST_UNREACH ICMP_HOST_ISOLATED 0 0.000
ICMP_DEST_UNREACH ICMP_NET_ANO 0 0.000
ICMP_DEST_UNREACH ICMP_HOST_ANO 0 0.000
ICMP_DEST_UNREACH ICMP_NET_UNR_TOS 0 0.000
ICMP_DEST_UNREACH ICMP_HOST_UNR_TOS 0 0.000
ICMP_DEST_UNREACH ICMP_PKT_FILTERED 0 0.000
ICMP_DEST_UNREACH ICMP_PREC_VIOLATION 0 0.000
ICMP_DEST_UNREACH ICMP_PREC_CUTOFF 0 0.000
ICMP_REDIRECT ICMP_REDIR_NET 0 0.000
ICMP_REDIRECT ICMP_REDIR_HOST 0 0.000
ICMP_REDIRECT ICMP_REDIR_NETTOS 0 0.000
ICMP_REDIRECT ICMP_REDIR_HOSTTOS 0 0.000
ICMP_TIME_EXCEEDED ICMP_EXC_TTL 14 0.458
ICMP_TIME_EXCEEDED ICMP_EXC_FRAGTIME 2 0.065
# ICMPv6 Type Code Packets Percentage
ICMP6_ECHOREQUEST - 0 0.000
ICMP6_ECHOREPLY - 0 0.000
ICMP6_PKT_TOO_BIG - 0 0.000
ICMP6_DEST_UNREACH ICMP6_NO_ROUTE 0 0.000
ICMP6_DEST_UNREACH ICMP6_COMM_PROHIBIT 0 0.000
ICMP6_DEST_UNREACH ICMP6_BEYOND_SCOPE 0 0.000
ICMP6_DEST_UNREACH ICMP6_ADDR_UNREACH 0 0.000
ICMP6_DEST_UNREACH ICMP6_PORT_UNREACH 0 0.000
ICMP6_DEST_UNREACH ICMP6_SR_FAILED 0 0.000
ICMP6_DEST_UNREACH ICMP6_REJECT 0 0.000
ICMP6_DEST_UNREACH ICMP6_ERROR_HDR 0 0.000
ICMP6_TIME_EXCEEDED ICMP6_EXC_HOPS 0 0.000
ICMP6_TIME_EXCEEDED ICMP6_EXC_FRAGTIME 0 0.000
ICMP6_PARAM_PROBLEM ICMP6_ERR_HDR 0 0.000
ICMP6_PARAM_PROBLEM ICMP6_UNRECO_NEXT_HDR 0 0.000
ICMP6_PARAM_PROBLEM ICMP6_UNRECO_IP6_OPT 0 0.000
ICMP6_RTER_ADVERT - 5 45.455
ICMP6_NBOR_SOLICIT - 3 27.273
ICMP6_NBOR_ADVERT - 3 27.273Now lets find all hosts sending ICMP messages:
$ tawk 'icmp()' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto icmpStat icmpTCcnt icmpBFTypH_TypL_Code icmpTmGtw icmpEchoSuccRatio icmpPFindex
A 59 0x0000000200004001 1022171701.692762 1022171701.692762 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 201.116.148.149 mx "Uninet S.A. de C.V." 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 0
A 896 0x0000000200004001 1022171701.812425 1022171701.812425 0.000000 1 3 eth:ipv4:icmp 00:80:48:d7:ed:7a 00:d0:02:6d:78:00 0x0800 138.212.189.88 jp "ASAHI KASEI CORPORATION" 0 201.116.161.83 mx "Uninet S.A. de C.V." 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 893
A 1073 0x0000000200004001 1022171701.889357 1022171701.889357 0.000000 1 3 eth:ipv4:icmp 00:48:54:7a:04:0f 00:d0:02:6d:78:00 0x0800 138.212.184.71 jp "ASAHI KASEI CORPORATION" 0 146.208.9.41 us "Keysight Technologies" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1056
A 1181 0x0000000200004001 1022171701.956543 1022171701.956543 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:80:48:b3:22:c4 0x0800 201.118.86.105 mx "Uninet S.A. de C.V." 0 138.212.189.66 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000008_0x0002 0x00000000 0 1170
A 1208 0x0000000200004001 1022171701.980834 1022171701.980834 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:80:48:b3:22:c4 0x0800 138.213.40.91 -- "--" 0 138.212.189.66 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1184
A 1236 0x0000000200004001 1022171702.009674 1022171702.009674 0.000000 1 3 eth:ipv4:icmp 00:48:54:7a:04:0f 00:d0:02:6d:78:00 0x0800 138.212.184.71 jp "ASAHI KASEI CORPORATION" 0 36.237.77.156 tw "Data Communication Busi" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1226
A 1561 0x0000000200004001 1022171702.247453 1022171702.247453 0.000000 1 3 eth:ipv4:icmp 00:04:76:22:07:90 00:d0:02:6d:78:00 0x0800 138.212.186.88 jp "ASAHI KASEI CORPORATION" 0 201.19.77.72 br "Telemar Norte Leste S.A" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1559
A 1576 0x0000000200004001 1022171702.265015 1022171702.265015 0.000000 1 3 eth:ipv4:icmp 00:08:a1:1d:3f:f1 00:d0:02:6d:78:00 0x0800 138.212.191.25 jp "ASAHI KASEI CORPORATION" 0 19.50.144.156 us "Ford Motor Company" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1574
A 1721 0x0000000200004001 1022171702.396273 1022171702.396273 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:24:eb 00:d0:02:6d:78:00 0x0800 138.212.190.25 jp "ASAHI KASEI CORPORATION" 0 19.6.20.159 us "Ford Motor Company" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1714
A 1744 0x0000000200004001 1022171702.417049 1022171702.417049 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 65.171.40.80 us "Sprint" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1743
A 1753 0x0000000200004001 1022171702.423157 1022171702.423157 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 193.108.29.243 lv "Infoserv-Riga Ltd" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1751
A 1823 0x0000000200004001 1022171702.510250 1022171702.510250 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 138.213.33.28 -- "--" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1818
A 1880 0x0000000200004000 1022171722.772690 1022171722.785414 0.012724 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:da:37:f6:03 0x0800 193.133.161.22 gb "Verizon UK Limited" 0 138.212.191.75 jp "ASAHI KASEI CORPORATION" 0 1 0x01 9 0x00000000_0x00000008_0x0008 0x00000000 0 7708
B 1880 0x0000000200004001 1022171702.597916 1022171702.597916 0.000000 1 3 eth:ipv4:icmp 00:50:da:37:f6:03 00:d0:02:6d:78:00 0x0800 138.212.191.75 jp "ASAHI KASEI CORPORATION" 0 193.133.161.22 gb "Verizon UK Limited" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1879
A 1908 0x0000000200004001 1022171702.623420 1022171702.623420 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:fc:44:99:fd 0x0800 201.74.106.234 br "CLARO S.A." 0 138.212.187.11 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1875
A 1988 0x0000000200004001 1022171702.721365 1022171702.721365 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:80:48:b3:22:c4 0x0800 139.97.6.149 fi "ELISA" 0 138.212.189.66 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1976
A 1997 0x0000000200004001 1022171702.739522 1022171702.739522 0.000000 1 3 eth:ipv4:icmp 00:80:48:d7:ed:7a 00:d0:02:6d:78:00 0x0800 138.212.189.88 jp "ASAHI KASEI CORPORATION" 0 216.218.79.22 -- "--" 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 1996
A 2038 0x0000000200004001 1022171702.768754 1022171702.768754 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 201.108.14.212 mx "Uninet S.A. de C.V." 0 1 0x01 1 0x00000000_0x00000008_0x0008 0x00000000 0 2012
A 2064 0x0000000200004000 1022171702.799287 1022171702.799287 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:a0:c9:1e:a4:19 0x0800 70.101.52.210 us "Frontier Communications" 0 138.212.184.246 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x99bb0002 1 0
B 2064 0x0000000200004001 1022171702.799877 1022171702.799877 0.000000 1 3 eth:ipv4:icmp 00:a0:c9:1e:a4:19 00:d0:02:6d:78:00 0x0800 138.212.184.246 jp "ASAHI KASEI CORPORATION" 0 70.101.52.210 us "Frontier Communications" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x99bb0002 0 0
A 2065 0x0000000200004000 1022171702.800596 1022171702.800596 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:10:5a:64:e9:36 0x0800 70.101.52.210 us "Frontier Communications" 0 138.212.184.247 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x99bc0002 1 0
B 2065 0x0000000200004001 1022171702.800830 1022171702.800830 0.000000 1 3 eth:ipv4:icmp 00:10:5a:64:e9:36 00:d0:02:6d:78:00 0x0800 138.212.184.247 jp "ASAHI KASEI CORPORATION" 0
...By scrolling to the right you see the icmpBFTypH_TypL_Code bit field. So we are interested in ICMP_HOST_UNREACH and ICMP_PORT_UNREACH. So the 3rd should be 3
$ tawk '{ split($icmpBFTypH_TypL_Code, A, "_"); if (bitsanyset(A[3], 0x3)) print }' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto icmpStat icmpTCcnt icmpBFTypH_TypL_Code icmpTmGtw icmpEchoSuccRatio icmpPFindex
A 1181 0x0000000200004001 1022171701.956543 1022171701.956543 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:80:48:b3:22:c4 0x0800 201.118.86.105 mx "Uninet S.A. de C.V." 0 138.212.189.66 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000008_0x0002 0x00000000 0 1170
A 2064 0x0000000200004000 1022171702.799287 1022171702.799287 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:a0:c9:1e:a4:19 0x0800 70.101.52.210 us "Frontier Communications" 0 138.212.184.246 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x99bb0002 1 0
B 2064 0x0000000200004001 1022171702.799877 1022171702.799877 0.000000 1 3 eth:ipv4:icmp 00:a0:c9:1e:a4:19 00:d0:02:6d:78:00 0x0800 138.212.184.246 jp "ASAHI KASEI CORPORATION" 0 70.101.52.210 us "Frontier Communications" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x99bb0002 0 0
A 2065 0x0000000200004000 1022171702.800596 1022171702.800596 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:10:5a:64:e9:36 0x0800 70.101.52.210 us "Frontier Communications" 0 138.212.184.247 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x99bc0002 1 0
B 2065 0x0000000200004001 1022171702.800830 1022171702.800830 0.000000 1 3 eth:ipv4:icmp 00:10:5a:64:e9:36 00:d0:02:6d:78:00 0x0800 138.212.184.247 jp "ASAHI KASEI CORPORATION" 0 70.101.52.210 us "Frontier Communications" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x99bc0002 0 0
A 2067 0x0000000200004000 1022171702.801985 1022171702.801985 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:fc:0c:d2:07 0x0800 70.101.52.210 us "Frontier Communications" 0 138.212.184.244 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x99bd0002 1 0
B 2067 0x0000000200004001 1022171702.803416 1022171702.803416 0.000000 1 3 eth:ipv4:icmp 00:50:fc:0c:d2:07 00:d0:02:6d:78:00 0x0800 138.212.184.244 jp "ASAHI KASEI CORPORATION" 0 70.101.52.210 us "Frontier Communications" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x99bd0002 0 0
A 2740 0x0000000200004000 1022171703.870541 1022171703.870541 0.000000 1 3 eth:ipv4:icmp 00:50:da:3e:19:97 00:d0:02:6d:78:00 0x0800 138.212.189.44 jp "ASAHI KASEI CORPORATION" 0 201.98.147.38 mx "Uninet S.A. de C.V." 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x00006ac3 1 0
B 2740 0x0000000200004001 1022171703.898733 1022171703.898733 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:da:3e:19:97 0x0800 201.98.147.38 mx "Uninet S.A. de C.V." 0 138.212.189.44 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x00006ac3 0 0
A 2764 0x0000000200004000 1022171703.912653 1022171703.912653 0.000000 1 3 eth:ipv4:icmp 00:50:da:3e:19:97 00:d0:02:6d:78:00 0x0800 138.212.189.44 jp "ASAHI KASEI CORPORATION" 0 217.12.211.19 -- "--" 0 1 0x01 1 0x00000000_0x00000100_0x0001 0x00006ac4 1 0
B 2764 0x0000000200004001 1022171703.918949 1022171703.918949 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:da:3e:19:97 0x0800 217.12.211.19 -- "--" 0 138.212.189.44 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000001_0x0001 0x00006ac4 0 0
A 1581 0x0000000200004000 1022171702.276213 1022171704.296096 2.019883 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:50:04:7f:c6:e4 0x0800 200.83.66.22 cl "VTR BANDA ANCHA S.A." 0 138.212.188.197 jp "ASAHI KASEI CORPORATION" 0 1 0x01 3 0x00000000_0x00000100_0x0001 0x0004cf6c 1 0
B 1581 0x0000000200004001 1022171702.276503 1022171704.296913 2.020410 1 3 eth:ipv4:icmp 00:50:04:7f:c6:e4 00:d0:02:6d:78:00 0x0800 138.212.188.197 jp "ASAHI KASEI CORPORATION" 0 200.83.66.22 cl "VTR BANDA ANCHA S.A." 0 1 0x01 3 0x00000000_0x00000001_0x0001 0x0004cf6c 0 0
A 3112 0x0000000200004001 1022171704.596259 1022171704.596259 0.000000 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:80:48:b3:22:c4 0x0800 200.9.115.105 -- "--" 0 138.212.189.66 jp "ASAHI KASEI CORPORATION" 0 1 0x01 1 0x00000000_0x00000800_0x0001 0x00000000 0 3101
A 3673 0x0000000200004001 1022171706.008768 1022171706.008768 0.000000 1 3 eth:ipv4:icmp 00:10:5a:77:8d:e4 00:d0:02:6d:78:00 0x0800 138.212.190.107 jp "ASAHI KASEI CORPORATION" 0 55.54.217.39 us "Headquarters" 0 1 0x01 1 0x00000000_0x00000008_0x0002 0x00000000 0 2194
A 1750 0x0000100200004000 1022171702.420004 1022171706.422258 4.002254 1 3 eth:ipv4:icmp 00:04:76:25:92:3b 00:d0:02:6d:78:00 0x0800 138.212.189.177 jp "ASAHI KASEI CORPORATION" 0 138.212.109.236 jp "Asahi Kasei Networks Co" 0 1 0x01 5 0x00000000_0x00000100_0x0001 0x00043ce9 1 0
B 1750 0x0000000200004001 1022171702.420110 1022171706.422380 4.002270 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:04:76:25:92:3b 0x0800 138.212.109.236 jp "Asahi Kasei Networks Co" 0 138.212.189.177 jp "ASAHI KASEI CORPORATION" 0 1 0x01 5 0x00000000_0x00000001_0x0001 0x00043ce9 0 0
A 3844 0x0000000200008000 1022171706.464670 1022171706.464670 0.000000 1 3 eth:ipv6:icmpv6 00:80:48:cd:88:83 00:60:08:2c:ca:8e 0x86dd 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 0 fe80::31e1:c7ff:d5fa:684c 12 "Link-local" 0 58 0x01 1 0x00000100_0x00000000_0x0001 0x00000000 0 0
B 3844 0x0000000200008001 1022171706.464331 1022171706.464331 0.000000 1 3 eth:ipv6:icmpv6 00:60:08:2c:ca:8e 00:80:48:cd:88:83 0x86dd fe80::31e1:c7ff:d5fa:684c 12 "Link-local" 0 2001:70e8:d3ce:e200:deaf:b9ff:1f0e:becf -- "--" 0 58 0x01 1 0x00000080_0x00000000_0x0001 0x00000000 0 0
A 1969 0x0000100200004000 1022171702.703042 1022171706.951177 4.248135 1 3 eth:ipv4:icmp 00:01:02:07:2e:fd 00:d0:02:6d:78:00 0x0800 138.212.189.172 jp "ASAHI KASEI CORPORATION" 0 219.41.251.166 -- "--" 0 1 0x01 6 0x00000000_0x00000100_0x0001 0x000259cd 1 0
B 1969 0x0000000200004001 1022171702.709337 1022171706.957427 4.248090 1 3 eth:ipv4:icmp 00:d0:02:6d:78:00 00:01:02:07:2e:fd 0x0800 219.41.251.166 -- "--"
...The bitfields are useful for selecting flows, but if you like a bit more human readability, set ICMP_TC_MD to 0, recompile and rerun T2, as indicated below.
Rebuild icmpDecode and rerun T2.
$ t2build icmpDecode
...
BUILD SUCCESSFUL
$ t2 -r ~/data/annoloc2.pcap -w ~/results
$ tawk '$icmpTCcnt > 0' annoloc2_flows.txt | tcol
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto tcpStates icmpStat icmpTCcnt icmpType_Code icmptmgtw icmpEchoSuccRatio icmpPFindex
A 59 0x0000000200004001 1022171701.692762 1022171701.692762 0.000000 1 3 eth:ipv4:icmp 0 138.212.187.10 jp "asahi kasei corpora" 0 201.116.148.149 mx "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 0
A 896 0x0000000200004001 1022171701.812425 1022171701.812425 0.000000 1 3 eth:ipv4:icmp 0 138.212.189.88 jp "asahi kasei corpora" 0 201.116.161.83 mx "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 893
A 1073 0x0000000200004001 1022171701.889357 1022171701.889357 0.000000 1 3 eth:ipv4:icmp 0 138.212.184.71 jp "asahi kasei corpora" 0 146.208.9.41 us "arin" 0 1 0x00 0x01 1 3_3 0x00000000 0 1056
A 1181 0x0000000200004001 1022171701.956543 1022171701.956543 0.000000 1 3 eth:ipv4:icmp 0 201.118.86.105 mx "--" 0 138.212.189.66 jp "asahi kasei corpora" 0 1 0x00 0x01 1 3_1 0x00000000 0 1170
A 1208 0x0000000200004001 1022171701.980834 1022171701.980834 0.000000 1 3 eth:ipv4:icmp 0 138.213.40.91 ff "apnic" 0 138.212.189.66 jp "asahi kasei corpora" 0 1 0x00 0x01 1 3_3 0x00000000 0 1184
A 1236 0x0000000200004001 1022171702.009674 1022171702.009674 0.000000 1 3 eth:ipv4:icmp 0 138.212.184.71 jp "asahi kasei corpora" 0 36.237.77.156 tw "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 1226
A 1561 0x0000000200004001 1022171702.247453 1022171702.247453 0.000000 1 3 eth:ipv4:icmp 0 138.212.186.88 jp "asahi kasei corpora" 0 201.19.77.72 br "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 1559
A 1576 0x0000000200004001 1022171702.265015 1022171702.265015 0.000000 1 3 eth:ipv4:icmp 0 138.212.191.25 jp "asahi kasei corpora" 0 19.50.144.156 us "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 1574
A 1722 0x0000000200004001 1022171702.396273 1022171702.396273 0.000000 1 3 eth:ipv4:icmp 0 138.212.190.25 jp "asahi kasei corpora" 0 19.6.20.159 us "searched the apnic " 0 1 0x00 0x01 1 3_3 0x00000000 0 1715
A 1745 0x0000000200004001 1022171702.417049 1022171702.417049 0.000000 1 3 eth:ipv4:icmp 0 138.212.187.10 jp "asahi kasei corpora" 0 65.171.40.80 ff "sprint" 0 1 0x00 0x01 1 3_3 0x00000000 0 1744
A 1754 0x0000000200004001 1022171702.423157 1022171702.423157 0.000000 1 3 eth:ipv4:icmp 0 138.212.187.10 jp "asahi kasei corpora" 0 193.108.29.243 lv "ripencc" 0 1 0x00 0x01 1 3_3 0x00000000 0 1752
A 1824 0x0000000200004001 1022171702.510250 1022171702.510250 0.000000 1 3 eth:ipv4:icmp 0 138.212.187.10 jp "asahi kasei corpora" 0 138.213.33.28 ff "apnic" 0 1 0x00 0x01 1 3_3 0x00000000 0 1819
A 1881 0x0000000200004001 1022171722.772690 1022171722.785414 0.012724 1 3 eth:ipv4:icmp 0 193.133.161.22 gb "--" 0 138.212.191.75 jp "asahi kasei corpora" 0 1 0x00 0x01 9 3_3;3_3;3_3;3_3;3_3;3_3;3_3;3_3;3_3 0x00000000 0 7889
B 1881 0x0000000200004001 1022171702.597916 1022171702.597916 0.000000 1 3 eth:ipv4:icmp 0 138.212.191.75 jp "asahi kasei corpora" 0 193.133.161.22 gb "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 1880
A 1909 0x0000000200004001 1022171702.623420 1022171702.623420 0.000000 1 3 eth:ipv4:icmp 0 201.74.106.234 br "--" 0 138.212.187.11 jp "asahi kasei corpora" 0 1 0x00 0x01 1 3_3 0x00000000 0 1876
A 1990 0x0000000200004001 1022171702.721365 1022171702.721365 0.000000 1 3 eth:ipv4:icmp 0 139.97.6.149 fi "elisa oyj" 0 138.212.189.66 jp "asahi kasei corpora" 0 1 0x00 0x01 1 3_3 0x00000000 0 1978
A 1999 0x0000000200004001 1022171702.739522 1022171702.739522 0.000000 1 3 eth:ipv4:icmp 0 138.212.189.88 jp "asahi kasei corpora" 0 216.218.79.22 us "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 1998
A 2040 0x0000000200004001 1022171702.768754 1022171702.768754 0.000000 1 3 eth:ipv4:icmp 0 138.212.187.10 jp "asahi kasei corpora" 0 201.108.14.212 mx "--" 0 1 0x00 0x01 1 3_3 0x00000000 0 2014
...Add layer 2/4 information
Information about MACs and ports which helps you decoding certain number can be added:
- macRecorder (records all mac pairs during a connection, pkt counts and MAC decoding)
- portClassifer (human readable ports)
Unload icmpDecode and load both plugins
$ t2build -u icmpDecode
...
$ t2build macRecorder portClassifier
...
BUILD SUCCESSFUL
$ t2 -r ~/data/annoloc2.pcap -w ~/results
...
$In the flow file below you will now see from the macRecorder plugin all MAC addresses including packet counts per flow. If redundant routing is presents you will see minimum two MAC pairs per flow. It is also useful to detect broken network cards, then you see several random MAC pairs. In the case of redundant routing the packet counts should be almost equal, if not then something is wrong. Moreover the manufacturer of the interface card is listed, so that the user does not need to look it up on the web. The portClassifier is somewhat misleading, as it does not really classifies, but instead transforms the port number into a human readable string, such as https for port 443 in our case.
$ tcol annoloc2_flows.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType ethVlanID srcIP srcIPCC srcIPWho srcPort dstIP dstIPCC dstIPWho dstPort l4Proto macPairs srcMac_dstMac_numP srcManuf_dstManuf dstPortClassN dstPortClass
A 59 0x0000000200004000 1022171701.692762 1022171701.692762 0.000000 1 3 eth:ipv4:icmp 00:80:48:b3:22:ef 00:d0:02:6d:78:00 0x0800 138.212.187.10 jp "ASAHI KASEI CORPORATION" 0 201.116.148.149 mx "Uninet S.A. de C.V." 0 1 1 00:80:48:b3:22:ef_00:d0:02:6d:78:00_1 CompexUs_Ditech 0 unknown
A 108 0x0000000200004000 1022171701.700133 1022171701.700133 0.000000 1 3 eth:ipv4:udp 00:00:1c:b6:1a:53 00:d0:02:6d:78:00 0x0800 138.212.184.165 jp "ASAHI KASEI CORPORATION" 8889 19.112.107.128 us "Ford Motor Company" 2001 17 1 00:00:1c:b6:1a:53_00:d0:02:6d:78:00_1 BellTech_Ditech 2001 wizard
A 138 0x0000000000004000 1022171701.700983 1022171701.700983 0.000000 1 3 eth:ipv4:tcp 00:01:02:b8:58:8a 00:d0:02:6d:78:00 0x0800 138.212.189.36 jp "ASAHI KASEI CORPORATION" 1044 205.25.217.73 us "Registration" 29981 6 1 00:01:02:b8:58:8a_00:d0:02:6d:78:00_1 3Com_Ditech 29981 unknown
A 193 0x0000000000004000 1022171701.704267 1022171701.704267 0.000000 1 3 eth:ipv4:tcp 00:48:54:7a:06:6a 00:d0:02:6d:78:00 0x0800 138.212.190.87 jp "ASAHI KASEI CORPORATION" 1068 70.128.194.122 us "AT&T Corp." 1863 6 1 00:48:54:7a:06:6a_00:d0:02:6d:78:00_1 DigitalS_Ditech 1863 msnp
A 245 0x0000000200004000 1022171701.706591 1022171701.706591 0.000000 1 3 eth:ipv4:udp 00:04:76:24:0e:f4 00:d0:02:6d:78:00 0x0800 138.212.188.99 jp "ASAHI KASEI CORPORATION" 7778 83.221.58.33 -- "--" 2009 17 1 00:04:76:24:0e:f4_00:d0:02:6d:78:00_1 3Com_Ditech 2009 whosockami
A 262 0x0000080200028000 1022171701.707777 1022171701.707777 0.000000 1 4 eth:ipv4:ipv6:UNK(168) 00:d0:02:6d:78:00 00:60:08:2c:ca:8e 0x86dd cfb6:1c18:5010:faf0:7f66:0:101:80a -- "--" 0 6c2:6a7f:1:384b::c100 -- "--" 0 168 1 00:d0:02:6d:78:00_00:60:08:2c:ca:8e_1 Ditech_3com 0 unknown
A 103 0x0000000200004000 1022171701.699999 1022171701.847857 0.147858 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:d0:b7:e8:9e:bb 0x0800 200.8.254.121 ve "Corporación Telemic C." 1174 138.212.190.162 jp "ASAHI KASEI CORPORATION" 6020 6 1 00:d0:02:6d:78:00_00:d0:b7:e8:9e:bb_2 Ditech_Intel 6020 x11
B 103 0x0000000200004001 1022171701.707779 1022171701.709106 0.001327 1 3 eth:ipv4:tcp 00:d0:b7:e8:9e:bb 00:d0:02:6d:78:00 0x0800 138.212.190.162 jp "ASAHI KASEI CORPORATION" 6020 200.8.254.121 ve "Corporación Telemic C." 1174 6 1 00:d0:b7:e8:9e:bb_00:d0:02:6d:78:00_2 Intel_Ditech 6020 x11
A 267 0x0000000000004000 1022171701.709116 1022171701.709116 0.000000 1 3 eth:ipv4:tcp 00:d0:02:6d:78:00 00:50:fc:0e:21:56 0x0800 209.171.12.143 ca "TELUS Communications In" 4987 138.212.185.230 jp "ASAHI KASEI CORPORATION" 41250 6 1 00:d0:02:6d:78:00_00:50:fc:0e:21:56_1 Ditech_EdimaxTe 41250 unknown
...Now you got a quick insight into basic plugins. You can now start using T2 on your own pcaps or look into other tutorials about specifics of traffic mining, or specific plugins. Have fun!!
Operational mode switching: ETH, IPv4/6, SCTP
T2 can operate in several operational modes. default is dual IP stack + L2 Ethernet flow production. In order to accelerate T2 it can be switched into IPv4 or IPv6 mode or only into a plain L2 flow/packet producer depending on your demands or your network.
Search for user defines in networkHeaders.h and have a look at the default settings:
// Constant Definition
// user defines
#define IPV6_ACTIVATE 2 // 0: IPv4 only, 1: IPv6 only, 2: dual mode
#define ETH_ACTIVATE 1 // 0: No Ethernet flows,
// 1: Activate Ethernet flows,
// 2: Also use Ethernet addresses for IPv4/6 flows
#define SCTP_ACTIVATE 0 // 1: activate SCTP streams -> Flows
#define SCTP_STATFINDEX 1 // 1: if SCTP_ACTIVATE = 1 then (1: findex constant for all SCTP streams in a packet 0: findex increments)
#define MULTIPKTSUP 0 // multi-packet suppression
#define T2_PRI_HDRDESC 1 // 1: keep track of the headers traversed
#define T2_HDRDESC_AGGR 1 // 1: aggregate repetitive headers, e.g., vlan{2}
#define T2_HDRDESC_LEN 128 // max length of the headers descriptionMoreover SCTP to flow transformation is supported. Which is by default disabled, because it requires additional code, the standard admin does not need. The researcher or protocol expert might need that functionality, so set SCTP_ACTIVATE to 1. The constant SCTP_STATFINDEX controls whether all SCTP streams sorted into several flows with the same flow Index or different incrementing flow indexes.
compile all plugins, as you may have plugins which implement the SCTP flow segregation, e.g. sctpDecode.
$ t2build -R
...
$and run T2 with your SCTP pcap or run it on an interface where SCTP traffic is present. If you are more interested in SCTP check out the sctp tutorial.
Now you got a quick insight into T2 functionality, basic plugin operations and workflow. You can now start using T2 on your own pcaps or look into other tutorials about specifics of your interest. Have fun!!