DPI: Deep Packet Inspection
DPI nDPI
Introduction
This tutorial details the different features of T2 concerning Deep Packet Inspection (DPI) T2 implements a wrapper for the well known nDPI being widely used by researchers and technicians. Hence, T2 provides the user with a highly effective selection mechanism based on L7 applications. So producing training and test files for AI experiments is now very easy.
Prerequisites
For this tutorial, it is assumed the user has a basic knowledge of Tranalyzer and that the file t2_aliases has been sourced in ~/.bashrc or ~/.bash_aliases as follows (Refer to How to install Tranalyzer for more details):
# $HOME/.bashrc
if [ -f "$T2HOME/scripts/t2_aliases" ]; then
. "$T2HOME/scripts/t2_aliases" # Note the leading '.'
fiMake sure to replace $T2HOME with the actual path, e.g., $HOME/tranalyzer2-0.9.1):
Dependencies
LGPL Crypto library:
Ubuntu sudo apt-get install libgcrypt20-devArch sudo pacman -S libgcryptGentoo sudo emerge libgcryptopenSUSE sudo zypper install libgcrypt-develRed Hat/Fedora sudo dnf install libgcrypt-develorsudo yum install libgcrypt-develmacOS brew install libgcrypt
Preparation
First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:
t2build -e -y
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied
Then compile the following plugins:
t2build tranalyzer2 basicFlow tcpStates portClassifier nDPI txtSink
...
BUILD SUCCESSFUL
If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:
mkdir ~/data ~/results
The anonymized sample PCAP used in this tutorial can be downloaded here: faf-exercise.pcap.
Please save it in your ~/data folder.
Now you are all set for T2 flow based nDPI experiments.
Flow and packet based nDPI
For network admins or researchers the L7 type of the traffic is great interest. So to select flows with this feature makes it very easy to weed out (un)interesting traffic, reduce the amount of flows or label flows for later AI training and testing. For the latter the nDPI plugin supplies beside the human readable also a numerical output.
To begin let us look into the directory.
nDPI
ls
autogen.sh clean.sh configure.ac COPYING default.config doc Makefile.am meson.build new_ndpi_prepatch.sh prototex README src t2plconf tests
Important to note is the new_ndpi_prepatch.sh script, which fetches the newest version of nDPI. Please refer to the documentation under the doc folder. Now let us look into the src directory
ls src/
Makefile.am nDPI nDPI.c nDPI.h
Besides the nDPI plugin files there is the nDPI/ folder with all the C code and libraries from the open source. Open the nDPI.h file to look at the config.
vi src/nDPI.h
...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
#define NDPI_OUTPUT_NUM 0 // Output a numerical classification
#define NDPI_OUTPUT_STR 1 // Output a textual classification
#define NDPI_OUTPUT_STATS 1 // Output nDPI protocol distribution in a separate file
#define NDPI_GUESS_UNKNOWN 1 // Try guessing protocol if not sure on flow terminate
/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */
/* No env / runtime configuration flags available for nDPI */
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...We leave the numerical classification off, which is useful for machine learning, as we like to compare nDPI to the L4 ports meaning provided by portClassifier. But if you like you can switch it on. If nDPI is not sure about the classification T2 helps a bit on flow terminate. This feature is enabled by default. If you changed the config, you need to rebuild nDPI, otherwise you can run T2 right away:
t2 -s -r ~/data/faf-exercise.pcap -w ~/results================================================================================ Tranalyzer 0.9.1 (Anteater), Cobra. PID: 22346, SID: 666 ================================================================================ [INF] Creating flows for L2, IPv4, IPv6 Active plugins: 01: basicFlow, 0.9.1 02: portClassifier, 0.9.1 03: nDPI, 0.9.1 04: tcpStates, 0.9.1 05: txtSink, 0.9.1 [INF] IPv4 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 481438 (481.44 K) [INF] IPv6 Ver: 5, Rev: 09082023, Range Mode: 0, subnet ranges loaded: 41486 (41.49 K) Processing file: /home/user/data/faf-exercise.pcap Link layer type: Ethernet [EN10MB/1] Snapshot length: 65535 (65.53 K) Dump start: 1258544215.037210000 sec (Wed 18 Nov 2009 11:36:55 GMT) Dump stop : 1258594491.683288000 sec (Thu 19 Nov 2009 01:34:51 GMT) Total dump duration: 50276.646078000 sec (13h 57m 56s) Finished processing. Elapsed time: 0.146227673 sec Finished unloading flow memory. Time: 0.146245162 sec Percentage completed: 100.00% Number of processed packets: 5902 (5.90 K) Number of processed bytes: 4993414 (4.99 M) Number of raw bytes: 4993414 (4.99 M) Number of pcap bytes: 5087870 (5.09 M) Number of IPv4 packets: 5902 (5.90 K) [100.00%] Number of A packets: 1986 (1.99 K) [33.65%] Number of B packets: 3916 (3.92 K) [66.35%] Number of A bytes: 209315 (209.31 K) [4.19%] Number of B bytes: 4784099 (4.78 M) [95.81%] <A packet load>: 105.40 <B packet load>: 1221.68 (1.22 K) -------------------------------------------------------------------------------- nDPI: Number of flows classified: 72 [100.00%] tcpStates: Aggregated tcpStatesAFlags=0x4a -------------------------------------------------------------------------------- Headers count: min: 3, max: 3, avg: 3.00 Number of TCP packets: 5902 (5.90 K) [100.00%] Number of TCP bytes: 4993414 (4.99 M) [100.00%] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Number of processed flows: 72 Number of processed IPv4 flows: 72 [100.00%] Number of processed A flows: 36 [50.00%] Number of processed B flows: 36 [50.00%] Number of request flows: 36 [50.00%] Number of reply flows: 36 [50.00%] Total A/B flow asymmetry: 0.00 Total req/rply flow asymmetry: 0.00 Number of processed A+B packets/A+B flows: 81.97 Number of processed A packets/A flows: 55.17 Number of processed B packets/ B flows: 108.78 Number of processed total packets/s: 0.12 Number of processed A+B packets/s: 0.12 Number of processed A packets/s: 0.04 Number of processed B packets/s: 0.08 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <Number of processed flows/s>: 0.00 <Bandwidth>: 792 b/s <Raw bandwidth>: 795 b/s Max number of flows in memory: 18 [0.01%] Memory usage: 0.05 GB [0.07%] Aggregated flowStat=0x0400000000004000 [INF] IPv4 flows
Open the flow file in your results folder. The end report states that nDPI was able to classify all flows. So let’s look into the flow file.
tcol ~/data/faf-exercise_flow.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType vlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto dstPortClassN dstPortClass nDPIclass tcpStatesAFlags
A 1 0x0400000000004000 1258544215.037210000 1258544215.372742000 0.335532000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1258 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 1 0x0400000000004001 1258544215.202900000 1258544215.537951000 0.335051000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1258 6 80 http "HTTP" 0x00
A 2 0x0400000000004000 1258544216.385370000 1258544216.723144000 0.337774000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1259 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 2 0x0400000000004001 1258544216.551313000 1258544216.888595000 0.337282000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1259 6 80 http "HTTP" 0x00
A 3 0x0400000000004000 1258544216.908284000 1258544217.008468000 0.100184000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1260 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 3 0x0400000000004001 1258544216.915576000 1258544217.008019000 0.092443000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 80 http "HTTP" 0x00
A 4 0x0400000000004000 1258544217.003718000 1258544217.348506000 0.344788000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1261 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 4 0x0400000000004001 1258544217.169421000 1258544217.513942000 0.344521000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1261 6 80 http "HTTP" 0x00
A 5 0x0400000000004000 1258544217.349751000 1258544217.413719000 0.063968000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1262 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 5 0x0400000000004001 1258544217.357036000 1258544217.413505000 0.056469000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1262 6 80 http "HTTP" 0x00
A 6 0x0400000000004000 1258544217.408963000 1258544217.754495000 0.345532000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1263 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 6 0x0400000000004001 1258544217.574652000 1258544217.919686000 0.345034000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1263 6 80 http "HTTP" 0x00
A 7 0x0400000000004000 1258544217.755746000 1258544217.791475000 0.035729000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1264 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 7 0x0400000000004001 1258544217.763049000 1258544217.791016000 0.027967000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1264 6 80 http "HTTP" 0x00
A 8 0x0400000000004000 1258544217.786474000 1258544218.129260000 0.342786000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1265 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 8 0x0400000000004001 1258544217.952162000 1258544218.294696000 0.342534000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1265 6 80 http "HTTP" 0x00
A 9 0x0400000000004000 1258544218.130258000 1258544218.166240000 0.035982000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1266 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 9 0x0400000000004001 1258544218.137543000 1258544218.165782000 0.028239000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1266 6 80 http "HTTP" 0x00
A 11 0x0400000000004000 1258562478.204828000 1258562509.633367000 31.428539000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1384 63.245.221.11 us "Mozilla" 80 6 80 http "HTTP" 0x00
B 11 0x0400000000004001 1258562478.266384000 1258562509.653978000 31.387594000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla" 80 192.168.1.104 07 "Private network" 1384 6 80 http "HTTP" 0x00
A 10 0x0400000000004000 1258562467.749142000 1258562509.633370000 41.884228000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1379 63.245.221.11 us "Mozilla" 80 6 80 http "HTTP" 0x00
B 10 0x0400000000004001 1258562467.754689000 1258562509.653962000 41.899273000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 63.245.221.11 us "Mozilla" 80 192.168.1.104 07 "Private network" 1379 6 80 http "HTTP" 0x00
A 12 0x0400000000004000 1258563573.941668000 1258563576.594009000 2.652341000 1 3 eth:ipv4:tcp 00:0b:db:63:5b:d4 00:19:e3:e7:5d:23 0x0800 192.168.1.103 07 "Private network" 1397 192.168.1.1 07 "Private network" 25 6 25 smtp "SMTP" 0x00
B 12 0x0400000000004001 1258563573.941709000 1258563576.594045000 2.652336000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:63:5b:d4 0x0800 192.168.1.1 07 "Private network" 25 192.168.1.103 07 "Private network" 1397 6 25 smtp "SMTP" 0x08
...Same for the packet file:
tcol ~/data/faf-exercise_flow.txt
%dir flowInd flowStat timeFirst timeLast duration numHdrDesc numHdrs hdrDesc srcMac dstMac ethType vlanID srcIP srcIPCC srcIPOrg srcPort dstIP dstIPCC dstIPOrg dstPort l4Proto dstPortClassN dstPortClass nDPIclass tcpStatesAFlags
A 1 0x0400000000004000 1258544215.037210000 1258544215.372742000 0.335532000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1258 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 1 0x0400000000004001 1258544215.202900000 1258544215.537951000 0.335051000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1258 6 80 http "HTTP" 0x00
A 2 0x0400000000004000 1258544216.385370000 1258544216.723144000 0.337774000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1259 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 2 0x0400000000004001 1258544216.551313000 1258544216.888595000 0.337282000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1259 6 80 http "HTTP" 0x00
A 3 0x0400000000004000 1258544216.908284000 1258544217.008468000 0.100184000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1260 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 3 0x0400000000004001 1258544216.915576000 1258544217.008019000 0.092443000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1260 6 80 http "HTTP" 0x00
A 4 0x0400000000004000 1258544217.003718000 1258544217.348506000 0.344788000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1261 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 4 0x0400000000004001 1258544217.169421000 1258544217.513942000 0.344521000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1261 6 80 http "HTTP" 0x00
A 5 0x0400000000004000 1258544217.349751000 1258544217.413719000 0.063968000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1262 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 5 0x0400000000004001 1258544217.357036000 1258544217.413505000 0.056469000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1262 6 80 http "HTTP" 0x00
A 6 0x0400000000004000 1258544217.408963000 1258544217.754495000 0.345532000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1263 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 6 0x0400000000004001 1258544217.574652000 1258544217.919686000 0.345034000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1263 6 80 http "HTTP" 0x00
A 7 0x0400000000004000 1258544217.755746000 1258544217.791475000 0.035729000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1264 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 7 0x0400000000004001 1258544217.763049000 1258544217.791016000 0.027967000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1264 6 80 http "HTTP" 0x00
A 8 0x0400000000004000 1258544217.786474000 1258544218.129260000 0.342786000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1265 77.67.44.206 gb "Akamai Technologies" 80 6 80 http "HTTP" 0x00
B 8 0x0400000000004001 1258544217.952162000 1258544218.294696000 0.342534000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 77.67.44.206 gb "Akamai Technologies" 80 192.168.1.104 07 "Private network" 1265 6 80 http "HTTP" 0x00
A 9 0x0400000000004000 1258544218.130258000 1258544218.166240000 0.035982000 1 3 eth:ipv4:tcp 00:0b:db:4f:6b:10 00:19:e3:e7:5d:23 0x0800 192.168.1.104 07 "Private network" 1266 198.189.255.75 us "California State University" 80 6 80 http "HTTP" 0x00
B 9 0x0400000000004001 1258544218.137543000 1258544218.165782000 0.028239000 1 3 eth:ipv4:tcp 00:19:e3:e7:5d:23 00:0b:db:4f:6b:10 0x0800 198.189.255.75 us "California State University" 80 192.168.1.104 07 "Private network" 1266 6 80 http "HTTP" 0x00
...If you scroll to the right you will notice the nDPIclass output classifying the traffic.
For this simple traffic type most of the ports match actually the meaning except for the FTP-Data flow.
As NDPI_OUTPUT_STATS is enabled nDPI supplies a separate traffic type statistics file shown below.
tcol ~/results/faf-exercise_nDPI.txt
# Protocol ID Packets Bytes Description
1 22 [ 0.37%] 2595 [ 0.05%] FTP_CONTROL
3 894 [ 15.15%] 148980 [ 2.98%] SMTP
7 742 [ 12.57%] 608762 [ 12.19%] HTTP
175 4615 [ 78.19%] 4537458 [ 90.87%] FTP_DATAThe file can be sorted and manipulated with the protStat script.
To sort the file by packets, run:
protStat ~/results/faf-exercise_nDPI.txt
Protocol ID Packets Bytes Description
175 4615 [ 78.19%] 4537458 [ 90.87%] FTP_DATA
3 894 [ 15.15%] 148980 [ 2.98%] SMTP
7 742 [ 12.57%] 608762 [ 12.19%] HTTP
1 22 [ 0.37%] 2595 [ 0.05%] FTP_CONTROLAnd to sort it by bytes, run it with the -b option as follows:
protStat -b ~/results/faf-exercise_nDPI.txt | tcol
Protocol ID Packets Bytes Description
175 4615 [ 78.19%] 4537458 [ 90.87%] FTP_DATA
7 742 [ 12.57%] 608762 [ 12.19%] HTTP
3 894 [ 15.15%] 148980 [ 2.98%] SMTP
1 22 [ 0.37%] 2595 [ 0.05%] FTP_CONTROLNow run t2 on the other pcaps or your own ones and see how nDPI performs.
It still has its problems with encryption, as expected.
Conclusion
If you changed the configuration of the plugin, do not forget to reset it for the following tutorials:
t2conf --reset nDPI && t2build -R
Have fun!