Tutorial: IP anonymization

Sometimes it is necessary to hide your personal or public addresses from anybody you share flow or packet files with. T2 implements a very simplistic mechanism for this purpose, it just blocks the output in the specific files.

Preparation

Before we start we need to prepare T2. If you did not complete the tutorials before just follow the procedure described below.

First I recommend to set T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins. Just as a precaution if you have some old plugins or files there. If you like to keep them, please copy them away.

$ t2build -e
Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? y
Plugin folder emptied
$
$ t2build tranalyzer2 basicFlow basicStats connStat txtSink
...
BUILD SUCCESSFUL

$

If you did not create a separate data and results directory yet, please do it now in another cmd window, that facilitates your workflow:

$ mkdir ~/data ~/results
$

Download the sample pcap if did not do it already: faf-exercise.pcap. Now you’re all set.

ANONYM_IP

The control for IP anonymization is located in the core, tranalyzer.h. You can see the current value using t2conf -G option:

$ t2conf tranalyzer2 -G ANONYM_IP
ANONYM_IP = 0
$ 

set it to 1, recompile the core and all plugins and run t2 on the pcap:

$ t2conf tranalyzer2 -D ANONYM_IP=1 && t2build -R
...
$ t2 -r ~/data/faf-exercise.pcap -w ~/results/ -s
================================================================================
Tranalyzer 0.8.9 (Anteater), Tarantula. PID: 22207
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.9
    02: basicStats, 0.8.9
    03: connStat, 0.8.9
    04: txtSink, 0.8.9
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406027 (406.03 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 50973 (50.97 K)
Processing file: /home/stefan/tranalyzer-website/tranalyzer/download/data/faf-exercise.pcap
Link layer type: Ethernet [EN10MB/1]
Dump start: 1258544215.037210 sec (Wed 18 Nov 2009 11:36:55 GMT)
Dump stop : 1258594491.683288 sec (Thu 19 Nov 2009 01:34:51 GMT)
Total dump duration: 50276.646078 sec (13h 57m 56s)
Finished processing. Elapsed time: 0.170411 sec
Finished unloading flow memory. Time: 0.170437 sec
Percentage completed: 100.00%
Number of processed packets: 5902 (5.90 K)
Number of processed bytes: 4993414 (4.99 M)
Number of raw bytes: 4993414 (4.99 M)
Number of pcap bytes: 5087870 (5.09 M)
Number of IPv4 packets: 5902 (5.90 K) [100.00%]
Number of A packets: 1986 (1.99 K) [33.65%]
Number of B packets: 3916 (3.92 K) [66.35%]
Number of A bytes: 209315 (209.31 K) [4.19%]
Number of B bytes: 4784099 (4.78 M) [95.81%]
Average A packet load: 105.40
Average B packet load: 1221.68 (1.22 K)
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
connStat: Number of unique source IPs: 21
connStat: Number of unique destination IPs: 19
connStat: Number of unique source/destination IPs connections: 10
connStat: Max unique number of source IP / destination port connections: 18
connStat: IP prtcon/sdcon, prtcon/scon: 1.800000, 0.857143
connStat: Source IP with max connections: 192.168.1.104: 2 connections
connStat: Destination IP with max connections: 192.168.1.1: 2 connections
--------------------------------------------------------------------------------
Headers count: min: 3, max: 3, average: 3.00
Number of TCP packets: 5902 (5.90 K) [100.00%]
Number of TCP bytes: 4993414 (4.99 M) [100.00%]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of processed   flows: 73
Number of processed A flows: 37 [50.68%]
Number of processed B flows: 36 [49.32%]
Number of request     flows: 36 [49.32%]
Number of reply       flows: 37 [50.68%]
Total   A/B    flow asymmetry: 0.01
Total req/rply flow asymmetry: -0.01
Number of processed   packets/flows: 80.85
Number of processed A packets/flows: 53.68
Number of processed B packets/flows: 108.78
Number of processed total packets/s: 0.12
Number of processed A+B   packets/s: 0.12
Number of processed A     packets/s: 0.04
Number of processed   B   packets/s: 0.08
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of average processed flows/s: 0.00
Average full raw bandwidth: 795 b/s
Average full bandwidth : 792 b/s
Max number of flows in memory: 18 [0.01%]
Memory usage: 0.03 GB [0.05%]
Aggregate flow status: 0x0400000000004000
[INF] IPv4 flows
$

So you still see IP addresses in the report summary, in the flow and packet files all IP addresses, teredo, L2TP or GRE entries are also removed. Yes it is coarse, but if will be more elaborate in the future.

$ cd ~/results
$ tcol faf-exercise_flows.txt
%dir  flowInd  flowStat            timeFirst          timeLast           duration   numHdrDesc  numHdrs  hdrDesc       srcMac             dstMac             ethType  ethVlanID  srcPort  dstPort  l4Proto  numPktsSnt  numPktsRcvd  numBytesSnt  numBytesRcvd  minPktSz  maxPktSz  avePktSize  stdPktSize  minIAT  maxIAT    aveIAT        stdIAT       pktps      bytps     pktAsm       bytAsm       connSip  connDip  connSipDip  connSipDprt  connF
A     1        0x0400000000004000  1258544215.037210  1258544215.372742  0.335532   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1258     80       6        6           7            367          3547          0         367       61.16667    113.5846    0       0.166651  0.055922      0.06351066   17.88205   1093.785  -0.07692308  -0.8124681   2        1        10          18           9
B     1        0x0400000000004001  1258544215.202900  1258544215.537951  0.335051   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1258     6        7           6            3547         367           0         1380      506.7143    474.1748    0       0.167371  0.04786443    0.06323512   20.89234   10586.45  0.07692308   0.8124681    1        2        9           17           17
A     2        0x0400000000004000  1258544216.385370  1258544216.723144  0.337774   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1259     80       6        5           4            322          464           0         322       64.4        106.4982    0       0.16665   0.06755479    0.06471679   14.8028    953.3001  0.1111111    -0.1806616   2        1        8           16           8
B     2        0x0400000000004001  1258544216.551313  1258544216.888595  0.337282   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1259     6        4           5            464          322           0         464       116         150.9982    0       0.169645  0.0843205     0.0658242    11.85951   1375.703  -0.1111111   0.1806616    1        2        7           15           15
A     3        0x0400000000004000  1258544216.908284  1258544217.008468  0.100184   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1260     80       6        18          73           319          95603         0         319       17.72222    61.73868    0       0.013738  0.005565778   0.00363383   179.6694   3184.141  -0.6043956   -0.9933488   2        1        8           14           7
B     3        0x0400000000004001  1258544216.915576  1258544217.008019  0.092443   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1260     6        73          18           95603        319           0         1380      1309.63     274.7284    0       0.021251  0.001266342   0.003059902  789.6758   1034183   0.6043956    0.9933488    1        2        7           13           13
A     4        0x0400000000004000  1258544217.003718  1258544217.348506  0.344788   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1261     80       6        5           4            323          466           0         323       64.6        106.829     0       0.166899  0.0689576     0.06384765   14.50166   936.8075  0.1111111    -0.1812421   2        1        6           12           6
B     4        0x0400000000004001  1258544217.169421  1258544217.513942  0.344521   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1261     6        4           5            466          323           0         466       116.5       151.649     0       0.177128  0.08613025    0.06695305   11.61032   1352.603  -0.1111111   0.1812421    1        2        5           11           11
A     5        0x0400000000004000  1258544217.349751  1258544217.413719  0.063968   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1262     80       6        9           26           320          30820         0         320       35.55556    84.1992     0       0.009743  0.007107555   0.002012674  140.6953   5002.501  -0.4857143   -0.9794477   2        1        6           10           5
B     5        0x0400000000004001  1258544217.357036  1258544217.413505  0.056469   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1262     6        26          9            30820        320           0         1380      1185.385    425.1664    0       0.017243  0.002171885   0.00369694   460.4296   545786.2  0.4857143    0.9794477    1        2        5           9            9
A     6        0x0400000000004000  1258544217.408963  1258544217.754495  0.345532   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1263     80       6        5           4            320          460           0         320       64          105.8368    0       0.166898  0.0691064     0.06375818   14.47044   926.1082  0.1111111    -0.1794872   2        1        4           8            4
B     6        0x0400000000004001  1258544217.574652  1258544217.919686  0.345034   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1263     6        4           5            460          320           0         460       115         149.6965    0       0.177889  0.0862585     0.06702126   11.59306   1333.202  -0.1111111   0.1794872    1        2        3           7            7
A     7        0x0400000000004000  1258544217.755746  1258544217.791475  0.035729   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1264     80       6        6           7            317          5268          0         317       52.83333    98.10986    0       0.010241  0.005954833   0.002865114  167.9308   8872.345  -0.07692308  -0.8864816   2        1        4           6            3
B     7        0x0400000000004001  1258544217.763049  1258544217.791016  0.027967   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1264     6        7           6            5268         317           0         1380      752.5714    567.7107    0       0.017745  0.003995285   0.005150571  250.295    188364.9  0.07692308   0.8864816    1        2        3           5            5
A     8        0x0400000000004000  1258544217.786474  1258544218.129260  0.342786   1           3        eth:ipv4:tcp  00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800              1265     80       6        5           4            323          466           0         323       64.6        106.829     0       0.16665   0.0685572     0.06394027   14.58636   942.2788  0.1111111    -0.1812421   2        1        2           4            2
B     8        0x0400000000004001  1258544217.952162  1258544218.294696  0.342534   1           3        eth:ipv4:tcp  00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800              80       1265     6        4           5            466          323           0         466       116.5       151.649     0       0.175146  0.08563349    0.06663121   11.67767   1360.449  -0.1111111   0.1812421    1        2        1           3            3

Same applies to the packet file.

$ tcol faf-exercise_packets.txt
%pktNo  flowInd  flowStat            time               pktIAT     flowDuration  numHdrs  hdrDesc       ethVlanID  srcMac             dstMac             ethType  srcPort  dstPort  l4Proto  pktLen  l7Len  l7Content
1       1        0x0400000000004000  1258544215.037210  0.000000   0.000000      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   1258     80       6        66      0      
2       1        0x0400000000004001  1258544215.202900  0.000000   0.000000      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   80       1258     6        62      0      
3       1        0x0400000000004000  1258544215.203358  0.166148   0.166148      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   1258     80       6        64      0      
4       1        0x0400000000004000  1258544215.203850  0.000492   0.166640      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   1258     80       6        425     367    GET /softw/90/update/avg9infoavi.ctf HTTP/1.1\r\nUser-Agent: AVGINET9-WXPPX86 90 AVI=270.14.71/2510 BUILD=707 LOC=1033 LIC=9I-ASXNN-X4WGW-M0XFR-T84VX-3VX02 DIAG=51E OPF=0 PCA=\r\nHost: backup.avg.cz\r\nAccept: */*\r\nAccept-Encoding: identity,deflate,gzip\r\nIf-Modified-Since: Tue, 17 Nov 2009 20:41:10 GMT\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nx-avg-id:78-175947826\r\n\r\n
5       1        0x0400000000004001  1258544215.370055  0.167155   0.167155      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   80       1258     6        1434    1380   HTTP/1.1 200 OK\r\nDate: Wed, 18 Nov 2009 11:39:48 GMT\r\nServer: Apache\r\nLast-Modified: Wed, 18 Nov 2009 09:04:15 GMT\r\nETag: "15c007-cea-478a186e401c0"\r\nAccept-Ranges: bytes\r\nContent-Length: 3306\r\nConnection: close\r\nContent-Type: text/plain\r\n\r\nAVG CTF Index File;ver(10)\r\nbin(u7avi1777ff.bin)grp(avi:1777)tm(0911180855)pri(2)len(6637487)\r\nbin(u7avi1777u1323ff.bin)grp(avi:1777)dep(avi:1323)tm(0911180855)pri(2)len(590030)\r\nbin(u7avi1777u1705ff.bin)grp(avi:1777)dep(avi:1705)tm(0911180855)pri(2)len(95323)\r\nbin(u7iavi2511ff.bin)grp(iavi:2511)tm(0911180855)pri(2)len(45641416)\r\nbin(u7iavi2511u2251fo.bin)grp(iavi:2511)dep(iavi:2251)tm(0911180855)pri(2)len(6175342)\r\nbin(u7iavi2511u2353fn.bin)grp(iavi:2511)dep(iavi:2353)tm(0911180855)pri(2)len(4506824)\r\nbin(u7iavi2511u2404fm.bin)grp(iavi:2511)dep(iavi:2404)tm(0911180855)pri(2)len(3342546)\r\nbin(u7iavi2511u2431fl.bin)grp(iavi:2511)dep(iavi:2431)tm(0911180855)pri(2)len(2609643)\r\nbin(u7iavi2511u2458fk.bin)grp(iavi:2511)dep(iavi:2458)tm(0911180855)pri(2)len(1380674)\r\nbin(u7iavi2511u2480fj.bin)grp(iavi:2511)dep(iavi:2480)tm(0911180855)pri(2)len(681743)\r\nbin(u7iavi2511u2490fi.bin)grp(iavi:2511)dep(iavi:2490)tm(0911180855)pri(2)len(555496)\r\nbin(u7iavi2511u2500fh.bin)grp(iavi:2511)dep(iavi:2500)tm(0911180855)pri(2)len(332906)\r\nbin(u7iavi2511u2505fh.bin)grp(iavi:2511)dep(iavi:2505)tm(0911180855)pri(2)len(204942)\r\nbin(u7iavi2511u2508
6       1        0x0400000000004001  1258544215.370067  0.000012   0.167167      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   80       1258     6        375     321    fg.bin)grp(iavi:2511)dep(iavi:2508)tm(0911180855)pri(2)len(72274)\r\nbin(u7iavi2511u2510ff.bin)grp(iavi:2511)dep(iavi:2510)tm(0911180855)pri(2)len(30540)\r\nbin(u9ichjw4qt.bin)grp(ichjw:4)pri(2)tm(0907131859)len(113488)\r\nbin(u9ichjw4u2ia.bin)grp(ichjw:4)dep(ichjw:2)pri(2)tm(0907131859)len(94470)\r\nbin(u9ichjw4u3gq.bin)grp(ic
7       1        0x0400000000004000  1258544215.370501  0.166651   0.333291      3        eth:ipv4:tcp             00:0b:db:4f:6b:10  00:19:e3:e7:5d:23  0x0800   1258     80       6        64      0      
8       1        0x0400000000004001  1258544215.370560  0.000493   0.167660      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   80       1258     6        1434    1380   hjw:4)dep(ichjw:3)pri(2)tm(0907131859)len(13313)\r\nbin(u9idat249b243tc.bin)grp(idat:249)dif(idat:243)pri(2)tm(0911051401)len(621430)\r\nbin(u9idat249b245bp.bin)grp(idat:249)dif(idat:245)pri(2)tm(0911051401)len(197648)\r\nbin(u9idat249b246mj.bin)grp(idat:249)dif(idat:246)pri(2)tm(0911051400)len(159289)\r\nbin(u9idat249b247uh.bin)grp(idat:249)dif(idat:247)pri(2)tm(0911051400)len(95680)\r\nbin(u9idat249le.bin)grp(idat:249)pri(2)tm(0911051400)len(1927074)\r\nbin(u9ifw42jx.bin)grp(ifw:42)pri(2)tm(0911131400)len(541853)\r\nbin(u9ifw42u38we.bin)grp(ifw:42)dep(ifw:38)pri(2)tm(0911131400)len(123575)\r\nbin(u9ifw42u39ke.bin)grp(ifw:42)dep(ifw:39)pri(2)tm(0911131400)len(92745)\r\nbin(u9ifw42u40dc.bin)grp(ifw:42)dep(ifw:40)pri(2)tm(0911131400)len(67292)\r\nbin(u9ifw42u41tz.bin)grp(ifw:42)dep(ifw:41)pri(2)tm(0911131400)len(18393)\r\nbin(x8all234yc.bin)grp(xplph:0012;xplsb:0099;xplsb2:0118;xplsc:0149)tm(0911180700)pri(2)len(1146352)\r\nbin(x8xplph_12gj.bin)grp(xplph:12)tm(0910280700)pri(2)len(4551)\r\nbin(x8xplsb2_118c8.bin)grp(xplsb2:118)tm(0911180700)pri(2)len(4989)\r\nbin(x8xplsb_9946.bin)grp(xplsb:99)tm(0911160700)pri(2)len(985460)\r\nbin(x8xplsb_99d9546.bin)grp(xplsb:99)dif(xplsb:95)tm(0911160700)pri(2)len(32267)\r\nbin(x8xplsb_99d9646.bin)grp(xplsb:99)dif(xplsb:96)tm(0911160700)pri(2)len(3505)\r\nbin(x8xplsb_99d9746.bin)grp(xplsb:99)dif(xplsb:97)tm(0911160700)pri(2)len(644)\r\nbin(x8xplsb_99d9846.bin
9       1        0x0400000000004001  1258544215.370571  0.000011   0.167671      3        eth:ipv4:tcp             00:19:e3:e7:5d:23  00:0b:db:4f:6b:10  0x0800   80       1258     6        520     466    )grp(xplsb:99)dif(xplsb:98)tm(0911160700)pri(2)len(581)\r\nbin(x8xplsc_149c8.bin)grp(xplsc:149)tm(0911180700)pri(2)len(151922)\r\nbin(x8xplsc_149d145c8.bin)grp(xplsc:149)dif(xplsc:145)tm(0911180700)pri(2)len(2561)\r\nbin(x8xplsc_149d146c8.bin)grp(xplsc:149)dif(xplsc:146)tm(0911180700)pri(2)len(1921)\r\nbin(x8xplsc_149d147c8.bin)grp(xplsc:149)dif(xplsc:147)tm(0911180700)pri(2)len(1767)\r\nbin(x8xplsc_149d148c8.bin)grp(xplsc:149)dif(xplsc:148)tm(0911180700)pri(2)len(1411)\r\n

It is a very rudimentary anonymization and it does not currently include the other plugins providing IP addresses, e.g. dnsDecode. The Anteater knows that removing IP’s is not the solution, but it worked for us in an urgent job. And then there is content anonymization, a large challenge. Be patient he’s working on it.

Do not forget to reset the configuration for the next tutorials:

$ t2conf tranalyzer2 -D ANONYM_IP=1 && t2build -R
...
$

or use the new command:

$ t2conf --reset && t2build -R
...
$

Have fun.