Geolocation and whois behind it

geoip MaxMind. t2whois whois

Introduction

This tutorial details the different features of T2 concerning geolocation and the determination of the organization behind an IP address. There are two options:

Core, plugin T2 geolocation and organization
geoip Open source geolocation GeoIP/MaxMind DB

Note that the standard geoip DB library functions are a bit slower than T2 native geolocation. And the open-source MaxMind DB does not contain the organization behind an IP address. As we do our own geolocation and organizational research, the anteater provides you with the newest info available.

In contrast to earlier versions, the subnet part has moved to the core since 0.8.8, providing services now to all plugins, also to the ones you want to write in future. Hence, no dependencies to the basicFlow plugin are necessary anymore.

In the next chapters we will discuss the T2 geolocation, the geoip plugin as a MaxMind wrapper is discussed in the next tutorial: Geolocation MaxMind DB: geoip, t2mmdb.

Preparation

First, restore T2 into a pristine state by removing all unnecessary or older plugins from the plugin folder ~/.tranalyzer/plugins:

t2build -e -y

Are you sure you want to empty the plugin folder '/home/wurst/.tranalyzer/plugins' (y/N)? yes
Plugin folder emptied

Then compile the following plugins:

t2build tranalyzer2 basicFlow basicStats tcpStates connStat txtSink

...
BUILD SUCCESSFUL

If you did not create a separate data and results directory yet, please do it now in another bash window, that facilitates your workflow:

mkdir ~/data ~/results

The anonymized sample PCAP used in this tutorial can be downloaded here: faf-exercise.pcap.

Please extract it under your data folder.

Now you are all set for T2 IP label experiments.

Subnet and IP labeling

T2 provides its own geo labeling and IP identification service, so no need anymore to lookup a MaxMind DB or whois for every IP address. The files necessary are always updated with each version of T2. The bzip2 subnet files for IPv4/6 are extracted by the autogen.sh script or by t2build using the programs under utils/. Note again, since version 0.8.8 the subnet config is shifted to the core, basicFlow only enables its flow output like all other plugins. So the geo service can now being used by any plugin you write in future without any complicated plugin dependencies. Moreover certain aggregation techniques are now possible, being described her: flow subnet aggregation. Nevertheless, you have now to be aware that the core controls this function.

The subnet files are now shifted from basicFlow to a utils folder in the root directory and are thus part of the core. Just type subnetutils and you will be moved to the said directory. This is how a pristine subnet folder looks like:

subnetutils

ls

corr6.c  ext6.c    mergec4.c  nett4.c  priv4.txt  rng6.c  sbm6.c   subnets4.txt.bz2  t2netID  vect4.c  whoCntryCds.txt  whoOrgCds.txt
ext4.c   Makefile  mergec6.c  nett6.c  priv6.txt  sbm4.c  subconv  subnets6.txt.bz2  tor      vect6.c  wholoc

The subnet function is now controlled in tranalyzer.h, the core config. The control of the subnet labeling is SUBNET_ON. Switching it off all subnet/geo labeling code will be removed. If you write your own plugin, this very switch must be included, see tutorial: write a geo label plugin

tranalyzer2

vi src/tranalyzer.h

...
#define SUBNET_ON       1 // Core control of subnet function for plugins
...

utils

vi subnetHL.h

...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */

#define SUBRNG       0 // IP range definition: 0: CIDR only, 1: Begin-End
#define CNTYCTY      0 // 1: add county, city
#define WHOADDR      0 // 1: add whois address info
#define SUB_MAP      1 // 1: mmap subnet, 0: normal read

#define CNTYLEN     14 // length of County record
#define CTYLEN      14 // length of City record
#define WHOLEN      30 // length of Organization record
#define ADDRLEN     30 // length of Address record

/* +++++++++++++++++++++ ENV / RUNTIME - conf Variables +++++++++++++++++++++ */

#define SUBNET_UNK "-" // Representation of unknown locations

/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...

The SUBRNG constant defines the search mode, either CIDR or ranges. The range mode has the advantage that any range can be defined by one single line whereas the CIDR notation would need many lines in the subnet file. We leave it at the default CIDR because it enables more precise geolocation in a certain range.

CNTYCTY defines the appearance/disappearance of County and City records in the generated binary subnet files as we do not want to load unnecessary data.

The WHOLEN, CNTYLEN and CTYLEN constants define the length of the County, City and WHOIS column respectively in the binary subnet file.

Since 0.8.8 the input of the subnet files is accelerated by memory mapping. The load time is significantly faster with 10^7 records. Nevertheless, if you prefer normal read or have an older system which does not support memory mapping set SUB_MAP=0 and recompile the core: t2build tranalyzer2. Since 0.8.9 we added the ADDSRLEN for an extra column, it is not used currently it just reminds me that in a later version there will be an additional address line. If you want to have one, the add your addresses to the WHOLEN, thus increasing the allocated space accordingly. It is demonstrated later in this tutorial.

Currently the supplied subnet file is small. Later larger files will be supplied to the open source in later versions.

However, the flow output is still controlled by the basicFlow plugin. New is the feature, that the flow may contain more than one IP, so BFO_SUBNET_IPLIST defines whether a list of IPs is produced or a masked IP. The latter is specific for the network aggregation mode.

basicFlow

vi src/basicFlow.h

...
/* ========================================================================== */
/* ------------------------ USER CONFIGURATION FLAGS ------------------------ */
/* ========================================================================== */
...
#define BFO_SUBNET_IPLIST      0 // 0: Display only the IP masked by SRCIP[46]CMSK and DSTIP[46]CMSK
                                 // 1: Display a list of IP aggregated

#define BFO_SUBNET_TEST        1 // Enable subnet test on inner IP
#define BFO_SUBNET_TEST_GRE    0 // Enable subnet test on GRE addresses
#define BFO_SUBNET_TEST_L2TP   0 // Enable subnet test on L2TP addresses
#define BFO_SUBNET_TEST_TEREDO 0 // Enable subnet test on Teredo addresses

#define BFO_SUBNET_ASN  0 // Output Autonomous System Numbers (ASN)
#define BFO_SUBNET_LL   0 // Output position (latitude, longitude and reliability)
#define BFO_SUBNET_ORG  1 // Output Organization
#define BFO_SUBNET_HEX  0 // Output the country code and organization information as one 32-bit hex number
...
/* ========================================================================== */
/* ------------------------- DO NOT EDIT BELOW HERE ------------------------- */
/* ========================================================================== */
...

BFO_SUBNET_TEST activates the subnet output in the flow file. It is switched on by default. The switches for GRE, L2TP and TEREDO activate geo-labeling for these specific protocol addresses. We leave them off because the pcaps in this tutorial do not contain any of these encapsulations, and it reduces clutter.

In order to be comparable with the geoip plugin output we switch on the Autonomous Systems Numbers (ASN) and the longitude/latitude output as indicated below. BFO_SUBNET_ORG controls the organization output. The BFO_SUBNET_HEX toggles between a human readable whois output or a hex coded one, which can be a powerful selection mechanism when processing large flow files. We leave this option off for now. To produce the CNTYCTY columns in the binary subnet file you need to recompile using the -f option.

t2conf tranalyzer2 -D CNTYCTY=1

t2conf basicFlow -D BFO_SUBNET_ASN=1 -D BFO_SUBNET_LL=1

t2build -R -f

t2 -r ~/data/faf-exercise.pcap -w ~/results

================================================================================
Tranalyzer 0.8.14 (Anteater), Tarantula. PID: 54749
================================================================================
[INF] Creating flows for L2, IPv4, IPv6
Active plugins:
    01: basicFlow, 0.8.14
    02: basicStats, 0.8.14
    03: tcpStates, 0.8.14
    04: connStat, 0.8.14
    05: txtSink, 0.8.14
[INF] IPv4 Ver: 5, Rev: 16122020, Range Mode: 0, subnet ranges loaded: 406105 (406.11 K)
[INF] IPv6 Ver: 5, Rev: 17122020, Range Mode: 0, subnet ranges loaded: 51345 (51.34 K)
...
--------------------------------------------------------------------------------
basicStats: Biggest L3 talker: 143.166.11.10 (US): 3101 (3.10 K) [52.54%] packets
basicStats: Biggest L3 talker: 143.166.11.10 (US): 4436320 (4.44 M) [88.84%] bytes
tcpStates: Aggregated tcpStatesAFlags=0x4a
connStat: Number of unique source IPs: 25
connStat: Number of unique destination IPs: 26
connStat: Number of unique source/destination IPs connections: 10
connStat: Max unique number of source IP / destination port connections: 18
connStat: IP prtcon/sdcon, prtcon/scon: 1.800000, 0.720000
connStat: Source IP with max connections: 192.168.1.104: 2 connections
connStat: Destination IP with max connections: 77.67.44.206 (GB): 1 connections
--------------------------------------------------------------------------------
...

Note that biggest talkers and connectors are now labeled with a country tag, if one is found.

Let’s print the essential columns of the flow file relevant to geolocation and whois.

tawk '{ print wildcard("^(src|dst)IP") }' ~/results/faf-exercise_flows.txt | sort -Vru -k1,1 | tcol

srcIP           srcIPASN  srcIPCC  srcIPCnty  srcIPCty         srcIPOrg                       srcIPLat_Lng_relP      dstIP          dstIPASN  dstIPCC  dstIPCnty  dstIPCty  dstIPOrg               dstIPLat_Lng_relP
198.189.255.75  2152      us       "ca"       "long beach"     "California State University"  33.76962_-118.1926_80  192.168.1.104  0         07       "-"        "-"       "Private network"      666_666_-1
192.168.1.105   0         07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         07       "-"        "-"       "Private network"      666_666_-1
192.168.1.104   0         07       "-"        "-"              "Private network"              666_666_-1             77.67.44.206   3257      gb       "England"  "London"  "Akamai Technologies"  51.50853_-0.12574_80
192.168.1.103   0         07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         07       "-"        "-"       "Private network"      666_666_-1
192.168.1.102   0         07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         07       "-"        "-"       "Private network"      666_666_-1
192.168.1.1     0         07       "-"        "-"              "Private network"              666_666_-1             192.168.1.103  0         07       "-"        "-"       "Private network"      666_666_-1
143.166.11.10   3614      us       "tx"       "round rock"     "Dell"                         30.51748_-97.67207_80  192.168.1.105  0         07       "-"        "-"       "Private network"      666_666_-1
77.67.44.206    3257      gb       "England"  "London"         "Akamai Technologies"          51.50853_-0.12574_80   192.168.1.104  0         07       "-"        "-"       "Private network"      666_666_-1
63.245.221.11   395642    us       "ca"       "mountain view"  "Mozilla Corporation"          38.6409_-121.5228_80   192.168.1.104  0         07       "-"        "-"       "Private network"      666_666_-1

Because we do not like to waste memory, some of the columns are cut. You can increase the values in the CNTYLEN and CTYLEN constants and redo the steps above.

Tor address labeling

By default IPv4/6 TOR addresses are integrated in the subnet file by the subconv script under utils/subnet when t2build or autogen.sh are invoked. You can switch it off by editing the autogen.sh file and removing the -t option of subconv. Below a flow file is shown where Tor addresses are present, I currently do not have an anonymized pcap for you to play with. I’m on it.

t2 -r ~/data/wurst.pcap -w ~/results

...
Number of average processed flows/s: 2.36
Average full raw bandwidth: 17117 b/s (17.12 Kb/s)
Average full bandwidth : 17117 b/s (17.12 Kb/s)
Max number of flows in memory: 6 [0.00%]
Memory usage: 0.13 GB [0.19%]
Aggregated flowStat=0x0500000000004000
[INF] IPv4 flows
[INF] Tor addresses

Note that the end report indicates that Tor addresses are present. In the flow file Tor addresses will be labeled by a TOR,, or just select all Tor traffic with the TORADD bit in flowStat as shown below.

tawk 'bitsanyset($flowStat, 0x0100000000000000) { print $dir, $flowInd, $flowStat, wildcard("^(src|dst)IP") }' ~/results/wurst_flows.txt | tcol

%dir  flowInd  flowStat            srcIP         srcIPASN  srcIPCC  srcIPCnty  srcIPCty     srcIPOrg                      srcIPLat_Lng_relP  dstIP         dstIPASN  dstIPCC  dstIPCnty  dstIPCty     dstIPOrg                      dstIPLat_Lng_relP
A     2        0x0500000000004000  192.168.7.4   0         04       "-"        "-"          "Private network"             666_666_-1         5.189.181.61  51167     de       "Bavaria"  "Nuremberg"  "GD(dontpanic)Contabo GmbH"   11.1617_49.405_80
B     2        0x0500000000004001  5.189.181.61  51167     de       "Bavaria"  "Nuremberg"  "GD(dontpanic)Contabo GmbH"   11.1617_49.405_80  192.168.7.4   0         04       "-"        "-"          "Private network"             666_666_-1

Since the 0.8.8 version, the srcIPOrg field contains the following information about the Tor node: TypeCode(Nickname)Domain/Organization

TypeCode Meaning
G Guard, or Entry interface
E Exit interface
D Directory interface

Hex code labeling

As mentioned above T2 supports hex code labeling, which is a powerful flow selection mechanism, as integer AND operations are much faster than strings compares. Open basicFlow.h and set BFO_SUBNET_HEX to 1, rebuild all and rerun t2, as indicated below:

t2conf basicFlow -D BFO_SUBNET_HEX=1 && t2build basicFlow

t2 -r ~/data/faf-exercise.pcap -w ~/results/

Now the strings are gone and replaced by 32 bit hex numbers. Now you can select all flows of a certain country and/or organization with a simple tawk script. Let’s select all srcIP and dstIP columns to see how it looks like now:

tawk '{ print wildcard("^(src|dst)IP") }' ~/results/faf-exercise_flows.txt | sort -Vru -k1,1 | tcol

srcIP           srcIPASN  srcIPCOC    srcIPCC  srcIPCnty  srcIPCty         srcIPOrg                       srcIPLat_Lng_relP      dstIP          dstIPASN  dstIPCOC    dstIPCC  dstIPCnty  dstIPCty  dstIPOrg               dstIPLat_Lng_relP
198.189.255.75  2152      0x8480549f  us       "ca"       "long beach"     "California State University"  33.76962_-118.1926_80  192.168.1.104  0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
192.168.1.105   0         0x0481c2a7  07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
192.168.1.104   0         0x0481c2a7  07       "-"        "-"              "Private network"              666_666_-1             77.67.44.206   3257      0x350011ed  gb       "England"  "London"  "Akamai Technologies"  51.50853_-0.12574_80
192.168.1.103   0         0x0481c2a7  07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
192.168.1.102   0         0x0481c2a7  07       "-"        "-"              "Private network"              666_666_-1             192.168.1.1    0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
192.168.1.1     0         0x0481c2a7  07       "-"        "-"              "Private network"              666_666_-1             192.168.1.103  0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
143.166.11.10   3614      0x848091cf  us       "tx"       "round rock"     "Dell"                         30.51748_-97.67207_80  192.168.1.105  0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
77.67.44.206    3257      0x350011ed  gb       "England"  "London"         "Akamai Technologies"          51.50853_-0.12574_80   192.168.1.104  0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1
63.245.221.11   395642    0x84817339  us       "ca"       "mountain view"  "Mozilla Corporation"          38.6409_-121.5228_80   192.168.1.104  0         0x0481c2a7  07       "-"        "-"       "Private network"      666_666_-1

The 32 bit binary coding is shown below:

cccc cccc cTww wwww wwww wwww wwww wwww

c:	country code
T:	TOR Notification bit
w:	WHOIS code

The code to text resolution can be found under utils/subnet

subnetutils

ls who*

whoCntryCds.txt  whoOrgCds.txt

Let’s see all flows from any organization coming from USA, from whoCntryCds.txt: 0x84800000

tawk 'and(strtonum($srcIPCOC), 0xff800000) == 0x84800000 || hdr() { print wildcard("^(src|dst)IP") }' ~/results/faf-exercise_flows.txt | sort -Vru -k1,1 | tcol

srcIP           srcIPASN  srcIPCOC    srcIPCC  srcIPCnty  srcIPCty         srcIPOrg                       srcIPLat_Lng_relP      dstIP          dstIPASN  dstIPCOC    dstIPCC  dstIPCnty  dstIPCty  dstIPOrg           dstIPLat_Lng_relP
198.189.255.75  2152      0x8480549f  us       "ca"       "long beach"     "California State University"  33.76962_-118.1926_80  192.168.1.104  0         0x0481c2a7  07       "-"        "-"       "Private network"  666_666_-1
143.166.11.10   3614      0x848091cf  us       "tx"       "round rock"     "Dell"                         30.51748_-97.67207_80  192.168.1.105  0         0x0481c2a7  07       "-"        "-"       "Private network"  666_666_-1
63.245.221.11   395642    0x84817339  us       "ca"       "mountain view"  "Mozilla Corporation"          38.6409_-121.5228_80   192.168.1.104  0         0x0481c2a7  07       "-"        "-"       "Private network"  666_666_-1

In srcIPCC or dstIPCC, the bit 0x00400000 indicates a TOR address or you can select TOR flows just with the flowStat bit 0x0100000000000000 as indicated below in traffic I generated on my computer.

tawk 'bitsanyset($flowStat, 0x0100000000000000) { print $dir, $flowInd, $flowStat, wildcard("^(src|dst)IP") }' ~/results/torwurst_flows.txt | tcol

%dir  flowInd  flowStat            srcIP        srcIPASN  srcIPCOC    srcIPCC  srcIPCnty  srcIPCty  srcIPOrg           srcIPLat_Lng_relP  dstIP          dstIPASN  dstIPCOC    dstIPCC  dstIPCnty        dstIPCty   dstIPOrg                          dstIPLat_Lng_relP
A     1        0x0500000000004100  10.20.7.153  0         0x0301c2a7  04       "-"        "-"       "Private network"  666_666_-1         51.15.246.170  12876     0x61c1a07e  fr       "Noord-Holland"  "Haarlem"  "GD(mitsuha)mitsuha.katawaredok"  2.3387_48.8582_80

As mentioned above the dst/src IP code of the B/A flow 0x29c07396 has the Tor bit set, thus a Tor address and the whole flow is Tor labelled in flowStat. As homework try now to select all Tor flows in faf-exercise.pcap using srcIPCC. Are there any?

But how do you find out what is behind the srcIPCOC or dstIPCOC (aka netID) code? Just use t2netID Let’s try it with the codes above:

t2netID 0x0301c2a7 0x61c1a07e

0x0301c2a7
	Country     : IPv4 private (04)
	Organization: private network

0x61c1a07e
	Country     : Netherlands (nl)
	Organization: online sas
	Tor address : yes

Internal whois: subnet your own

Which admin was not asking himself WHO, WHERE and WHY the fuck is somebody doing what he is doing, or how to find an in-house IP 10.23.4.5? Yeah, I did lot’s and got weary to lookup Excel sheets, logs or if I was lucky, DBs. Now you try to do that on 1000 addresses and hand over a report in no time.

As the private IPv4/6 address space is hopefully only listed inside your organization we need to build our own subnet file. Building one is fairly easy if IP to location and organization is available as a tab or csv file. So that you can expand the current subnet files or rewrite them, T2 is shipped with the .txt version and including scripts to convert them to the T2 compatible binary version. That is the reason, why the initial build of core takes a bit longer.

Since version 0.8.8 the subnet files are located in the utils/subnet directory. The original subnets[46].txt.bz2 is the decompressed subnets[46].txt file, which contains all information. Together with the TOR information the temporary _HL.txt and _HLP.txt files created, containing all search and routing relevant items. According to the user config then a tailored binary format _HLP.bin is produced which the anteater then loads on invocation.

Below you see the subnet directory after compilation:

subnetutils

ls

corr6.c  ext4.d  ext6.d     mergec6.c  nett4.d  nett6.d    rng6    sbm4    sbm6    subconv           subnets4_HL.txt   subnets6_HLP.bin  subnets6.txt      tor      vect4.d  vect6.d          whoOrgCds.txt
ext4     ext6    Makefile   nett4      nett6    priv4.txt  rng6.c  sbm4.c  sbm6.c  subnets4_HLP.bin  subnets4.txt      subnets6_HLP.txt  subnets6.txt.bz2  vect4    vect6    whoCntryCds.txt
ext4.c   ext6.c  mergec4.c  nett4.c    nett6.c  priv6.txt  rng6.d  sbm4.d  sbm6.d  subnets4_HLP.txt  subnets4.txt.bz2  subnets6_HL.txt   t2netID           vect4.c  vect6.c  wholoc

Here you see all the intermittently build subnet text file versions, which is beneficial if you want to troubleshoot the subnet file generation process.

Open subnets4.txt, the IPv6 is built in a similar fashion. Since version 0.8.10 needs version 5, IPv4 and 6 are now alike. New is the more elaborate description of multicast addresses, and some updates.

lsx subnets4.txt

#	5	16122020
# IPCIDR	IPrange	CtryWhoCode	ASN	Uncert	Latitude	Longitude	Country	County	City	Org
# Begin IPv4 private address space
10.0.0.0/8	10.0.0.0-10.255.255.255	0x0301c2a7	0	-1.0	666.000000	666.000000	04	-	-	Private network
14.0.0.0/8	14.0.0.0-14.255.255.255	0x00000000	0	-1.0	666.000000	666.000000	03	-	-	Public data networks
24.0.0.0/8	24.0.0.0-24.255.255.255	0x00000000	0	-1.0	666.000000	666.000000	09	-	-	Cable television networks
127.0.0.0/8	127.0.0.0-127.255.255.255	0x01014fe7	0	-1.0	666.000000	666.000000	01	-	-	Loopback
100.64.0.0/10	100.64.0.0-100.127.255.255	0x0702041f	0	-1.0	666.000000	666.000000	20	-	-	Shared address space
169.254.0.0/16	169.254.0.0-169.254.255.255	0x02014965	0	-1.0	666.000000	666.000000	02	-	-	Link-local
172.16.0.0/12	172.16.0.0-172.31.255.255	0x0381c2a7	0	-1.0	666.000000	666.000000	05	-	-	Private network
192.0.0.0/24	192.0.0.0-192.0.0.255	0x0401c2a7	0	-1.0	666.000000	666.000000	06	-	-	Private network
192.0.2.0/24	192.0.2.0-192.0.2.255	0x07823fb8	0	-1.0	666.000000	666.000000	21	-	-	TEST-NET-1
192.88.99.0/24	192.88.99.0-192.88.99.255	0x0b011c29	0	-1.0	666.000000	666.000000	60	-	-	IPv6 to IPv4 relay
192.168.0.0/16	192.168.0.0-192.168.255.255	0x0481c2a7	0	-1.0	666.000000	666.000000	07	-	-	Private network
198.18.0.0/15	198.18.0.0-198.19.255.255	0x0501c2a7	0	-1.0	666.000000	666.000000	08	-	-	Private network
198.51.100.0/24	198.51.100.0-198.51.100.255	0x08023fb9	0	-1.0	666.000000	666.000000	22	-	-	TEST-NET-2
203.0.113.0/24	203.0.113.0-203.0.113.255	0x08823fba	0	-1.0	666.000000	666.000000	23	-	-	TEST-NET-3
224.0.0.0/4	224.0.0.0-239.255.255.255	0x06017598	0	-1.0	666.000000	666.000000	10	-	-	Multicast
224.0.0.1/32	224.0.0.1-224.0.0.1	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	All Systems on this Subnet
224.0.0.2/32	224.0.0.2-224.0.0.2	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	All Routers on this Subnet
224.0.0.4/32	224.0.0.4-224.0.0.4	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	DVMRP Routers
224.0.0.5/32	224.0.0.5-224.0.0.5	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	OSPFIGP All Routers
224.0.0.6/32	224.0.0.6-224.0.0.6	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	OSPFIGP Designated Routers
224.0.0.7/32	224.0.0.7-224.0.0.7	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	ST Routers
224.0.0.8/32	224.0.0.8-224.0.0.8	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	ST Hosts
224.0.0.9/32	224.0.0.9-224.0.0.9	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	RIP2 Routers
224.0.0.10/32	224.0.0.10-224.0.0.10	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	EIGRP Routers
224.0.0.11/32	224.0.0.11-224.0.0.11	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	Mobile-Agents
224.0.0.12/32	224.0.0.12-224.0.0.12	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	DHCP Server / Relay Agent
224.0.0.13/32	224.0.0.13-224.0.0.13	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	All PIM Routers
224.0.0.14/32	224.0.0.14-224.0.0.14	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	RSVP-ENCAPSULATION
224.0.0.15/32	224.0.0.15-224.0.0.15	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	all-cbt-routers
224.0.0.16/32	224.0.0.16-224.0.0.16	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	designated-sbm
224.0.0.17/32	224.0.0.17-224.0.0.17	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	all-sbms
224.0.0.18/32	224.0.0.18-224.0.0.18	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	VRRP
...
224.252.0.0/14	224.252.0.0-224.255.255.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	DIS Transient Groups
233.0.0.0/9	233.0.0.0-233.251.255.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	GLOP Block
233.252.0.0/24	233.252.0.0-233.252.0.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	MCAST-TEST-NET
233.252.1.0/27	233.252.1.0-233.252.1.31	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	Pico
233.252.2.0/23	233.252.2.0-233.252.7.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	Tradition
233.252.8.0/22	233.252.8.0-233.252.11.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	BVMF_MKT_DATA
233.252.12.0/23	233.252.12.0-233.252.13.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	blizznet-tv-services
233.252.14.0/23	233.252.14.0-233.252.17.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	BVMF_MKT_DATA_2
234.0.0.0/8	234.0.0.0-234.255.255.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	Unicast-Prefix-based Multicast Addr
239.0.0.0/8	239.0.0.0-239.255.255.255	0x00000000	0	-1.0	666.000000	666.000000	10	-	-	Organization-Local Scope
240.0.0.0/4	240.0.0.0-255.255.255.254	0x0901dd04	0	-1.0	666.000000	666.000000	24	-	-	Reserved
255.255.255.255/32	255.255.255.255-255.255.255.255	0x06804ca0	0	-1.0	666.000000	666.000000	11	-	-	Broadcast
# End IPv4 private address space
1.0.0.0/24	1.0.0.0-1.0.0.255	0x8480205e	13335	80.000000	34.052231	-118.243683	us	California	Los Angeles	APNIC Research and Development
1.0.1.0/24	1.0.1.0-1.0.1.255	0x260062cc	0	80.000000	26.061390	119.306107	cn	Fujian	Fuzhou	CHINANET FUJIAN PROVINCE NETWORK
1.0.2.0/23	1.0.2.0-1.0.3.255	0x260062cc	0	80.000000	26.061390	119.306107	cn	Fujian	Fuzhou	CHINANET FUJIAN PROVINCE NETWORK
1.0.4.0/24	1.0.4.0-1.0.4.255	0x148284d8	56203	80.000000	-37.813999	144.963318	au	Victoria	Melbourne	Wirefreebroadband Pty Ltd
1.0.8.0/21	1.0.8.0-1.0.15.255	0x260062ce	0	80.000000	23.116671	113.250000	cn	Guangdong	Guangzhou	CHINANET Guangdong province network
1.0.16.0/24	1.0.16.0-1.0.16.255	0x4780fc29	2519	80.000000	35.689507	139.691696	jp	Tokyo	Tokyo	i2ts
1.0.32.0/19	1.0.32.0-1.0.63.255	0x260062ce	0	80.000000	23.116671	113.250000	cn	Guangdong	Guangzhou	CHINANET Guangdong province network
1.0.64.0/18	1.0.64.0-1.0.127.255	0x4780b092	18144	80.000000	34.385281	132.455276	jp	Hiroshima	Hiroshima	Energia Communications
1.0.128.0/18	1.0.128.0-1.0.191.255	0x7c00a471	9737	80.000000	18.790380	98.984680	th	Chiang Mai	Chiang Mai	Dynamic IP Address for residential Broadband Customers
1.0.192.0/18	1.0.192.0-1.0.255.255	0x7c00a471	9737	80.000000	9.945610	99.078468	th	Chumphon	Lang Suan	Dynamic IP Address for residential Broadband Customers
1.1.0.0/24	1.1.0.0-1.1.0.255	0x260062cc	0	80.000000	26.061390	119.306107	cn	Fujian	Fuzhou	CHINANET FUJIAN PROVINCE NETWORK
1.1.1.0/24	1.1.1.0-1.1.1.255	0x8480205e	13335	80.000000	34.052231	-118.243683	us	California	Los Angeles	APNIC Research and Development
1.1.2.0/23	1.1.2.0-1.1.3.255	0x260062cc	0	80.000000	26.061390	119.306107	cn	Fujian	Fuzhou	CHINANET FUJIAN PROVINCE NETWORK
...

As our pcap only contains IPv4 addresses we restrict the following tutorial to the IPv4 subnet file. Just change the 4 to 6 and you have the IPv6 issue covered.

You can now write your own subnet file or modify the original one. It is advisable to make a copy of the subnets4.txt to have an easy way to restore the default.

Let’s define the 192.168.x.y network a bit more precisely by adding two more lines describing the Knoedelrutschen company with one /24, one /28 and /26 network. Using the C program ipcalc, you can transform arbitrary network ranges to CIDR. There is also a perl program, but it is too slow if large quantities need to be processed.

ipcalc 192.168.1.100-192.168.1.127

deaggregate 192.168.1.100 - 192.168.1.127
192.168.1.100/30
192.168.1.104/29
192.168.1.112/28

To facilitate the process a subnet addition can be downloaded from out webpage knoedelsub.txt.

cat knoedelsub.txt

# Begin Knoedelrutschen Company internal network
192.168.1.0/24  192.168.1.0-192.168.1.255       0x2f8fffff      0       -1.0    666.000000      666.000000      eu      Ruelps  -       Knoedelrutschen Inc
# Begin Knoedelrutschen Company internal sub networks
192.168.1.0/28  192.168.1.0-192.168.1.15        0x328ff666      0       1.5     48.856892       2.350850        fr      ile-de-France   Paris   Managers, Eifeltower, over paid
192.168.1.64/26 192.168.1.64-192.168.1.127      0x328ff888      0       0.01    46.947990       7.459672        ch      Bern    Bern    Warp Drive Guys, Knödelstrasse 5
192.168.1.100/30        192.168.1.100-192.168.1.127     0x230ff889      0       0.02    46.47990        7.9672  ch      Bern    Bern    HW Guys, Wurstgasse 4
192.168.1.104/29        192.168.1.100-192.168.1.127     0x230ff890      0       0.02    46.48000        7.9672  ch      Bern    Bern    Sales, Wurstgasse 5
192.168.1.112/28        192.168.1.100-192.168.1.127     0x230ff891      0       0.01    46.48111        7.9672  ch      Bern    Bern    SW Guys, Wurstgasse 6
# End Knoedelrutschen company internal sub networks
# End Knoedelrutschen Company internal network

Note that every column of a record has to separated by a tab \t.

Now append knoedelsub.txt to your subnets4.txt under utils/subnet:

cp subnets4.txt.bz2 subnets4c.txt.bz2

cp whoOrgCds.txt whoOrgCdsc.txt

cat knoedelsub.txt >> subnets4.txt

bzip2 -cf subnets4.txt > subnets4.txt.bz2

awk -F"\t" '!/^#/ { printf "0x%08x\t%s\n", and(strtonum($3), 0x007fffff), $11 }\' knoedelsub.txt | LC_COLLATE=C sort -t$'\t' -k1,1 -V >> whoOrgCds.txt

tail whoOrgCds.txt

0x00026d92	zzi d.o.o.
0x00026d93	zzone on line philippines inc
0x00026d94	zzoomm plc
0x00026d95	zzzipnet
0x000ff666	Managers, Eifeltower, over paid
0x000ff888	Warp Drive Guys, Knödelstrasse 5
0x000ff889	HW Guys, Wurstgasse 4
0x000ff890	Sales, Wurstgasse 5
0x000ff891	SW Guys, Wurstgasse 6
0x000fffff	Knoedelrutschen Inc

Entries are sorted to also enable binary search. The whoCntryCds.txt is good, but if you want to invent a new country, feel free to add a new code.

t2netID 0x230ff889

0x230ff889:
	Country     : Switzerland (ch)
	Organization: HW Guys, Wurstgasse 4

In order to create a new subnet binary we need use the script subconv or simply compile t2 with the -f option. Because autogen.sh decompresses the subnets4.txt.bz2 and thus overwrites the subnet file we need first to bzip2 your subnets4.txt and then rebuild the core and all plugins which implement the subnet functions with the -f -R option, if a config was changed in subnetHL.h in the utils directory. subnetHL.h is included by any plugin which likes to use the subnet labeling service from the core. If nothing had been changed, then the compilation of the core is sufficient.

Nevertheless the safest way to reconstruct the binary and ship it to the ~/.tranalyzer/plugins/ folder is to recompile all plugins in use. Then rerun t2 with the pcap:

bzip2 -cf subnets4.txt > subnets4.txt.bz2

t2build -f -R

t2 -r ~/data/faf-exercise.pcap -w ~/results/

Now open the flow file and you will see your IP labeling.

tawk '{ print wildcard("^(src|dst)IP") }' ~/results/faf-exercise_flows.txt | sort -Vru -k1,1 | tcol

srcIP           srcIPASN  srcIPCC  srcIPCnty        srcIPCty         srcIPOrg                          srcIPLat_Lng_relP      dstIP          dstIPASN  dstIPCC  dstIPCnty        dstIPCty  dstIPOrg                          dstIPLat_Lng_relP
198.189.255.75  2152      us       "ca"             "long beach"     "California State University"     33.76962_-118.1926_80  192.168.1.104  0         ch       "Bern"           "Bern"    "Sales, Wurstgasse 5"             46.48_7.9672_0.02
192.168.1.105   0         ch       "Bern"           "Bern"           "Sales, Wurstgasse 5"             46.48_7.9672_0.02      192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.104   0         ch       "Bern"           "Bern"           "Sales, Wurstgasse 5"             46.48_7.9672_0.02      77.67.44.206   3257      gb       "England"        "London"  "Akamai Technologies"             51.50853_-0.12574_80
192.168.1.103   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.102   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.1     0         fr       "ile-de-France"  "Paris"          "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5   192.168.1.103  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02
143.166.11.10   3614      us       "tx"             "round rock"     "Dell"                            30.51748_-97.67207_80  192.168.1.105  0         ch       "Bern"           "Bern"    "Sales, Wurstgasse 5"             46.48_7.9672_0.02
77.67.44.206    3257      gb       "England"        "London"         "Akamai Technologies"             51.50853_-0.12574_80   192.168.1.104  0         ch       "Bern"           "Bern"    "Sales, Wurstgasse 5"             46.48_7.9672_0.02
63.245.221.11   395642    us       "ca"             "mountain view"  "Mozilla Corporation"             38.6409_-121.5228_80   192.168.1.104  0         ch       "Bern"           "Bern"    "Sales, Wurstgasse 5"             46.48_7.9672_0.02

As we are using the CIDR mode, let’s now test the range mode. So open utils.h and set SUBRNG to 1 or simply use the t2conf command below. T2 now selects the 2nd column in the subnet file and the resulting binary subnet file will be smaller. Just do a wc -l on a CIDR and Range subnet file. Why in our case? How could you compensate for that difference? Think a little.

In any case, sometime the range mode makes your life easier because you can define ranges as you wish, not as CIDR dictates.

t2conf tranalyzer2 -D SUBRNG=1

t2build -f -R

t2 -r ~/data/faf-exercise.pcap -w ~/results/

If you look into the flow file, you will now discover that the engineers are now all at Wurstgasse 4. Why? Think.

tawk '{ print wildcard("^(src|dst)IP") }' ~/results/faf-exercise_flows.txt | sort -Vru -k1,1 | tcol

srcIP           srcIPASN  srcIPCC  srcIPCnty        srcIPCty         srcIPOrg                          srcIPLat_Lng_relP      dstIP          dstIPASN  dstIPCC  dstIPCnty        dstIPCty  dstIPOrg                          dstIPLat_Lng_relP
198.189.255.75  2152      us       "ca"             "long beach"     "California State University"     33.76962_-118.1926_80  192.168.1.104  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02
192.168.1.105   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.104   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    77.67.44.206   3257      gb       "England"        "London"  "Akamai Technologies"             51.50853_-0.12574_80
192.168.1.103   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.102   0         ch       "Bern"           "Bern"           "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02    192.168.1.1    0         fr       "ile-de-France"  "Paris"   "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5
192.168.1.1     0         fr       "ile-de-France"  "Paris"          "Managers, Eifeltower, over pai"  48.85689_2.35085_1.5   192.168.1.103  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02
143.166.11.10   3614      us       "tx"             "round rock"     "Dell"                            30.51748_-97.67207_80  192.168.1.105  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02
77.67.44.206    3257      gb       "England"        "London"         "Akamai Technologies"             51.50853_-0.12574_80   192.168.1.104  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02
63.245.221.11   395642    us       "ca"             "mountain view"  "Mozilla Corporation"             38.6409_-121.5228_80   192.168.1.104  0         ch       "Bern"           "Bern"    "HW Guys, Wurstgasse 4"           46.4799_7.9672_0.02

If you are only interested in processing certain IPs instead of a pcap and you have no network connection, or do not want to overload whois and get blocked, or MaxMind DB is not enough for you, the t2whois is your tool of choice.

t2whois

Suppose you want to write your own subnet file or just test a few IPs without using whois or any other DB, t2whois allows you to query the anteater DB.

It is compiled along side with tranalyzer2. And it then supplies the subnet data according to the configuration in subnetHL.h. Try the following commands to get acquainted with t2whois.

t2whois -h

Usage:
    t2whois [OPTION...] [INPUT...]

Input:
    -               If no input is provided, read from stdin
    ip              Read IP address(es) directly from the command line
    -r file         Read IP address(es) from 'file'

Optional arguments:
    -D              Run as a server/daemon on 127.0.0.1:6666
    -a              Server address
    -p              Server port

    -d file         Binary subnet file to use for IPv4
    -e file         Binary subnet file to use for IPv6

    -o field(s)     Field(s) to output (in order). Many fields can be selected
                    by using multiple '-o' options or by separating the fields
                    with a comma, e.g., -o field1,field2. Valid field names are
                    ip, netmask, net, mask, range, who, country, county, city,
                    asn, lat, lng, prec, id

    -q              Do not display an interactive prompt when reading from stdin

    -k file         Generate a KML 'file'

    -l              Output one line per IP
    -H              Do not output the header with -l option
    -t char         Start character(s) for column header (-l option) ["%"]

    -s char         Column separator for output ["\t"]

Help and documentation arguments:
    -L              Describe the available fields and exit
    -V              Show info about the database (version, ...) and exit
    -h              Show help options and exit

Let’s try a simple case with two addresses on the command line:

t2whois 77.67.44.206 63.245.221.11 192.168.1.104

IP              77.67.44.206
Network/Mask    77.67.44.192/28
Range           77.67.44.192 - 77.67.44.207
Organization    Akamai Technologies
Country         gb
County          England
City            London
ASN             3257
Latitude        51.508530
Longitude       -0.125740
Precision       80.000000
NetID           0x350011ed

IP              63.245.221.11
Network/Mask    63.245.208.0/20
Range           63.245.208.0 - 63.245.223.255
Organization    Mozilla Corporation
Country         us
County          ca
City            mountain view
ASN             395642
Latitude        38.640900
Longitude       -121.522797
Precision       80.000000
NetID           0x84817339

IP              192.168.1.104
Network/Mask    192.168.0.0/16
Range           192.168.0.0 - 192.168.255.255
Organization    Private network
Country         07
County          -
City            -
ASN             0
Latitude        666.000000
Longitude       666.000000
Precision       -1.000000
NetID           0x0481c2a7

Now we like the output in a line to post process it or to pipe into a tool such as tcol

t2whois -l 77.67.44.206 63.245.221.11 | tcol

%IP            Network/Mask     Range                          Organization         Country  County   City           ASN     Latitude   Longitude    Precision  NetID
77.67.44.206   77.67.44.192/28  77.67.44.192 - 77.67.44.207    Akamai Technologies  gb       England  London         3257    51.508530  -0.125740    80.000000  0x350011ed
63.245.221.11  63.245.208.0/20  63.245.208.0 - 63.245.223.255  Mozilla Corporation  us       ca       mountain view  395642  38.640900  -121.522797  80.000000  0x84817339

If you want the interactive mode, just invoke t2whois without any argument:

t2whois

[INF] Enter an IPv4/6 address, 'header', 'help' or 'quit' to exit

>>> 88.67.56.56
IP              88.67.56.56
Network/Mask    88.67.48.0/20
Range           88.67.48.0 - 88.67.63.255
Organization    ARCOR AG
Country         de
County          Baden-Wurttemb
City            Stuttgart
ASN             3209
Latitude        48.782318
Longitude       9.177020
Precision       80.000000
NetID           0x2a8022a7

>>>

And if you want to lookup all public hosts in your flow file:

tawk '!privip($srcIP) { print $srcIP } !privip($dstIP) { print $dstIP }' ~/results/faf-exercise_flows.txt | sort -u | t2whois -l | tcol

%IP             Network/Mask     Range                          Organization                 Country  County   City           ASN     Latitude   Longitude    Precision  NetID
143.166.11.10   143.166.0.0/16   143.166.0.0 - 143.166.255.255  Dell                         us       tx       round rock     3614    30.517477  -97.672066   80.000000  0x848091cf
198.189.255.75  198.189.0.0/16   198.189.0.0 - 198.189.255.255  California State University  us       ca       long beach     2152    33.769615  -118.192574  80.000000  0x8480549f
63.245.221.11   63.245.208.0/20  63.245.208.0 - 63.245.223.255  Mozilla Corporation          us       ca       mountain view  395642  38.640900  -121.522797  80.000000  0x84817339
77.67.44.206    77.67.44.192/28  77.67.44.192 - 77.67.44.207    Akamai Technologies          gb       England  London         3257    51.508530  -0.125740    80.000000  0x350011ed

If you like to select only certain columns:

t2whois -L

The fields available are:
	ip         	IP
	netmask    	Network/Mask
	net        	Network
	mask       	Mask
	range      	Range
	org        	Organization
	country    	Country
	county     	County
	city       	City
	asn        	ASN
	lat        	Latitude
	lng        	Longitude
	prec       	Precision
	netid      	NetID

tawk -H '{ print $srcIP }' ~/results/faf-exercise_flows.txt | sort -u | t2whois -l -o ip,netmask,asn,country,org | tcol

%IP             Network/Mask     ASN     Country  Organization
143.166.11.10   143.166.0.0/16   3614    us       Dell
192.168.1.1     192.168.0.0/16   0       07       Private network
192.168.1.102   192.168.0.0/16   0       07       Private network
192.168.1.103   192.168.0.0/16   0       07       Private network
192.168.1.104   192.168.0.0/16   0       07       Private network
192.168.1.105   192.168.0.0/16   0       07       Private network
198.189.255.75  198.189.0.0/16   2152    us       California State University
63.245.221.11   63.245.208.0/20  395642  us       Mozilla Corporation
77.67.44.206    77.67.44.192/28  3257    gb       Akamai Technologies

Why are all HW Guys now? Right! By the time you should realize that if you are in range mode, only the range is checked, not the CIDR address. So switch back to CIDR mode, recompile and rerun t2:

t2conf tranalyzer2 -D SUBRNG=0

t2build -f -R

t2 -r ~/data/faf-exercise.pcap -w ~/results/

tawk -H '{ print $srcIP }' ~/results/faf-exercise_flows.txt | sort -u | t2whois -l -o ip,netmask,asn,country,org | tcol

%IP             Network/Mask      ASN     Country  Organization
143.166.11.10   143.166.0.0/16    3614    us       Dell
192.168.1.1     192.168.1.0/28    0       fr       Managers, Eifeltower, over pai
192.168.1.102   192.168.1.100/30  0       ch       HW Guys, Wurstgasse 4
192.168.1.103   192.168.1.100/30  0       ch       HW Guys, Wurstgasse 4
192.168.1.104   192.168.1.104/29  0       ch       Sales, Wurstgasse 5
192.168.1.105   192.168.1.104/29  0       ch       Sales, Wurstgasse 5
198.189.255.75  198.189.0.0/16    2152    us       California State University
63.245.221.11   63.245.208.0/20   395642  us       Mozilla Corporation
77.67.44.206    77.67.44.192/28   3257    gb       Akamai Technologies

Ahhhh, now the Sales people are back.

Now we load different subnet binaries. Because they are very big, loading them every time is cumbersome. So a client server solution is the way to go. We copy our binaries in the configuration we like to a server directory, let’s pick the ~/data. Then invoke t2whois in one bash window.

cp .tranalyzer/plugins/subnets4_HLP.bin ~/data

cp .tranalyzer/plugins/subnets6_HLP.bin ~/data

t2whois -D -d ~/data/subnets4_HLP.bin -e ~/data/subnets6_HLP.bin

[INF] Server listening on 127.0.0.1:6666
[INF] New client: 127.0.0.1:56864 4

Now open in another bash window the client with netcat:

nc 127.0.0.1 6666

88.67.56.56
IP              88.67.56.56
Network/Mask    88.67.48.0/20
Range           88.67.48.0 - 88.67.63.255
Organization    ARCOR AG
Country         de
County          Baden-Wurttemb
City            Stuttgart
ASN             3209
Latitude        48.782318
Longitude       9.177020
Precision       80.000000
NetID           0x2a8022a7

Let us finish this section with an example of t2whois -k option which can be used to generate a KML file.

tawk -H '{ print shost(); print dhost() }' ~/results/faf-exercise_flows.txt | t2whois -k ~/results/faf-exercise.kml

ls ~/results | grep -F .kml

faf-exercise.kml

The faf-exercise.kml file contains information about each IP (as specified with t2whois -o option) and its location (latitude, longitude). This KML (Keyhole Markup Language) file can then be loaded in, e.g., Google Maps or Google Earth, and will display each IP at its exact location.

t2whois can generate KML files to display addresses location…
… and additional information is also readily available

You can also load your own subnet file(s) using the -e or -d options. Try t2whois --help for more information.

t2locate

t2locate allows you to find the city nearest to geodesic coordinates. First you need to build the DB. We do not do it at installation because it takes ca 30min. So move to the \$T2HOME/scripts/t2locate directory and invoke the update_db script. It is good practice to read the README.md file while waiting.

tran

cd scripts/t2locate

ls

README.md  scripts  t2locate  update_db

./update_db

-2021-09-24 13:45:09--  http://download.geonames.org/export/dump/allCountries.zip
Resolving download.geonames.org (download.geonames.org)... 5.9.152.54
Connecting to download.geonames.org (download.geonames.org)|5.9.152.54|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 370984486 (354M) [application/zip]
Saving to: ‘db.zip’

db.zip                                                      100%[========================================================================================================================================>] 353.80M  9.29MB/s    in 46s

2021-09-24 13:46:25 (7.73 MB/s) - ‘db.zip’ saved [370984486/370984486]

Extracting data...
Done
Preprocessing data...
Importing data into database... Done

t2locate -h

Usage:
    t2locate [OPTIONS] [LATITUDE LONGITUDE]

Get a nearby location based on the coordinates supplied

Optional arguments:
    -f, --file=FILE           Read coordinates from FILE
                              (one lat/long pair per line)
    -o, --output=FILE         Write the output to FILE instead of stdout
    -d, --database=DIR        Absolute path to database folder
    -s, --separator=FS        Column separator for the output (default: '\t')
    -i, --input-separator=FS  Column separator for the input (default: '\t')

Help and documentation arguments:
    -h, --help                Show this help, then exit

t2locate 34.052231 -118.243683

34.052231	-118.243683	US	California	Bowie County			Los Angeles Police Department - Headquarters	100m

Or try to pipe a file (Note that the values are separated by a tab (\t)):

cat nudelverlaufskritik.txt

34.052231       -118.243683
44.052231       118.243683
11.052231       18.243683

t2locate -f nudelverlaufskritik.txt

34.052231    -118.243683    US    California                          Bowie County     Los Angeles Police Department - Headquarters    100m
44.052231    118.243683     CN    Inner Mongolia Autonomous Region    Laofangshen      Laofangshen                                     10km
11.052231    18.243683      TD    Guera Region                        Koulia                                                           10km

Conclusion

Right! This is all for now. And don’t forget to reset the configuration of T2 for the next tutorials:

mv subnets4c.txt.bz2 subnets4.txt.bz2

mv whoOrgCdsc.txt whoOrgCds.txt

t2conf --reset tranalyzer2 basicFlow && t2build -R -f

Have fun!