Tutorial: t2py - Control and Operate T2 with Python

Introduction

t2py is a python library which can be used to control and operate Tranalyzer2. It can be used as an alternative to t2conf, t2build and other scripts.

Caveats

This library is still experimental and at an early stage of development. As such, the API may be subject to change.

Bug reports, feature requests, feedback and suggestions are welcome and can be addressed directly to Andy.

Dependencies

Required Dependencies

Optional Dependencies

The following dependencies are only required for specific operations:

  • pandas is only required for the to_pandas() functions in T2.py and T2Utils.py.
  • pdoc3 is only required to generate the API documentation as HTML.

Getting Started

Setup

If you installed Tranalyzer with the setup.sh script (or are sourcing t2_aliases in your ~/.bashrc file), then you should be ready to use t2py:

$ t2py
>>>

Otherwise, you must tell Python where to find the t2py module. This is achieved with the PYTHONPATH environment variable:

  1. Make sure $T2HOME exists:

  2. If it does not exist, make sure to set it:

  3. In order to be able to import the t2py module in your current terminal, run the following command:

  4. If you want this change to be permanent, add the following line to your ~/.bashrc (or ~/.zshrc, …):

Now you are all set to start using t2py!

Available Modules

t2py provides the following modules:

  • T2Utils: provide wrappers around Tranalyzer scripts and utilities
  • T2Plugin: represent a Tranalyzer2 plugin
  • T2: manage a session (set of plugins, configuration changes, flow file, …)

Import

The simplest way to get started is to use the t2py alias, which will start an interactive Python session and import all the modules:

$ t2py
>>>

If you want to work on specific plugins, you can give the plugin names to t2py and the requested T2Plugin will be automatically created for you:

$ t2py myPlugin1 myPlugin2
>>>

If you want to do it the hard way, first start an interactive Python session, then import the T2Plugin module and finally create your plugins as T2Plugin objects:

$ python3
>>>

You can import each module separately, several or all of them at the same time:

The following sections simply list the available variables and methods. For more details and examples, refer to the t2py API documentation.

T2Utils.py: simple wrapper around Tranalyzer2 scripts and utilities

For more details, refer to the t2py.T2Utils API.

>>> from t2py import T2Utils

# Read-Only Properties

>>> T2Utils.T2HOME    # -> str
>>> T2Utils.T2PLHOME  # -> str

>>> T2Utils.T2BUILD   # -> str
>>> T2Utils.T2CONF    # -> str
>>> T2Utils.T2FM      # -> str
>>> T2Utils.T2PLUGIN  # -> str

# Static Functions

>>> T2Utils.apply_config(
...     plugin: str,
...     infile: str = None
... )
>>> T2Utils.build(
...     plugin: Union[str, List[str]],
...     plugin_folder: str = None,
...     force_rebuild: bool = False,
...     debug: bool = False
... )
>>> T2Utils.clean(
...     plugin: str
... )
>>> T2Utils.create_pcap_list(
...     pcaps: List[str],
...     outfile: str = None
... )
>>> T2Utils.create_plugin_list(
...     plugins: List[str],
...     outfile: str = None
... )
>>> T2Utils.generate_config(
...     plugin: str,
...     outfile: str = None
... )
>>> T2Utils.get_config(
...     plugin: str,
...     name: str,
...     infile: str = None
... ) -> Any
>>> T2Utils.get_config_from_source(
...     plugin: str,
...     name: str
... ) -> Any
>>> T2Utils.get_default(
...     plugin: str,
...     name: str
... ) -> Any
>>> T2Utils.list_config(
...     plugin: str
... ) -> List[str]
>>> T2Utils.list_plugins(
...     infile: str = None
... ) -> List[str]
>>> T2Utils.load_plugins(
...     plugin: Union[str, List[str]] = None
... )
>>> T2Utils.network_interfaces() -> List[str]
>>> T2Utils.plugin_description(
...     plugin: str
... ) -> str
>>> T2Utils.plugin_number(
...     plugin: str
... ) -> str
>>> T2Utils.plugins(
...     category: Union[str, List[str]] = None
... ) -> List[str]
>>> T2Utils.reset_config(
...     plugin: Union[str, List[str]],
...     name: Union[str, List[str]] = None,
...     outfile: str = None
... )
>>> T2Utils.run_tranalyzer(
...     pcap: str = None,
...     iface: str = None,
...     pcap_list: Union[str, List[str]] = None,
...     output_prefix: str = None,
...     log_file: bool = False,
...     packet_mode: bool = False,
...     plugin_folder: str = None,
...     loading_list: str = None,
...     plugins: List[str] = None,
...     bpf: str = None,
...     t2_exec: str = None,
...     timeout: int = None
... )
>>> T2Utils.set_config(
...     plugin: str,
...     name: str,
...     value: Any,
...     outfile: str = None
... )
>>> T2Utils.set_config(
...     plugin: str,
...     name_val: Dict[str, Any],
...     outfile: str = None
... )
>>> T2Utils.set_default(
...     plugin: Union[str, List[str]],
...     name: Union[str, List[str]] = None,
...     outfile: str = None
... )
>>> T2Utils.t2_exec(
...     debug: bool = False
... ) -> str
>>> T2Utils.to_json_array(
...     infile: str,
...     delimiter: str = '\t'
... ) -> List[Dict[str, Any]]
>>> T2Utils.to_pandas(
...     infile: str,
...     delimiter: str = None
... ) -> pandas.core.frame.DataFrame
>>> T2Utils.to_pdf(
...     pcap: str = None,
...     flow_file: str = None,
...     prefix: str = None,
...     config: bool = True,
...     reset_config: bool = True,
...     open_pdf: bool = True
... )
>>> T2Utils.unload(
...     plugin: Union[str, List[str]],
...     plugin_folder: str = None
... )
>>> T2Utils.valid_plugin_names() -> List[str]

T2Plugin.py: represent a plugin

For more details, refer to the t2py.T2Plugin API.

T2.py: manage several plugins and run T2, convert/display flow file, …

For more details, refer to the t2py.T2 API.

>>> from t2py import T2

>>> t2 = T2()

# Read-Only Properties

>>> t2.default_plugin_folder  # -> str
>>> t2.plugins                # -> Dict[str, T2Plugin]
>>> t2.t2_exec                # -> str
>>> t2.tranalyzer2            # -> T2Plugin

# Read/Write Properties

>>> t2.plugin_folder          # -> str
>>> t2.loading_list           # -> str

# Functions

>>> t2.add_output_format(
...     extension: Union[str, List[str]]
... )
>>> t2.add_plugin(
...     plugin: str
... )
>>> t2.add_plugins(
...     plugins: List[str]
... )
>>> t2.apply_changes()
>>> t2.build(
...     plugin: Union[str, List[str]] = None,
...     plugin_folder: str = None,
...     force_rebuild: bool = False,
...     debug: bool = False
... )
>>> t2.clean(
...     plugin: Union[str, List[str]] = None
... )
>>> t2.clear_plugins()
>>> t2.create_plugin_list(
...     plugins: List[str] = None,
...     outfile: str = None
... )
>>> t2.discard_changes()
>>> t2.flow_file() -> str
>>> t2.flow_file_json() -> str
>>> t2.flow_file_txt() -> str
>>> t2.flows() -> List[Dict[str, Any]]
>>> t2.flows_json() -> List[Dict[str, Any]]
>>> t2.flows_txt() -> List[Dict[str, Any]]
>>> t2.headers() -> str
>>> t2.headers_file() -> str
>>> t2.list_plugins()
>>> t2.log() -> str
>>> t2.log_file() -> str
>>> t2.packet_file() -> str
>>> t2.packets() -> List[Dict[str, Any]]
>>> t2.print_flows()
>>> t2.print_flows_json()
>>> t2.print_flows_txt()
>>> t2.print_headers()
>>> t2.print_log()
>>> t2.print_packets()
>>> t2.print_report()
>>> t2.remove_plugin(
...     plugin: str
... )
>>> t2.remove_plugins(
...     plugins: List[str]
... )
>>> t2.report() -> str
>>> t2.reset()
>>> t2.run(
...     pcap: str = None,
...     iface: str = None,
...     pcap_list: Union[str, List[str]] = None,
...     output_prefix: str = None,
...     packet_mode: bool = False,
...     plugin_folder: str = None,
...     plugins: List[str] = None,
...     loading_list: str = None,
...     bpf: str = None,
...     rebuild: bool = False,
...     timeout: int = None
... )
>>> t2.set_plugins(
...     plugins: List[str] = None
... )
>>> t2.status()
>>> t2.to_pandas(
...     infile: str = None,
...     delimiter: str = None
... ) -> pandas.core.frame.DataFrame
>>> t2.unload(
...     plugin: Union[str, List[str]] = None,
...     plugin_folder: str = None
... )

# In addition, each plugin is accessible as a T2Plugin object:

>>> t2.myPlugin

API Documentation

For the complete documentation with examples, refer to the t2py API documentation.

Getting Help

To list the variables and functions available, run one of the following commands:

The documentation for each module can be accessed as follows:

Sample Sessions using t2py

Let’s see how to use t2py for real!

Simple Operations with the T2Utils Module

In this example, we will be configuring, building and performing some simple operations with the T2Utils module.

First, open a terminal and start a t2py session:

$ t2py
>>>

Ok, let’s start by configuring and building the basicStats plugin:

Let’s build tranalyzer2 and add the basicFlow, tcpStates and txtSink plugins, so we produce something useful!

We are now ready to run tranalyzer2 against a PCAP file! We just have to tell T2 which plugins to load, to activate the packet mode and to instruct T2 to save the results in the ~/results folder:

Let’s reset the configuration of the basicStats plugin:

Plugin Configuration with the T2Plugin Module

In this example, we will be working with the basicStats plugin.

First, open a terminal and start a t2py session:

$ t2py basicStats
>>>

Let us make sure the plugin was successfully created:

First of all, let us inspect some basic properties of the plugin, namely its name, description and number:

Second, let us inspect the available flags and their default values:

Let us inspect and then toggle the BS_AGGR_CNT and BS_VAR and BS_STDDEV flags:

>>> basicStats.BS_AGGR_CNT
0
>>> basicStats.BS_AGGR_CNT = 1
>>> basicStats.BS_AGGR_CNT
1
>>> basicStats.BS_VAR
0
>>> basicStats.BS_VAR = 1
>>> basicStats.BS_VAR
1
>>> basicStats.BS_STDDEV
1
>>> basicStats.BS_STDDEV = 0
>>> basicStats.BS_STDDEV
0
>>>

Note that we could have used the special values 'yes' and 'no' instead of 0 and 1:

Let us inspect our (pending) changes:

Looks good! Let’s apply our changes!

Let us now inspect our pending changes again:

OK, no more pending changes! But let us make sure the changes have been applied:

Excellent, now we can build the plugin:

Now we can either run T2 with the T2 module, the T2Utils module or via a standard shell command:

$ t2 -r /home/user/data/annoloc2.pcap -w /home/user/results/
...
$

When we do not need the basicStats plugin anymore, we can unload it (remove it from the plugin folder) and clean up the temporary building files:

It is a good habit to restore a plugin configuration to its default once finished:

Let us inspect the changes using the status() function this time:

OK, let us now apply the changes:

Tranalyzer2 Execution Session with the T2 Module

In this example, we will be working with Tranalyzer2, a set of plugins and a pcap file.

First, open a terminal and start a t2py session:

$ t2py
>>>

All the parameters can be specified when the T2 object is created or later by using the appropriate properties or functions as illustrated in this example.

Let us start by selecting some plugins:

OK, now it is time to choose an appropriate output format:

Nice, the required sink plugins were automatically added!

Each plugin is represented by a T2Plugin object, and can be indepently configured as discussed in the previous Plugin Configuration with the T2Plugin Module section.

As a reminder, let us activate BS_AGGR_CNT in basicStats:

Let us now configure tranalyzer2 itself! It is also available in the T2 object as a T2Plugin object:

OK, time to start thinking about running tranalyzer! We want to select a PCAP file and output the flow file in the ~/results/ folder:

Now we are almost set! We still have to apply our changes and then run tranalyzer.

But before that, let us inspect all the changes pending, using the status() function on the T2 object:

Looks good! Let us apply the changes and build everything:

Time to run tranalyzer! Let us do a run with the packet mode and a BPF filter:

Now it is time to analyse the output! Let’s start with the header and log files:

The log (report) file can be accessed with t2.log()/t2.print_log() or t2.report()/t2.print_report():

Now the good stuff, namely the flow file:

>>> t2.flow_file()
'/home/user/results/annoloc2_flows.json'
>>> t2.flows()
[..., {'dir': 'B', 'flowInd': 1030, 'flowStat': '0x0400000200004001', 'timeFirst': '1022171701.877349', 'timeLast': '1022171726.639232', 'duration': '24.761883', 'numHdrDesc': 1, 'numHdrs': [3], 'hdrDesc': ['eth:ipv4:tcp'], 'srcMac': ['00:01:02:b4:36:56'], 'dstMac': ['00:d0:02:6d:78:00'], 'ethType': '0x0800', 'srcIP': '138.212.187.109', 'srcIPCC': 'jp', 'srcIPOrg': 'ASAHI KASEI CORPORATION', 'srcPort': 80, 'dstIP': '133.26.84.187', 'dstIPCC': 'jp', 'dstIPOrg': 'Meiji University', 'dstPort': 4766, 'l4Proto': 6, 'numPktsSnt': 2729, 'numPktsRcvd': 1692, 'numPktsRTAggr': 4421, 'numBytesSnt': 3970812, 'numBytesRcvd': 0, 'numBytesRTAggr': 3970812, 'minPktSz': 0, 'maxPktSz': 1460, 'avePktSize': 1455.0428466796875, 'stdPktSize': 70.65760803222656, 'minIAT': 0.0, 'maxIAT': 0.48004499077796936, 'aveIAT': 0.009073597379028797, 'stdIAT': 0.03961425647139549, 'pktps': 110.209716796875, 'bytps': 160359.859375, 'pktAsm': 0.23456232249736786, 'bytAsm': 1.0, 'tcpFStat': '0x0040', 'ipMindIPID': 1, 'ipMaxdIPID': 8, 'ipMinTTL': 64, 'ipMaxTTL': 64, 'ipTTLChg': 0, 'ipToS': '0x00', 'ipFlags': '0x1840', 'ipOptCnt': 0, 'ipOptCpCl_Num': ['0x00', '0x00000000'], 'ip6OptCntHH_D': [0, 0], 'ip6OptHH_D': ['0x00000000', '0x00000000'], 'tcpISeqN': 1167186881, 'tcpPSeqCnt': 2653, 'tcpSeqSntBytes': 4535488, 'tcpSeqFaultCnt': 64, 'tcpPAckCnt': 0, 'tcpFlwLssAckRcvdBytes': 0, 'tcpAckFaultCnt': 2, 'tcpInitWinSz': 6432, 'tcpAveWinSz': 6432.0, 'tcpMinWinSz': 6432, 'tcpMaxWinSz': 6432, 'tcpWinSzDwnCnt': 0, 'tcpWinSzUpCnt': 0, 'tcpWinSzChgDirCnt': 0, 'tcpWinSzThRt': 0.0, 'tcpFlags': '0x98', 'tcpAnomaly': '0xb800', 'tcpOptPktCnt': 0, 'tcpOptCnt': 0, 'tcpOptions': '0x00000000', 'tcpMSS': 0, 'tcpWS': 1, 'tcpMPTBF': '0x0000', 'tcpMPF': '0x00', 'tcpMPAID': 0, 'tcpMPdssF': '0x00', 'tcpTmS': 0, 'tcpTmER': 0, 'tcpEcI': 0.0, 'tcpUtm': 0.0, 'tcpBtm': '0.000000', 'tcpSSASAATrip': 0.0, 'tcpRTTAckTripMin': 0.0, 'tcpRTTAckTripMax': 0.4159089922904968, 'tcpRTTAckTripAve': 0.0032711897511035204, 'tcpRTTAckTripJitAve': 0.018527768552303314, 'tcpRTTSseqAA': 0.018186353147029877, 'tcpRTTAckJitAve': 0.043254632502794266, 'tcpStatesAFlags': '0x03'}]
>>>

Let’s now extract all the unique source/destination IP pairs involved in an ICMP flow!

Note that for the _flows.txt file, all values are treated as strings, so to match ICMP, you would need to write flow['l4Proto'] == '1' instead.

Let’s start investigating the biggest talkers in terms of flows with Panda:

One last example displaying the biggest talkers in terms of ICMP flows:

Subject to Change

This library is still experimental and at an early stage of development. As such, the API may be subject to change.

Below is a list of functions or properties which may be removed in a future version.

Feel free to send us bug reports, feature requests, feedback and suggestions!

T2Plugin

  • The list_config() function may be removed as the same result can be obtained with the flags property.
  • The set_default() function may be removed as the same result can be obtained with the reset() function.
  • The get_default() function may be removed as the same result can be obtained with the default property.

T2

Bug Reports, Feature Requests, Feedback and Suggestions

Feel free to send us bug reports, feature requests, feedback and suggestions!