Alphabetical List of TAWK Functions

tawk

Introduction

This sections presents an overview of all tawk functions available.

  • Parameters between brackets are optional,
  • IPs can be given as string ("1.2.3.4"), hexadecimal (0xffffffff) or int (4294967295),
  • Network masks can be given as string ("255.255.255.0"), hexadecimal (0xffffff00) or CIDR notation (24),
  • Networks can be given as string, hexadecimal or int, e.g., "1.2.3.4/24" or "0x01020304/255.255.255.0",
  • String functions can be made case insensitive by adding the suffix i, e.g., streq (case sensitive) becomes streqi (case insensitive),

  • More details and examples can be found for every function by running tawk -d funcname.

  • Examples are listed here and can be loaded with tawk -e option.

  • t2nfdump functions are listed here and can be loaded with tawk -n option.

Tawk functions

A: abs, aggr, aggrrep, ah

abs(v) Return the absolute value of v.
aggr(fields[,val[,num]]) Perform aggregation of fields and store the sum of val.
fields and val can be tab separated lists of fields, e.g., $srcIP4 "\t" $dstIP4.
Results are sorted according to the first value of val.
If val is omitted, the empty string or equal to "flows" or "packets"
(case insensitive), count the number of records (flows or packets).
If num is omitted or 0, return the full list,
If num > 0 return the top num results,
If num < 0 return the bottom num results.
aggrrep(fields[,val[,num[,ign_e[,sep]]]]) Perform aggregation of the repetitive fields and store the sum of val.
val can be a tab separated lists of fields, e.g., $numBytesSnt "\t" $numPktsSnt.
Results are sorted according to the first value of val.
If val is omitted, the empty string or equal to "flows" or "packets"
(case insensitive), count the number of records (flows or packets).
If num is omitted or 0, return the full list,
If num > 0 return the top num results,
If num < 0 return the bottom num results.
If ign_e is omitted or 0, consider all values, otherwise ignore empty values.
sep can be used to change the separator character (default: ";")
ah([p]) Return true if the protocol is equal to 51 (AH).

B: base64, base64d, bfeq, bitsallset, bitsanyset, bitshift

base64(s) Encode a string s as base64.
base64d(s) Decode a base64 encoded string s.
bfeq(val1,val2) Return true if the hexadecimal numbers val1 and val2 are equal.
bitsallset(val,mask) Return true if all the bits set in mask are also set in val.
bitsanyset(val,mask) Return true if one of the bits set in mask is also set in val.
bitshift(n[,t[,d[,b]]]) Shift a byte or a list of bytes n to the left or right by a given number of bits t.
To shift to the left, set d to 0 (default), to shift to the right, set d != 0
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45)

C: chomp, contains, cvalcontains

chomp(s) Remove leading and trailing spaces from a string.
contains(val,txt) Return true if val contains the substring txt.
cvalcontains(val,item) Alias for valcontains(val, "_", item).

D: dhcp, dhost, diff, dnet, dns, dport

dhcp() Return true if the flow contains DHCP traffic.
dhost([ip|net]) Return true if the destination IP is equal to ip or belongs to net.
If ip is omitted, return the destination IP.
diff(file[,mode]) Compare file and the input, and print the name of the columns which differ.
The mode parameter can be used to control the format of the output.
dnet([ip|net]) Alias for dhost([ip|net]).
dns() Return true if the flow contains DNS traffic.
dport([p]) Return true if the destination port appears in p (comma or semicolon separated)
Ranges may also be specified using a dash, e.g., dport("80-90").
If p is omitted, return the destination port.

E: esp

esp([p]) Return true if the protocol is equal to 50 (ESP).

F: ffsplit, flow, follow_stream

ffsplit([s[,k[,h]]]) Split the input file into smaller more manageable files.
The files to create can be specified as argument to the function (one comma separated string).
If no argument is specified, create one file per column whose name ends with Stat,
e.g., dnsStat, and one for pwxType (pw).
If k > 0, then only print relevant fields and those controlled by h, a comma separated list of
fields to keep in each file, e.g., "srcIP,dstIP".
flow([f]) Return all flows whose index appears in f (comma or semicolon separated).
Ranges may also be specified using a dash, e.g., flow("1-3")
If f is omitted, return the flow index.
follow_stream(f[,of[,d[,pf[,r[,nc]]]]]) Return the payload of the flow with index f.
of can be used to change the output format:
- 0: Payload only [default],
- 1: prefix each payload with packet/flow info,
- 2: JSON,
- 3: Reconstruct (pipe the output to xxd -p -r to reproduce the binary file).
d can be used to only extract a specific direction ("A" or "B") [default: "" (A and B)].
pf can be used to change the payload format:
- 0: ASCII [default],
- 1: Hexdump,
- 2: Raw/Binary,
- 3: Base64.
r can be used to prevent the analysis of TCP sequence numbers (no TCP reassembly and reordering).
nc can be used to print the data without colors.

G: gre

gre([p]) Return true if the protocol is equal to 47 (GRE).

H: haspreffix, hassuffix, hdr, host, hrnum, http

hasprefix(val,pre) Return true if val begins with the prefix pre.
hassuffix(val,suf) Return true if val finishes with the suffix suf.
hdr() Use this function in your tests to keep the header (column names).
host([ip|net]) Return true if the source or destination IP is equal to ip or belongs to net.
If ip is omitted, return the source and destination IP.
hrnum(num[,mode[,suffix]]) Convert the number num to its human readable form.
http() Return true if the flow contains HTTP traffic.

I: icmp, icmp6, igmp, isfloat, isint, isip, isip6, isiphex, isipnum, isipstr, isnum, isuint, ip, ip2mask, ip2num, ip2hex, ip2str, ip62str, ip6compress, ip6expand, ipinnet, ipinrange, ipv4, ipv6

icmp([p]) Return true if the protocol is equal to 1 (ICMP).
icmp6([p]) Return true if the protocol is equal to 58 (ICMPv6).
igmp([p]) Return true if the protocol is equal to 2 (IGMP).
isfloat(v) Return true if v is a floating point number.
isint(v) Return true if v is an integer.
isip(v) Return true if v is an IPv4 address in hexadecimal, numerical or dotted decimal notation.
isip6(v) Return true if v is an IPv6 address.
isiphex(v) Return true if v is an IPv4 address in hexadecimal notation.
isipnum(v) Return true if v is an IPv4 address in numerical (int) notation.
isipstr(v) Return true if v is an IPv4 address in dotted decimal notation.
isnum(v) Return true if v is a number (signed, unsigned or floating point).
isuint(v) Return true if v is an unsigned integer.
ip() Return true if the flow contains IPv4 or IPv6 traffic.
ip2mask(ip) Convert an IP address to a network mask (int).
ip2num(ip) Convert an IP address to a number.
ip2hex(ip) Convert an IPv4 address to hex.
ip2str(ip) Convert an IPv4 address to string.
ip62str(ip) Convert an IPv6 address to string.
ip6compress(ip) Compress an IPv6 address.
ip6expand(ip[,trim]) Expand an IPv6 address.
If trim != 0, remove leading zeros.
ipinnet(ip,net[,mask]) Test whether an IP address belongs to a given network.
ipinrange(ip,low,high) Test whether an IP address lies between two addresses.
ipv4() Return true if the flow contains IPv4 traffic.
ipv6() Return true if the flow contains IPv6 traffic.

J: join, json

join(a,s) Convert an array to string, separating each value with s.
json([s]) Convert the string s to JSON. The first record is used as column names. If s is omitted, convert the entire row ($0)

L: localtime, loopback, lstrip

localtime(t) Convert UNIX timestamp to string (localtime).
loopback(ip) Return true if ip is a loopback address.
lstrip(s) Remove leading spaces from a string.

M: mask2ip, mask2ip6, mask2ip6str, mask2ipstr, max, max2, max3, mcast, mean, min, min2, min3

mask2ip(m) Convert a network mask (int) to an IPv4 address (int).
mask2ip6(m) Convert a network mask (int) to an IPv6 address (int).
mask2ip6str(m) Convert a network mask (int) to an IPv6 address (string).
mask2ipstr(m) Convert a network mask (int) to an IPv4 address (string).
max(c) Keep track of the max value of a column c.
The result can be accessed with get_max(c) or printed with print_max([c]).
max2(a,b) Return the maximum value between a and b.
max3(a,b,c) Return the maximum value between a, b and c.
mcast(ip) Return true if ip is a multicast address.
mean(c) Compute the mean value of a column c.
The result can be accessed with get_mean(c) or printed with print_mean([c]).
min(c) Keep track of the min value of a column c.
The result can be accessed with get_min(c) or printed with print_min([c]).
min2(a,b) Return the minimum value between a and b.
min3(a,b,c) Return the minimum value between a, b and c.

N: net, nibble_swap, not

net([ip|net]) Alias for host([ip|net]).
nibble_swap(n[,b]) Swap the nibbles of a byte or of a list of bytes n.
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45)
not(q) Return the logical negation of a query q.
This function can be used to keep the header when negating a query.

P: packet, port, printerr, privip, proto, proto2str

packet([p]) Return all packets whose index appears in p (comma or semicolon separated).
Ranges may also be specified using a dash, e.g., packet("1-3")
If p is omitted, return the packet number.
port([p]) Return true if the source or destination port appears in p (comma or semicolon separated)
Ranges may also be specified using a dash, e.g., port("80-90").
If p is omitted, return the source and destination ports.
printerr(s) Print the string s in red with an added newline.
privip(ip) Return true if ip is a private IP.
proto([p]) Return true if the protocol number appears in p (comma or semicolon separated).
Ranges may also be specified using a dash, e.g., proto("1-3").
If p is omitted, return the protocol number.
proto2str(p) Return the string representation of the protocol number p.
If p is omitted, return the string representation of the protocol.

R: rstrip, rsvp, rvalcontains

rstrip(s) Remove trailing spaces from a string.
rsvp([p]) Return true if the protocol is equal to 46 (RSVP).
rvalcontains(val,item) Alias for valcontains(val, ";", item).

S: sctp, shark, shost, snet, splitc, splitr, sport, streq, strip, strisempty, strneq

sctp([p]) Return true if the protocol is equal to 132 (SCTP).
shark(q) Query flow files according to Wireshark’s syntax.
shost([ip|net]) Return true if the source IP is equal to ip or belongs to net.
If ip is omitted, return the source IP.
snet([ip|net]) Alias for shost([ip|net]).
splitc(val[,num[,osep]]) Split compound values. Alias for t2split(val, "_", num, osep).
splitr(val[,num[,osep]]) Split repetitive values. Alias for t2split(val, ";", num, osep).
sport([p]) Return true if the source port appears in p (comma or semicolon separated)
Ranges may also be specified using a dash, e.g., sport("80-90").
If p is omitted, return the source port.
streq(val1,val2) Return true if val1 is equal to val2.
strip(s) Remove leading and trailing spaces from a string.
strisempty(val) Return true if val is an empty string.
strneq(val1,val2) Return true if val1 and val2 are not equal.

T: t2rsort, t2sort, t2split, t2whois, tcp, tcpflags, texscape, timestamp, tobits, tuple2, tuple3, tuple4, tuple5, tuple6

t2rsort(col[,num[,type]]) Sort the file in reverse order according to col.
(Multiple column numbers can be specified by using ; as separator, e.g., 1 ";" 2)
If num is omitted or 0, return the full list,
If num > 0 return the top num results,
If num < 0 return the bottom num results.
type can be used to specify the type of data to sort:
"ip", "num" or "str" (default is based on the first matching record).
t2sort(col[,num[,type[,rev]]]) Sort the file according to col.
(Multiple column numbers can be specified by using ; as separator, e.g., 1 ";" 2)
If num is omitted or 0, return the full list,
If num > 0 return the top num results,
If num < 0 return the bottom num results.
type can be used to specify the type of data to sort:
"ip", "num" or "str" (default is based on the first matching record).
If rev > 0, sort in reverse order (alternatively, use the t2rsort() function).
t2split(val,sep[,num[,osep]]) Split values according to sep.
If num is omitted or 0, val is split into osep separated columns.
If num > 0, return the num repetition.
If num < 0, return the num repetition from the end, e.g., -1 for last element.
Multiple num can be specified, e.g., "1;-1;2".
Output separator osep, defaults to OFS.
t2whois(ip[,o_opt]) Wrapper to call t2whois from tawk.
ip must be a valid IPv4 or IPv6 address.
o_opt is passed verbatim to t2whois -o option
(run t2whois -L for more details).
tcp([p]) Return true if the protocol is equal to 6 (TCP).
tcpflags([val]) If val is specified, return true if the specified flags are set.
If val is omitted, return a string representation of the TCP flags.
texscape(s) Escape the string s to make it LaTeX compatible.
timestamp(t) Convert date to UNIX timestamp.
tobits(u,[b]) Convert the unsigned integer u to its binary representation.
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45)
tuple2() Return the 2 tuple (source IP and destination IP).
tuple3() Return the 3 tuple (source IP, destination IP and port).
tuple4() Return the 4 tuple (source IP and port, destination IP and port).
tuple5() Return the 5 tuple (source IP and port, destination IP and port, protocol).
tuple6() Return the 6 tuple (source IP and port, destination IP and port, proto, VLANID).

U: udp, unquote, urldecode, utc

udp([p]) Return true if the protocol is equal to 17 (UDP).
unquote(s) Remove leading and trailing quotes from a string.
urldecode(url) Decode the encoded URL url.
utc(t) Convert UNIX timestamp to string (UTC).

V: valcontains

valcontains(val,sep,item) Return true if one item of val split by sep is equal to item.

W: wildcard

wildcard(expr) Print all columns whose name matches the regular expression expr.
If expr is preceded by an exclamation mark, return all columns whose name does NOT match expr.

Alphabetical List of TAWK Examples

Collection of examples using tawk functions. More advanced examples can be found in t2fm.

Those functions require the use of tawk -e option.

dnsZT() Return all flows where a DNS zone transfer was performed.
exeDL([n]) Return the top N EXE downloads.
httpHostsURL([f]) Return all HTTP hosts and a list of the files hosted (sorted alphabetically.
If f > 0, print the number of times a URL was requested.
nonstdports() Return all flows running protocols over non-standard ports.
passivedns() Extract all DNS server replies from a flow file.
The following information is reported for each reply:
FirstSeen, LastSeen, Type (A or AAAA), TTL, Query, Answer, Organization, Country, AS number.
passwords([val[,num]]) Return information about hosts sending authentication in cleartext.
If val is omitted or equal to "flows", count the number of flows.
Otherwise, sum up the values of val.
If num is omitted or 0, returns the full list,
If num > 0 return the top num results,
If num < 0 return the bottom num results.
postQryStr([n]) Return the top N POST requests with query strings.
ssh() Return the SSH connections.
topDnsA([n]) Return the top N DNS answers.
topDnsIp4([n]) Return the top N DNS answers IPv4 addresses.
topDnsIp6([n]) Return the top N DNS answers IPv6 addresses.
topDnsQ([n]) Return the top N DNS queries.
topHttpMimesST([n]) Return the top HTTP content-type (type/subtype).
topHttpMimesT([n]) Return the top HTTP content-type (type only).
topSLD([n]) Return the top N second-level domains queried (google.com, yahoo.com, …).
topTLD([n]) Return the top N top-level domains (TLD) queried (.com, .net, …).

Alphabetical List of TAWK nfdump functions

Collection of functions for tawk allowing access to specific fields using a syntax similar as nfdump.

Those functions require the use of tawk -n option.

bpp() Bytes per package
bps() Bits per second
byt() Bytes - default input
da() Destination Address
dap() Destination Address:Port
dp() Destination Port
flg() TCP Flags
ibyt() Input Bytes
ipkt() Input Packets
mpls() MPLS labels 1-10
mpls1() MPLS label 1
mpls2() MPLS label 2
mpls3() MPLS label 3
mpls4() MPLS label 4
mpls5() MPLS label 5
mpls6() MPLS label 6
mpls7() MPLS label 7
mpls8() MPLS label 8
mpls9() MPLS label 9
mpls10() MPLS label 10
obyt() Output Bytes
oextended() nfdump extended output format (-o extended)
oline() nfdump line output format (-o line)
olong() nfdump long output format (-o long)
opkt() Output Packets
pkt() Packets - default input
pps() Packets per second
pr() Protocol
sa() Source Address
sap() Source Address:Port
sp() Source Port
td() Duration
te() End Time - last seen
ts() Start Time - first seen