Alphabetical List of TAWK Functions
tawkContents
- Introduction
- Tawk functions
- A: abs, aggr, aggrrep, ah
- B: base64, base64d, bfeq, bitsallset, bitsanyset, bitshift
- C: chomp, contains, cvalcontains
- D: dhcp, dhost, diff, dnet, dns, dport
- E: esp
- F: ffsplit, flow, follow_stream
- G: gre
- H: haspreffix, hassuffix, hdr, host, hrnum, http
- I: icmp, icmp6, igmp, isfloat, isint, isip, isip6, isiphex, isipnum, isipstr, isnum, isuint, ip, ip2mask, ip2num, ip2hex, ip2str, ip62str, ip6compress, ip6expand, ipinnet, ipinrange, ipv4, ipv6
- J: join, json
- L: localtime, loopback, lstrip
- M: mask2ip, mask2ip6, mask2ip6str, mask2ipstr, max, max2, max3, mcast, mean, min, min2, min3
- N: net, nibble_swap, not
- P: packet, port, printerr, privip, proto, proto2str
- R: rstrip, rsvp, rvalcontains
- S: sctp, shark, shost, snet, splitc, splitr, sport, streq, strip, strisempty, strneq
- T: t2rsort, t2sort, t2split, t2whois, tcp, tcpflags, texscape, timestamp, tobits, tuple2, tuple3, tuple4, tuple5, tuple6
- U: udp, unquote, urldecode, utc
- V: valcontains
- W: wildcard
- Alphabetical List of TAWK Examples
- Alphabetical List of TAWK nfdump functions
Introduction
This sections presents an overview of all tawk functions available.
- Parameters between brackets are optional,
- IPs can be given as string (
"1.2.3.4"
), hexadecimal (0xffffffff
) or int (4294967295
), - Network masks can be given as string (
"255.255.255.0"
), hexadecimal (0xffffff00
) or CIDR notation (24
), - Networks can be given as string, hexadecimal or int, e.g.,
"1.2.3.4/24"
or"0x01020304/255.255.255.0"
, - String functions can be made case insensitive by adding the suffix
i
, e.g.,streq
(case sensitive) becomesstreqi
(case insensitive), More details and examples can be found for every function by running
tawk -d funcname
.- Examples are listed here and can be loaded with
tawk -e
option. t2nfdump
functions are listed here and can be loaded withtawk -n
option.
Tawk functions
A: abs, aggr, aggrrep, ah
abs(v) |
Return the absolute value of v . |
aggr(fields[,val[,num]]) |
Perform aggregation of fields and store the sum of val . |
fields and val can be tab separated lists of fields, e.g., $srcIP4 "\t" $dstIP4 . |
|
Results are sorted according to the first value of val . |
|
If val is omitted, the empty string or equal to "flows" or "packets" |
|
(case insensitive), count the number of records (flows or packets). | |
If num is omitted or 0 , return the full list, |
|
If num > 0 return the top num results, |
|
If num < 0 return the bottom num results. |
|
aggrrep(fields[,val[,num[,ign_e[,sep]]]]) |
Perform aggregation of the repetitive fields and store the sum of val . |
val can be a tab separated lists of fields, e.g., $numBytesSnt "\t" $numPktsSnt . |
|
Results are sorted according to the first value of val . |
|
If val is omitted, the empty string or equal to "flows" or "packets" |
|
(case insensitive), count the number of records (flows or packets). | |
If num is omitted or 0 , return the full list, |
|
If num > 0 return the top num results, |
|
If num < 0 return the bottom num results. |
|
If ign_e is omitted or 0 , consider all values, otherwise ignore empty values. |
|
sep can be used to change the separator character (default: ";" ) |
|
ah([p]) |
Return true if the protocol is equal to 51 (AH). |
B: base64, base64d, bfeq, bitsallset, bitsanyset, bitshift
base64(s) |
Encode a string s as base64. |
base64d(s) |
Decode a base64 encoded string s . |
bfeq(val1,val2) |
Return true if the hexadecimal numbers val1 and val2 are equal. |
bitsallset(val,mask) |
Return true if all the bits set in mask are also set in val . |
bitsanyset(val,mask) |
Return true if one of the bits set in mask is also set in val . |
bitshift(n[,t[,d[,b]]]) |
Shift a byte or a list of bytes n to the left or right by a given number of bits t . |
To shift to the left, set d to 0 (default), to shift to the right, set d != 0 |
|
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45 ) |
C: chomp, contains, cvalcontains
chomp(s) |
Remove leading and trailing spaces from a string. |
contains(val,txt) |
Return true if val contains the substring txt . |
cvalcontains(val,item) |
Alias for valcontains(val, "_", item) . |
D: dhcp, dhost, diff, dnet, dns, dport
dhcp() |
Return true if the flow contains DHCP traffic. |
dhost([ip|net]) |
Return true if the destination IP is equal to ip or belongs to net . |
If ip is omitted, return the destination IP. |
|
diff(file[,mode]) |
Compare file and the input, and print the name of the columns which differ. |
The mode parameter can be used to control the format of the output. |
|
dnet([ip|net]) |
Alias for dhost([ip|net]) . |
dns() |
Return true if the flow contains DNS traffic. |
dport([p]) |
Return true if the destination port appears in p (comma or semicolon separated) |
Ranges may also be specified using a dash, e.g., dport("80-90") . |
|
If p is omitted, return the destination port. |
E: esp
esp([p]) |
Return true if the protocol is equal to 50 (ESP). |
F: ffsplit, flow, follow_stream
ffsplit([s[,k[,h]]]) |
Split the input file into smaller more manageable files. |
The files to create can be specified as argument to the function (one comma separated string). | |
If no argument is specified, create one file per column whose name ends with Stat , |
|
e.g., dnsStat , and one for pwxType (pw ). |
|
If k > 0 , then only print relevant fields and those controlled by h , a comma separated list of |
|
fields to keep in each file, e.g., "srcIP,dstIP" . |
|
flow([f]) |
Return all flows whose index appears in f (comma or semicolon separated). |
Ranges may also be specified using a dash, e.g., flow("1-3") |
|
If f is omitted, return the flow index. |
|
follow_stream(f[,of[,d[,pf[,r[,nc]]]]]) |
Return the payload of the flow with index f . |
of can be used to change the output format: |
|
- 0 : Payload only [default], |
|
- 1 : prefix each payload with packet/flow info, |
|
- 2 : JSON, |
|
- 3 : Reconstruct (pipe the output to xxd -p -r to reproduce the binary file). |
|
d can be used to only extract a specific direction ("A" or "B" ) [default: "" (A and B)]. |
|
pf can be used to change the payload format: |
|
- 0 : ASCII [default], |
|
- 1 : Hexdump, |
|
- 2 : Raw/Binary, |
|
- 3 : Base64. |
|
r can be used to prevent the analysis of TCP sequence numbers (no TCP reassembly and reordering). |
|
nc can be used to print the data without colors. |
G: gre
gre([p]) |
Return true if the protocol is equal to 47 (GRE). |
H: haspreffix, hassuffix, hdr, host, hrnum, http
hasprefix(val,pre) |
Return true if val begins with the prefix pre . |
hassuffix(val,suf) |
Return true if val finishes with the suffix suf . |
hdr() |
Use this function in your tests to keep the header (column names). |
host([ip|net]) |
Return true if the source or destination IP is equal to ip or belongs to net . |
If ip is omitted, return the source and destination IP. |
|
hrnum(num[,mode[,suffix]]) |
Convert the number num to its human readable form. |
http() |
Return true if the flow contains HTTP traffic. |
I: icmp, icmp6, igmp, isfloat, isint, isip, isip6, isiphex, isipnum, isipstr, isnum, isuint, ip, ip2mask, ip2num, ip2hex, ip2str, ip62str, ip6compress, ip6expand, ipinnet, ipinrange, ipv4, ipv6
icmp([p]) |
Return true if the protocol is equal to 1 (ICMP). |
icmp6([p]) |
Return true if the protocol is equal to 58 (ICMPv6). |
igmp([p]) |
Return true if the protocol is equal to 2 (IGMP). |
isfloat(v) |
Return true if v is a floating point number. |
isint(v) |
Return true if v is an integer. |
isip(v) |
Return true if v is an IPv4 address in hexadecimal, numerical or dotted decimal notation. |
isip6(v) |
Return true if v is an IPv6 address. |
isiphex(v) |
Return true if v is an IPv4 address in hexadecimal notation. |
isipnum(v) |
Return true if v is an IPv4 address in numerical (int) notation. |
isipstr(v) |
Return true if v is an IPv4 address in dotted decimal notation. |
isnum(v) |
Return true if v is a number (signed, unsigned or floating point). |
isuint(v) |
Return true if v is an unsigned integer. |
ip() |
Return true if the flow contains IPv4 or IPv6 traffic. |
ip2mask(ip) |
Convert an IP address to a network mask (int). |
ip2num(ip) |
Convert an IP address to a number. |
ip2hex(ip) |
Convert an IPv4 address to hex. |
ip2str(ip) |
Convert an IPv4 address to string. |
ip62str(ip) |
Convert an IPv6 address to string. |
ip6compress(ip) |
Compress an IPv6 address. |
ip6expand(ip[,trim]) |
Expand an IPv6 address. |
If trim != 0 , remove leading zeros. |
|
ipinnet(ip,net[,mask]) |
Test whether an IP address belongs to a given network. |
ipinrange(ip,low,high) |
Test whether an IP address lies between two addresses. |
ipv4() |
Return true if the flow contains IPv4 traffic. |
ipv6() |
Return true if the flow contains IPv6 traffic. |
J: join, json
join(a,s) |
Convert an array to string, separating each value with s . |
json([s]) |
Convert the string s to JSON. The first record is used as column names. If s is omitted, convert the entire row ($0 ) |
L: localtime, loopback, lstrip
localtime(t) |
Convert UNIX timestamp to string (localtime). |
loopback(ip) |
Return true if ip is a loopback address. |
lstrip(s) |
Remove leading spaces from a string. |
M: mask2ip, mask2ip6, mask2ip6str, mask2ipstr, max, max2, max3, mcast, mean, min, min2, min3
mask2ip(m) |
Convert a network mask (int) to an IPv4 address (int). |
mask2ip6(m) |
Convert a network mask (int) to an IPv6 address (int). |
mask2ip6str(m) |
Convert a network mask (int) to an IPv6 address (string). |
mask2ipstr(m) |
Convert a network mask (int) to an IPv4 address (string). |
max(c) |
Keep track of the max value of a column c . |
The result can be accessed with get_max(c) or printed with print_max([c]) . |
|
max2(a,b) |
Return the maximum value between a and b . |
max3(a,b,c) |
Return the maximum value between a , b and c . |
mcast(ip) |
Return true if ip is a multicast address. |
mean(c) |
Compute the mean value of a column c . |
The result can be accessed with get_mean(c) or printed with print_mean([c]) . |
|
min(c) |
Keep track of the min value of a column c . |
The result can be accessed with get_min(c) or printed with print_min([c]) . |
|
min2(a,b) |
Return the minimum value between a and b . |
min3(a,b,c) |
Return the minimum value between a , b and c . |
N: net, nibble_swap, not
net([ip|net]) |
Alias for host([ip|net]) . |
nibble_swap(n[,b]) |
Swap the nibbles of a byte or of a list of bytes n . |
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45 ) |
|
not(q) |
Return the logical negation of a query q . |
This function can be used to keep the header when negating a query. |
P: packet, port, printerr, privip, proto, proto2str
packet([p]) |
Return all packets whose index appears in p (comma or semicolon separated). |
Ranges may also be specified using a dash, e.g., packet("1-3") |
|
If p is omitted, return the packet number. |
|
port([p]) |
Return true if the source or destination port appears in p (comma or semicolon separated) |
Ranges may also be specified using a dash, e.g., port("80-90") . |
|
If p is omitted, return the source and destination ports. |
|
printerr(s) |
Print the string s in red with an added newline. |
privip(ip) |
Return true if ip is a private IP. |
proto([p]) |
Return true if the protocol number appears in p (comma or semicolon separated). |
Ranges may also be specified using a dash, e.g., proto("1-3") . |
|
If p is omitted, return the protocol number. |
|
proto2str(p) |
Return the string representation of the protocol number p . |
If p is omitted, return the string representation of the protocol. |
R: rstrip, rsvp, rvalcontains
rstrip(s) |
Remove trailing spaces from a string. |
rsvp([p]) |
Return true if the protocol is equal to 46 (RSVP). |
rvalcontains(val,item) |
Alias for valcontains(val, ";", item) . |
S: sctp, shark, shost, snet, splitc, splitr, sport, streq, strip, strisempty, strneq
sctp([p]) |
Return true if the protocol is equal to 132 (SCTP). |
shark(q) |
Query flow files according to Wireshark’s syntax. |
shost([ip|net]) |
Return true if the source IP is equal to ip or belongs to net . |
If ip is omitted, return the source IP. |
|
snet([ip|net]) |
Alias for shost([ip|net]) . |
splitc(val[,num[,osep]]) |
Split compound values. Alias for t2split(val, "_", num, osep) . |
splitr(val[,num[,osep]]) |
Split repetitive values. Alias for t2split(val, ";", num, osep) . |
sport([p]) |
Return true if the source port appears in p (comma or semicolon separated) |
Ranges may also be specified using a dash, e.g., sport("80-90") . |
|
If p is omitted, return the source port. |
|
streq(val1,val2) |
Return true if val1 is equal to val2 . |
strip(s) |
Remove leading and trailing spaces from a string. |
strisempty(val) |
Return true if val is an empty string. |
strneq(val1,val2) |
Return true if val1 and val2 are not equal. |
T: t2rsort, t2sort, t2split, t2whois, tcp, tcpflags, texscape, timestamp, tobits, tuple2, tuple3, tuple4, tuple5, tuple6
t2rsort(col[,num[,type]]) |
Sort the file in reverse order according to col . |
(Multiple column numbers can be specified by using ; as separator, e.g., 1 ";" 2 ) |
|
If num is omitted or 0 , return the full list, |
|
If num > 0 return the top num results, |
|
If num < 0 return the bottom num results. |
|
type can be used to specify the type of data to sort: |
|
"ip" , "num" or "str" (default is based on the first matching record). |
|
t2sort(col[,num[,type[,rev]]]) |
Sort the file according to col . |
(Multiple column numbers can be specified by using ; as separator, e.g., 1 ";" 2 ) |
|
If num is omitted or 0 , return the full list, |
|
If num > 0 return the top num results, |
|
If num < 0 return the bottom num results. |
|
type can be used to specify the type of data to sort: |
|
"ip" , "num" or "str" (default is based on the first matching record). |
|
If rev > 0 , sort in reverse order (alternatively, use the t2rsort() function). |
|
t2split(val,sep[,num[,osep]]) |
Split values according to sep . |
If num is omitted or 0 , val is split into osep separated columns. |
|
If num > 0 , return the num repetition. |
|
If num < 0 , return the num repetition from the end, e.g., -1 for last element. |
|
Multiple num can be specified, e.g., "1;-1;2" . |
|
Output separator osep , defaults to OFS . |
|
t2whois(ip[,o_opt]) |
Wrapper to call t2whois from tawk . |
ip must be a valid IPv4 or IPv6 address. |
|
o_opt is passed verbatim to t2whois -o option |
|
(run t2whois -L for more details). |
|
tcp([p]) |
Return true if the protocol is equal to 6 (TCP). |
tcpflags([val]) |
If val is specified, return true if the specified flags are set. |
If val is omitted, return a string representation of the TCP flags. |
|
texscape(s) |
Escape the string s to make it LaTeX compatible. |
timestamp(t) |
Convert date to UNIX timestamp. |
tobits(u,[b]) |
Convert the unsigned integer u to its binary representation. |
Set b to 16 to force interpretation as hexadecimal, e.g., interpret 45 as 69 (0x45 ) |
|
tuple2() |
Return the 2 tuple (source IP and destination IP). |
tuple3() |
Return the 3 tuple (source IP, destination IP and port). |
tuple4() |
Return the 4 tuple (source IP and port, destination IP and port). |
tuple5() |
Return the 5 tuple (source IP and port, destination IP and port, protocol). |
tuple6() |
Return the 6 tuple (source IP and port, destination IP and port, proto, VLANID). |
U: udp, unquote, urldecode, utc
udp([p]) |
Return true if the protocol is equal to 17 (UDP). |
unquote(s) |
Remove leading and trailing quotes from a string. |
urldecode(url) |
Decode the encoded URL url . |
utc(t) |
Convert UNIX timestamp to string (UTC). |
V: valcontains
valcontains(val,sep,item) |
Return true if one item of val split by sep is equal to item . |
W: wildcard
wildcard(expr) |
Print all columns whose name matches the regular expression expr . |
If expr is preceded by an exclamation mark, return all columns whose name does NOT match expr . |
Alphabetical List of TAWK Examples
Collection of examples using tawk
functions. More advanced examples can be found in t2fm.
Those functions require the use of tawk -e
option.
dnsZT() |
Return all flows where a DNS zone transfer was performed. |
exeDL([n]) |
Return the top N EXE downloads. |
httpHostsURL([f]) |
Return all HTTP hosts and a list of the files hosted (sorted alphabetically. |
If f > 0 , print the number of times a URL was requested. |
|
nonstdports() |
Return all flows running protocols over non-standard ports. |
passivedns() |
Extract all DNS server replies from a flow file. |
The following information is reported for each reply: | |
FirstSeen, LastSeen, Type (A or AAAA), TTL, Query, Answer, Organization, Country, AS number. | |
passwords([val[,num]]) |
Return information about hosts sending authentication in cleartext. |
If val is omitted or equal to "flows" , count the number of flows. |
|
Otherwise, sum up the values of val . |
|
If num is omitted or 0 , returns the full list, |
|
If num > 0 return the top num results, |
|
If num < 0 return the bottom num results. |
|
postQryStr([n]) |
Return the top N POST requests with query strings. |
ssh() |
Return the SSH connections. |
topDnsA([n]) |
Return the top N DNS answers. |
topDnsIp4([n]) |
Return the top N DNS answers IPv4 addresses. |
topDnsIp6([n]) |
Return the top N DNS answers IPv6 addresses. |
topDnsQ([n]) |
Return the top N DNS queries. |
topHttpMimesST([n]) |
Return the top HTTP content-type (type/subtype). |
topHttpMimesT([n]) |
Return the top HTTP content-type (type only). |
topSLD([n]) |
Return the top N second-level domains queried (google.com, yahoo.com, …). |
topTLD([n]) |
Return the top N top-level domains (TLD) queried (.com, .net, …). |
Alphabetical List of TAWK nfdump functions
Collection of functions for tawk
allowing access to specific fields using a syntax similar as nfdump
.
Those functions require the use of tawk -n
option.
bpp() |
Bytes per package |
bps() |
Bits per second |
byt() |
Bytes - default input |
da() |
Destination Address |
dap() |
Destination Address:Port |
dp() |
Destination Port |
flg() |
TCP Flags |
ibyt() |
Input Bytes |
ipkt() |
Input Packets |
mpls() |
MPLS labels 1-10 |
mpls1() |
MPLS label 1 |
mpls2() |
MPLS label 2 |
mpls3() |
MPLS label 3 |
mpls4() |
MPLS label 4 |
mpls5() |
MPLS label 5 |
mpls6() |
MPLS label 6 |
mpls7() |
MPLS label 7 |
mpls8() |
MPLS label 8 |
mpls9() |
MPLS label 9 |
mpls10() |
MPLS label 10 |
obyt() |
Output Bytes |
oextended() |
nfdump extended output format (-o extended ) |
oline() |
nfdump line output format (-o line ) |
olong() |
nfdump long output format (-o long ) |
opkt() |
Output Packets |
pkt() |
Packets - default input |
pps() |
Packets per second |
pr() |
Protocol |
sa() |
Source Address |
sap() |
Source Address:Port |
sp() |
Source Port |
td() |
Duration |
te() |
End Time - last seen |
ts() |
Start Time - first seen |