What is Zeek    

Open Source Software By Vern Paxson to create “Rich logs” with information about network traffic. This is a data-driven sensor to collect data, aggregate the data onto a log collector and gain insight into your network traffic. Scalable to create many workers to collect data and processors. Can also be deployed on low cost off the shelf hardware. Cluster configuration supports high bandwidth networks easily Zeek likes to think of itself as an intelligence framework to analyze network traffic.

Get data from a sensor

The focus of this article is not data collection, but analyst use to use data from packet captures that have been collected. `analyze -s “” -e “” sensor | zeek -r - 

Zeek Logs

Zeek writes a variety of logs based on what it detects.

• conn.log Initial IP/Protocol Connections
• pe.log Portable Executable files found
• known_hosts.log New hosts seen in the past hour
• known_services.log New services seen in the past hour
• dpd.log Dynamic protocol detection
• weird.log Anomalous Activity
• loaded_script.log Scripts loaded upon start/restart
• reporter.log Severety of issues with Zeek
• sotfwarel.log Version numbers of vulnerable application layer software
• Protocols (http, dns, ssl, smtp, etc.)

Viewing Logs

In order to view the log files created, you will generally use cat to display the log and then pipe that to zeek-cut to select the values you want displayed.

Example Command

cat conn.log | zeek-cut -u ts uid id.orig_h id.orig_p id.resp_h id.resp_p orig_bytes resp_bytes

• cat conn.log Start by using cat to display the contents of the log. Any log file you see will represent some aspect of traffic on your network.
• zeek-cut   Piped to zeek-cut, we can begin to select the fields we want. Each log file has different fields, so having a reference from zeek.org or another source is helpful.
• -u option to print the time as a UTC timestamp.
• ts displays the time stamp.
• uid displays the uniquely generated ID for this connection. This ID will be the same across logs during this capture/time period.
• id.orig_h displays the source IP address (originating host IP)
• id.orig_p displays the source port (originating port)
• id.resp_h displays the destination IP address (responding host IP)
• id.resp_p displays the destination port (responding port)
• orig_bytes displays bytes sent from the originating host
• resp_bytes displays bytes sent from the responding host

zeek-cut Options

• -c Include the first format header block in the output.
• -C Include all format header blocks in the output.
• -m Include the first format header blocks in the output in minimal view.
• -M Include all format header blocks in the output in minimal view.
• -d Convert time values into human-readable format.
• -D <fmt> Like -d, but specify format for time (see strftime(3) for syntax).
• mrarlen@gmail_com
• -F <ofs> Sets a different output field separator character.
• -h Show help.
• -n Print all fields except those specified.
• -u Like -d, but print timestamps in UTC instead of local time.
• -U <fmt> Like -D, but print timestamps in UTC instead of local time.

Sample Fields Available from various logs

conn.log

#fields ts  uid id.orig_h   id.orig_p   id.resp_h   id.resp_p   version cipher  curve   server_name resumed  last_alert  next_protocol   established ssl_history cert_chain_fps  client_cert_chain_fps   sni_matches_cert

files.log

#fields ts  fuid tx_hosts    rx_hosts    conn_uids   source  depth   analyzers   mime_type   filename duration    local_orig  is_orig seen_bytes  total_bytes missing_bytes   overflow_bytes  timedout parent_fuid  md5 sha1 sha256  extracted   extracted_cutoff    extracted_size

SSL.log

#fields ts  uid id.orig_h   id.orig_p   id.resp_h   id.resp_p   version cipher  curve   server_name resumed  last_alert  next_protocol   established ssl_history cert_chain_fps  client_cert_chain_fps   sni_matches_cert

Sample Queries

How many packets were sent and received by connection with source port 36499 cat conn.log | zeek-cut orig_pkts resp_pkts id.orig_p | grep 36499

What is the server that was connected to in the SSL log? cat ssl.log | zeek-cut id.org_h id.orig_p id.resp_h id.resp_p server_name

Identify the file that was transferred from a host of interest. cat files.log | zeek-cut fuid tx_hosts rx_hosts conn_uids

What is the UID of a Portable Executable file that was downloaded? `cat pe.log | zeek-cut -d ts id

What is the web traffic like from a host? cat http.log | zeek-cut -d ts id.orig_h id.orig_p id.resp_h id.resp_p method host uri | grep <IP-Address>

Describe the traffic from a host of interest cat http.log | zeek-cut id.orig_h id.resp_h method user_agent | grep <IP-Address>

What IP addresses and sites did a host of interest connect to? cat dns.log | zeek-cut -d ts id.orig_h id.resp_h id.resp_p proto query answers| grep <IP-Address>

Describe the email exchanges at the time of the incident. cat smtp.log | zeek-cut -d ts uid id.orig_h id.orig_p id.resp_h id.resp_p helo mailfrom rcptto subject

What is SiLK?

• System for Internet Level Knowledge, SiLK is an Open Source software project from Carnegie Mellon.
• Single point to send all NetFlow data
• Can convert large packet data in order to analyze the data.
• Network behavioral analysis once data is converted to numeric.
• Perfect for networks that cannot collect packets.

Commands to Work with a SiLK Repository

**Note, commands are usually chained together.

rwfilter

Allows you to pull data out of the repository and filter it.

rwcut

Allows you to select fields from your filter and display it to or write to a file.

rwstats

generate basic statistics on data

rwcount

aggregate flows into bins

rwuniq

Use any set of fields as a filter to the flows

A Sample Command

`rwfilter –type=all –start-date ‘year-mm-dd:hh:mm:ss’ –end-date ‘year-mm-dd:hh:mm:ss’ –proto=0-255 –pass=stdout | rwcut –fields <0-Xorfieldname> –count=10

• --type=all (determines the sensor/file as the source of data)
• --start-date (Start date and time)
• --end-date (End date and time)
• --proto=0-255 (Criteria to filter the data by. In this case Protocol. Designating 0-255 is one way to get data from all flows.)
• --pass=stdout (Options are Pass or Fail. Pass sends the data you filtered to output for the next command to be run. Fail selects the data other than the filter you selected. Example: –proto=53 –fail=stdout would give you all flows that were NOT dns flows.)
• | (pipe the output to the next tool.)
• rwcut (This is a great tool to cut the data and show certain fields. Examples below.)
• --fields (Select the fields to display. Could be a range like –fields=0-10 or seperate items –fields=sip,sport,dip,dport.)
• --count (The number of items to display.)
• Looking for initial SYN and a following ACK. --flags-initial=S/SA --flags-session=A/A

Real Examples you may use

Find flows related to TCP port 443:

`rwfilter –type=all –start-date ‘year-mm-dd:hh:mm:ss’ –end-date ‘year-mm-dd:hh:mm:ss’ –proto=6 –dport=443 –pass=stdout | rwcut –fields sip,sport, dip,dport,bytes –no-columns

Find flows that are not UDP 53

`rwfilter –type=all –start-date__ –end-date ___ –proto=17 –aport=0-52,54-255 –pass=stdout | rwcut –fields sip,sport, dip,dport,bytes –no-columns

Pass is our network. Fail is things leaving the network.

`rwfilter –type=all –start-date ‘year-mm-dd:hh:mm:ss’ –end-date ‘year-mm-dd:hh:mm:ss’ scidr=10.10.0.0/16 –proto=6 –pass=stdout | rwfilter –dcidr=10.10.0.0/16 –fail=stdout –input-pipe=stdin | rwcut –fields sip,sport, dip,dport,bytes –no-columns

All flows for a given time

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –print-stat

TCP Flows

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=6 –print-stat

All Hosts connecting to port 60000

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date 2018/10/31 –proto=6 –flags-initial=S/SA –dport 60000 –pass=stdout | rwstats –count 200 –fields sip

Find all flows where SYN is present with or without ECN

`rwfilter –type=all –start-date=year-mm-dd:hh:mm:ss –end-date=year-mm-dd:hh:mm:ss –proto=6 –flags-all=S/SFRPAU –pass=stdout | rwcut –fields sip,sport,dip,dport,packets,flags | head -21

Find all the records for two hosts.

`rwfilter –type=all –start-date=year-mm-dd:hh:mm:ss –end-date=year-mm-dd:hh:mm:ss –any-address=172.28.30.5 –pass=stdout | rwfilter –input-pipe=stdin –any-address=192.225.158.2 –pass=stdout | rwcut –fields sip,sport,dip,dport,packets,flags,protocol

Top Talkers

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwstats –fields=sip –count=5

Top Talkers from External Sources

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –fail=stdout | rwstats –fields=sip –count=5

Top Talkers from Internal Sources

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwstats –fields=sip –count=5

Top Talkers Source to Destination Port

Source IPs to what ports?

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwstats –fields=sip,dport –count=5
fields create a single bin of data you’re looking for.

Top Talkers Source Bytes

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwstats –fields=sip –bytes –count=5
–values = bytes (bytes, flows, packets)

Sort Protocols by bytes

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwstats –fields=protocol –bytes –count=50

Find All protocols in the data

`rwfilter –type=all –scidr=10.10.0.0/16 –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss5 –proto=0-255 –pass=stdout | rwuniq – fields=protocol

How many flows are there between dates?

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=0-255 –print-statistics

How many TCP FLows?

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=6 –print-statistics

How many IP Protocols

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=0-255 –pass=stdout | rwuniq –fields proto

What is the Port Number that received the most connection attempts?

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=6 –flags-initial=S/SA –pass=stdout | rwstats –fields dport –count 10

Which host transferred the most bytes? Top Talker

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=0-255 –pass=stdout | rwstats –fieds=sip –bytes –count=10

How many different ports?

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=0-255 –saddress=172.16.60.32 –pass=stdout | rwstats –fieds=dport –bytes –count=20

What is the timestamp of the largest single flow?

`rwfilter –type=all –start-date year-mm-dd:hh:mm:ss –end-date year-mm-dd:hh:mm:ss –proto=0-255 –pass=stdout | rwstats –bytes –fields=stime,sip –count=5

What is tshark?

tshark is the command line version of Wireshark. It’s faster than Wireshark in processing packets, but not as performant as tcpdump.

Notes and Quirks about tshark

• tshark like wireshark will do some processing of packet data. For example, if you look at ICMP error messages, the first 64 bytes of the original packet are included in the packet, so tshark can find that data where tcpdump would not find it with a similar filter.
  • tcpdump-n -r icmp-error.pcap’net 10' | wc-l
  • tshark-n -r icmp-error.pcap-Y ‘ip.addr== 10.0.0.0/8’ | wc-l
• tshark -c # will look through the # of packets specified. It will NOT give you the first # results based on your query/filter. This is different than tcpdump -c # which will give you # results of the query/filter. z

tshark Options

• -q (quiet display, reduce extra display info)
• -r (read a pcap file)
• -n (no DNS resolution)
• -Y ‘wireshark-filter’ (add a wireshark filter)
• -w (write a pcap file)
• -T fields -e <fields> (-T determines the type of output. -e which fields to display)
  • tcp.strem
  • eth.src
  • eth.dst
  • tcp.dstport
  • tcp.srcport
  • ip.src
  • ip.dst
  • dns.id
  • dns.qry.name
  • dns.a
• -z <statistics>
  • http
  • tree
  • http_req
  • follow,tcp,ascii, <#>

Sample Commands

Using tshark with ICMP

Using tshark to get HTTP information

tshark -r <pcap-file> -n -q -z http,tree tshark -r -n -q -z http_req,tree

Using tshark to find and follow a stream

tshark -n -r <pcap-file> -Y 'tcp.srcport == 5678 and tcp.dstport == 80' -T fields -e tcp.stream | uniq tshark -n -r -q -z follow,tcp,ascii,98 | more

Using tshark with DNS

`tshark -n -r -Y ‘ip.addr == 192.168.11.175 and ip.addr == 192.168.11.26 and udp.port == 53’ -T fields -e eth.src -e eth.dst -e ip.src -e ip.dst -e dns.id -e dns.qry.name -e dns.a

Bonus

When you get a base64 encoded payload, you can create a file and use this to decode it. `base64 –decode -i attachment.txt > /tmp/attachment.bin

Use tcpdump to find and isolate packets/sessions/events of interest. Use Wireshark to inspect details.