Real-Time Zeek Analytics

Rule Function

Zeek is an open source network security monitoring tool, consisting of a suite of monitoring apps divided into modules. This rule accepts syslog messages in JSON format comprised of Zeek log messages originating from these various modules. The rule then sets a set of basic user tags (indicated below) and composes a LogZilla event message consisting of certain of the incoming JSON data fields (Zeek data).

The Zeek modules (and whether they are processed by this rule) are:

Processed	Module	Description
☑	`conn`	TCP/UDP/ICMP connections
☑	`dce_rpc`	Distributed Computing Environment/RPC
☑	`dhcp`	DHCP leases
	`dnp3`	DNP3 requests and replies
☑	`dns`	DNS activity
☑	`dpd`	Dynamic protocol detection
☑	`files`	File analysis results
☑	`ftp`	FTP activity
☑	`http`	HTTP requests and replies
	`irc`	IRC commands and responses
☑	`kerberos`	Kerberos
	`modbus`	Modbus commands and responses
	`modbus_register_change`	Tracks changes to Modbus holding registers
	`mysql`	MySQL
☑	`ntlm`	NT LAN Manager (NTLM)
☑	`ntp`	Network Time Protocol
	`radius`	RADIUS authentication attempts
	`rdp`	RDP
	`rfb`	Remote Framebuffer (RFB)
☑	`sip`	SIP
	`smb_cmd`	SMB commands
☑	`smb_files`	SMB files
☑	`smb_mapping`	SMB trees
☑	`smtp`	SMTP transactions
	`snmp`	SNMP messages
	`socks`	SOCKS proxy requests
☑	`ssh`	SSH connections
☑	`ssl`	SSL/TLS handshake info
☑	`stats`	Memory/event/packet/lag statistics
☑	`syslog`	Syslog messages
☑	`tunnel`	Tunneling protocol events
☑	`weird`	Unexpected network-level activity
☑	`x509`	X.509 certificate info

Vendor Information

User Tags

There are many potential user tags available based on the Zeek data fields. The following table indicates both used and potential user tags:

Used	Zeek Field	Tag Name
☑	`domain`	`Domain`
☑	`id.orig_h`	`SrcIP`
☑	`id.orig_p`	`SrcPort`
☑	`id.resp_h`	`DstIP`
☑	`id.resp_p`	`DstPort`
☑	`operation`	`Operation`
☑	`rcode_name`	`rCode Name`
☑	`status_msg`	`Status Message`
	`dce_rpc`	`Distributed Computing Environment/RPC`
	`_node`	`Zeek Node`
	`_system_name"`	`Zeek System Name`
	`aa`	`Zeek AA`
	`action`	`Action`
	`actions`	`Action`
	`analyzer`	`Threat`
	`answers`	`DNS Answer`
	`assigned_addr`	`DHCP Assigned IP`
	`basic_constraints_ca`	`Basic Constraints CA`
	`certificate_issuer`	`SSL Cert Issuer`
	`certificate_key_alg`	`SSL Key Alg`
	`certificate_key_type`	`SSL Key Type`
	`certificate_sig_alg`	`SSL Sig Alg`
	`certificate_subject`	`SSL Subject`
	`cipher`	`SSL Cipher`
	`cipher_alg`	`SSL Cipher Alg`
	`client`	`SSL Client`
	`client_addr`	`SSL Client Addr`
	`client_cert_subject`	`SSL Cert Subj`
	`client_fqdn`	`SSL Client FQDN`
	`client_message`	`Client Message`
	`compression_alg`	`Compression Alg`
	`content_type`	`Content Type`
	`direction`	`Direction`
	`domain`	`Domain`
	`domainname`	`Domain Name`
	`dst`	`Destination`
	`endpoint`	`Endpoint`
	`error_msg`	`Error Message`
	`extracted`	`Extracted`
	`extracted_cutoff`	`Extracted Cutoff`
	`failure_reason`	`Failure Reason`
	`file_desc`	`File Desc`
	`file_mime_type`	`Mime Type`
	`forwardable`	`Forwardable`
	`helo`	`SMTP Helo`
	`host`	`Host`
	`host_key_alg`	`Host Key Alg`
	`host_name`	`Host Name`
	`host_p`	`Host P`
	`info_msg`	`Info Message`
	`is_orig`	`Is Orig`
	`issuer`	`Issuer`
	`kex_alg`	`Key Exchange Alg`
	`local_orig`	`Local Orig`
	`local_resp`	`Local Resp`
	`mac_alg`	`MAC Alg`
	`mailfrom`	`Mail From`
	`method`	`Method`
	`mime_type`	`Mime Type`
	`mode`	`Mode`
	`msg`	`Message`
	`msg_types`	`Message Type`
	`n`	`Zeek N`
	`name`	`Name`
	`named_pipe`	`Named Pipe`
	`native_file_system`	`Native FS`
	`num_exts`	`Num Exts`
	`orig_filenames`	`Orig Filename`
	`orig_mime_types`	`Orig Mime Type`
	`origin`	`Origin`
	`p`	`Zeek P`
	`password`	`Password`
	`path`	`Path`
	`peer`	`Peer`
	`peer_descr`	`Peer Desc`
	`precision`	`Precision`
	`prev_name`	`Previous Name`
	`proto`	`Protocol`
	`proxied`	`Proxied`
	`qclass`	`qClass`
	`qclass_name`	`qClass Name`
	`qtype_name`	`qType Name`
	`query`	`Query`
	`ra`	`Zeek RA`
	`rcptto`	`Rcpt To`
	`rd`	`Zeek RD`
	`ref_id`	`Referer ID`
	`referer`	`Referer`
	`remote_location_city`	`City`
	`remote_location_country_code`	`Country Code`
	`remote_location_region`	`Region`
	`renewable`	`Renewable`
	`reply_to`	`Reply To`
	`uest_from`	`Request From`
	`request_path`	`Path`
	`request_to`	`Request To`
	`request_type`	`Request Type`
	`resp_filenames`	`File Name`
	`resp_mime_types`	`Mime Type`
	`response_from`	`Resp. From`
	`response_path`	`Resp. Path`
	`response_to`	`Resp. To`
	`resumed`	`Resumed`
	`root_disp`	`Root Disposition`
	`san_dns`	`SAN DNS`
	`san_email`	`SAN Email`
	`san_ip`	`SAN IP`
	`san_uri`	`SAN URI`
	`seq`	`Sequence`
	`server`	`Server`
	`server_addr`	`Server Addr.`
	`server_cert_subject`	`SSL Cert Subj`
	`server_dns_computer_name`	`DNS Name`
	`server_message`	`Server Message`
	`server_name`	`Server Name`
	`server_nb_computer_name`	`Netbios Name`
	`server_tree_name`	`Tree Name`
	`service`	`Service`
	`share_type`	`Share Type`
	`source`	`Source`
	`stratum`	`Stratum`
	`sub`	`Subject`
	`subject`	`Subject`
	`success`	`Success`
	`tags`	`Tags`
	`tc`	`Zeek TC`
	`times_accessed`	`Times Accessed`
	`times_changed`	`Times Changed`
	`times_created`	`Times Created`
	`times_modified`	`Times Modified`
	`unparsed_version`	`Unparsed Version`
	`uri`	`URI`
	`user_agent`	`User Agent`
	`username`	`User`
	`validation_status`	`Validation Status`
	`version`	`Version`
	`version_addl`	`Version Addl`
	`version_major`	`Version Major`
	`version_minor`	`Version Minor`
	`version_minor2`	`Version Minor2`
	`warning`	`Warning`
	`z`	`Zeek Z`

HC Tags

SrcIP
DstIP
Domain
Status Message

Log Examples

conn entry corresponding to a basic UDP packet communication

{
  "ts": 1591367999.305988,
  "uid": "CMdzit1AMNsmfAIiQc",
  "id.orig_h": "192.168.4.76",
  "id.orig_p": 36844,
  "id.resp_h": "192.168.4.1",
  "id.resp_p": 53,
  "proto": "udp",
  "service": "dns",
  "duration": 0.06685185432434082,
  "orig_bytes": 62,
  "resp_bytes": 141,
  "conn_state": "SF",
  "missed_bytes": 0,
  "history": "Dd",
  "orig_pkts": 2,
  "orig_ip_bytes": 118,
  "resp_pkts": 2,
  "resp_ip_bytes": 197
}

ssl entry corresponding to a client initiating an SSL connection

{
  "ts": 1598377391.921726,
  "uid": "CsukF91Bx9mrqdEaH9",
  "id.orig_h": "192.168.4.49",
  "id.orig_p": 56718,
  "id.resp_h": "13.32.202.10",
  "id.resp_p": 443,
  "version": "TLSv12",
  "cipher": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
  "curve": "secp256r1",
  "server_name": "www.taosecurity.com",
  "resumed": false,
  "next_protocol": "h2",
  "established": true,
  "cert_chain_fuids": [
    "F2XEvj1CahhdhtfvT4",
    "FZ7ygD3ERPfEVVohG9",
    "F7vklpOKI4yX9wmvh",
    "FAnbnR32nIIr2j9XV"
  ],
  "client_cert_chain_fuids": [],
  "subject": "CN=www.taosecurity.com",
  "issuer": "CN=Amazon,OU=Server CA 1B,O=Amazon,C=US"
}

ssh entry corresponding to an inbound SSH connection

{
  "ts": "2020-09-16T13:29:23.245216Z",
  "uid": "CzEmsljW9ooL0WnBd",
  "id.orig_h": "35.196.195.158",
  "id.orig_p": 53160,
  "id.resp_h": "192.168.4.37",
  "id.resp_p": 22,
  "version": 2,
  "auth_success": true,
  "auth_attempts": 1,
  "direction": "INBOUND",
  "client": "SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2",
  "server": "SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3",
  "cipher_alg": "[email protected]",
  "mac_alg": "[email protected]",
  "compression_alg": "none",
  "kex_alg": "curve25519-sha256",
  "host_key_alg": "ecdsa-sha2-nistp256",
  "host_key": "a3:41:03:32:1f:8c:8e:82:92:9f:62:8c:38:82:d3:74",
  "hasshVersion": "1.0",
  "hassh": "ec7378c1a92f5a8dde7e8b7a1ddf33d1",
  "hasshServer": "b12d2871a1189eff20364cf5333619ee",
  "cshka": "[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa",
  "hasshAlgorithms": "curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c;[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected];[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,[email protected],zlib",
  "sshka": "ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519",
  "hasshServerAlgorithms": "curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1;[email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected];[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1;none,[email protected]"
}

Zeek Server-side configuration

Zeek configuration

The syslog-ng config in the section below expects the Zeek logs to be sent in JSON format. On the Zeek side, you may need to enable this log type.

edit /opt/zeek/share/zeek/site/local.zeek and set:

# Output in JSON format
@load policy/tuning/json-logs.zeek

Run zeekctl deploy to deploy the change
check /opt/zeek/logs/current/conn.log to make sure it's in JSON format, for example:

# tail -1 /opt/zeek/logs/current/conn.log
{"ts":1641949013.6772,"uid":"CX1l7X34hCbGkWGlB6","id.orig_h":"192.168.10.107","id.orig_p":36278,"id.resp_h":"192.168.10.255","id.resp_p":32412,"proto":"udp","conn_state":"OTH","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"CC","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0}

Syslog-ng Configuration on Zeek

For use with Zeek log files this LogZilla rule requires that syslog be configured to read those Zeek log files and forward the log messages to LogZilla.

WARNING: You may also need to enable the syslog-ng-mod-extra package which provides the syslog-ng type output driver used in the zeek2logzilla.conf below. However, newer versions of syslog-ng have the module in the base package.

For example, if installing on a debian based system:

apt install syslog-ng-mod-extra

The syslog-ng configuration for the machine hosting the Zeek log files should be as follows:

# This is for your *zeek* server (not the LogZilla server)
# filename: /etc/syslog-ng/conf.d/zeek2logzilla.conf
# Zeek log format should look like:
# {"ts":1641946189.335886,"uid":"Ce6ul9J1tSYJNyRga","id.orig_h":"192.168.10.98","id.orig_p":755,"id.resp_h":"192.168.10.99","id.resp_p":2049,"proto":"tcp","conn_state":"OTH","local_orig":true,"local_resp":true,"missed_bytes":0,"history":"CC","orig_pkts":0,"orig_ip_bytes":0,"resp_pkts":0,"resp_ip_bytes":0}
#
# Global Options
options {
  flush_lines(100);
  threaded(yes);
  use_dns(yes);
  use_fqdn (no);
  keep_hostname (yes);
  dns-cache-size(2000);
  dns-cache-expire(87600);
};

# Define log sources
# WARNING: DO NOT USE the zeek symlinked directory
# (/opt/zeek/logs/current by default)
# If you do, then when zeek is restarted
# syslog-ng will try to follow/watch the old files
# and not the new ones
source s_zeek_logs {
    wildcard-file(
        base-dir("/opt/zeek/spool/zeek")
        filename-pattern("*.log")
        flags(no-parse)
    );
};

# Set destination (logzilla)
# REPLACE the host "logzilla" below
# with the actual hostname or IP of your LZ server
# test and make sure you can ping/reach the host
destination d_logzilla {
  syslog-ng(server("logzilla") port(514));
};

log {
  source(s_zeek_logs);
  parser { json-parser (prefix(".JSON.")); };
  rewrite { set("zeek" value(".JSON._source_type")); };
  rewrite { set("$(basename ${FILE_NAME})" value(".JSON._source")); };
  destination(d_logzilla);
  flags(flow-control);
};