Hunting on Sysmon events with Jupyter Notebooks - Environment setup
Updated: Jan 2
This post it's the first of a series where we will explore the capabilities of Sysmon and how we can leverage this tool together with Jupyter notebooks and big data Python libraries like Pandas to hunt for indicators of compromise or behaviors which are the result of the activity of advanced adversaries in a network. In this post we will provide some background information about Sysmon and the steps needed to install and configure a Sysmon hunt environment for research and development purposes.
About Sysmon
System Monitor (Sysmon) is a tool developed by Mark Russinovich at Microsoft. It is a Windows driver and service that logs system activity and saves it to the Windows event log. It provides detailed information about process creations, network connections, changes to files, among many other things. Currently, Sysmon provides up to 23 different types of events that are useful for monitoring various aspects of Windows system behavior. The Sysmon agent can be installed in persistent mode and will restart every time the system reboots. The Sysmon events are generated in Windows devices and can be collected in a central location to be used subsequently for continuous monitoring, Threat Hunting, and Incident Response.
Sysmon includes the following capabilities:
Logs process creation with full command line for both current and parent processes.
Records the hash of process image files using SHA1 (the default), MD5, SHA256 or IMPHASH.
Multiple hashes can be used at the same time.
Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs.
Include a session GUID in each events to allow correlation of events on same logon session.
Logs loading of drivers or DLLs with their signatures and hashes.
Logs opens for raw read access of disks and volumes
Optionally logs network connections, including each connection’s source process, IP addresses, port numbers, hostnames and port names.
Detects changes in file creation time to understand when a file was really created. Modification of file create timestamps is a technique commonly used by malware to cover its tracks.
Automatically reload configuration if changed in the registry.
Rule filtering to include or exclude certain events dynamically.
Generates events from early in the boot process to capture activity made by even sophisticated kernel-mode malware.
More information about Sysmon it's available in Microsoft's website.
Sysmon provides good theoretical coverage of the MITRE ATT&CK matrix, as shown in the diagram below.
Developed by @olafhartong (Follow him on Twitter)
It is possible to enhance further the coverage with additional attack correlation logic targeting specific Sysmon events. For example, using the Sysmon event ID 1 (Process creation), we can correlate different anomalies in the command line used by the system's running processes. Some examples are the execution of PowerShell commands with suspicious command line parameters or obfuscation and the execution of internal Windows commands with specific parameters to gather system information.
Installation of the hunting environment
To start exploring and hunting Sysmon data, you need to set up an environment to generate this data, store it, and process it. Our research and development environment uses Winlogbeat to forward the Sysmon logs, Logstash for filtering/parsing and an Azure data lake to store them. We have a virtual Jupyter Lab environment hosted in a powerful ESXi server for hunting purposes. A simpler setup can leverage Winlogbeat and Logstash on top of VMWare Workstation or VirtualBox to store and consume the log files locally in your own computer. Below we are proposing a relatively simple setup that can be reproduced in a commodity computer with enough RAM and CPU.
Requirements
We recommend using a laptop/workstation with a minimum of 16GB of RAM and a 64-bit x86 Intel or AMD Processor from 2011 or later.
VMWare Workstation or VirtualBox
A newly installed and updated Ubuntu 18.04 virtual machine
Enabled WSL and Ubuntu 18.04 environment installed
In our examples, we use VMWare Workstation as a virtualization engine, but theoretically, everything described in this article should work using VirtualBox as well.
Logstash server configuration
You can install Logstash in Ubuntu 18.04 following the Elastic guide. After the installation, follow the steps below to configure Logstash and prepare it for Jupyter.
1. Configure Logstash
Create a folder to store the Sysmon logs received by Logstash
$ sudo mkdir /var/sysmon-logs
$ sudo chmod 777 /var/sysmon-logsCreate a new Logstash configuration file for Winlogbeat
$ sudo vim /etc/logstash/conf.d/winlogbeat.confInsert the lines below:
input {
beats {
port => 5044
}
}
output {
if [agent][type] == "winlogbeat" {
file {
path => "/var/sysmon-logs/winlogbeat-%{+YYYY-MM-dd-HH}.json"
}
}
}
With this configuration Logstash will create local log files that will be rotated on hourly basis.
Restart the Logstash service
$ sudo systemctl restart logstash2. Create an anonymous share to access the Sysmon logs stored by Logstash
Install samba-server
$ sudo apt install tasksel
$ sudo tasksel install samba-serverNext, add the following lines into the Samba configuration file using your favorite text editor
$ sudo nano /etc/samba/smb.conf[sysmon-logs]
comment = Public anonymous share for Sysmon logs
path = /var/sysmon-logs
browsable =yes
create mask = 0660
directory mask = 0771
writable = yes
guest ok = yesRestart the Samba service
$ sudo systemctl restart smbdTest the access to the new share
$ smbclient -L localhostThe output should look similar to the one below.
3. Map the new share as a drive in your host system
Note: In this example we are using Windows 10 as host system.
Map the share in a new drive
Use the IP address allocated to your own Logstash virtual server. Use the credentials below to connect:
Username: anonymous
Password: anonymous
Install Sysmon and Winlogbeat
In this section we will cover the steps needed to install and configure Sysmon and Winlogbeat in the Windows system that will be the target for monitoring/hunting.
1. Install Sysmon
Download Sysmon from here and extract the contents of the zip file
Download the SwiftOnSecurity configuration file for Sysmon from here and save it in the same folder as Sysmon
Open a Windows command prompt as admin and execute the command below to install Sysmon
> Sysmon64.exe -accepteula -i sysmonconfig-export.xml2. Install Winlogbeat
Download the Winlogbeat zip file from the downloads page
Extract the contents into C:\Program Files
Rename the winlogbeat-<version> directory to Winlogbeat
Open a PowerShell prompt as an Administrator (right-click on the PowerShell icon and select Run As Administrator).
From the PowerShell prompt, run the following commands to install the service.
PS C:\Users\Administrator> cd 'C:\Program Files\Winlogbeat'
PS C:\Program Files\Winlogbeat> .\install-service-winlogbeat.ps1Configure Winlogbeat
Edit the file C:\Program Files\Winlogbeat\winlogbeat.yml and update it according to the pictures below.
Start the Winlogbeat service
PS C:\Program Files\Winlogbeat> Start-Service winlogbeatInstall Jupyter Lab
If you haven't done it, follow this guide to install WSL in your Windows 10 host system. After WSL it's installed, go to the Microsoft store and install Ubuntu 18.04.
Open a new Ubuntu WSL console and follow the steps below to install a new Jupyter Lab environment.
Create a new Python 3 virtual environment for Jupyter
$ cd ~
$ virtualenv -p python3.6 jupyterActivate the new environment
$ source jupyter/bin/activateInstall all the required Python libraries
(jupyter)$ pip3 install jupyterlab pandas plotly scipy sklearn matplotlibGenerate Jupyter configuration file
(jupyter)$ jupyter notebook --generate-configEdit the config file and set the password for Jupyter
(jupyter)$ vim /home/[USER]/.jupyter/jupyter_notebook_config.py## Hashed password to use for web authentication.
#
# To generate, type in a python/IPython shell:
#
# from notebook.auth import passwd; passwd()
#
# The string should be of the form type:salt:hashed-password.
c.NotebookApp.password = u'HASHED PASSWORD'Install Jupyter Lab extension for data visualization using Plotly
(jupyter)$ jupyter labextension install jupyterlab-plotly@4.8.1
(jupyter)$ jupyter labextension install plotlywidget@4.8.1Start Jupyter
(jupyter)$ jupyter lab
Open a new browser window and navigate to the URL below to access Jupyter
Login to the web console using the password created previously
If the installation went well you should be able to access the Jupyter console.
Next steps
In our next post, we will start using our new hunt environment to explore and hunt on Sysmon data. We will focus on process execution events (Sysmon event ID 1), and we will provide some useful hints on how to use Python playbooks to catch advanced adversaries.
Stay tuned!
