Hunt for Threats Using Notebooks in Azure Sentinel

By Ashwin Venugopal -


 Access Azure Sentinel Data with External Tools

Azure Sentinel's foundation is based on the Log Analytics Data Store, which is capable of combining the high-performance querying, dynamic schema, and scales to massive data volumes. The Azure portal and all the other Azure Sentinel tools always uses a standard API to access this data store which is also available for external tools such as Python and PowerShell. There are two libraries that can be used to simplify API access:
  1. kqlmagic
  2. msticpy

kqlmagic

The kqlmagic library provides the easy to implement API wrapper to run KQL queries.

msticpy

msticpy, also known as the Microsoft Threat Intelligence Python Security Tools, is a set of Python tools intended to be used for security investigations and hunting. Many of the tools originated as code Jupyter notebooks solve a problem as part of security investigation. Some of the tools are only useful in notebooks, but many others can be used from the Python command line or imported into your code.

The package addresses three central needs for security investigators and hunters:

  • Acquiring and enriching data
  • Analyzing data
  • Visualizing data

msticpy can query using KQL; the library also provides predefined queries for Azure Sentinel, Microsoft 365 Defender for Endpoint, and the Microsoft Security Graph. 

Hunt with Notebooks

You can create and share documents that contains live code, equations, visualizations, and explanatory text by using a Jupyter notebook which can be used for data cleaning and transformation, numerical simulation, statistical modeling, machine learning, and much more. It can also combine full programmability with a vast library collection for machine learning, visualization, and data analysis. All these attributes makes Jupyter a useful tool for security investigation and hunting. 

Several notebooks, which are developed by some of the Microsoft's security analysts, are packaged with Azure Sentinel. Notebooks have two components:

  • The browser-based interface where you enter and run queries as well as code and where the execution results are displayed.
  • The kernel is responsible for parsing and executing the code itself.   

Many popular Python libraries are readily used by Azure Sentinel libraries such as pandas, matplotlib, bokeh, etc. There are a wide variety of other Python packages for you to choose from covering areas such as:
  • Visualizations and graphics
  • Data processing and analysis
  • Statistics and numerical computing
  • Machine learning and deep learning  
The msticpy package is used in many of the included notebooks and its tools are explicitly designed to help with creating notebooks for hunting and investigation.  

Create a Notebook

To start with Notebooks use the "Getting Started Guide For Azure Sentinel ML Notebooks" notebook.
  1. In the Azure Sentinel Workspace, select Notebooks (preview).
  2. Select a "Getting Started Guide For Azure Sentinel ML Notebooks". Then select Launch notebook in the bottom right.
  3. Next, you need to select an Azure ML workspace. Select Create new.
  4. In the subscription box, select your subscription.
  5. Select Create a new Resource group and choose a name for the same.
  6. In the workspace detail section- Give workspace a unique name, choose your region, Save your storage account, key vault, as well as Application insights information, and the container registry option can remain as None.
  7. At the bottom of the page, select Review + Create. Then on the next page, select create. It will take a moment to deploy the workspace.
  8. After the deployment is finished, select the Go top resource button.
  9. Select the Launch studio button that appears in the center of the screen.
  10. On the Welcome to the studio page, select your directory and subscription as well as the workspace you just created. Then select Get started. 
  11. Select the Notebooks button on the toolbar on the left side of your screen.
  12. A new area showing your files will appear. In the Users/yourname/ select the A Getting Started Guide For Azure Sentinel ML Notebooks file. 
  13. Next to the Compute instance selector at the top of the screen, select the ..., and select New Compute.
  14. Choose your compute settings.
  15. Name your Compute instance and select the Create button at the bottom of the screen. 
  16. In the top right of the notebook, select a kernel to use.    
If you cannot complete the steps above, you can view the steps on its GitHub page instead. 












Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different...
Read more

Deployment (Part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Azure Resources Microsoft Sentinel needs following resources to be created: Subscription (if a dedicated subscription(s) will be used) Resource group(s) Log Analytics workspace(s) Automation rules/playbook Alert rules Workbooks Microsoft Sentinel offers hundreds of alert rules, workbooks, and automation playbook templates along with hunting scripts. The templates can be used to activate/deploy schedule alerts, create customized dashboards, create automation playbooks and perform threat-hunting activities. Generally, once deployed, the resources created have to be adjusted to match the existing environment, configure local credentials, etc. Methods of deployment: Manual- Administrator can manually configures the Microsoft Sentinel resources with the help of Azure portal. Any manual process has the inherent risks of human operator error, lack of compliance with potential change control procedures, and u...
Read more

Project Resourcing (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here Engineering - SIEM The SIEM engineers configures Microsoft Sentinel, including Log Analytics, Logic Apps, workbooks, and playbooks. They are also responsible for the following high-level tasks: Initial configuration of Azure tenant, including provisioning required resources, assigning access roles, and configuring workspace parameters like log retention, resource tagging, and blueprints.  Deployment and configuration of syslog/CEF log collection agents in appropriate locations to collect logs from on-premises devices. Generally, it will be joint effort of both system engineer and SIEM engineer, involving configuration and potential troubleshooting.  Working with system owners to enable log forwarding and configuring any required parsing of log data in Log Analytics.  Working with security operations to create and deploy KQL analytic rules to offer detections for SOC/Computer Security Incident Response Team (CSRIT) use. Tuning...
Read more