Hunt for Threats Using Notebooks in Azure Sentinel

By Ashwin Venugopal -


 Access Azure Sentinel Data with External Tools

Azure Sentinel's foundation is based on the Log Analytics Data Store, which is capable of combining the high-performance querying, dynamic schema, and scales to massive data volumes. The Azure portal and all the other Azure Sentinel tools always uses a standard API to access this data store which is also available for external tools such as Python and PowerShell. There are two libraries that can be used to simplify API access:
  1. kqlmagic
  2. msticpy

kqlmagic

The kqlmagic library provides the easy to implement API wrapper to run KQL queries.

msticpy

msticpy, also known as the Microsoft Threat Intelligence Python Security Tools, is a set of Python tools intended to be used for security investigations and hunting. Many of the tools originated as code Jupyter notebooks solve a problem as part of security investigation. Some of the tools are only useful in notebooks, but many others can be used from the Python command line or imported into your code.

The package addresses three central needs for security investigators and hunters:

  • Acquiring and enriching data
  • Analyzing data
  • Visualizing data

msticpy can query using KQL; the library also provides predefined queries for Azure Sentinel, Microsoft 365 Defender for Endpoint, and the Microsoft Security Graph. 

Hunt with Notebooks

You can create and share documents that contains live code, equations, visualizations, and explanatory text by using a Jupyter notebook which can be used for data cleaning and transformation, numerical simulation, statistical modeling, machine learning, and much more. It can also combine full programmability with a vast library collection for machine learning, visualization, and data analysis. All these attributes makes Jupyter a useful tool for security investigation and hunting. 

Several notebooks, which are developed by some of the Microsoft's security analysts, are packaged with Azure Sentinel. Notebooks have two components:

  • The browser-based interface where you enter and run queries as well as code and where the execution results are displayed.
  • The kernel is responsible for parsing and executing the code itself.   

Many popular Python libraries are readily used by Azure Sentinel libraries such as pandas, matplotlib, bokeh, etc. There are a wide variety of other Python packages for you to choose from covering areas such as:
  • Visualizations and graphics
  • Data processing and analysis
  • Statistics and numerical computing
  • Machine learning and deep learning  
The msticpy package is used in many of the included notebooks and its tools are explicitly designed to help with creating notebooks for hunting and investigation.  

Create a Notebook

To start with Notebooks use the "Getting Started Guide For Azure Sentinel ML Notebooks" notebook.
  1. In the Azure Sentinel Workspace, select Notebooks (preview).
  2. Select a "Getting Started Guide For Azure Sentinel ML Notebooks". Then select Launch notebook in the bottom right.
  3. Next, you need to select an Azure ML workspace. Select Create new.
  4. In the subscription box, select your subscription.
  5. Select Create a new Resource group and choose a name for the same.
  6. In the workspace detail section- Give workspace a unique name, choose your region, Save your storage account, key vault, as well as Application insights information, and the container registry option can remain as None.
  7. At the bottom of the page, select Review + Create. Then on the next page, select create. It will take a moment to deploy the workspace.
  8. After the deployment is finished, select the Go top resource button.
  9. Select the Launch studio button that appears in the center of the screen.
  10. On the Welcome to the studio page, select your directory and subscription as well as the workspace you just created. Then select Get started. 
  11. Select the Notebooks button on the toolbar on the left side of your screen.
  12. A new area showing your files will appear. In the Users/yourname/ select the A Getting Started Guide For Azure Sentinel ML Notebooks file. 
  13. Next to the Compute instance selector at the top of the screen, select the ..., and select New Compute.
  14. Choose your compute settings.
  15. Name your Compute instance and select the Create button at the bottom of the screen. 
  16. In the top right of the notebook, select a kernel to use.    
If you cannot complete the steps above, you can view the steps on its GitHub page instead. 












Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more