Managing & Collecting Data (part 2)

By Ashwin Venugopal -

 




To read part 1, please click here



Configuring Log Analytics Storage Options

After determining the amount of data you can ingest as well as store in Log Analytics daily, you can affect the cost of data usage easily as the amount of data you store and retain directly impacts the cost. By exploring the Log Analytics workspace you can easily view the current usage and costs by choosing Usage and estimated costs from the General menu.

Now you can see a dashboard of information with the pricing tier and current costs on the left-hand side as well as graphs on the right-hand side to present the variation in consumption on a daily basis for the last 31 days. You can also explore following two options along the top menu bar:

  • Daily cap- It can enable or disable the ability to limit the data ingestion into the Log Analytics workspace per-day but it can pose a risk of security information loss valuable for detecting threats across your environment. This should be used only for non-production environments.

  • Data retention- It helps you to configure the number of days data can be retained in the Log Analytics workspace. Although by default the number of days is 31 days, but after enabling the Microsoft Sentinel the default free amount becomes 90 days. If you want to increase beyond 90 days, you will be charged with a set fee per GB per month.

Reviewing Alternative Storage Options

The following solutions can be considered as an alternate way for long-term storage, outside of Log Analytics:
  • Azure Blob Storage- A query can created in Azure Monitor to choose a data to move from the Log Analytics workspace and send it to the required Azure Storage Account which helps you to filter the information by type. After the defined query, Azure Automation can be used to run PowerShell as well as load the result as a CSV file while copying it to Azure Blob Storage. 

  • Azure SQL- The Azure Blob Storage's data can be ingested into Azure SQL enabling another method for searching and analyzing the data. Azure Data Factory is used to connect Azure Blob Storage location to Azure SQL Database/Data Warehouse, automating the ingestion of any new data. 

  • Azure Data Lake Storage Gen2- Instead of using azure Blob Storage, you can use this method to access the data through open source platforms like HDInsight, Hadoop, Cloudera, Azure Databricks, and Hortonworks. This solution is easy to manage, performance increasing as well as cost-effective.     








To read part 1, please click here



Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different...
Read more

Deployment (Part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Azure Resources Microsoft Sentinel needs following resources to be created: Subscription (if a dedicated subscription(s) will be used) Resource group(s) Log Analytics workspace(s) Automation rules/playbook Alert rules Workbooks Microsoft Sentinel offers hundreds of alert rules, workbooks, and automation playbook templates along with hunting scripts. The templates can be used to activate/deploy schedule alerts, create customized dashboards, create automation playbooks and perform threat-hunting activities. Generally, once deployed, the resources created have to be adjusted to match the existing environment, configure local credentials, etc. Methods of deployment: Manual- Administrator can manually configures the Microsoft Sentinel resources with the help of Azure portal. Any manual process has the inherent risks of human operator error, lack of compliance with potential change control procedures, and u...
Read more

Deployment (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 3, please click  here Microsoft Sentinel Content Hub Content in Microsoft Sentinel includes any of the following types: Data connectors offer log ingestion from different sources into Microsoft Sentinel. Parsers give log formatting/transformation into ASIM formats, supporting usage across various Microsoft Sentinel content types and scenarios. Workbooks provide monitoring, visualization, and interactively with data in Microsoft Sentinel, highlighting meaningful insights for users.  Analytics rules give alerts that point to relevant SOC actions via incidents.  Hunting Queries are used by SOC teams to proactively hunt for threats in Microsoft Sentinel. Notebooks help SOC teams in using advanced hunting features in Jupyter and Azure Notebooks. Watchlists support the ingestion of specific data for enhanced threat detection and reduced alert fatigue.  Playbooks and Azure Logic Apps Custom Connectors provide featu...
Read more