Ingesting Data Into Azure (Part 1 of 2)

By Ashwin Venugopal -

 





To read part 2, please click here





Understanding Tooling for the Manual Ingestion of Data

This is the list of the options to bring data into your datastores or directly into your ML pipelines:
  • Azure Storage Explorer- It's an interactive application that permits you to upload data to and control datastores, like storage accounts and managed disks. This one is the easiest tool for managing storage accounts and can be found here- https://azure.microsoft.com/en-us/features/storage-explorer/#overview.

  • Azure CLI- We can do anything with the help of CLI, including the creation and uploading of blob into storage account. The proper commands to upload blobs can be found here- https://docs.microsoft.com/en-us/cli/azure/storage/blob.

  • AzCopy- This one is also designed to copy blobs or files to a storage account and it is not much different from Azure CLI in performance. Its download link and description is here- https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10.

  • The Azure Portal- There is a web interface present directly in the Azure portal to upload or change data and if you navigate the storage account, the inbuilt storage browser can be used to upload blobs and files directly via the web interface which is also true for any of the database technologies.

  • RDBMS Management Tooling- Any typical management tool can be used to configure, create, and change tables as well as schemas in a relational database. For an SQL database and Synapse, it is SQL Server Management Studio, for PostgreSQL, it's pgAdmin, and for MySQL, it would be MySQL Workbench.

  • Azure Data Studio- It can help you to connect with any Microsoft SQL database, Synapse, PostgreSQL database in Azure, and Azure Data Explorer. It's a multiplication tool same as the typical management tooling and can be described here- https://docs.microsoft.com/en-us/sql/azure-data-studio/download-azure-data-studio?view=sql-server-ver15.

  • Azure Machine Learning Designer (Import Data)- You can also use the Import Data component present in the Machine Learning designer to add data ad hoc to your pipeline, instead of Azure Machine Learning datastore. It's information can be found here- https://docs.microsoft.com/en-us/azure/machine-learning/component-reference/import-data.




To read part 2, please click here


























Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different...
Read more

Deployment (Part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Azure Resources Microsoft Sentinel needs following resources to be created: Subscription (if a dedicated subscription(s) will be used) Resource group(s) Log Analytics workspace(s) Automation rules/playbook Alert rules Workbooks Microsoft Sentinel offers hundreds of alert rules, workbooks, and automation playbook templates along with hunting scripts. The templates can be used to activate/deploy schedule alerts, create customized dashboards, create automation playbooks and perform threat-hunting activities. Generally, once deployed, the resources created have to be adjusted to match the existing environment, configure local credentials, etc. Methods of deployment: Manual- Administrator can manually configures the Microsoft Sentinel resources with the help of Azure portal. Any manual process has the inherent risks of human operator error, lack of compliance with potential change control procedures, and u...
Read more

Design Planning (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Complex Organizational Structures SIEM can be an expensive security control, hence, it is common multiple businesses to contribute to the overall expense. Various organizational units may require a level of access to specific dashboards or sets of data. Microsoft Sentinel offers the ability to assign table-level permissions and limit the level of access to the minimum required to perform requisite job functions.  Role-based Access Control (RBAC) Requirements Microsoft Sentinel offers an extensive list of Azure built-in roles that can be used to provide granular access according to the job requirements and permitted level of access. Some of them are various Microsoft Sentinel dedicated roles: Microsoft Sentinel Contributor- Can perform all engineering-related configuration, such as creating alert rules, configuring data connectors, and additional similar tasks.  Microsoft Sentinel Reader- Can...
Read more