Ingesting Data & Managing Datasets (Part 1 of 2)

By Ashwin Venugopal -

 




To read part 2, please click here






Choosing Data Storage Solutions for Azure Machine Learning

If you want to start training an ML model on remote compute targets like VM, then, you have to ensure that all the executables can access the training data efficiently; all the more for the people who want to access the data in parallel for experimentation, labeling, and training from multiple environments as well as multiple machines. In order to achieve this, we have to manage the data efficiently for different types usages in Azure.

Organizing Data in Azure Machine Learning

In Azure Machine Learning, data is managed as datasets and data storage as datastores.

Datastore is an abstraction of a physical data storage system that is used to link the existing storage system to an Azure Machine Learning workspace. You will have to provide the connection as well as authentication details to connect the existing storage to the workspace by creating a datastore after which the data storage can be easily accessed by the users via the datastore object. This makes it easy to provide access to data storage to your developers, data engineers, and scientists who are collaborating in an Azure machine Learning workspace. At present, the following services can be connected as datastores to a workspace:

  1. Azure Blob containers
  2. Azure file share
  3. Azure Data Lake
  4. Azure Data Lake Gen2
  5. Azure SQL Database
  6. Azure Database for PostgreSQL
  7. Databricks File System
  8. Azure Database for MySQL   

Dataset is an abstraction of data in general and Azure Machine Learning supports two types of data formats-

Tabular Datasets, that are used to define tabular data like from comma or delimiter-separated files, or from paraquet and JSON files, or from SQL queries. They can also be used and defined directly from their publicly available URLs (which is called Direct Dataset), just like the fetching of data via URLs similar to the other popular libraries such as pandas and requests. 

File Datasets, that are used to define any binary data from files and folders, like images, audio, and video data. 

Both of them can be easily registered in your workspace (also called as registered datasets) and are available in your Azure ML Studio under Datasets.

Understanding the Default Storage Accounts of Azure Machine Learning

Default datastore is an Azure Blob storage account, that is created automatically with Azure Machine Learning when you set up the initial workspace, and it is internally used to store all the snapshots, logs, figures, models, and much more while executing the experiment runs. 

The Default datastore in your Azure workspace can be accessed as well as used similar to the custom datastores by creating a datastore reference, and after successfully accessing the datastore and connecting the custom datastores, we have to ensure the efficient data storage for the various ML use cases.  






To read part 2, please click here






















































Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different groups of computers, re
Read more

Project Resourcing (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here Engineering - SIEM The SIEM engineers configures Microsoft Sentinel, including Log Analytics, Logic Apps, workbooks, and playbooks. They are also responsible for the following high-level tasks: Initial configuration of Azure tenant, including provisioning required resources, assigning access roles, and configuring workspace parameters like log retention, resource tagging, and blueprints.  Deployment and configuration of syslog/CEF log collection agents in appropriate locations to collect logs from on-premises devices. Generally, it will be joint effort of both system engineer and SIEM engineer, involving configuration and potential troubleshooting.  Working with system owners to enable log forwarding and configuring any required parsing of log data in Log Analytics.  Working with security operations to create and deploy KQL analytic rules to offer detections for SOC/Computer Security Incident Response Team (CSRIT) use. Tuning of alert rule parameter
Read more

Design Planning (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Complex Organizational Structures SIEM can be an expensive security control, hence, it is common multiple businesses to contribute to the overall expense. Various organizational units may require a level of access to specific dashboards or sets of data. Microsoft Sentinel offers the ability to assign table-level permissions and limit the level of access to the minimum required to perform requisite job functions.  Role-based Access Control (RBAC) Requirements Microsoft Sentinel offers an extensive list of Azure built-in roles that can be used to provide granular access according to the job requirements and permitted level of access. Some of them are various Microsoft Sentinel dedicated roles: Microsoft Sentinel Contributor- Can perform all engineering-related configuration, such as creating alert rules, configuring data connectors, and additional similar tasks.  Microsoft Sentinel Reader- Can query the log data stor
Read more