Ingesting Data & Managing Datasets (Part 1 of 2)

By Ashwin Venugopal -

 




To read part 2, please click here






Choosing Data Storage Solutions for Azure Machine Learning

If you want to start training an ML model on remote compute targets like VM, then, you have to ensure that all the executables can access the training data efficiently; all the more for the people who want to access the data in parallel for experimentation, labeling, and training from multiple environments as well as multiple machines. In order to achieve this, we have to manage the data efficiently for different types usages in Azure.

Organizing Data in Azure Machine Learning

In Azure Machine Learning, data is managed as datasets and data storage as datastores.

Datastore is an abstraction of a physical data storage system that is used to link the existing storage system to an Azure Machine Learning workspace. You will have to provide the connection as well as authentication details to connect the existing storage to the workspace by creating a datastore after which the data storage can be easily accessed by the users via the datastore object. This makes it easy to provide access to data storage to your developers, data engineers, and scientists who are collaborating in an Azure machine Learning workspace. At present, the following services can be connected as datastores to a workspace:

  1. Azure Blob containers
  2. Azure file share
  3. Azure Data Lake
  4. Azure Data Lake Gen2
  5. Azure SQL Database
  6. Azure Database for PostgreSQL
  7. Databricks File System
  8. Azure Database for MySQL   

Dataset is an abstraction of data in general and Azure Machine Learning supports two types of data formats-

Tabular Datasets, that are used to define tabular data like from comma or delimiter-separated files, or from paraquet and JSON files, or from SQL queries. They can also be used and defined directly from their publicly available URLs (which is called Direct Dataset), just like the fetching of data via URLs similar to the other popular libraries such as pandas and requests. 

File Datasets, that are used to define any binary data from files and folders, like images, audio, and video data. 

Both of them can be easily registered in your workspace (also called as registered datasets) and are available in your Azure ML Studio under Datasets.

Understanding the Default Storage Accounts of Azure Machine Learning

Default datastore is an Azure Blob storage account, that is created automatically with Azure Machine Learning when you set up the initial workspace, and it is internally used to store all the snapshots, logs, figures, models, and much more while executing the experiment runs. 

The Default datastore in your Azure workspace can be accessed as well as used similar to the custom datastores by creating a datastore reference, and after successfully accessing the datastore and connecting the custom datastores, we have to ensure the efficient data storage for the various ML use cases.  






To read part 2, please click here






















































Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more