Ingesting Data & Managing Datasets (Part 2 of 2)

By Ashwin Venugopal -

 






To read part 1, please click here




Exploring Options for Storing Training Data in Azure

Database systems can be classified according to the type of data and data access into the following types:
  • Relational Database Management Systems (RDBMSs)- They are generally used to store normalized transactional data via B-tree-based ordered indices and the joining of multiple rows with multiple columns may lead to typical queries filter, group, and aggregate. Azure can support various RDBMSs, such as Azure SQL Database, as well as Azure Database for PostgreSQL and MySQL.

  • NoSQL- These are Key-value-based storage systems used for storing de-normalized data with hash-based or ordered indices. Typical queries can access a single record via a collection distributed according to a partition key. Azure can support various NoSQL-based services like Azure cosmos DB and Azure Table storage.

Hence, both the database technologies can be used to store data for machine learning, according to your use cases. While RDBMSs are good for storing training data for machine learning, NoSQL systems are great for storing lookup data like training labels, or ML results such as recommendations, predictions, or feature vectors. 

Creating a Datastore & Ingesting Data

Creating Blob Storage & Connecting it with the Azure ML Workspace

Creating Blob Storage

  • Firstly, navigate a terminal of your choice then, login to Azure, and check if you are working on correct subscription. 

  • In order to create a storage account, we have to explore the available options and required settings by running this command-

 $ az storage account create -h

  • Now, we create our storage account and choose a globally unique name for it.

  • Finally, we have to create a container in our new blob storage.

Now, after completing this part, we can connect the storage to our Azure ML workspace.

Creating a Datastore in Azure ML

  • First of all, we have to understand all the requirements to create a datastore via this command-

az ml datastore create -h

Its output will help us to understand that the name of the resource group, name of the ML workspace, and a YAML file is required. 

  • For YAML file, we will navigate through https://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-datastore-blob, to locate the required schema of our file with some examples, in which the most secured are the ones with limited access via an SAS token.

  • After that we can either download the blobdatastore.yml file from the GitHub repository or create a file with the same name.

  • Now, we have to generate an SAS token as well as the permissions to add, create, delete, list, read, and write in the mlfiles container. Users either choose an expiration data that's far enough in the future or normally select a much shorter expiry to rotate this key accordingly.

  • Navigate to the directory the YAML file is in to finally create the datastore in Azure ML workspace. 

These above steps ensures that we have registered a datastore connected to our blob storage using an SAS token.








To read part 1, please click here


























Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more