Ingesting Data Into Azure (Part 2 of 2)

By Ashwin Venugopal -

 







To read part 1, please click here






Understanding Tooling for Automated Ingestion & Transformation of Data

There are some services that can help us to automatically transform as well as move data and can also integrate easily with the pipelines and MLOps in Azure Machine Learning.

Azure Data Factory

It is an enterprise-ready solution for moving and transforming data in Azure that also allows you to connect with hundreds of different sources and create pipelines to transform the integrated data, calling multiple other services in Azure. It can help you to create pipelines, data flows, datasets, and power queries:

  • Pipelines- They are the main attraction of Azure Data Factory. Complex pipelines can be created by calling multiple services to pull data from a source, transform it, and store it in a sink.

  • Datasets- As they are used in pipelines as a source or a sink, you have to specify a connection to a particular data in a datastore that you want to read from or write to in the end before building a pipeline.

  • Data Flows- They permits you to do the actual processing or transformation of data within Data Factory itself, instead of calling a different service to do the heavy lifting.

  • Power Query- It helps you to do Data exploration with DAX inside the Data Factory, which is generally possible only with Power BI or Excel otherwise.

Azure Synapse Spark Pools

We already know that Azure Databricks and Azure Synapse allows you to run Spark jobs in Spark pool and as Apache Spark allows you to transform as well as preprocess extremely large datasets via the distributive nature of the node pool underneath, this tool can be very helpful in taking apart and filtering out the datasets even before starting the actual machine learning process.

We can also run notebooks from any one of the Azure Data factory or the integration engine in azure Synapse, thus having access to these services automatically. Besides, we can also add Synapse Spark Pool as a so-called Linked Service in the Azure Machine Learning workspace allowing us the access of both the ML compute targets and Spark pool as a target for computations via the Azure Machine Learning SDK; offering us another good option for building a clean end-to-end MLOps workflow.

Copying Data to Blob storage

We will follow the steps given below and work with the Melbourne Housing dataset, created by Anthony Pino, available at- https://www.kaggle.com/anthonypino/melbourne-housing-market, to make our file available in our mldemoblob datastore:
  • Firstly, download the melb_data.csv file from https://www.kaggle.com/dansbecker/melbourne-housing-snapshot, and store it in a suitable folder on your device.

  • Now, you have to navigate the folder and run the following command in the CLI while replacing the storage account name with your own-

az storage blob upload \
-- account-name mldemoblob8765 \
-- file ./melb_data.csv \
-- container-name mlfiles \
-- name melb_data.csv
  • In order to verify this, you have to install Azure Storage Explorer and then login to your Azure account in that application; after that, navigate to your storage account and open the mlfiles container. You will see your file where it should be. You can also just drag and drop your file here, creating a blob file automatically, according to your comfort.

  • Finally, check the application itself. For example, when you right-click on the container, you can choose Get Shared Access Signature, opening a wizard allowing you to create a SAS token directly here, instead of using the command line.

Now, our raw dataset file is available in our storage account and therefore in our ML datastore.







To read part 1, please click here
















Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more