Using Datasets in Azure Machine Learning

By Ashwin Venugopal -

 









Creating New Datasets

Although there are multiple ways to create new datasets, they are mainly differentiated between tabular and file datasets having different constructors, according to the type of dataset you would like to create:
  1. Dataset.Tabular.from_* for tabular datasets
  2. Dataset.file.from_* for file-based datasets 

Tabular dataset can also be further divided either into a Direct dataset where the data is being accessed from the original location via a public URL; or stored on either the default or a custom datastore.

A Dataset object can be accessed or passed around in the current environment through its object reference, but, it can also be registered as well as accessed through the dataset name called a registered dataset.

Exploring Data in Datasets

There are many ways to explore the registered datasets in Azure ML. In tabular ones, a dataset can be loaded and analyzed programmatically in an Azure Machine Learning workspace and after having a reference to the dataset, it can be converted into an actual in-memory pandas DataFrame or a lazy-loaded Spark or Dask DataFrame by calling any one of the following methods:
  1. to_pandas_dataframe () to create an in-memory pandas DataFrame
  2. to_spark_dataframe () to create a lazily loaded Spark DataFrame
  3. to_dask_dataframe () to create lazily loaded Dask DataFrame 

Lazy datasets are the ones that only loads some data to memory when explicitly needed; while non-lazy ones can simply load all the data into memory and hence are limited by the available memory.

Now, since you have successfully loaded the DataFrame, you can run your favorite pandas methods to explore the datasets. 

Using External Datasets with Open Datasets

If you want to improve the prediction performance of any ML model, then, you should add additional information to your training data by joining external datasets to the training data. It is easier to join them when you work with transactional data containing dates, to create additional features for the training dataset and hence improve prediction performance. Some of the derived features for dates are weekdays, weekends, time to or since weekends, holidays, time to or since holidays, sports events, concerts, and whatnots. You can also join additional country-specific data, like- population data, economic data, sociological data, health data, labor data, and much more; which generally offers you extra insights to improve your model's performance.

Open datasets is a service that provides access to curated datasets for transportation, health and genomics, labor and economics, population and safety, categories and common datasets, etc. that can used to boost your model's performance.

However, Azure Open Datasets allows you to access curated datasets more conveniently in the form of Azure Machine Learning datasets right from within your Azure Machine Learning workspace. 
















Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more