Performing Data Analysis On a Tabular Dataset

By Ashwin Venugopal -

 









Initial Exploration & Cleansing of the Melbourne Housing Dataset

Here, we will look at the contents of a data loaded  from a data store that is registered in Azure Machine Learning and start doing some basic cleaning regarding the raw data:
  • Download the given packages via Python PIP either separately or using the requirements file you can find in GitHub's repository- pandas, seaborn, plotly, scikit-learn, numpy, missingno, umap-learn, and statsmodels.

  • You can either create a new Jupyter notebook or follow along with the one mentioned before.

  • Now, connect to your ML workspace via the configuration file.

  • After that, retrieve the data from your defined ML data store, yourname, and load the dataset into a tabular dataset object.

  • Since, the methods offered by tabular dataset object are not enough, we have to convert it into pandas DataFrame, providing the first look of our data.

  • Now, we can look at the so-called shape of the datasets, which will show us how many columns and how many rows the dataset contains- raw_df.shape.

  • Finally, we have to run the code in order to look at the unique values, the number of missing values, and the data type of each feature.

  • Now, we can start by removing some of the not so important features by sticking with our original DataFrame, called raw_df, and create a new one called df. This will also allow us to add the removed features at any time and since, every row in a DataFrame has an index, so even if we filter out the rows, we can still match the original values. 

  • After that, we can rename some columns to increase our understanding of them.

  • Look for the duplicates using keep to False showing each row with a duplicate, so that we can simply remove one of them. However, we can also use the attribute called inplace to directly overwrite the current DataFrame.

  • Check the categorical features that seem to have missing categories and concentrate on the one with a lot of entries. This code will show the list of unique values in the column- df['CouncilArea'].unique().

  • Finally, we can check if we can get rid of either suburbs or postcodes by creating a DataFrame showing the number of assigned suburbs as well as postcodes. We can look for the ones that have been mapped to multiple suburbs to find the respective list.

Up till now,  we have done some basic exploration and base pruning of our dataset.




















































Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more