Performing Data Analysis On a Tabular Dataset

By Ashwin Venugopal -

 









Initial Exploration & Cleansing of the Melbourne Housing Dataset

Here, we will look at the contents of a data loaded  from a data store that is registered in Azure Machine Learning and start doing some basic cleaning regarding the raw data:
  • Download the given packages via Python PIP either separately or using the requirements file you can find in GitHub's repository- pandas, seaborn, plotly, scikit-learn, numpy, missingno, umap-learn, and statsmodels.

  • You can either create a new Jupyter notebook or follow along with the one mentioned before.

  • Now, connect to your ML workspace via the configuration file.

  • After that, retrieve the data from your defined ML data store, yourname, and load the dataset into a tabular dataset object.

  • Since, the methods offered by tabular dataset object are not enough, we have to convert it into pandas DataFrame, providing the first look of our data.

  • Now, we can look at the so-called shape of the datasets, which will show us how many columns and how many rows the dataset contains- raw_df.shape.

  • Finally, we have to run the code in order to look at the unique values, the number of missing values, and the data type of each feature.

  • Now, we can start by removing some of the not so important features by sticking with our original DataFrame, called raw_df, and create a new one called df. This will also allow us to add the removed features at any time and since, every row in a DataFrame has an index, so even if we filter out the rows, we can still match the original values. 

  • After that, we can rename some columns to increase our understanding of them.

  • Look for the duplicates using keep to False showing each row with a duplicate, so that we can simply remove one of them. However, we can also use the attribute called inplace to directly overwrite the current DataFrame.

  • Check the categorical features that seem to have missing categories and concentrate on the one with a lot of entries. This code will show the list of unique values in the column- df['CouncilArea'].unique().

  • Finally, we can check if we can get rid of either suburbs or postcodes by creating a DataFrame showing the number of assigned suburbs as well as postcodes. We can look for the ones that have been mapped to multiple suburbs to find the respective list.

Up till now,  we have done some basic exploration and base pruning of our dataset.




















































Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different groups of computers, re
Read more

Project Resourcing (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here Engineering - SIEM The SIEM engineers configures Microsoft Sentinel, including Log Analytics, Logic Apps, workbooks, and playbooks. They are also responsible for the following high-level tasks: Initial configuration of Azure tenant, including provisioning required resources, assigning access roles, and configuring workspace parameters like log retention, resource tagging, and blueprints.  Deployment and configuration of syslog/CEF log collection agents in appropriate locations to collect logs from on-premises devices. Generally, it will be joint effort of both system engineer and SIEM engineer, involving configuration and potential troubleshooting.  Working with system owners to enable log forwarding and configuring any required parsing of log data in Log Analytics.  Working with security operations to create and deploy KQL analytic rules to offer detections for SOC/Computer Security Incident Response Team (CSRIT) use. Tuning of alert rule parameter
Read more

Design Planning (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Complex Organizational Structures SIEM can be an expensive security control, hence, it is common multiple businesses to contribute to the overall expense. Various organizational units may require a level of access to specific dashboards or sets of data. Microsoft Sentinel offers the ability to assign table-level permissions and limit the level of access to the minimum required to perform requisite job functions.  Role-based Access Control (RBAC) Requirements Microsoft Sentinel offers an extensive list of Azure built-in roles that can be used to provide granular access according to the job requirements and permitted level of access. Some of them are various Microsoft Sentinel dedicated roles: Microsoft Sentinel Contributor- Can perform all engineering-related configuration, such as creating alert rules, configuring data connectors, and additional similar tasks.  Microsoft Sentinel Reader- Can query the log data stor
Read more