Understanding End-to-End Machine Learning Process (Part 4 of 5)

By Ashwin Venugopal -

 





To read part 1, please click here
To read part 2, please click here
To read part 3, please click here
To read part 5, please click here





Defining Labels & Engineering Features


Now, we have to create and transform features, typically referred to as feature engineering while creating labels when missing.

Labeling

It is also known as annotation and although it's the least exciting part of an ML project, it's the most important one in the whole process. Labeling data requires deep insight and understanding of the context of the dataset as well as the prediction process. Proper labels will greatly help in improving the prediction performance and also helps in studying the dataset deeply. Mislabeling might lead to label noise that can affect the performance of every downstream process in the ML pipeline and it should be avoided. 

Some other techniques and tooling are also available to make the labelling process faster due to the fact that ML algorithm can be used both for the desired project and learning of the ways to label data. Such models can start proposing labels during your manual annotation of the dataset.

Feature Engineering

Now, we can transform or add new features with the help of the knowledge gathered from previous steps. Typically, we have to perform one of the following actions:

  1. Feature creation- Create new features from a given set of features or from additional information sources.
  2. Feature transformation- Transforms single features to make them useful and stable for the utilized ML algorithm.
  3. Feature extraction- Create derived features from the original data.
  4. Feature selection- Choose the most prominent and predictive features. 

Training Labels

The following steps defines the process of training an ML model:

  • Define Your ML Task- First of all, we have to define the ML task we are facing, which is generally defined by the business decision behind your use case. You also choose among supervised as well as unsupervised learning methods along with the other categories according to the amount of labeled data.

  • Pick a Suitable Model- This might be logistical regression, a gradient-boosted ensemble tree, or a DNN, just to name a few popular ML model choices and this choice must depend on the training infrastructure as well as the shape and type of data.

  • Pick or Implement a Loss Function & an Optimizer- While experimenting your data, you might have found a strategy to test your model performance and should have picked a data split, loss function, and optimizer already. However, if its not done yet, then, you have to evaluate what you want to measure and optimize at this point.

  • Pick a Dataset Split- Splitting your data into different sets (like training, validation, and test sets) offers you extra insight into the performance of your training and optimization process and also allows you to avoid overfitting your model to your training data.

  • Train a Simple Model Using Cross-Validation- After all the preceding are made, you can train your ML model via cross-validation on a training and validation set, without leaking training data into validation. Next, you have to interpret the error metric of the validation runs.

  • Tune the Model- In the end, you can either tune the outcome of the model by working with the so-called hyperparameters of a model, do model stacking or the other advanced methods, or you might have to go back to the initial data and work on that before training the model again. 












To read part 1, please click here
To read part 2, please click here
To read part 3, please click here
To read part 5, please click here



















































Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Planning for Implementing SAP Solutions on Azure (Part 2 of 5)

By Ashwin Venugopal -
Image
  To read part 1 please click  here To read part 3 please click  here To read part 4 please click  here To read part 5 please click  here Load Balancing The load balancer uses probe ports to determine as well as route the traffic to an active DBMS node, but of there's any failover then the most common SAP application architecture can automatically reconnect against the private virtual IP address without the need for the SAP application to reconfigure it, while the load balancer redirect the traffic against the private virtual IP address without any need for the SAP application to reconfigure. The loaf balancer also offers an option of DirectServerReturn which if configured, can help in directing the traffic from the DBMS VM to the SAP application layer directly to the application layer without routing through the load balancer. But, if it isn't configured, then the return traffic to the SAP application layer is routed through the load balancer. You can also use Azure Load Balan
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more