Understanding End-to-End Machine Learning Process (Part 2 of 5)

By Ashwin Venugopal -

 





To read part 1, please click here
To read part 3, please click here
To read part 4, please click here
To read part 5, please click here






Classifying ML Algorithms

There are three main types of ML algorithms:

  • Supervised Learning- In this one, models are trained with a so-called labeled dataset i.e. we will also know about the required output along with the required input. It's divided into two groups- classification problems works with discrete results having output as a class or group, for example- identifying fraud in money transactions or doing object detection in images; whereas regression problems works with continuous results with output as a certain value, for example- forecasting prices for houses or the stock market or predicting population growth. However, this learning requires labeling the whole dataset which is generally a tedious task.

  • Unsupervised Learning- As the name suggests, here, models are trained on unlabeled dataset which refers to self-organized learning to find patterns in data, known as clustering. For example- filtering of spam emails from inbox or movie recommendations or clothing a person might like to watch or purchase. The learning algorithms are generally used in real-time scenarios where data is processed directly. This type of learning doesn't require labelling of the dataset.

  • Reinforcement Learning- Here, algorithms learn by reacting to a given environment on their own, just like the humans learn while growing-up. For example- the technique of teaching a dog to behave. However, technically, this is realized through a so-called agent guided by a policy map to decide the probability of taking action within a specific state. For example- training navigation control for a robot or an AI opponent for a game.

Discovering the End-to-End ML Process

In order to get a clear picture of the required steps to create a high-quality ML model, we can refer to the following diagram:




Hence, the above figure defines the following distinct steps:

  1. Excavating data and sources.
  2. Preparing and cleaning data.
  3. Defining labels and engineering features
  4. Training models
  5. Deploying models

Although these steps shows a single ML project, but. if you are dealing with a lot of projects and data, then, you will have to adopt some form of automation and operationalization, referred as MLOps.





To read part 1, please click here
To read part 3, please click here
To read part 4, please click here
To read part 5, please click here
















































Comments

Post a Comment

Popular posts from this blog

Query, Visualize, & Monitor Data in Azure Sentinel

By Ashwin Venugopal -
Image
  Azure Sentinel Workbooks Several ready for use templates are provided by the Azure Sentinel that can be used to create your own workbook and then modify them as needed for Contoso. Most of the data connectors it uses to ingest data come with their own workbooks, but better insight can be obtained by simply looking into the data that is being ingested by using tables and visualizations, including bar and pie charts. You can also make your own workbook easily by using this data from the beginning instead of using the predefined templates. Workbook Page You can easily access the workbook page from the Azure Sentinel from the navigation pane which consists of the: Workbook header- You can add a new workbook and review the saved workbooks as well as templates that are available on the workbook page. Templates section- You can access existing workbook templates on the Templates tab. You can save some of the workbooks for quick access and they will appear on the My Workbooks tab.  From the
Read more

Work with String Data Using KQL Statements

By Ashwin Venugopal -
Image
  Extract Data from Unstructured String Fields Unstructured string fields often contains the Security log data and requires parsing to extract data. There are multiple ways of pulling information from string fields in KQL and the two primary operators used are extract and parse.  Extract It gets a match for a regular expression from a text string. You can convert the extracted substring to the indicated type.  Arguments regex- A regular expression. captureGroup- A positive int constant indicating the capture group to extract. 0 stands for the entire match, 1 for the value matched by the first '('parenthesis')' in the regular expression, 2 or more for subsequent parenthesis.  text- A string to search. typeLiteral- An optional type literal. If provided, the extracted substring is converted to this type.  Returns If regex finds a match in text- the substring matched against the indicated capture group captureGroup, optionally converted to typeLiteral. If there's no mat
Read more

Threat Hunting in Microsoft Sentinel (part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Introducing the Microsoft Sentinel Hunting Page Threat hunting is a series activities performed while investigating and as there are no guidelines to perform it, the available tools on the Microsoft Sentinel can be introduced to perform better in your investigations. Firstly, you have to choose the Hunting link in the Microsoft Sentinel navigation menu to get to the Hunting page which is divided into the following sections: The header bar- It is located at the top of the page as usual and contains Refresh as well as timespan drop down menu. The New Query button can help you with creating a new query and the Run all queries button can run all the hunting queries in the background while also updating the hunting query list section with the number of results found.  The summary bar- Here, you can view the total number of queries available to run, the total number of bookmarks, the number of results from running live
Read more