Understanding End-to-End Machine Learning Process (Part 2 of 5)

By Ashwin Venugopal -

 





To read part 1, please click here
To read part 3, please click here
To read part 4, please click here
To read part 5, please click here






Classifying ML Algorithms

There are three main types of ML algorithms:

  • Supervised Learning- In this one, models are trained with a so-called labeled dataset i.e. we will also know about the required output along with the required input. It's divided into two groups- classification problems works with discrete results having output as a class or group, for example- identifying fraud in money transactions or doing object detection in images; whereas regression problems works with continuous results with output as a certain value, for example- forecasting prices for houses or the stock market or predicting population growth. However, this learning requires labeling the whole dataset which is generally a tedious task.

  • Unsupervised Learning- As the name suggests, here, models are trained on unlabeled dataset which refers to self-organized learning to find patterns in data, known as clustering. For example- filtering of spam emails from inbox or movie recommendations or clothing a person might like to watch or purchase. The learning algorithms are generally used in real-time scenarios where data is processed directly. This type of learning doesn't require labelling of the dataset.

  • Reinforcement Learning- Here, algorithms learn by reacting to a given environment on their own, just like the humans learn while growing-up. For example- the technique of teaching a dog to behave. However, technically, this is realized through a so-called agent guided by a policy map to decide the probability of taking action within a specific state. For example- training navigation control for a robot or an AI opponent for a game.

Discovering the End-to-End ML Process

In order to get a clear picture of the required steps to create a high-quality ML model, we can refer to the following diagram:




Hence, the above figure defines the following distinct steps:

  1. Excavating data and sources.
  2. Preparing and cleaning data.
  3. Defining labels and engineering features
  4. Training models
  5. Deploying models

Although these steps shows a single ML project, but. if you are dealing with a lot of projects and data, then, you will have to adopt some form of automation and operationalization, referred as MLOps.





To read part 1, please click here
To read part 3, please click here
To read part 4, please click here
To read part 5, please click here
















































Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different...
Read more

Deployment (Part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Azure Resources Microsoft Sentinel needs following resources to be created: Subscription (if a dedicated subscription(s) will be used) Resource group(s) Log Analytics workspace(s) Automation rules/playbook Alert rules Workbooks Microsoft Sentinel offers hundreds of alert rules, workbooks, and automation playbook templates along with hunting scripts. The templates can be used to activate/deploy schedule alerts, create customized dashboards, create automation playbooks and perform threat-hunting activities. Generally, once deployed, the resources created have to be adjusted to match the existing environment, configure local credentials, etc. Methods of deployment: Manual- Administrator can manually configures the Microsoft Sentinel resources with the help of Azure portal. Any manual process has the inherent risks of human operator error, lack of compliance with potential change control procedures, and u...
Read more

Project Resourcing (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here Engineering - SIEM The SIEM engineers configures Microsoft Sentinel, including Log Analytics, Logic Apps, workbooks, and playbooks. They are also responsible for the following high-level tasks: Initial configuration of Azure tenant, including provisioning required resources, assigning access roles, and configuring workspace parameters like log retention, resource tagging, and blueprints.  Deployment and configuration of syslog/CEF log collection agents in appropriate locations to collect logs from on-premises devices. Generally, it will be joint effort of both system engineer and SIEM engineer, involving configuration and potential troubleshooting.  Working with system owners to enable log forwarding and configuring any required parsing of log data in Log Analytics.  Working with security operations to create and deploy KQL analytic rules to offer detections for SOC/Computer Security Incident Response Team (CSRIT) use. Tuning...
Read more