Performing Data Analysis & Visualization (Part 3 of 3)

By Ashwin Venugopal -

 




To read part 1, please click here
To read part 2, please click here








Exploring & Analyzing File Datasets

We need to look out for the following aspects while exploring and analyzing datasets:
  • Uniformity- If all the images in a dataset are not of same size, then, they should be rescaled which might require centering pixel values per channel, may be followed by some form of normalization.

  • Augmentation- Here, we diversify the dataset without taking on new data (new images), which can be very useful while dealing with small dataset and typically involves horizontal as well as vertical flipping, cropping, and rotating among other transformation.

However, if you want to take pictures as uniformly as possible to cover a lot of various scenarios, you have to consider following aspects:

  • Camera Type- We might require same type of camera to take pictures in the same format across the globe. 

  • Environmental Conditions- Lighting, temperature, humidity, etc. can also influence the electronics in the camera.

  • Positioning- Position of the angle while taking picture can also influence the image quality.

These are only some of the points to consider while taking a picture.

There are also other forms of files like Sound Files, that can be used to build a speech-to-text model to convert whatever we say into text. We can also use Fourier Transformations to decompose our sound files, while considering the following aspects:

  • Recording Hardware- A voice assistant at home, contains probably the same microphone for everyone, while the one on mobile phones might have different microphones. 

  • Environment- We might need to record voices in different environments which may have different sound spectrum (like, the spectrum would be different in a tram as compared to a recording booth).

  • Pronunciation- The ML algorithm in your brain may have a hard time deciphering different pronunciations (especially dialects) and so does the actual ML model.

Hence, summarize that for file datasets we don't have as many options to statistically eliminate the problems, so, we must concentrate on taking good and clean samples that simulate the kind of realistic environment we would get when the model is running in production. 








To read part 1, please click here
To read part 2, please click here








Comments

Post a Comment

Popular posts from this blog

Deployment (Part 3)

By Ashwin Venugopal -
Image
  To read part 1, please click  here To read part 2, please click  here Microsoft Monitoring Agent (MMA) MMA is used across multiple Microsoft solutions and has undergone a few name changes as its features and functionality have evolved. If it is being considered to be used, then following areas should be covered during a Microsoft Sentinel deployment project: Consider the timelines and the strategy for a migration to AMA (Azure Monitor Agent) before the MMA end-of-life.  Consider the central deployment and management methodology for MMA. Determine which endpoints are in scope for deployment as these affect the log ingestion volume significantly.  Azure Monitor Agent (AMA) Organizations with MMA as the main Microsoft Sentinel agent should start planning the migration to AMA as soon as possible to avoid a rushed migration. One of the main advantages of AMA is the ability to control the log collection policy centrally and apply different policies against different...
Read more

Deployment (Part 1)

By Ashwin Venugopal -
Image
  To read part 2, please click  here To read part 3, please click  here Azure Resources Microsoft Sentinel needs following resources to be created: Subscription (if a dedicated subscription(s) will be used) Resource group(s) Log Analytics workspace(s) Automation rules/playbook Alert rules Workbooks Microsoft Sentinel offers hundreds of alert rules, workbooks, and automation playbook templates along with hunting scripts. The templates can be used to activate/deploy schedule alerts, create customized dashboards, create automation playbooks and perform threat-hunting activities. Generally, once deployed, the resources created have to be adjusted to match the existing environment, configure local credentials, etc. Methods of deployment: Manual- Administrator can manually configures the Microsoft Sentinel resources with the help of Azure portal. Any manual process has the inherent risks of human operator error, lack of compliance with potential change control procedures, and u...
Read more

Project Resourcing (Part 2)

By Ashwin Venugopal -
Image
  To read part 1, please click  here Engineering - SIEM The SIEM engineers configures Microsoft Sentinel, including Log Analytics, Logic Apps, workbooks, and playbooks. They are also responsible for the following high-level tasks: Initial configuration of Azure tenant, including provisioning required resources, assigning access roles, and configuring workspace parameters like log retention, resource tagging, and blueprints.  Deployment and configuration of syslog/CEF log collection agents in appropriate locations to collect logs from on-premises devices. Generally, it will be joint effort of both system engineer and SIEM engineer, involving configuration and potential troubleshooting.  Working with system owners to enable log forwarding and configuring any required parsing of log data in Log Analytics.  Working with security operations to create and deploy KQL analytic rules to offer detections for SOC/Computer Security Incident Response Team (CSRIT) use. Tuning...
Read more