Custom Named Entity Recognition (Part 3)

By Ashwin Venugopal -

 




Label Your Data

Properly labeling or tagging your data is a crucial step in developing a custom entity extraction model. Labels serve to indicate instances of particular entities within the text that are used for training the model. Three things to focus on are:

  • Consistency - Label your data the same way across all files for training. Consistency allows your model to learn without any conflicting inputs.

  • Precision - Label your entities consistently, without unnecessary extra words. Precision ensures only the correct data is included in your extracted entity.

  • Completeness - Label your data completely, and don't miss any entities. Completeness helps your model always recognize the entities present.

How To Label Your Data ?

Language Studio offers a straightforward approach for annotating your data. It enables you to view the file, mark the start and end of your entities, and specify their type.

Each label you identify is saved in a file located in your storage account alongside your dataset, formatted as an auto-generated JSON file. This file is then utilized by the model to learn how to identify custom entities. You can input this file when setting up your project (for instance, if you are bringing in labels from a different project), but it must conform to the accepted custom NER data formats.

Train and Evaluate Your Model

Training and assessing your model is a cyclical process that involves incorporating more data and labels into your training dataset to enhance the model's accuracy. To identify which data and labels require refinement, Language Studio offers scoring on the View model details page located in the left-hand pane.

Individual entities and your overall model score are broken down into three metrics to explain how they're performing and where they need to improve:

  • Precision- The ratio of successful entity recognitions to all attempted recognitions. A high score means that as long as the entity is recognized, it's labeled correctly.

  • Recall- The ratio of successful entity recognitions to the actual number of entities in the document. A high score means it finds the entity or entities well, regardless of if it assigns them the right label.

  • F1 Score- Combination of precision and recall providing a single scoring metric.

Scores are available both per entity and for the model as a whole. You may find an entity scores well, but the whole model doesn't.

How To Interpret Metrics ?

Our model should ideally perform well in both precision and recall, which indicates that entity recognition is effective. A low score for both metrics indicates that the model is having trouble identifying items in the document and, when it does, it is not confidently assigning the right label.

If precision is low but recall is high, it means that the model recognizes the entity well but doesn't label it as the correct entity type.

If precision is high but recall is low, it means that the model doesn't always recognize the entity, but when the model extracts the entity, the correct label is applied.

Confusion Matrix

The Confusion matrix is located on a separate tab at the top of the same View model information page. This view gives a comprehensive picture of the model and highlights its shortcomings by providing a visual table of all the entities and their respective performances.

The confusion matrix allows you to visually identify where to add data to improve your model's performance.

Conclusion

We have successfully learnt about labelling data as well as training and evaluating our model.








Comments

Post a Comment

Popular posts from this blog

Information Protection Scanner: Resolve Issues with Information Protection Scanner Deployment

By Ashwin Venugopal -
Image
  About If you are experiencing problems with the Microsoft Purview Information Protection scanner, check the health of your deployment by utilizing the Start-ScannerDiagnosticsPowerShell cmdlet to start the scanner diagnostic tool. The diagnostics tool checks the following details and then creates a log files with the results: Whether the database is up to date. Whether network URLs are accessible. Whether there's a valid authentication token and policy can be acquired. Whether the profile is defined in the Azure portal. Whether offline/online configuration exists and can be acquired. Whether the rules configured are valid.  The Start-ScannerDiagnostics command does not perform a comprehensive check of prerequisites. If you encounter problems with the scanner, you need to verify that your system meets the scanner's requirements, and that your scanner's configuration and installation are finished.  Verify Scanning Details Per Scanner Node and Repository Run the Get-ScanSt...
Read more

Azure AI Search plugin in Microsoft Security Copilot (Preview)

By Ashwin Venugopal -
Image
  Introduction Azure AI Search is a service that allows to reliably extract content search, extract, and understand at scale. It is typically used to search for documents, data surveys, and chat-style Copilot apps over proprietary data. Indexing and querying are Azure AI Search's two main workloads. While querying allows to search the content from index, indexing loads the content into the search engine and makes it searchable. By connecting to one of the Azure AI Search indexes from within Copilot, the Microsoft Security Copilot integration for Azure enables to use the content in Copilot to produce contextualized, pertinent, and targeted responses.  Requirement Before using the Azure AI Search plugin in Copilot, the desired Azure AI Search index to connect to Copilot is configured as follows: The text field must be searchable.  The title field must be filterable. The vector-field must use text-embedding-ada-002. Integrated vectorization can be used to setup an index in t...
Read more

How AMI Store & Restore Works?

By Ashwin Venugopal -
Image
  CreateStoreImageTask The CreateStoreImageTask API stores an AMI as a single object in an S3 bucket. The API initiates a task that collects all the data from the AMI and its associated snapshots, subsequently using an S3 multipart upload to store this data in an S3 object. It gathers all elements of the AMI, including most of the non-Region-specific metadata and all EBS snapshots included in the AMI, and consolidates them into a single object in S3. During the upload process, the data is compressed to minimize the space utilized in S3, which means that the S3 object could be smaller than the total sizes of the snapshots in the AMI. If the account making this API call has access to AMI and snapshot tags, they will be retained. The S3 object shares the same ID as the AMI, except it has a .bin extension. Additionally, the S3 metadata tags for the object include the AMI name, AMI description, AMI registration date, AMI owner account, and a timestamp indicating when the store operation...
Read more