Analyzers and Tokenizers

 

Improve an Index With Analyzers and Tokenized Terms

By default, Azure AI Search is set up to examine text and find useful tokens for your index. The appropriate tokens guaranty that users may swiftly locate the documents they require. An ideal index is often produced by the default configuration. However, you might wish to specify exactly how text is processed if you have uncommon or unique fields.

Analyzers in AI Search

Text is retrieved by AI Search when it indexes your material. That text must be processed in order to create a helpful index with terms that aid users in finding documents. For example:

• The text should be broken into words, often by using whitespace and punctuation characters as delimiters.

• Stopwords, such as "the" and "it", should be removed because users don't search for them.

• Words should be reduced to their root form. For example, past tense words, such as "ran", should be replaced with present tense words, such as "run".

Analyzers carry out this type of processing in AI Search. The Lucene analyzer is used by default if you don't specify an analysis for a field. Because it can handle a wide variety of languages and provide valuable tokens for your index, the default Lucene analyzer is a solid option for the majority of fields.

Alternatively, you can specify one of the analyzers that are built into AI Search. Built-in analyzers are of two types:

• Language analyzers- Use an integrated language analyzer if you require sophisticated features for particular languages, such as entity recognition, word decompounding, and lemmatization. Microsoft offers fifty analyzers for various languages.

• Specialized analyzers- These language-neutral analyzers are applied to specific fields like product IDs or zip codes. For instance, you can specify a regular expression to match token separators using the PatternAnalyzer.

What is a Custom Analyzer?

You have a lot of possibilities with the built-in analyzers, but occasionally a field requires an analyzer with odd behavior. You can make a custom analyzer in these situations. A custom analyzer consists of:

• Character filters- These filters process a string before it reaches the tokenizer.

• Tokenizers- These components divide the text into tokens to be added to the index.

• Token filters- These filters remove or modify the tokens emitted by the tokenizer.

Character Filters

Before the text is divided into tokens, some processes may need to be finished. These actions are made possible by character filters. There are three character filters that you can use:

• html_strip- This filter removes HTML constructs such as tags and attributes.

• Mapping- This filter enables you to specify mappings that replace one string with another. For example, you could specify a mapping that replaces TX with Texas.

• pattern_replace- This filter enables you to specify a regular expression that identifies patterns in the input text and how matching text should be replaced.

Conclusion

We have successfully learnt about analyzers and tokenizers.